Testing: Evaluate Jest alternatives

[aduth]

Previously: #1382

The ramifications of the Facebook patents grant decision extend beyond using React. In #1382, we ported from a Mocha test framework to using Jest. Jest is covered by the same patents grant:

https://github.com/facebook/jest/blob/master/PATENTS

Therefore, like React, we are in the position to be moving away from Jest. This could mean:

• Returning to Mocha
• Evaluating other options which provide similar benefits to Jest, as outlined in Tests: Setup Jest as an alternative test runner #1382
• Adopting the QUnit testing approach used elsewhere in core

This ought to be the topic of a JavaScript Core Slack meeting, though the next scheduled meeting will be occupied primarily by discussion of frameworks and hooks.

EDIT: react-test-renderer is another lib that has patent grant.

[aaronjorbin]

A bit of historical background:
While core uses QUnit in some areas now, when the original tests were setup there, the decision was made to name that group of tests qunit so that core wasn't tied to only using QUnit in the future. Therefore, I think a decision on using QUnit should be based on QUnit's strengths and benefits regardless to its use in core today.

[ntwb]

I was chuffed to discover AVA now includes snapshot testing, I'm leaning toward AVA for this reason for the ESLint and Stylelint repos where I currently use Jest snapshots.

Gutenberg explored Jest snapshots previously in #1841 and #2018 though there is no usage of Jest snapshots currently

[gziolo]

It looks like Ava no longer uses jest-snapshots as a dependency since avajs/ava#1341 has landed. This is great, because it no longer depends on code that has patent clause :)

[gziolo]

Useful articles which should help with explorations of Jest alternative:

• https://medium.com/dailyjs/javascript-test-runners-benchmark-3a78d4117b4
• https://mo.github.io/2017/06/05/javascript-unit-testing.html
• https://medium.com/powtoon-engineering/a-complete-guide-to-testing-javascript-in-2017-a217b4cd5a2a

There is also this repository https://github.com/vitalets/js-testrunners-bench which contains JavaScript test-runners benchmarks. Results can be found here. We want to have Babel support so this is the most accurate benchmark which covers our use case. Ava doesn't seem to perform well there ...

[gziolo]

I figured out that @BE-Webdesign has already spent some time adding missing tests to the existing blocks, so it would be nice to hear his opinion here.

[BE-Webdesign]

I don't understand the legal stuff fully, so these comments will be disregarding that aspect.

Jest probably makes the most sense for this project still, or any largeish project for that matter, purely on ease of setup.

Mocha is great and with extra configuration, it can be made to run as fast as Jest. You can do snapshots in Mocha as well.

Ava: have not used it, I am sure it would be fine as well.

Tape is an option that I like a lot, but would most likely involve a much larger rewrite of the testing suite we already have, as it is more bare bones. It also might require refactoring some of our code. Nobody likes tape, and at this stage in the project, it would be a bad fit.

QUnit, is already used by core, so that could definitely make sense to use that. The only difference is we should not do the tests in HTML like core does and instead use the CLI. QUnit uses TAP, like tape, so one of the benefits of tape is bound up in QUnit.

There are probably other great options out there, but really it comes down to Mocha and Jest. Ever since Jest got rid of the auto mock everything setup, it has had a significant edge over Mocha. Factoring in the patent issues, which again I cannot comment on really, I would say going back to Mocha makes the most sense.

Mocha is something most JS devs are comfortable with. We can do snapshotting and mimic many of Jest's features and performance characteristics, but it would require extra setup. It is hard to beat Jest on ease of use, performance, and ease of setup ( the three things that matter in a test runner ), but if the React decision impacts this as well, I think Mocha would be the clear choice, personally.

[NiluK]

Jest is now MIT Licensed: https://code.facebook.com/posts/300798627056246

[youknowriad]

you know what, If there's something I'm really happy about it being relicensed, it's Jest, even more than React

[ntwb]

Thanks @BE-Webdesign for the insightful reply 👍

I've not used Mocha much myself, nice to see snapshots appearing in more test frameworks, it's a fantastic feature...

I've also not looked at QUnit apart from cores HTML usage, CLI mode would be best here...

react-test-renderer is another lib that has patent grant.

I'm taking a calculated punt that this will also become MIT licensed as it's in the base React repo

Jest, I do like Jest, a lot, it's watch mode is sweet as 👌 .

Now that Jest is becoming MIT licensed though the general consensus here still appears to favor Jest just as it was previously in #1382, we should probably stick with Jest.

[BE-Webdesign]

So are we sticking with Jest? Please reply 'Yes' and close this issue out 😄.

[ntwb]

I don't see any reason to change now that Jest is (or soon to be) MIT licensed.

That said, let's throw it out there in it the #core-js discussion in ~36 hours from now

[gziolo]

I think we are good with Jest with the latest announcement from FB. Personally, I'm more than happy with it. We still didn't explore the power of snapshots, so it can become even more useful for us. We can wait until the #core-js discussion, but it probably is going to be dominated by the topic of framework-agnostic blocks. Anyway, we can decide tomorrow. I'm all for closing this issue :)

[BE-Webdesign]

Closing in favor of Jest, re open if something comes up. Talking about this in JS meeting can resolve it as well.