Are you suffering from Selenium-itis?

An exploration into the different methods of testing React components/applications and whether Selenium might be the silver bullet.

Presented at London React User Group during their Jan 2016 meetup.

Transcript

1. Are you suffering from Selenium- itis? January 2016 seleniumitis /sɪˈliːnɪəmʌɪtɪs/ noun the fear of web automation oliverwoodings
2. ▸ Front end developer at Qubit ▸ Part of the Opentag dev team ▸ Have been exploring testing React for 1.5 years: Who am I? react-test-tree react-driver react-page-objects
3. Evolution of testing React components
4. ▸ All we had were the official React TestUtils ▸ Powerful, but low level = loads of boilerplate In the beginning...
5. ✓ An implementation of the Page Object pattern for React ✓ Used refs to describe a component tree ✓ Reduces the amount of boilerplate in tests Adds boilerplate back by having to write Page Objects Pattern is better for larger entities; React components are small, don't need an API to describe them QubitProducts/react-page-objects
6. QubitProducts/react-page-objects ▸ Basic usage ▸ With custom Page Object
7. ✓ Evolution of react-page-objects - borrows the idea of using refs to navigate the render tree ✓ Has none of the boilerplate of TestUtils or react-page- objects ✓ Supports stubbing, context changing and mounting X Have to add refs to anything you want to test X Library reaches inside the React internals - hacky and not liked by the React team QubitProducts/react-test-tree
8. QubitProducts/react-test-tree
9. QubitProducts/react-test-tree ▸ Basic usage ▸ Advanced
10. ✓ jQuery for React ✓ Very rich set of selectors (including selecting by component type and props) ✓ Is extendable ✓ Can mount into the DOM or shallowRender X Not had a big uptake in the community yet jquense/teaspoon
11. ✓ Adds JSX equality checks to expect.js ✓ Allows you to assert that the entire output of a component is what you were expecting ✓ Has versions for chai and other assertion libraries X Not really a testing framework, just a utility algolia/expect-jsx
12. ✓ Massive uptake - 2.5k stars ✓ Supports 3 methods of rendering (shallow, DOM, static) ✓ Doesn't require refs to be added for testing ✓ Clean, extensive API with jQuery-style selectors X Only designed to run in Node - tricky to run in the browser X Selectors are a little limited compared to teaspoon airbnb/enzyme
13. The testing triangle
14. The testing triangle E2E Functional Unit
15. Fitting into the testing triangle E2E Functional Unit enzyme react-test-tree expect-jsx teaspoon
16. The importance of E2E/Functional "2 unit tests, 0 functional"
17. Why are we ignoring E2E? ▸ Web automation inherently designed to be indifferent to framework/library choices ▸ Already plenty of tools for E2E, most of them using Selenium WebDriver - webdriver.io - Nightwatch.js - Nemo.js ▸ Pain in the arse to implement/run
18. Framework-indifference can be negative ▸ Web automations frequently throw false negatives ▸ Often caused by arbitrary timeouts - e.g. click a button, wait for the page to reload - how long do you wait for? ▸ There is no way to solve this without being framework- agnostic
19. Framework-indifference is also positive ▸ Being ignorant to the app's framework makes the test more 'user-like' ▸ Allows E2E tests to survive across code refactoring/framework changes ▸ Could there be a middle ground?
20. Building a 'smarter' web automation system
21. Enhancing Selenium WebDriver 1. Use WebDriver as a base for launching browsers and injecting code 2. Enhance WebDriver with framework-agnostic plugins: ▸ Select React elements (enzyme/teaspoon) ▸ Do equality checks against components (expect-jsx) ▸ Navigate around the app (react-router) ▸ Wait for actions (redux)
22. Devs also need to be able to choose their own test runners and assertion libraries Requirements Modern API Library indifferent Fast, simple, reliable For devs to want to use web automation it needs to be simple, fast and reliable The API should be simple and embrace modern JS features such as async/await
23. What might this look like?
24. ✓ Let's you write context-aware web automations ✓ Will optionally download and run Selenium Standalone automagically ✓ Does not care what runner/framework you use ✓ Zero-config N.B: Currently just a proof-of-concept - functioning release expected soon QubitProducts/react-driver
25. 1. Include registration module
26. 2. Write automations with modern API
27. 3. Run using CLI wrapper
28. Where is React testing headed?
29. 1. The rise of enzyme ▸ First testing library to be massively adopted by the community ▸ Over 2000 stars, almost 1.5k downloads a day ▸ Has a clear vision for its future development
30. 2. Generative Testing ▸ As components become more functional, the simplicity of testing them increases ▸ Hypothesis: If a component is perfectly functional and its props are perfectly described, every possible input/output can be deterministically generated and tested.
31. 3. Generative Regression Testing ▸ Generative testing requires you to be able to assert that an output is correct given an arbitrary generated input ▸ Complex components could have many different possible inputs/outputs ▸ What if we adopt the principles of regression testing?
32. 3. Generative Regression Testing 1 2 3 4 Implement component Generate initial test spec based on props Update component Run generative tests against spec
33. 3. Generative Regression Testing ▸ Regression testing without the false negatives ▸ Testing against JSX output, not a screenshot ▸ However it is still hard to implement 'well' ▸ Where is the test spec stored? ▸ How do you quickly and easily update the spec? ▸ How many variants do you generate? ▸ How does the user control what is generated (are PropTypes enough)?
34. We're hiring! http://eng.qubit.com