Detox: Gray Box End to End Testing Framework for Mobile Apps

High velocity native mobile development requires us to adopt continuous integration workflows, which means our reliance on manual QA has to drop significantly. The most difficult part of automated testing on mobile is the tip of the testing pyramid – E2E. The core problem with E2E tests is flakiness – tests are usually not deterministic. React Native makes flakiness worse due to the async nature of the bridge. We believe the only way to tackle flakiness head on is by moving from blackbox testing to graybox testing and that’s where detox comes into play. The library synchronizes with the lifecycle of the app, including React Native core, making tests finally deterministic.

Transcript

1. medium/@rotemmiz github.com/rotemmiz @rotemmiz rotemm@wix.com Detox Rotem Mizrachi-Meidan Gray Box End to End Testing Framework for Mobile Apps September 2017
2. Wix Engineering Locations Hi. I am Rotem Software Engineer at Wix.com
3. The Wix App § Written from scratch in React Native § Started a 18 months ago § 6 product groups, and growing § 40 developers, 25 modules § 70% JavaScript, 30% Native code § Release train (2 platforms every week)
4. § 14 person-days for full regression (300 tests) on one device. § Running only a subset of the full regression ~70 tests (3 days) § Obviously can’t run this on every release Manual Tests Manual Tests
5. QA Doesn't Scale ▪ QA Test Suite grows with every new feature, with every new regression QA Has More Work Every Week
6. Automated tests are the future! That’s nothing new though…
7. redux-testkit, vanilla tests with remx Unit Tests Component Tests UI Automation Tests E2E Tests enzyme, enzyme-drivers
8. Why is E2E Unreliable?
9. § Tests sometimes fail for no apparent reason § Tests are nondeterministic tasks run in different order inside the app § Unclear when the app finished handling user interaction § Users have to deal with synchronization manually Flakiness sleep(); sleep();
10. ... expectElementToNotExist: async testId => { let element; for (let i = 0; i < 20; i++) { element = await driver.getElementIfExists(testId, 50); if (element) { // if it's still there check back in a little bit await driver.sleep(50); } else { element = undefined; break; } } expect({element, testId}).not.toBeAValidElement(); }, ... sleep(a_lot);
11. Flakiness q - probability of a failed test n - number of tests ✓ ✘
12. § UIAutomator (Android) § UIAutomation (iOS) § Interact with the app from the outside, just like a user Appium
13. § Flaky Results differ locally and in CI, fail with no reason § Slow sleeps, Instruments without delays § Complicated setup, API, maintenance
14. Black Box “Black-box testing is a method of software testing that examines the functionality of an application without peering into its internal structures or workings.” - Wikipedia
15. Black Box Query only resources/APIs that are provided by the operating system. In this case, UIAutomator is a viewer that inspects layout hierarchy, supplied by Android OS.
16. JS Reconciler (Virtual DOM) Native Rendering The Bridge Javascript Thread JSCore (VM) Main Thread (UI) Native Rendering in Brief React
17. Black box was a dead end, we needed a different approach
18. Detox
19. Gray box, not black box
20. Run on any thread including the main (UI) thread Access to memory read the application state Runs in the same process You don’t test what you ship test code is part of the app
21. Uses EarlGrey and Espresso
22. Query internal resources: ▪ Main thread queue ▪ Thread execution ▪ Network stack (in flight requests) ▪ Write your own custom resource monitor Gray Box Sync Mechanisms expect / perform wait No Yes Is app idle ? onTransitionToIdle()
23. Does not rely on WebDriver
24. Simple API
25. § Debuggable § Async § Cross platform § Test Runner Independent describe('Login flow', () => { it('should login successfully', async () => { await device.reloadReactNative(); await expect(element(by.id('email'))).toBeVisible(); await element(by.id('email')).typeText('john@example.com'); await element(by.id('password')).typeText('123456'); await element(by.text('Login')).tap(); await expect(element(by.text('Welcome'))).toBeVisible(); await expect(element(by.id('email'))).toNotExist(); }); });
26. React Native support
27. Evaluating expectations on the device
28. Test Runner Test Runner Tester Tester Detox Server Detox Server Testee Testee EarlGrey / Espresso EarlGrey / Espresso Action / Expectation Invocation protocol (JSON) Websocket relay Invocation isIdle() Invocation result Invocation result Websocket relay Pass / Fail Serial, blocky invocation
29. redux-testkit, vanilla tests with remx Unit Tests Component Tests UI Automation Tests E2E Tests enzyme, enzyme-drivers Detox
30. react-native-repackager An extension for react-native packager Extending the packager’s ability to override bundled files with any other file, essentially creating an easy way to mock environments in react-native github.com/wix/react-native-repackager
31. redux-testkit, vanilla tests with remx Unit Tests Component Tests UI Automation Tests E2E Tests enzyme, enzyme-drivers Detox + react-native-repackager Detox
32. It’s Fast About 5-10 times faster than black box solutions x5-x10
35. Simon Racz
39. Detox is a TDD Project § 100% coverage § Detox’s E2E API is tested with Detox § It is designed to accept contribution
40. http://github.com/wix/detox Contributions Welcome npm install detox --save-dev yarn add detox -D
41. Thanks. medium/@rotemmiz github.com/rotemmiz @rotemmiz rotemm@wix.com medium.com/@rotemmiz