No More Flaky Tests

Do you play flaky automation tests bingo? Sometimes they pass, sometimes they fail, but you lose all the time. You hate them, they drive you crazy and you would give an arm and a leg to fix them. Flaky tests are the bane of fast-paced development. They drain considerable amount of resources and energy, and all they produce is sorrow.

As time passes, you start to ignore the flaky tests. And with them you ignore the tests that fail for legitimate reasons. Soon regression bugs start to creep in production. You abandon your test automation efforts and revert back to manual regression tests only. This puts even more strain on your team. Development velocity is reduced dramatically.

There are four main causes for high level automation tests flakiness: application architecture, usage of shared resources, async operations and external dependencies. In this talk I’ll present practical, time-tested examples on how to deal with non-determinism in your API or UI tests.

According to recent paper by University of Illinois, the majority of flaky tests, 78%, are flaky the first time they are written. Learn how to identify them on the spot. You’ll also learn how to use continuous improvement techniques to fix any flakiness caused by changes in the application you’re testing.

Stop playing whack-a-mole and start using systematic approach to find and fix the root causes of your tests failures.

Transcript

1. No More Flaky Tests @EmanuilSlavov emanuil.slavov@gmail.com
2. What is a flaky test? A test that produces ‘random’ result when ran multiple times agains the same application. @EmanuilSlavov
3. Why bother? @EmanuilSlavov
4. Adding 1 test a day. Assuming 5% of tests are flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
5. Adding 1 test a day. Assuming 5% of tests are flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
6. Why listen to this presentation? Google’s flaky test rate: 1.5%* Komfo’s flaky test rate: 0.13% *Blog Post: Flaky Tests at Google and How We Mitigate Them @EmanuilSlavov
7. How to detect a flaky test? After the tests run is done, rerun the failing ones, if any. If the failing ones pass on the second try, then they are flaky. @EmanuilSlavov
8. Where do they occur? Unit Tests - run only in memory API Tests - fully deployed system UI Tests - fully deployed system @EmanuilSlavov
9. Flaky Tests Kata
10. Immediately investigate the flaky test Consider deleting the test if fix is impossible The cause may may not be the test @EmanuilSlavov
11. 78%of all tests are flaky the first time they are written* *An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014 When you are done with the fix… @EmanuilSlavov
12. for i in {1..100}; do if ! execute_test ; then break; fi; done; single test execution command stop if the test fails even once run it 100 times @EmanuilSlavov
13. The Ideal Test At the right level Independent Tests one thing only @EmanuilSlavov
14. What causes flaky tests? @EmanuilSlavov
15. Lack of APIs forces testing only through the UI Most business logic in not in the UI Architecture @EmanuilSlavov
17. 24%of our test failures were due to shared resources Shared Resources @EmanuilSlavov
18. 39%of the test failures were due to async causes Handling of Async Operations @EmanuilSlavov
19. result = @user.search(items) result.first[‘channel_id'].should == channel_id sleeping(1).seconds.between_tries.failing_after(5).tries do end async operation assert retry up to 5 times, in case of exception @EmanuilSlavov
20. 3rd Party Services Unreliable Network Throttling Costly Can’t Control Output @EmanuilSlavov
21. Application Facebook Paypal Amazon S3 @EmanuilSlavov
22. Facebook Application Paypal Amazon S3 Proxy @EmanuilSlavov
23. Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage @EmanuilSlavov
24. So we decided to create our own. @EmanuilSlavov
25. github.com/emanuil/nagual Project Nagual @EmanuilSlavov
26. Test Data Management @EmanuilSlavov
27. The time needed to create data for a single test: And then the test starts Call 12 API endpoints Modify data in 11 tables This setup takes about 1.2 seconds @EmanuilSlavov
28. The Trouble with UI tests
29. Too many moving parts The only “true” testing?!? Dev/QA division, outsource UI is made for human consumption @EmanuilSlavov
30. AUTOMATION ≠ SELENIUM @EmanuilSlavov
31. UI Tests Unit Tests API Tests UI Tests @EmanuilSlavov
32. If you still need to write UI tests…. @EmanuilSlavov
33. Basic Fixes @EmanuilSlavov
34. Don’t use XPath locators No record and replay Smart Wait for UI elements Wait for all Ajax calls completion @EmanuilSlavov
35. Advanced Fixes @EmanuilSlavov
36. Skip Login 5 seconds 22 seconds Use token in the URL, cookie, browser plugins @EmanuilSlavov
37. No Setup in the UI Insert test data directly with API or in the DB @EmanuilSlavov
38. No Navigation Insert test data directly with API or in the DB @EmanuilSlavov
39. Conclusion Test at the right level Full control of the test environment Immediately investigate flaky tests Collaborate with developers UI test should have minimal interactions @EmanuilSlavov
40. Recommended Reading @EmanuilSlavov
41. Please, rate this talk. EmanuilSlavov.com @EmanuilSlavov speakerdeck.com/emanuil