No More Flaky Tests

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
November 16, 2017

No More Flaky Tests

Do you play flaky automation tests bingo? Sometimes they pass, sometimes they fail, but you lose all the time. You hate them, they drive you crazy and you would give an arm and a leg to fix them. Flaky tests are the bane of fast-paced development. They drain considerable amount of resources and energy, and all they produce is sorrow.

As time passes, you start to ignore the flaky tests. And with them you ignore the tests that fail for legitimate reasons. Soon regression bugs start to creep in production. You abandon your test automation efforts and revert back to manual regression tests only. This puts even more strain on your team. Development velocity is reduced dramatically.

There are four main causes for high level automation tests flakiness: application architecture, usage of shared resources, async operations and external dependencies. In this talk I’ll present practical, time-tested examples on how to deal with non-determinism in your API or UI tests.

According to recent paper by University of Illinois, the majority of flaky tests, 78%, are flaky the first time they are written. Learn how to identify them on the spot. You’ll also learn how to use continuous improvement techniques to fix any flakiness caused by changes in the application you’re testing.

Stop playing whack-a-mole and start using systematic approach to find and fix the root causes of your tests failures.



November 16, 2017


  1. No More Flaky Tests @EmanuilSlavov

  2. What is a flaky test? A test that produces ‘random’

    result when ran multiple times agains the same application. @EmanuilSlavov
  3. Why bother? @EmanuilSlavov

  4. Adding 1 test a day. Assuming 5% of tests are

    flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
  5. Adding 1 test a day. Assuming 5% of tests are

    flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
  6. Why listen to this presentation? Google’s flaky test rate: 1.5%*

    Komfo’s flaky test rate: 0.13% *Blog Post: Flaky Tests at Google and How We Mitigate Them @EmanuilSlavov
  7. How to detect a flaky test? After the tests run

    is done, rerun the failing ones, if any. If the failing ones pass on the second try, then they are flaky. @EmanuilSlavov
  8. Where do they occur? Unit Tests - run only in

    memory API Tests - fully deployed system UI Tests - fully deployed system @EmanuilSlavov
  9. Flaky Tests Kata

  10. Immediately investigate the flaky test Consider deleting the test if

    fix is impossible The cause may may not be the test @EmanuilSlavov
  11. 78%of all tests are flaky the first time they are

    written* *An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014 When you are done with the fix… @EmanuilSlavov
  12. for i in {1..100}; do if ! execute_test ; then

    break; fi; done; single test execution command stop if the test fails even once run it 100 times @EmanuilSlavov
  13. The Ideal Test At the right level Independent Tests one

    thing only @EmanuilSlavov
  14. What causes flaky tests? @EmanuilSlavov

  15. Lack of APIs forces testing only through the UI Most

    business logic in not in the UI Architecture @EmanuilSlavov
  16. None
  17. 24%of our test failures were due to shared resources Shared

    Resources @EmanuilSlavov
  18. 39%of the test failures were due to async causes Handling

    of Async Operations @EmanuilSlavov
  19. result = result.first[‘channel_id'].should == channel_id sleeping(1).seconds.between_tries.failing_after(5).tries do end async

    operation assert retry up to 5 times, in case of exception @EmanuilSlavov
  20. 3rd Party Services Unreliable Network Throttling Costly Can’t Control Output

  21. Application Facebook Paypal Amazon S3 @EmanuilSlavov

  22. Facebook Application Paypal Amazon S3 Proxy @EmanuilSlavov

  23. Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary

    Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage @EmanuilSlavov
  24. So we decided to create our own. @EmanuilSlavov

  25. Project Nagual @EmanuilSlavov

  26. Test Data Management @EmanuilSlavov

  27. The time needed to create data for a single test:

    And then the test starts Call 12 API endpoints Modify data in 11 tables This setup takes about 1.2 seconds @EmanuilSlavov
  28. The Trouble with UI tests

  29. Too many moving parts The only “true” testing?!? Dev/QA division,

    outsource UI is made for human consumption @EmanuilSlavov
  30. AUTOMATION ≠ SELENIUM @EmanuilSlavov

  31. UI Tests Unit Tests API Tests UI Tests @EmanuilSlavov

  32. If you still need to write UI tests…. @EmanuilSlavov

  33. Basic Fixes @EmanuilSlavov

  34. Don’t use XPath locators No record and replay Smart Wait

    for UI elements Wait for all Ajax calls completion @EmanuilSlavov
  35. Advanced Fixes @EmanuilSlavov

  36. Skip Login 5 seconds 22 seconds Use token in the

    URL, cookie, browser plugins @EmanuilSlavov
  37. No Setup in the UI Insert test data directly with

    API or in the DB @EmanuilSlavov
  38. No Navigation Insert test data directly with API or in

    the DB @EmanuilSlavov
  39. Conclusion Test at the right level Full control of the

    test environment Immediately investigate flaky tests Collaborate with developers UI test should have minimal interactions @EmanuilSlavov
  40. Recommended Reading @EmanuilSlavov

  41. Please, rate this talk. @EmanuilSlavov