Upgrade to Pro — share decks privately, control downloads, hide ads and more …

No More Flaky Tests

November 16, 2017

No More Flaky Tests

Do you play flaky automation tests bingo? Sometimes they pass, sometimes they fail, but you lose all the time. You hate them, they drive you crazy and you would give an arm and a leg to fix them. Flaky tests are the bane of fast-paced development. They drain considerable amount of resources and energy, and all they produce is sorrow.

As time passes, you start to ignore the flaky tests. And with them you ignore the tests that fail for legitimate reasons. Soon regression bugs start to creep in production. You abandon your test automation efforts and revert back to manual regression tests only. This puts even more strain on your team. Development velocity is reduced dramatically.

There are four main causes for high level automation tests flakiness: application architecture, usage of shared resources, async operations and external dependencies. In this talk I’ll present practical, time-tested examples on how to deal with non-determinism in your API or UI tests.

According to recent paper by University of Illinois, the majority of flaky tests, 78%, are flaky the first time they are written. Learn how to identify them on the spot. You’ll also learn how to use continuous improvement techniques to fix any flakiness caused by changes in the application you’re testing.

Stop playing whack-a-mole and start using systematic approach to find and fix the root causes of your tests failures.


November 16, 2017

More Decks by emanuil

Other Decks in Programming


  1. What is a flaky test? A test that produces ‘random’

    result when ran multiple times agains the same application. @EmanuilSlavov
  2. Adding 1 test a day. Assuming 5% of tests are

    flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
  3. Adding 1 test a day. Assuming 5% of tests are

    flaky. Increasing time to fix a test - from 1 to 11 minutes. 0 35 70 105 140 JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC @EmanuilSlavov
  4. Why listen to this presentation? Google’s flaky test rate: 1.5%*

    Komfo’s flaky test rate: 0.13% *Blog Post: Flaky Tests at Google and How We Mitigate Them @EmanuilSlavov
  5. How to detect a flaky test? After the tests run

    is done, rerun the failing ones, if any. If the failing ones pass on the second try, then they are flaky. @EmanuilSlavov
  6. Where do they occur? Unit Tests - run only in

    memory API Tests - fully deployed system UI Tests - fully deployed system @EmanuilSlavov
  7. Immediately investigate the flaky test Consider deleting the test if

    fix is impossible The cause may may not be the test @EmanuilSlavov
  8. 78%of all tests are flaky the first time they are

    written* *An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014 When you are done with the fix… @EmanuilSlavov
  9. for i in {1..100}; do if ! execute_test ; then

    break; fi; done; single test execution command stop if the test fails even once run it 100 times @EmanuilSlavov
  10. Lack of APIs forces testing only through the UI Most

    business logic in not in the UI Architecture @EmanuilSlavov
  11. Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary

    Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage @EmanuilSlavov
  12. The time needed to create data for a single test:

    And then the test starts Call 12 API endpoints Modify data in 11 tables This setup takes about 1.2 seconds @EmanuilSlavov
  13. Too many moving parts The only “true” testing?!? Dev/QA division,

    outsource UI is made for human consumption @EmanuilSlavov
  14. Don’t use XPath locators No record and replay Smart Wait

    for UI elements Wait for all Ajax calls completion @EmanuilSlavov
  15. Skip Login 5 seconds 22 seconds Use token in the

    URL, cookie, browser plugins @EmanuilSlavov
  16. No Setup in the UI Insert test data directly with

    API or in the DB @EmanuilSlavov
  17. Conclusion Test at the right level Full control of the

    test environment Immediately investigate flaky tests Collaborate with developers UI test should have minimal interactions @EmanuilSlavov