No More Flaky Tests

Do you play flaky automation tests bingo? Sometimes they pass, sometimes they fail, but you lose all the time. You hate them, they drive you crazy and you would give an arm and a leg to fix them. Flaky tests are the bane of fast-paced development. They drain considerable amount of resources and energy, and all they produce is sorrow.

As time passes, you start to ignore the flaky tests. And with them you ignore the tests that fail for legitimate reasons. Soon regression bugs start to creep in production. You abandon your test automation efforts and revert back to manual regression tests only. This puts even more strain on your team. Development velocity is reduced dramatically.

There are four main causes for high level automation tests flakiness: application architecture, usage of shared resources, async operations and external dependencies. In this talk I’ll present practical, time-tested examples on how to deal with non-determinism in your API or UI tests.

According to recent paper by University of Illinois, the majority of flaky tests, 78%, are flaky the first time they are written. Learn how to identify them on the spot. You’ll also learn how to use continuous improvement techniques to fix any flakiness caused by changes in the application you’re testing.

Stop playing whack-a-mole and start using systematic approach to find and fix the root causes of your tests failures.

Transcript

1. No More Flaky Tests
   @EmanuilSlavov
   emanuil.slavov@gmail.com
   

   View Slide

2. What is a flaky test?
   A test that produces ‘random’ result when ran
   multiple times agains the same application.
   @EmanuilSlavov
   

   View Slide

3. Why bother?
   @EmanuilSlavov
   

   View Slide

4. Adding 1 test a day.
   Assuming 5% of tests are flaky.
   Increasing time to fix a test - from 1 to 11 minutes.
   0
   35
   70
   105
   140
   JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
   @EmanuilSlavov
   

   View Slide

5. Adding 1 test a day.
   Assuming 5% of tests are flaky.
   Increasing time to fix a test - from 1 to 11 minutes.
   0
   35
   70
   105
   140
   JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
   @EmanuilSlavov
   

   View Slide

6. Why listen to this presentation?
   Google’s flaky test rate: 1.5%*
   Komfo’s flaky test rate: 0.13%
   *Blog Post: Flaky Tests at Google and How We Mitigate Them
   @EmanuilSlavov
   

   View Slide

7. How to detect a flaky test?
   After the tests run is done, rerun the failing ones, if any.
   If the failing ones pass on the second try, then they are flaky.
   @EmanuilSlavov
   

   View Slide

8. Where do they occur?
   Unit Tests - run only in memory
   API Tests - fully deployed system
   UI Tests - fully deployed system
   @EmanuilSlavov
   

   View Slide

9. Flaky Tests Kata
   

   View Slide

10. Immediately investigate the flaky test
    Consider deleting the test if fix is impossible
    The cause may may not be the test
    @EmanuilSlavov
    

    View Slide

11. 78%of all tests are flaky the
    first time they are written*
    *An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014
    When you are done with the fix…
    @EmanuilSlavov
    

    View Slide

12. for i in {1..100}; do if ! execute_test ; then break; fi; done;
    single test execution command
    stop if the test fails even once
    run it 100 times
    @EmanuilSlavov
    

    View Slide

13. The Ideal Test
    At the right level
    Independent
    Tests one thing only
    @EmanuilSlavov
    

    View Slide

14. What causes flaky tests?
    @EmanuilSlavov
    

    View Slide

15. Lack of APIs forces testing only through the UI
    Most business logic in not in the UI
    Architecture
    @EmanuilSlavov
    

    View Slide

16. View Slide

17. 24%of our test failures were
    due to shared resources
    Shared Resources
    @EmanuilSlavov
    

    View Slide

18. 39%of the test failures were
    due to async causes
    Handling of Async Operations
    @EmanuilSlavov
    

    View Slide

19. result = @user.search(items)
    result.first[‘channel_id'].should == channel_id
    sleeping(1).seconds.between_tries.failing_after(5).tries do
    end
    async operation
    assert
    retry up to 5 times, in case of exception
    @EmanuilSlavov
    

    View Slide

20. 3rd Party Services
    Unreliable Network
    Throttling
    Costly
    Can’t Control Output
    @EmanuilSlavov
    

    View Slide

21. Application
    Facebook
    Paypal
    Amazon
    S3
    @EmanuilSlavov
    

    View Slide

22. Facebook
    Application Paypal
    Amazon
    S3
    Proxy
    @EmanuilSlavov
    

    View Slide

23. Transparent
    Fake SSL certs
    Dynamic Responses
    Local Storage
    Return Binary Data
    Regex URL match
    Existing Tools (March 2016)
    Stubby4J
    WireMock
    Wilma
    soapUI
    MockServer
    mounteback
    Hoverfly
    Mirage
    @EmanuilSlavov
    

    View Slide

24. So we decided to create
    our own.
    @EmanuilSlavov
    

    View Slide

25. github.com/emanuil/nagual
    Project Nagual
    @EmanuilSlavov
    

    View Slide

26. Test Data Management
    @EmanuilSlavov
    

    View Slide

27. The time needed to create data for a single test:
    And then the test starts
    Call 12 API endpoints
    Modify data in 11 tables
    This setup takes
    about 1.2 seconds
    @EmanuilSlavov
    

    View Slide

28. The Trouble with UI tests
    

    View Slide

29. Too many moving parts
    The only “true” testing?!?
    Dev/QA division, outsource
    UI is made for human consumption
    @EmanuilSlavov
    

    View Slide

30. AUTOMATION ≠ SELENIUM
    @EmanuilSlavov
    

    View Slide

31. UI Tests
    Unit Tests API Tests
    UI Tests
    @EmanuilSlavov
    

    View Slide

32. If you still need to write UI tests….
    @EmanuilSlavov
    

    View Slide

33. Basic Fixes
    @EmanuilSlavov
    

    View Slide

34. Don’t use XPath locators
    No record and replay
    Smart Wait for UI elements
    Wait for all Ajax calls completion
    @EmanuilSlavov
    

    View Slide

35. Advanced Fixes
    @EmanuilSlavov
    

    View Slide

36. Skip Login
    5 seconds
    22 seconds
    Use token in the URL, cookie, browser plugins
    @EmanuilSlavov
    

    View Slide

37. No Setup in the UI
    Insert test data directly with API or in the DB
    @EmanuilSlavov
    

    View Slide

38. No Navigation
    Insert test data directly with API or in the DB
    @EmanuilSlavov
    

    View Slide

39. Conclusion
    Test at the right level
    Full control of the test environment
    Immediately investigate flaky tests
    Collaborate with developers
    UI test should have minimal interactions
    @EmanuilSlavov
    

    View Slide

40. Recommended Reading
    @EmanuilSlavov
    

    View Slide

41. Please, rate this talk.
    EmanuilSlavov.com
    @EmanuilSlavov
    speakerdeck.com/emanuil
    

    View Slide