$30 off During Our Annual Pro Sale. View Details »

No More Flaky Tests

emanuil
November 16, 2017

No More Flaky Tests

Do you play flaky automation tests bingo? Sometimes they pass, sometimes they fail, but you lose all the time. You hate them, they drive you crazy and you would give an arm and a leg to fix them. Flaky tests are the bane of fast-paced development. They drain considerable amount of resources and energy, and all they produce is sorrow.

As time passes, you start to ignore the flaky tests. And with them you ignore the tests that fail for legitimate reasons. Soon regression bugs start to creep in production. You abandon your test automation efforts and revert back to manual regression tests only. This puts even more strain on your team. Development velocity is reduced dramatically.

There are four main causes for high level automation tests flakiness: application architecture, usage of shared resources, async operations and external dependencies. In this talk I’ll present practical, time-tested examples on how to deal with non-determinism in your API or UI tests.

According to recent paper by University of Illinois, the majority of flaky tests, 78%, are flaky the first time they are written. Learn how to identify them on the spot. You’ll also learn how to use continuous improvement techniques to fix any flakiness caused by changes in the application you’re testing.

Stop playing whack-a-mole and start using systematic approach to find and fix the root causes of your tests failures.

emanuil

November 16, 2017
Tweet

More Decks by emanuil

Other Decks in Programming

Transcript

  1. No More Flaky Tests
    @EmanuilSlavov
    [email protected]

    View Slide

  2. What is a flaky test?
    A test that produces ‘random’ result when ran
    multiple times agains the same application.
    @EmanuilSlavov

    View Slide

  3. Why bother?
    @EmanuilSlavov

    View Slide

  4. Adding 1 test a day.
    Assuming 5% of tests are flaky.
    Increasing time to fix a test - from 1 to 11 minutes.
    0
    35
    70
    105
    140
    JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
    @EmanuilSlavov

    View Slide

  5. Adding 1 test a day.
    Assuming 5% of tests are flaky.
    Increasing time to fix a test - from 1 to 11 minutes.
    0
    35
    70
    105
    140
    JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC
    @EmanuilSlavov

    View Slide

  6. Why listen to this presentation?
    Google’s flaky test rate: 1.5%*
    Komfo’s flaky test rate: 0.13%
    *Blog Post: Flaky Tests at Google and How We Mitigate Them
    @EmanuilSlavov

    View Slide

  7. How to detect a flaky test?
    After the tests run is done, rerun the failing ones, if any.
    If the failing ones pass on the second try, then they are flaky.
    @EmanuilSlavov

    View Slide

  8. Where do they occur?
    Unit Tests - run only in memory
    API Tests - fully deployed system
    UI Tests - fully deployed system
    @EmanuilSlavov

    View Slide

  9. Flaky Tests Kata

    View Slide

  10. Immediately investigate the flaky test
    Consider deleting the test if fix is impossible
    The cause may may not be the test
    @EmanuilSlavov

    View Slide

  11. 78%of all tests are flaky the
    first time they are written*
    *An Empirical Analysis of Flaky Tests, Qingzhou Luo et al., University of Illinois 2014
    When you are done with the fix…
    @EmanuilSlavov

    View Slide

  12. for i in {1..100}; do if ! execute_test ; then break; fi; done;
    single test execution command
    stop if the test fails even once
    run it 100 times
    @EmanuilSlavov

    View Slide

  13. The Ideal Test
    At the right level
    Independent
    Tests one thing only
    @EmanuilSlavov

    View Slide

  14. What causes flaky tests?
    @EmanuilSlavov

    View Slide

  15. Lack of APIs forces testing only through the UI
    Most business logic in not in the UI
    Architecture
    @EmanuilSlavov

    View Slide

  16. View Slide

  17. 24%of our test failures were
    due to shared resources
    Shared Resources
    @EmanuilSlavov

    View Slide

  18. 39%of the test failures were
    due to async causes
    Handling of Async Operations
    @EmanuilSlavov

    View Slide

  19. result = @user.search(items)
    result.first[‘channel_id'].should == channel_id
    sleeping(1).seconds.between_tries.failing_after(5).tries do
    end
    async operation
    assert
    retry up to 5 times, in case of exception
    @EmanuilSlavov

    View Slide

  20. 3rd Party Services
    Unreliable Network
    Throttling
    Costly
    Can’t Control Output
    @EmanuilSlavov

    View Slide

  21. Application
    Facebook
    Paypal
    Amazon
    S3
    @EmanuilSlavov

    View Slide

  22. Facebook
    Application Paypal
    Amazon
    S3
    Proxy
    @EmanuilSlavov

    View Slide

  23. Transparent
    Fake SSL certs
    Dynamic Responses
    Local Storage
    Return Binary Data
    Regex URL match
    Existing Tools (March 2016)
    Stubby4J
    WireMock
    Wilma
    soapUI
    MockServer
    mounteback
    Hoverfly
    Mirage
    @EmanuilSlavov

    View Slide

  24. So we decided to create
    our own.
    @EmanuilSlavov

    View Slide

  25. github.com/emanuil/nagual
    Project Nagual
    @EmanuilSlavov

    View Slide

  26. Test Data Management
    @EmanuilSlavov

    View Slide

  27. The time needed to create data for a single test:
    And then the test starts
    Call 12 API endpoints
    Modify data in 11 tables
    This setup takes
    about 1.2 seconds
    @EmanuilSlavov

    View Slide

  28. The Trouble with UI tests

    View Slide

  29. Too many moving parts
    The only “true” testing?!?
    Dev/QA division, outsource
    UI is made for human consumption
    @EmanuilSlavov

    View Slide

  30. AUTOMATION ≠ SELENIUM
    @EmanuilSlavov

    View Slide

  31. UI Tests
    Unit Tests API Tests
    UI Tests
    @EmanuilSlavov

    View Slide

  32. If you still need to write UI tests….
    @EmanuilSlavov

    View Slide

  33. Basic Fixes
    @EmanuilSlavov

    View Slide

  34. Don’t use XPath locators
    No record and replay
    Smart Wait for UI elements
    Wait for all Ajax calls completion
    @EmanuilSlavov

    View Slide

  35. Advanced Fixes
    @EmanuilSlavov

    View Slide

  36. Skip Login
    5 seconds
    22 seconds
    Use token in the URL, cookie, browser plugins
    @EmanuilSlavov

    View Slide

  37. No Setup in the UI
    Insert test data directly with API or in the DB
    @EmanuilSlavov

    View Slide

  38. No Navigation
    Insert test data directly with API or in the DB
    @EmanuilSlavov

    View Slide

  39. Conclusion
    Test at the right level
    Full control of the test environment
    Immediately investigate flaky tests
    Collaborate with developers
    UI test should have minimal interactions
    @EmanuilSlavov

    View Slide

  40. Recommended Reading
    @EmanuilSlavov

    View Slide

  41. Please, rate this talk.
    EmanuilSlavov.com
    @EmanuilSlavov
    speakerdeck.com/emanuil

    View Slide