Need for Speed (VelocityConf Amsterdam, 2016)

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why.

Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes.

Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution.

Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than three minutes after every code, database, or infrastructure change.

Transcript

1. NEED FOR SPEED
   accelerate tests from 3 hours to 3 minutes
   emo@komfo.com
   

   View Slide

2. With slow tests you’re
   shipping crap faster.
   

   View Slide

3. Everyone Loves Unit Tests
   

   View Slide

4. View Slide

5. The tests are slow
   The tests are unreliable
   The tests can’t exactly pinpoint the problem
   High Level Tests Problems
   

   View Slide

6. 3
   hours
   3
   minutes
   600 API tests
   

   View Slide

7. Before After
   The
   3 Minute
   Goal
   

   View Slide

8. It’s not about the numbers you’ll see
   or the techniques.
   It’s all about continuous improvement.
   

   View Slide

9. Key Steps
   

   View Slide

10. Dedicated
    Environment
    

    View Slide

11. Developer Developer Developer
    Created new dedicated
    test environment.
    Automated
    Tests
    MySQL
    Mongo
    Core API
    PHP/Java
    

    View Slide

12. Execution Time in Minutes
    180
    123
    New Environment
    

    View Slide

13. Empty Databases
    

    View Slide

14. Use empty databases
    Developer Developer Developer
    Automated
    Tests
    MySQL
    Mongo
    Core API
    PHP/Java
    

    View Slide

15. Tests need to setup all the data they need!
    

    View Slide

16. The time needed to create data for each test:
    And then the test starts
    Call 12 API endpoints
    Modify data in 11 tables
    Takes about 1.2 seconds
    

    View Slide

17. Get the current DB schema and apply it
    before the test run starts.
    Only the DB schema and config tables (~20) are
    needed.
    

    View Slide

18. 180
    123
    Execution Time in Minutes
    89
    Empty Databases
    

    View Slide

19. Simulate
    Dependencies
    

    View Slide

20. Problems with external dependencies
    Sketchy Internet
    Throttling API Calls
    Expiring Credentials
    

    View Slide

21. +Some
    More
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    Stub all external dependencies
    Core API
    

    View Slide

22. Transparent
    Fake SSL certs
    Dynamic Responses
    Local Storage
    Return Binary Data
    Regex URL match
    Existing Tools (March 2016)
    Stubby4J
    WireMock
    Wilma
    soapUI
    MockServer
    mounteback
    Hoverfly
    Mirage
    We created project Nagual,
    soon on Github.
    

    View Slide

23. Some of the tests still need to contact
    the real world.
    

    View Slide

24. 180
    123
    89
    Execution Time in Minutes
    65
    Stub Dependencies
    

    View Slide

25. Move to Containers
    

    View Slide

26. Elastic
    Search
    Etcd
    Log
    stash
    Redis
    MySQL
    Mongo
    Core API
    PHP/Java
    Automated
    Tests
    Single server
    

    View Slide

27. To cope with increasing complexity
    we created one container per service.
    But we we’re in for a surprise!
    

    View Slide

28. 180
    123
    89
    65
    Execution Time in Minutes
    104
    Using Containers
    

    View Slide

29. Run Databases
    in Memory
    

    View Slide

30. mysqld some_options --datadir /dev/shm
    Only in memory
    

    View Slide

31. 180
    123
    89
    65
    104
    Execution Time in Minutes
    61
    Run Databases in Memory
    

    View Slide

32. Don’t Clean
    Test Data
    

    View Slide

33. The cost to delete data after every test case
    Call 4 API endpoints
    Remove data from 23 tables
    Or, stop the container, the data evaporates
    Takes about 1.5 seconds
    

    View Slide

34. 180
    123
    89
    65
    104
    61
    Execution Time in Minutes
    46
    Don’t delete test data
    

    View Slide

35. Run in Parallel
    

    View Slide

36. We can do this because every tests creates its
    own test data and is independent.
    This should be your last resort, after you’ve
    exhausted all other options.
    

    View Slide

37. Execution Time (minutes)
    0
    4.5
    9
    13.5
    18
    Number of Threads
    4 6 8 10 12 14 16
    The Sweet Spot
    

    View Slide

38. Had to make some adjustments
    to accommodate the fast tests.
    

    View Slide

39. All timestamps had to be in milliseconds
    We’re adding milliseconds
    Twitter returns only seconds
    

    View Slide

40. try {
    
    
    
    }
    
    catch(Exception $exception) {

    usleep(rand(100, 500));

    $this->insertInTable($record); 

    }
    $this->insertInTable($record);
    // too fast tests, too much deadlocks
    

    View Slide

41. 180
    123
    89
    65
    104
    61
    46
    Execution Time in Minutes
    5
    Run in Parallel
    

    View Slide

42. Equalize Workload
    

    View Slide

43. Before
    Number of tests per thread
    0
    35
    70
    105
    140
    Thread #
    1 2 3 4 5 6 7 8 9 10
    

    View Slide

44. 180
    123
    89
    65
    104
    61
    46
    5
    Execution Time in Minutes
    3
    Equal Batches
    Run in Parallel
    Don’t delete test data
    Run Databases in Memory
    Using Containers
    Stub Dependencies
    Empty Databases
    New Environment
    

    View Slide

45. After Hardware Upgrade
    The Outcome
    2:15 min.
    1:38 min.
    

    View Slide

46. The tests are slow
    The tests are unreliable
    The tests can’t exactly pinpoint the problem
    High Level Tests Problems
    3 Minutes
    No external dependencies
    It’s cheap to run all tests after every commit
    

    View Slide

47. 51
    minutes
    12*
    minutes
    How about the UI tests?
    *Running in single thread
    

    View Slide

48. One more thing…
    

    View Slide

49. Log files - errors, exceptions
    After all the tests complete, check:
    Databases - wrong, unexpected data
    Execution time - load, performance issues
    

    View Slide

50. In a couple of years, running all your
    automated tests, after every code
    change, for less than 3 minutes, will be
    standard development practice.
    

    View Slide

51. How to Start
    

    View Slide

52. Create dedicated automation test environment
    Simulate external dependencies
    Your tests should create all the data they need
    Run in parallel and scale horizontally
    

    View Slide

53. Recommended Reading
    

    View Slide

54. EmanuilSlavov.com
    @EmanuilSlavov
    

    View Slide

55. View Slide

56. Slide #, Photo Credits
    1. https://www.flickr.com/photos/thomashawk
    2. https://www.flickr.com/photos/paulineguilmot
    10. https://www.flickr.com/photos/100497095@N02
    13. https://www.flickr.com/photos/andrewmalone
    19. https://www.flickr.com/photos/astrablog
    25. https://www.flickr.com/photos/foilman
    29. https://www.flickr.com/photos/missusdoubleyou
    32. https://www.flickr.com/photos/canonsnapper
    35. https://www.flickr.com/photos/anotherangle
    42. https://www.flickr.com/photos/-aismist
    

    View Slide