Need for Speed - Accelerate Tests From 3 Hours to 3 Minutes

All automated tests, other than unit, are slow and unreliable for the fast development pace every company craves. When a test fails, it’s hard to pinpoint the exact reason why. There are lots of external dependencies and factors outside of your control.

At Komfo, we had automated tests running for more than 3 hours every night in relatively large SaaS application. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable as feedback loop. At one point the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. We decided to stop this madness and after considerable effort and dedication, currently the same tests run for 3 minutes. This is the story of how we achieved nearly 60x faster tests.

We believe that in the near future this will be standard practice, much like unit tests or continuous integration now. In order for a company to stay competitive, _all_ existing automated tests — e.g. static code analysis, unit tests, API, UI, should complete in less than 5 minutes after _every_ code change.

The presentation is technical and touches on topics such as test automation framework design, hermetic servers, Docker containers, architecture for testability, test environments provisioning, DevOps collaboration, testing when depending on internal and external services, the joys and pitfalls of parallel execution.

Transcript

1. NEED FOR SPEED
   accelerate tests from 3 hours to 3 minutes
   emo@komfo.com
   

   View Slide

2. The tests are slow
   The tests are unreliable
   The tests can’t exactly pinpoint the problem
   High Level Tests Problems
   

   View Slide

3. This is Our Story
   

   View Slide

4. 600 API (REST) tests
   3 hours of execution time
   This was a big problem.
   

   View Slide

5. Before After
   

   View Slide

6. 3
   hours
   3
   minutes
   

   View Slide

7. 180
   Execution Time in Minutes
   

   View Slide

8. Key Steps
   

   View Slide

9. Use Dedicated
   Environment
   

   View Slide

10. Automated
    Tests
    Mongo
    MySQL
    Core API
    PHP/Java
    Developer Developer Developer
    Created new dedicated
    test environment!
    

    View Slide

11. 180
    123
    Execution Time in Minutes
    New Environment
    

    View Slide

12. Use Empty Databases
    

    View Slide

13. Automated
    Tests
    Core API
    PHP/Java
    Developer Developer Developer
    Use empty databases
    Mongo
    MySQL
    

    View Slide

14. Tests need to setup all the data they need!
    

    View Slide

15. The average cost to setup every test case
    Call 12 API endpoints
    Modify data in 11 tables
    And then the test starts
    

    View Slide

16. Dump DB schema on every test run and restore it
    Only need DB schema and config tables (~20)
    

    View Slide

17. 180
    123
    89
    Execution Time in Minutes
    Empty Databases
    

    View Slide

18. Simulate External
    Dependencies
    

    View Slide

19. Problems with external dependencies
    Sketchy Internet
    Throttling API Calls
    Expiring Credentials
    

    View Slide

20. MySQL
    Mongo
    Automated
    Tests
    +Some
    More
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    Stub all external dependencies
    Core API
    PHP/Java
    

    View Slide

21. External stubs - standalone fake services
    Internal stubs - triggered based on request type
    

    View Slide

22. Some of the tests still need to contact
    the real world.
    

    View Slide

23. 180
    123
    89
    65
    Execution Time in Minutes
    Stub Dependencies
    

    View Slide

24. Move to Containers
    

    View Slide

25. Elastic
    Search
    Etcd
    Log
    stash
    Redis
    MySQL
    Mongo
    Core API
    PHP/Java
    Automated
    Tests
    Single server
    

    View Slide

26. To cope with increasing complexity
    we created one container per service.
    But we we’re in for a surprise!
    

    View Slide

27. 180
    123
    89
    65
    104
    Execution Time in Minutes
    Using Containers
    

    View Slide

28. Run Databases
    in Memory
    

    View Slide

29. mysqld some_options --datadir /dev/shm
    Only in memory
    

    View Slide

30. 180
    123
    89
    65
    104
    61
    Execution Time in Minutes
    Run Databases in Memory
    

    View Slide

31. Don’t Clean
    Test Data
    

    View Slide

32. The cost to delete data after every test case
    Call 4 API endpoints
    Remove data from 23 tables
    Stop container, the data evaporates
    

    View Slide

33. 180
    123
    89
    65
    104
    61
    46
    Execution Time in Minutes
    Don’t delete test data
    

    View Slide

34. Run in Parallel
    

    View Slide

35. We can do this because every tests creates it’s
    own test data and is independent.
    This should be your last resort, after you’ve
    exhausted all other options.
    

    View Slide

36. Execution Time (minutes)
    0
    4.5
    9
    13.5
    18
    Number of Threads
    4 6 8 10 12 14 16
    The Sweet Spot
    

    View Slide

37. Had to make some adjustments
    to accommodate for the fast tests.
    

    View Slide

38. All timestamps had to be in milliseconds
    Enhancing them to milliseconds
    Twitter returns only seconds
    

    View Slide

39. try {
    
    
    
    }
    
    catch(Exception $exception) {

    usleep(rand(100, 500));

    $this->insertInTable($record); 

    }
    $this->insertInTable($record);
    // too fast tests, too much deadlocks
    

    View Slide

40. 180
    123
    89
    65
    104
    61
    46
    5
    Execution Time in Minutes
    Run in Parallel
    

    View Slide

41. Equalize Batch Sizes
    

    View Slide

42. Before
    Number of tests per batch
    0
    35
    70
    105
    140
    Batch #
    1 2 3 4 5 6 7 8 9 10
    

    View Slide

43. 180
    123
    89
    65
    104
    61
    46
    5
    3
    Execution Time in Minutes
    Equal Batches
    

    View Slide

44. The Outcome
    

    View Slide

45. We did even better than 3 minutes!!!
    

    View Slide

46. 51
    minutes
    12*
    minutes
    How about the UI tests?
    *Running in single thread
    

    View Slide

47. The tests are slow
    The tests are unreliable
    The tests can’t exactly pinpoint the problem
    High Level Tests Problems
    3 Minutes
    No external dependencies
    Cheap tests to run on every commit
    Awesomeness
    

    View Slide

48. One more thing…
    

    View Slide

49. Log files
    After every test run, check for errors
    DB tables
    Collections
    

    View Slide

50. Being able to run all your tests in
    less than 5 minutes after every
    code change is your target.
    In 3-5 years this will be standard
    practice.
    

    View Slide

51. What’s Next
    

    View Slide

52. Scale horizontally to keep the 3 min. threshold
    Automatic workarounds Docker bugs
    Compare CPU, Memory or DB consumption
    Run all tests after every DB schema change
    

    View Slide

53. How to Start
    

    View Slide

54. Create dedicated automation test environment
    Simulate external dependencies
    Your tests should create all the data they need
    Run in parallel and scale horizontally
    

    View Slide

55. Recommended Reading
    

    View Slide

56. EmanuilSlavov.com
    @EmanuilSlavov
    

    View Slide