Need for Speed - Accelerate Tests From 3 Hours to 3 Minutes

All automated tests, other than unit, are slow and unreliable for the fast development pace every company craves. When a test fails, it’s hard to pinpoint the exact reason why. There are lots of external dependencies and factors outside of your control.

At Komfo, we had automated tests running for more than 3 hours every night in relatively large SaaS application. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable as feedback loop. At one point the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. We decided to stop this madness and after considerable effort and dedication, currently the same tests run for 3 minutes. This is the story of how we achieved nearly 60x faster tests.

We believe that in the near future this will be standard practice, much like unit tests or continuous integration now. In order for a company to stay competitive, _all_ existing automated tests — e.g. static code analysis, unit tests, API, UI, should complete in less than 5 minutes after _every_ code change.

The presentation is technical and touches on topics such as test automation framework design, hermetic servers, Docker containers, architecture for testability, test environments provisioning, DevOps collaboration, testing when depending on internal and external services, the joys and pitfalls of parallel execution.

Transcript

1. NEED FOR SPEED accelerate tests from 3 hours to 3 minutes emo@komfo.com
2. The tests are slow The tests are unreliable The tests can’t exactly pinpoint the problem High Level Tests Problems
3. This is Our Story
4. 600 API (REST) tests 3 hours of execution time This was a big problem.
5. Before After
6. 3 hours 3 minutes
7. 180 Execution Time in Minutes
8. Key Steps
9. Use Dedicated Environment
10. Automated Tests Mongo MySQL Core API PHP/Java Developer Developer Developer Created new dedicated test environment!
11. 180 123 Execution Time in Minutes New Environment
12. Use Empty Databases
13. Automated Tests Core API PHP/Java Developer Developer Developer Use empty databases Mongo MySQL
14. Tests need to setup all the data they need!
15. The average cost to setup every test case Call 12 API endpoints Modify data in 11 tables And then the test starts
16. Dump DB schema on every test run and restore it Only need DB schema and config tables (~20)
17. 180 123 89 Execution Time in Minutes Empty Databases
18. Simulate External Dependencies
19. Problems with external dependencies Sketchy Internet Throttling API Calls Expiring Credentials
20. MySQL Mongo Automated Tests +Some More STUB STUB STUB STUB STUB STUB STUB Stub all external dependencies Core API PHP/Java
21. External stubs - standalone fake services Internal stubs - triggered based on request type
22. Some of the tests still need to contact the real world.
23. 180 123 89 65 Execution Time in Minutes Stub Dependencies
24. Move to Containers
25. Elastic Search Etcd Log stash Redis MySQL Mongo Core API PHP/Java Automated Tests Single server
26. To cope with increasing complexity we created one container per service. But we we’re in for a surprise!
27. 180 123 89 65 104 Execution Time in Minutes Using Containers
28. Run Databases in Memory
29. mysqld some_options --datadir /dev/shm Only in memory
30. 180 123 89 65 104 61 Execution Time in Minutes Run Databases in Memory
31. Don’t Clean Test Data
32. The cost to delete data after every test case Call 4 API endpoints Remove data from 23 tables Stop container, the data evaporates
33. 180 123 89 65 104 61 46 Execution Time in Minutes Don’t delete test data
34. Run in Parallel
35. We can do this because every tests creates it’s own test data and is independent. This should be your last resort, after you’ve exhausted all other options.
36. Execution Time (minutes) 0 4.5 9 13.5 18 Number of Threads 4 6 8 10 12 14 16 The Sweet Spot
37. Had to make some adjustments to accommodate for the fast tests.
38. All timestamps had to be in milliseconds Enhancing them to milliseconds Twitter returns only seconds
39. try { } catch(Exception $exception) {
 usleep(rand(100, 500));
 $this->insertInTable($record); 
 } $this->insertInTable($record); // too fast tests, too much deadlocks
40. 180 123 89 65 104 61 46 5 Execution Time in Minutes Run in Parallel
41. Equalize Batch Sizes
42. Before Number of tests per batch 0 35 70 105 140 Batch # 1 2 3 4 5 6 7 8 9 10
43. 180 123 89 65 104 61 46 5 3 Execution Time in Minutes Equal Batches
44. The Outcome
45. We did even better than 3 minutes!!!
46. 51 minutes 12* minutes How about the UI tests? *Running in single thread
47. The tests are slow The tests are unreliable The tests can’t exactly pinpoint the problem High Level Tests Problems 3 Minutes No external dependencies Cheap tests to run on every commit Awesomeness
48. One more thing…
49. Log files After every test run, check for errors DB tables Collections
50. Being able to run all your tests in less than 5 minutes after every code change is your target. In 3-5 years this will be standard practice.
51. What’s Next
52. Scale horizontally to keep the 3 min. threshold Automatic workarounds Docker bugs Compare CPU, Memory or DB consumption Run all tests after every DB schema change
53. How to Start
54. Create dedicated automation test environment Simulate external dependencies Your tests should create all the data they need Run in parallel and scale horizontally
55. Recommended Reading
56. EmanuilSlavov.com @EmanuilSlavov