Need for Speed (VelocityConf Amsterdam, 2016)

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why.

Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes.

Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution.

Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than three minutes after every code, database, or infrastructure change.

Transcript

1. NEED FOR SPEED accelerate tests from 3 hours to 3 minutes emo@komfo.com
2. With slow tests you’re shipping crap faster.
3. Everyone Loves Unit Tests
5. The tests are slow The tests are unreliable The tests can’t exactly pinpoint the problem High Level Tests Problems
6. 3 hours 3 minutes 600 API tests
7. Before After The 3 Minute Goal
8. It’s not about the numbers you’ll see or the techniques. It’s all about continuous improvement.
9. Key Steps
10. Dedicated Environment
11. Developer Developer Developer Created new dedicated test environment. Automated Tests MySQL Mongo Core API PHP/Java
12. Execution Time in Minutes 180 123 New Environment
13. Empty Databases
14. Use empty databases Developer Developer Developer Automated Tests MySQL Mongo Core API PHP/Java
15. Tests need to setup all the data they need!
16. The time needed to create data for each test: And then the test starts Call 12 API endpoints Modify data in 11 tables Takes about 1.2 seconds
17. Get the current DB schema and apply it before the test run starts. Only the DB schema and config tables (~20) are needed.
18. 180 123 Execution Time in Minutes 89 Empty Databases
19. Simulate Dependencies
20. Problems with external dependencies Sketchy Internet Throttling API Calls Expiring Credentials
21. +Some More STUB STUB STUB STUB STUB STUB STUB Stub all external dependencies Core API
22. Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage We created project Nagual, soon on Github.
23. Some of the tests still need to contact the real world.
24. 180 123 89 Execution Time in Minutes 65 Stub Dependencies
25. Move to Containers
26. Elastic Search Etcd Log stash Redis MySQL Mongo Core API PHP/Java Automated Tests Single server
27. To cope with increasing complexity we created one container per service. But we we’re in for a surprise!
28. 180 123 89 65 Execution Time in Minutes 104 Using Containers
29. Run Databases in Memory
30. mysqld some_options --datadir /dev/shm Only in memory
31. 180 123 89 65 104 Execution Time in Minutes 61 Run Databases in Memory
32. Don’t Clean Test Data
33. The cost to delete data after every test case Call 4 API endpoints Remove data from 23 tables Or, stop the container, the data evaporates Takes about 1.5 seconds
34. 180 123 89 65 104 61 Execution Time in Minutes 46 Don’t delete test data
35. Run in Parallel
36. We can do this because every tests creates its own test data and is independent. This should be your last resort, after you’ve exhausted all other options.
37. Execution Time (minutes) 0 4.5 9 13.5 18 Number of Threads 4 6 8 10 12 14 16 The Sweet Spot
38. Had to make some adjustments to accommodate the fast tests.
39. All timestamps had to be in milliseconds We’re adding milliseconds Twitter returns only seconds
40. try { } catch(Exception $exception) {
 usleep(rand(100, 500));
 $this->insertInTable($record); 
 } $this->insertInTable($record); // too fast tests, too much deadlocks
41. 180 123 89 65 104 61 46 Execution Time in Minutes 5 Run in Parallel
42. Equalize Workload
43. Before Number of tests per thread 0 35 70 105 140 Thread # 1 2 3 4 5 6 7 8 9 10
44. 180 123 89 65 104 61 46 5 Execution Time in Minutes 3 Equal Batches Run in Parallel Don’t delete test data Run Databases in Memory Using Containers Stub Dependencies Empty Databases New Environment
45. After Hardware Upgrade The Outcome 2:15 min. 1:38 min.
46. The tests are slow The tests are unreliable The tests can’t exactly pinpoint the problem High Level Tests Problems 3 Minutes No external dependencies It’s cheap to run all tests after every commit
47. 51 minutes 12* minutes How about the UI tests? *Running in single thread
48. One more thing…
49. Log files - errors, exceptions After all the tests complete, check: Databases - wrong, unexpected data Execution time - load, performance issues
50. In a couple of years, running all your automated tests, after every code change, for less than 3 minutes, will be standard development practice.
51. How to Start
52. Create dedicated automation test environment Simulate external dependencies Your tests should create all the data they need Run in parallel and scale horizontally
53. Recommended Reading
54. EmanuilSlavov.com @EmanuilSlavov
56. Slide #, Photo Credits 1. https://www.flickr.com/photos/thomashawk 2. https://www.flickr.com/photos/paulineguilmot 10. https://www.flickr.com/photos/100497095@N02 13. https://www.flickr.com/photos/andrewmalone 19. https://www.flickr.com/photos/astrablog 25. https://www.flickr.com/photos/foilman 29. https://www.flickr.com/photos/missusdoubleyou 32. https://www.flickr.com/photos/canonsnapper 35. https://www.flickr.com/photos/anotherangle 42. https://www.flickr.com/photos/-aismist