Need for Speed (VelocityConf Amsterdam, 2016)

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
November 07, 2016

Need for Speed (VelocityConf Amsterdam, 2016)

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why.

Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes.

Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution.

Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than three minutes after every code, database, or infrastructure change.

1f686da361195e15bb4e478397a4fc8f?s=128

emanuil

November 07, 2016
Tweet

Transcript

  1. 4.
  2. 5.

    The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems
  3. 8.

    It’s not about the numbers you’ll see or the techniques.

    It’s all about continuous improvement.
  4. 16.

    The time needed to create data for each test: And

    then the test starts Call 12 API endpoints Modify data in 11 tables Takes about 1.2 seconds
  5. 17.

    Get the current DB schema and apply it before the

    test run starts. Only the DB schema and config tables (~20) are needed.
  6. 21.

    +Some More STUB STUB STUB STUB STUB STUB STUB Stub

    all external dependencies Core API
  7. 22.

    Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary

    Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage We created project Nagual, soon on Github.
  8. 26.

    Elastic Search Etcd Log stash Redis MySQL Mongo Core API

    PHP/Java Automated Tests Single server
  9. 27.

    To cope with increasing complexity we created one container per

    service. But we we’re in for a surprise!
  10. 33.

    The cost to delete data after every test case Call

    4 API endpoints Remove data from 23 tables Or, stop the container, the data evaporates Takes about 1.5 seconds
  11. 34.
  12. 36.

    We can do this because every tests creates its own

    test data and is independent. This should be your last resort, after you’ve exhausted all other options.
  13. 37.

    Execution Time (minutes) 0 4.5 9 13.5 18 Number of

    Threads 4 6 8 10 12 14 16 The Sweet Spot
  14. 40.

    try { } catch(Exception $exception) {
 usleep(rand(100, 500));
 $this->insertInTable($record); 


    } $this->insertInTable($record); // too fast tests, too much deadlocks
  15. 41.

    180 123 89 65 104 61 46 Execution Time in

    Minutes 5 Run in Parallel
  16. 43.

    Before Number of tests per thread 0 35 70 105

    140 Thread # 1 2 3 4 5 6 7 8 9 10
  17. 44.

    180 123 89 65 104 61 46 5 Execution Time

    in Minutes 3 Equal Batches Run in Parallel Don’t delete test data Run Databases in Memory Using Containers Stub Dependencies Empty Databases New Environment
  18. 46.

    The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems 3 Minutes No external dependencies It’s cheap to run all tests after every commit
  19. 49.

    Log files - errors, exceptions After all the tests complete,

    check: Databases - wrong, unexpected data Execution time - load, performance issues
  20. 50.

    In a couple of years, running all your automated tests,

    after every code change, for less than 3 minutes, will be standard development practice.
  21. 52.

    Create dedicated automation test environment Simulate external dependencies Your tests

    should create all the data they need Run in parallel and scale horizontally
  22. 55.
  23. 56.

    Slide #, Photo Credits 1. https://www.flickr.com/photos/thomashawk 2. https://www.flickr.com/photos/paulineguilmot 10. https://www.flickr.com/photos/100497095@N02

    13. https://www.flickr.com/photos/andrewmalone 19. https://www.flickr.com/photos/astrablog 25. https://www.flickr.com/photos/foilman 29. https://www.flickr.com/photos/missusdoubleyou 32. https://www.flickr.com/photos/canonsnapper 35. https://www.flickr.com/photos/anotherangle 42. https://www.flickr.com/photos/-aismist