Need for Speed (VelocityConf Amsterdam, 2016)

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
November 07, 2016

Need for Speed (VelocityConf Amsterdam, 2016)

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why.

Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes.

Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution.

Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than three minutes after every code, database, or infrastructure change.

1f686da361195e15bb4e478397a4fc8f?s=128

emanuil

November 07, 2016
Tweet

Transcript

  1. NEED FOR SPEED accelerate tests from 3 hours to 3

    minutes emo@komfo.com
  2. With slow tests you’re shipping crap faster.

  3. Everyone Loves Unit Tests

  4. None
  5. The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems
  6. 3 hours 3 minutes 600 API tests

  7. Before After The 3 Minute Goal

  8. It’s not about the numbers you’ll see or the techniques.

    It’s all about continuous improvement.
  9. Key Steps

  10. Dedicated Environment

  11. Developer Developer Developer Created new dedicated test environment. Automated Tests

    MySQL Mongo Core API PHP/Java
  12. Execution Time in Minutes 180 123 New Environment

  13. Empty Databases

  14. Use empty databases Developer Developer Developer Automated Tests MySQL Mongo

    Core API PHP/Java
  15. Tests need to setup all the data they need!

  16. The time needed to create data for each test: And

    then the test starts Call 12 API endpoints Modify data in 11 tables Takes about 1.2 seconds
  17. Get the current DB schema and apply it before the

    test run starts. Only the DB schema and config tables (~20) are needed.
  18. 180 123 Execution Time in Minutes 89 Empty Databases

  19. Simulate Dependencies

  20. Problems with external dependencies Sketchy Internet Throttling API Calls Expiring

    Credentials
  21. +Some More STUB STUB STUB STUB STUB STUB STUB Stub

    all external dependencies Core API
  22. Transparent Fake SSL certs Dynamic Responses Local Storage Return Binary

    Data Regex URL match Existing Tools (March 2016) Stubby4J WireMock Wilma soapUI MockServer mounteback Hoverfly Mirage We created project Nagual, soon on Github.
  23. Some of the tests still need to contact the real

    world.
  24. 180 123 89 Execution Time in Minutes 65 Stub Dependencies

  25. Move to Containers

  26. Elastic Search Etcd Log stash Redis MySQL Mongo Core API

    PHP/Java Automated Tests Single server
  27. To cope with increasing complexity we created one container per

    service. But we we’re in for a surprise!
  28. 180 123 89 65 Execution Time in Minutes 104 Using

    Containers
  29. Run Databases in Memory

  30. mysqld some_options --datadir /dev/shm Only in memory

  31. 180 123 89 65 104 Execution Time in Minutes 61

    Run Databases in Memory
  32. Don’t Clean Test Data

  33. The cost to delete data after every test case Call

    4 API endpoints Remove data from 23 tables Or, stop the container, the data evaporates Takes about 1.5 seconds
  34. 180 123 89 65 104 61 Execution Time in Minutes

    46 Don’t delete test data
  35. Run in Parallel

  36. We can do this because every tests creates its own

    test data and is independent. This should be your last resort, after you’ve exhausted all other options.
  37. Execution Time (minutes) 0 4.5 9 13.5 18 Number of

    Threads 4 6 8 10 12 14 16 The Sweet Spot
  38. Had to make some adjustments to accommodate the fast tests.

  39. All timestamps had to be in milliseconds We’re adding milliseconds

    Twitter returns only seconds
  40. try { } catch(Exception $exception) {
 usleep(rand(100, 500));
 $this->insertInTable($record); 


    } $this->insertInTable($record); // too fast tests, too much deadlocks
  41. 180 123 89 65 104 61 46 Execution Time in

    Minutes 5 Run in Parallel
  42. Equalize Workload

  43. Before Number of tests per thread 0 35 70 105

    140 Thread # 1 2 3 4 5 6 7 8 9 10
  44. 180 123 89 65 104 61 46 5 Execution Time

    in Minutes 3 Equal Batches Run in Parallel Don’t delete test data Run Databases in Memory Using Containers Stub Dependencies Empty Databases New Environment
  45. After Hardware Upgrade The Outcome 2:15 min. 1:38 min.

  46. The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems 3 Minutes No external dependencies It’s cheap to run all tests after every commit
  47. 51 minutes 12* minutes How about the UI tests? *Running

    in single thread
  48. One more thing…

  49. Log files - errors, exceptions After all the tests complete,

    check: Databases - wrong, unexpected data Execution time - load, performance issues
  50. In a couple of years, running all your automated tests,

    after every code change, for less than 3 minutes, will be standard development practice.
  51. How to Start

  52. Create dedicated automation test environment Simulate external dependencies Your tests

    should create all the data they need Run in parallel and scale horizontally
  53. Recommended Reading

  54. EmanuilSlavov.com @EmanuilSlavov

  55. None
  56. Slide #, Photo Credits 1. https://www.flickr.com/photos/thomashawk 2. https://www.flickr.com/photos/paulineguilmot 10. https://www.flickr.com/photos/100497095@N02

    13. https://www.flickr.com/photos/andrewmalone 19. https://www.flickr.com/photos/astrablog 25. https://www.flickr.com/photos/foilman 29. https://www.flickr.com/photos/missusdoubleyou 32. https://www.flickr.com/photos/canonsnapper 35. https://www.flickr.com/photos/anotherangle 42. https://www.flickr.com/photos/-aismist