Need for Speed - Accelerate Tests From 3 Hours to 3 Minutes

1f686da361195e15bb4e478397a4fc8f?s=47 emanuil
November 10, 2015

Need for Speed - Accelerate Tests From 3 Hours to 3 Minutes

All automated tests, other than unit, are slow and unreliable for the fast development pace every company craves. When a test fails, it’s hard to pinpoint the exact reason why. There are lots of external dependencies and factors outside of your control.

At Komfo, we had automated tests running for more than 3 hours every night in relatively large SaaS application. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable as feedback loop. At one point the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. We decided to stop this madness and after considerable effort and dedication, currently the same tests run for 3 minutes. This is the story of how we achieved nearly 60x faster tests.

We believe that in the near future this will be standard practice, much like unit tests or continuous integration now. In order for a company to stay competitive, _all_ existing automated tests — e.g. static code analysis, unit tests, API, UI, should complete in less than 5 minutes after _every_ code change.

The presentation is technical and touches on topics such as test automation framework design, hermetic servers, Docker containers, architecture for testability, test environments provisioning, DevOps collaboration, testing when depending on internal and external services, the joys and pitfalls of parallel execution.

1f686da361195e15bb4e478397a4fc8f?s=128

emanuil

November 10, 2015
Tweet

Transcript

  1. NEED FOR SPEED accelerate tests from 3 hours to 3

    minutes emo@komfo.com
  2. The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems
  3. This is Our Story

  4. 600 API (REST) tests 3 hours of execution time This

    was a big problem.
  5. Before After

  6. 3 hours 3 minutes

  7. 180 Execution Time in Minutes

  8. Key Steps

  9. Use Dedicated Environment

  10. Automated Tests Mongo MySQL Core API PHP/Java Developer Developer Developer

    Created new dedicated test environment!
  11. 180 123 Execution Time in Minutes New Environment

  12. Use Empty Databases

  13. Automated Tests Core API PHP/Java Developer Developer Developer Use empty

    databases Mongo MySQL
  14. Tests need to setup all the data they need!

  15. The average cost to setup every test case Call 12

    API endpoints Modify data in 11 tables And then the test starts
  16. Dump DB schema on every test run and restore it

    Only need DB schema and config tables (~20)
  17. 180 123 89 Execution Time in Minutes Empty Databases

  18. Simulate External Dependencies

  19. Problems with external dependencies Sketchy Internet Throttling API Calls Expiring

    Credentials
  20. MySQL Mongo Automated Tests +Some More STUB STUB STUB STUB

    STUB STUB STUB Stub all external dependencies Core API PHP/Java
  21. External stubs - standalone fake services Internal stubs - triggered

    based on request type
  22. Some of the tests still need to contact the real

    world.
  23. 180 123 89 65 Execution Time in Minutes Stub Dependencies

  24. Move to Containers

  25. Elastic Search Etcd Log stash Redis MySQL Mongo Core API

    PHP/Java Automated Tests Single server
  26. To cope with increasing complexity we created one container per

    service. But we we’re in for a surprise!
  27. 180 123 89 65 104 Execution Time in Minutes Using

    Containers
  28. Run Databases in Memory

  29. mysqld some_options --datadir /dev/shm Only in memory

  30. 180 123 89 65 104 61 Execution Time in Minutes

    Run Databases in Memory
  31. Don’t Clean Test Data

  32. The cost to delete data after every test case Call

    4 API endpoints Remove data from 23 tables Stop container, the data evaporates
  33. 180 123 89 65 104 61 46 Execution Time in

    Minutes Don’t delete test data
  34. Run in Parallel

  35. We can do this because every tests creates it’s own

    test data and is independent. This should be your last resort, after you’ve exhausted all other options.
  36. Execution Time (minutes) 0 4.5 9 13.5 18 Number of

    Threads 4 6 8 10 12 14 16 The Sweet Spot
  37. Had to make some adjustments to accommodate for the fast

    tests.
  38. All timestamps had to be in milliseconds Enhancing them to

    milliseconds Twitter returns only seconds
  39. try { } catch(Exception $exception) {
 usleep(rand(100, 500));
 $this->insertInTable($record); 


    } $this->insertInTable($record); // too fast tests, too much deadlocks
  40. 180 123 89 65 104 61 46 5 Execution Time

    in Minutes Run in Parallel
  41. Equalize Batch Sizes

  42. Before Number of tests per batch 0 35 70 105

    140 Batch # 1 2 3 4 5 6 7 8 9 10
  43. 180 123 89 65 104 61 46 5 3 Execution

    Time in Minutes Equal Batches
  44. The Outcome

  45. We did even better than 3 minutes!!!

  46. 51 minutes 12* minutes How about the UI tests? *Running

    in single thread
  47. The tests are slow The tests are unreliable The tests

    can’t exactly pinpoint the problem High Level Tests Problems 3 Minutes No external dependencies Cheap tests to run on every commit Awesomeness
  48. One more thing…

  49. Log files After every test run, check for errors DB

    tables Collections
  50. Being able to run all your tests in less than

    5 minutes after every code change is your target. In 3-5 years this will be standard practice.
  51. What’s Next

  52. Scale horizontally to keep the 3 min. threshold Automatic workarounds

    Docker bugs Compare CPU, Memory or DB consumption Run all tests after every DB schema change
  53. How to Start

  54. Create dedicated automation test environment Simulate external dependencies Your tests

    should create all the data they need Run in parallel and scale horizontally
  55. Recommended Reading

  56. EmanuilSlavov.com @EmanuilSlavov