$30 off During Our Annual Pro Sale. View Details »

Need for Speed (VelocityConf Amsterdam, 2016)

emanuil
November 07, 2016

Need for Speed (VelocityConf Amsterdam, 2016)

All automated tests (other than unit) are too slow and unreliable for the fast development pace every company craves, and there are many external dependencies and factors outside of your control, so when these tests fail, it’s hard to pinpoint the exact reason why.

Komfo had automated tests running for more than three hours every night. The execution time just kept growing unrestricted, and the tests were getting more unstable and unusable due to a feedback loop. At one point, the continuous integration build for the tests was red for more than 20 days in a row. Regression bugs started to appear undetected in production. The Komfo team decided to put an end to this madness—after considerable effort and dedication, the same tests currently run for only three minutes.

Emanuil Slavov tells the story of how Komfo achieved 60x faster tests. Along the way, Emanuil touches on topics such as test automation framework design, hermetic servers, optimizing test environments for speed, architecture for testability, DevOps collaboration, virtual containers, reliable tests when depending on services outside of your control, and the joys and pitfalls of parallel execution.

Having automated tests run that fast completely changes the software development game. There is no need to wait for the nightly tests to complete, and there are no more smoke tests. You can run all tests, get fast feedback, and release instantly without falling behind. In the near future, this will be standard practice, much like infrastructure as code or continuous integration are today. In order for a company to stay competitive, all existing automated tests—static code analysis, unit tests, API, and UI—should complete in less than three minutes after every code, database, or infrastructure change.

emanuil

November 07, 2016
Tweet

More Decks by emanuil

Other Decks in Programming

Transcript

  1. NEED FOR SPEED
    accelerate tests from 3 hours to 3 minutes
    [email protected]

    View Slide

  2. With slow tests you’re
    shipping crap faster.

    View Slide

  3. Everyone Loves Unit Tests

    View Slide

  4. View Slide

  5. The tests are slow
    The tests are unreliable
    The tests can’t exactly pinpoint the problem
    High Level Tests Problems

    View Slide

  6. 3
    hours
    3
    minutes
    600 API tests

    View Slide

  7. Before After
    The
    3 Minute
    Goal

    View Slide

  8. It’s not about the numbers you’ll see
    or the techniques.
    It’s all about continuous improvement.

    View Slide

  9. Key Steps

    View Slide

  10. Dedicated
    Environment

    View Slide

  11. Developer Developer Developer
    Created new dedicated
    test environment.
    Automated
    Tests
    MySQL
    Mongo
    Core API
    PHP/Java

    View Slide

  12. Execution Time in Minutes
    180
    123
    New Environment

    View Slide

  13. Empty Databases

    View Slide

  14. Use empty databases
    Developer Developer Developer
    Automated
    Tests
    MySQL
    Mongo
    Core API
    PHP/Java

    View Slide

  15. Tests need to setup all the data they need!

    View Slide

  16. The time needed to create data for each test:
    And then the test starts
    Call 12 API endpoints
    Modify data in 11 tables
    Takes about 1.2 seconds

    View Slide

  17. Get the current DB schema and apply it
    before the test run starts.
    Only the DB schema and config tables (~20) are
    needed.

    View Slide

  18. 180
    123
    Execution Time in Minutes
    89
    Empty Databases

    View Slide

  19. Simulate
    Dependencies

    View Slide

  20. Problems with external dependencies
    Sketchy Internet
    Throttling API Calls
    Expiring Credentials

    View Slide

  21. +Some
    More
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    STUB
    Stub all external dependencies
    Core API

    View Slide

  22. Transparent
    Fake SSL certs
    Dynamic Responses
    Local Storage
    Return Binary Data
    Regex URL match
    Existing Tools (March 2016)
    Stubby4J
    WireMock
    Wilma
    soapUI
    MockServer
    mounteback
    Hoverfly
    Mirage
    We created project Nagual,
    soon on Github.

    View Slide

  23. Some of the tests still need to contact
    the real world.

    View Slide

  24. 180
    123
    89
    Execution Time in Minutes
    65
    Stub Dependencies

    View Slide

  25. Move to Containers

    View Slide

  26. Elastic
    Search
    Etcd
    Log
    stash
    Redis
    MySQL
    Mongo
    Core API
    PHP/Java
    Automated
    Tests
    Single server

    View Slide

  27. To cope with increasing complexity
    we created one container per service.
    But we we’re in for a surprise!

    View Slide

  28. 180
    123
    89
    65
    Execution Time in Minutes
    104
    Using Containers

    View Slide

  29. Run Databases
    in Memory

    View Slide

  30. mysqld some_options --datadir /dev/shm
    Only in memory

    View Slide

  31. 180
    123
    89
    65
    104
    Execution Time in Minutes
    61
    Run Databases in Memory

    View Slide

  32. Don’t Clean
    Test Data

    View Slide

  33. The cost to delete data after every test case
    Call 4 API endpoints
    Remove data from 23 tables
    Or, stop the container, the data evaporates
    Takes about 1.5 seconds

    View Slide

  34. 180
    123
    89
    65
    104
    61
    Execution Time in Minutes
    46
    Don’t delete test data

    View Slide

  35. Run in Parallel

    View Slide

  36. We can do this because every tests creates its
    own test data and is independent.
    This should be your last resort, after you’ve
    exhausted all other options.

    View Slide

  37. Execution Time (minutes)
    0
    4.5
    9
    13.5
    18
    Number of Threads
    4 6 8 10 12 14 16
    The Sweet Spot

    View Slide

  38. Had to make some adjustments
    to accommodate the fast tests.

    View Slide

  39. All timestamps had to be in milliseconds
    We’re adding milliseconds
    Twitter returns only seconds

    View Slide

  40. try {



    }

    catch(Exception $exception) {

    usleep(rand(100, 500));

    $this->insertInTable($record); 

    }
    $this->insertInTable($record);
    // too fast tests, too much deadlocks

    View Slide

  41. 180
    123
    89
    65
    104
    61
    46
    Execution Time in Minutes
    5
    Run in Parallel

    View Slide

  42. Equalize Workload

    View Slide

  43. Before
    Number of tests per thread
    0
    35
    70
    105
    140
    Thread #
    1 2 3 4 5 6 7 8 9 10

    View Slide

  44. 180
    123
    89
    65
    104
    61
    46
    5
    Execution Time in Minutes
    3
    Equal Batches
    Run in Parallel
    Don’t delete test data
    Run Databases in Memory
    Using Containers
    Stub Dependencies
    Empty Databases
    New Environment

    View Slide

  45. After Hardware Upgrade
    The Outcome
    2:15 min.
    1:38 min.

    View Slide

  46. The tests are slow
    The tests are unreliable
    The tests can’t exactly pinpoint the problem
    High Level Tests Problems
    3 Minutes
    No external dependencies
    It’s cheap to run all tests after every commit

    View Slide

  47. 51
    minutes
    12*
    minutes
    How about the UI tests?
    *Running in single thread

    View Slide

  48. One more thing…

    View Slide

  49. Log files - errors, exceptions
    After all the tests complete, check:
    Databases - wrong, unexpected data
    Execution time - load, performance issues

    View Slide

  50. In a couple of years, running all your
    automated tests, after every code
    change, for less than 3 minutes, will be
    standard development practice.

    View Slide

  51. How to Start

    View Slide

  52. Create dedicated automation test environment
    Simulate external dependencies
    Your tests should create all the data they need
    Run in parallel and scale horizontally

    View Slide

  53. Recommended Reading

    View Slide

  54. EmanuilSlavov.com
    @EmanuilSlavov

    View Slide

  55. View Slide

  56. Slide #, Photo Credits
    1. https://www.flickr.com/photos/thomashawk
    2. https://www.flickr.com/photos/paulineguilmot
    10. https://www.flickr.com/photos/100497095@N02
    13. https://www.flickr.com/photos/andrewmalone
    19. https://www.flickr.com/photos/astrablog
    25. https://www.flickr.com/photos/foilman
    29. https://www.flickr.com/photos/missusdoubleyou
    32. https://www.flickr.com/photos/canonsnapper
    35. https://www.flickr.com/photos/anotherangle
    42. https://www.flickr.com/photos/-aismist

    View Slide