Test-automation and CI in practice

Test-automation and CI in practice

This is an updated version of the "TDD in the real world" talk I did in 2015 for Reykjavík University, original available here: https://speakerdeck.com/steinnes/tdd-in-the-real-world

36f7b3921277b2ed27eb4798c18266e4?s=128

Steinn Eldjárn Sigurðarson

October 31, 2016
Tweet

Transcript

  1. Test-automation and CI in practice Steinn Eldjárn Sigurðarson

  2. What is software engineering?

  3. Why do we test? 3

  4. Why do we test? • Not to prove correctness 4

  5. Why do we test? • To facilitate changes

  6. Why do we test? • To increase confidence

  7. When do we test? 7

  8. When do we test? • Small projects?

  9. When do we test? • LARGE projects?

  10. Story: QuizUp 1.0 -> 2.0

  11. QuizUp 1.0 • Few client tests • A lot of

    server tests • Server tests often too large and slow • .. or worse: flaky • Fairly simple app
  12. QuizUp 1.0 • Several server deployments every single day •

    Client releases were a lot more scary • “Bricked” ~30k installs with one release
  13. QuizUp 2.0 • Huge set of new features • Massively

    more complex app!
  14. None
  15. None
  16. QuizUp 2.0 • Excellent test coverage of clients • Total

    rewrite — it was hard, but it worked! • An order of magnitude more complex client • Maintained development velocity
  17. Different types of tests • Unit tests? • Integration tests?

    • System tests? • End-to-end tests? • … the evil ice cream!?
  18. None
  19. Different types of tests • Usually better to test the

    smallest possible units • Small tests are fast, rarely break and are run often • Large tests with many “moving parts” are slower and tend to break more often due to external factors: • network issues • race conditions • random / rare factors
  20. Different types of tests • What happens if you write

    large/flaky tests, well..
  21. Different types of tests http://imgur.com/j11G35h

  22. Story: Takumi

  23. Platform for Influencer Campaigns

  24. None
  25. Takumi Backend • Then: • ~5100 LOC vs. ~4400 LOC

    of tests • 35 seconds to run entire test suite • Unit tests run in 2 seconds and cover 57% • Integration tests in 30-35 seconds, cover 84% • Combined coverage: 91% • 97 unit tests, 258 integration tests
  26. Takumi Backend • Now: • ~10400 LOC vs. ~10100 LOC

    of tests • Server tests run in 2-3 minutes • Unit tests take 15 seconds and cover 78% • Integration tests 2-3 minutes and cover 68% • Combined coverage: 89% • 570 unit tests, 242 integration tests
  27. Takumi Backend • 1652 commits since launch • 337 ->

    849 pull requests • hundreds of deployments • 109 migrations
  28. Takumi App(s) • Timeline • April 2015 - October 2015:

    0.x.x • October 2015 - June 2016: 1.x.x • June 2016 - now: 2.x.x
  29. Takumi App(s) • 0.x.z had some tests. The bad kind.

    • 1.x.z had no tests, was written because 0.x.z was unstable • 2.x.z has tests, bugs are TDD’d, still working on optimal test patterns for RN components
  30. TDD ALL THE THINGS?

  31. Nope. Not really.

  32. To TDD or not to TDD? • Often yields clear,

    concise and testable code • It took me ~8 years to understand the role of tests • You can go far without ever TDD-ing • When in doubt, alternate between coding and testing • Think of test code as the code which exposes the clarity, or lack thereof, of your real code
  33. To TDD or not to TDD? • If you end

    up with a huge suite of tests asserting minor interactions with a huge mock: no • Think of TDD-ing and RG-refactoring like weightlifting: • They build core strength which comes in handy during everyday life, improves your performance in your activity of choice and reduces the chance of injury. • (if done correctly!)
  34. CI / CD • What’s the purpose of CI? •

    Automate test runs to prevent regressions • Increase confidence for deployment or release
  35. CD? • What’s the purpose of CD? • Brilliant for

    shared development/staging environments • When done for large scale systems, rollout is usually slow and easily abortable • Great for development velocity, challenging for service reliability
  36. Example CI pipeline

  37. The Takumi “pipeline”

  38. Thank you!