Test-automation in practice

Updated version of my yearly talk about some real world testing / agile development examples and stories held as a guest at University of Reykjavík.

Transcript

1. Test-automation and CI in practice Steinn Eldjárn Sigurðarson
2. What is software engineering?
3. Why do we test? 3
4. Why do we test? • Not to prove correctness 4
5. Why do we test? • To facilitate changes
6. Why do we test? • To increase confidence
7. QuizUp 1.0 -> 2.0
8. QuizUp 1.0 • Few client tests • A lot of server tests • Server tests often too large and slow • .. or worse: flaky • Fairly simple app
9. QuizUp 1.0 • Several server deployments every single day • Client releases were a lot more scary • “Bricked” ~30k installs with one release
10. QuizUp 2.0 • Huge set of new features • Massively more complex app!
13. QuizUp 2.0 • Excellent test coverage of clients • Total rewrite — it was hard, but it worked! • An order of magnitude more complex client • Maintained development velocity
14. TAKUMI
15. Platform for Influencer Campaigns
17. Takumi Backend • At launch: • ~5100 LOC vs. ~4400 LOC of tests • 35 seconds to run entire test suite • Unit tests run in 2 seconds and cover 57% • Integration tests in 30-35 seconds, cover 84% • Combined coverage: 91% • 97 unit tests, 258 integration tests
18. Takumi Backend • Two years later: • ~16300 LOC vs. ~11900 LOC of tests • Server tests run in 2-3 minutes • Unit tests take 24 seconds and cover 77% • Integration tests take 2 minutes and cover 63% • Combined coverage: 83% • 940 unit tests, 197 integration tests
19. Takumi Backend • 1652 commits since launch • 337 -> 1334 pull requests • hundreds of deployments • 184 migrations
20. Takumi Mobile App • Three from-scratch implementations in a year • First prototype written in 5 months • Rewritten in 2 months before launch • Rewritten in React-Native 6 months after launch
21. Takumi Mobile App • First prototype had some tests • First rewrite had no tests, but a lot less code • Current version has tests, but we still rely on a lot of manual QA • Still learning how to test and TDD
22. CI / CD • What’s the purpose of CI? • Automate test runs to prevent regressions • Increase confidence for deployment or release
23. CD? • What’s the purpose of CD? • Brilliant for shared development/staging environments • When done for large scale systems, rollout is usually slow and easily abortable • Great for development velocity, challenging for service reliability
24. Example CI pipeline
25. The Takumi “pipeline”
26. Thank you!