Testing in Production: Dangerous, scary or better?

Slides from Test Bash Belfast 2017

Transcript

1. Prepared by Jon Hare-Winton Testing in Production: dangerous, scary… or better? Jon Hare-Winton @jonhw87 1
2. @jonhw87 2 Testing in Production: Dangerous, scary or better w • What is it? • Why? • What are the challenges? • Taking it further - an example • Lessons learnt
3. @jonhw87 3 Testing in Production: Dangerous, scary or better Traditional flow Develop Test Deploy Check LOCAL Test / staging PRODUCTION ???
4. @jonhw87 4 Testing in Production: Dangerous, scary or better w Could we…test…in prod?
5. @jonhw87 5 Testing in Production: Dangerous, scary or better w Test environments allow us to: • Test before live • Make a mess… • And not clear it up • Have total control
6. @jonhw87 6 Testing in Production: Dangerous, scary or better w Test environments can also be: • Maintenance heavy • Lower priority than prod • Isolated • Inaccurate
7. @jonhw87 7 Testing in Production: Dangerous, scary or better Traditional flow Develop Test Deploy Check LOCAL Test / staging PRODUCTION ???
8. @jonhw87 8 Testing in Production: Dangerous, scary or better w Monitoring What it can What it can’t API / HTTP responses Integrated user journey Machine status Traffic levels Crash reporting
9. @jonhw87 9 Testing in Production: Dangerous, scary or better w Rob Ewaschuk: ‘Do your users care if your MySQL servers are down? No, they care if their queries are failing’ https://docs.google.com/document/d/1XrXehEJ27vQLqsIpm8b5Wgtxd1Xv262t1Ywu6HBTcfg/
10. @jonhw87 10 Testing in Production: Dangerous, scary or better w Agile, devops etc... = moving faster
11. @jonhw87 11 Testing in Production: Dangerous, scary or better w But prod!?
12. @jonhw87 12 Testing in Production: Dangerous, scary or better w Fears for testing in production: • What if users notice? ◦ Something broke! ◦ A test while it’s running ◦ Test data ◦ Test residue ◦ Other evidence of test activity
13. @jonhw87 13 Testing in Production: Dangerous, scary or better w Production Testing = Keep it hidden!
14. @jonhw87 14 Testing in Production: Dangerous, scary or better w In practice...
15. @jonhw87 15 Testing in Production: Dangerous, scary or better w In the pipeline…but in production • When a test environment is not: ◦ Possible ◦ Practical ◦ Affordable • Keep it hidden = keep your release hidden? • Requires fast recovery
16. @jonhw87 16 Testing in Production: Dangerous, scary or better w …or as monitoring (synthetic monitoring)
17. @jonhw87 17 Testing in Production: Dangerous, scary or better w The tests: • Create a piece of content • Follow it through the entire stack • Take down the content • Confirm through the stack
18. @jonhw87 18 Testing in Production: Dangerous, scary or better w The challenges: • Hiding test content • Continual running • The tests • Visibility of results
19. @jonhw87 19 Testing in Production: Dangerous, scary or better w Hiding test content… ...in more places than you think
20. @jonhw87 20 Testing in Production: Dangerous, scary or better w Continuous running...technically
21. @jonhw87 21 Testing in Production: Dangerous, scary or better w
22. @jonhw87 22 Testing in Production: Dangerous, scary or better w
23. @jonhw87 23 Testing in Production: Dangerous, scary or better w Continuous running… ...and unexpected outcomes
24. @jonhw87 24 Testing in Production: Dangerous, scary or better w Continuous running… ...accuracy
25. @jonhw87 25 Testing in Production: Dangerous, scary or better w Example CI tests: 15 runs per day / 95% accuracy = <4 failures per week
26. @jonhw87 26 Testing in Production: Dangerous, scary or better w Synthetic monitoring tests: 3,000 runs per day / 95% accuracy = 150 failures per day
27. @jonhw87 27 Testing in Production: Dangerous, scary or better w Tests
28. @jonhw87 28 Testing in Production: Dangerous, scary or better w Better webdriver... • Assert on the API / Data • Everything conditional • No unnecessary UI actions
29. @jonhw87 29 Testing in Production: Dangerous, scary or better w Accuracy…. ...more accuracy…. ...and tolerance
30. @jonhw87 30 Testing in Production: Dangerous, scary or better w Visibility of results
31. @jonhw87 31 Testing in Production: Dangerous, scary or better w CI tests: Commit / merge Test Release Fix
32. @jonhw87 32 Testing in Production: Dangerous, scary or better w Synthetic monitoring tests: Continuous test
33. @jonhw87 33 Testing in Production: Dangerous, scary or better w
34. @jonhw87 34 Testing in Production: Dangerous, scary or better w
35. @jonhw87 35 Testing in Production: Dangerous, scary or better w Benefits: • Better awareness • Accurate alerting • Easier & faster investigation • Use beyond the team
36. @jonhw87 36 Testing in Production: Dangerous, scary or better w Lessons learnt • Not scary • Not dangerous • Better?
37. 37 Thanks! jon.hare-winton@theguardian.com @jonhw87 @jonhw87 Testing in Production: Dangerous, scary or better w