Testing in Production: Dangerous, scary or better?

We all run tests on safe, separate test environments. But what if let them loose on production systems? Is that dangerous, or is it worth the risk? This can seem daunting, but there are huge benefits. I’ll discuss some of the downsides to testing in isolated, unrepresentative test environments, before talking about my experiences of using Selenium to test in production as much as possible. I’ll demonstrate the benefits, like how we can turn automated tests into monitoring and early alerting systems for our production environments, and the steps we need to take to make sure our production tests don’t affect our users.

Transcript

1. Prepared by Jon Hare-Winton Testing in Production: dangerous, scary… or better? Jon Hare-Winton @jonhw87 1
2. @jonhw87 2 Testing in Production: Dangerous, scary or better w • What is it? • Why? • What are the challenges? • Taking it further - an example • Lessons learnt
3. @jonhw87 3 Testing in Production: Dangerous, scary or better Traditional flow Develop Test Deploy Check LOCAL Test / staging PRODUCTION ???
4. @jonhw87 4 Testing in Production: Dangerous, scary or better w Could we…test…in prod?
5. @jonhw87 5 Testing in Production: Dangerous, scary or better w Test environments allow us to: • Make a mess… • And not clear it up • Test before live • Have total control
6. @jonhw87 6 Testing in Production: Dangerous, scary or better w Test environments can also be: • Maintenance heavy • Lower priority than prod • Isolated • Inaccurate
7. @jonhw87 7 Testing in Production: Dangerous, scary or better Traditional flow Develop Test Deploy Check LOCAL Test / staging PRODUCTION ???
8. @jonhw87 8 Testing in Production: Dangerous, scary or better w Monitoring What it can What it can’t API / HTTP responses Integrated user journey Machine status Traffic levels Crash reporting
9. @jonhw87 9 Testing in Production: Dangerous, scary or better w
10. @jonhw87 10 Testing in Production: Dangerous, scary or better w But prod!?
11. @jonhw87 11 Testing in Production: Dangerous, scary or better w Fears for testing in production: • What if users notice? ◦ A test while it’s running ◦ Test data ◦ Test residue ◦ Other evidence of test activity
12. @jonhw87 12 Testing in Production: Dangerous, scary or better w Production Testing = Keep it hidden!
13. @jonhw87 13 Testing in Production: Dangerous, scary or better w In practice...
14. @jonhw87 14 Testing in Production: Dangerous, scary or better w In the pipeline...
15. @jonhw87 15 Testing in Production: Dangerous, scary or better w …or as monitoring
16. @jonhw87 16 Testing in Production: Dangerous, scary or better w
17. @jonhw87 17 Testing in Production: Dangerous, scary or better w The tests: • Create a piece of content • Follow it through the entire stack • Take down the content • Confirm through the stack
18. @jonhw87 18 Testing in Production: Dangerous, scary or better w
19. @jonhw87 19 Testing in Production: Dangerous, scary or better w The challenges: • Hiding test content • The tests • Continual running • Visibility of results
20. @jonhw87 20 Testing in Production: Dangerous, scary or better w Hiding test content
21. @jonhw87 21 Testing in Production: Dangerous, scary or better w Tests
22. @jonhw87 22 Testing in Production: Dangerous, scary or better w Continuous running
23. @jonhw87 23 Testing in Production: Dangerous, scary or better w
24. @jonhw87 24 Testing in Production: Dangerous, scary or better w Continuous running
25. @jonhw87 25 Testing in Production: Dangerous, scary or better w Continuous running
26. @jonhw87 26 Testing in Production: Dangerous, scary or better w Visibility of results
27. @jonhw87 27 Testing in Production: Dangerous, scary or better w
28. @jonhw87 28 Testing in Production: Dangerous, scary or better w
29. @jonhw87 29 Testing in Production: Dangerous, scary or better w Benefits: • Better awareness • Easier & faster investigation • Use beyond the team
30. @jonhw87 30 Testing in Production: Dangerous, scary or better w Lessons learnt • Not scary • Not dangerous • Better?
31. 31 Thanks! jon.hare-winton@theguardian.com @jonhw87 @jonhw87 Testing in Production: Dangerous, scary or better w
32. @jonhw87 32 Testing in Production: Dangerous, scary or better w If you want to do this: • Business critical flows / functions • Hide it! • Traditional flow vs monitoring