Testing in Production

How do you know your feature is working perfectly in production? If something breaks in production, how will you know? Will you wait for a user to report it to you? What do you do when your staging test results do not reflect current production behavior? In order to test proactively as opposed to reactively, try testing in production! You will have an increased accuracy of test results, your tests will run faster due to the elimination of bad data, and you will have higher confidence before releases. This can be accomplished through feature flagging, canary releases, setting up a proper CI/CD pipeline, and data cleanup. You will leave this talk with strategies to mitigate risk, to better your understanding of the steps to get there, and to shift your company’s testing culture, so you can provide the best possible experience to your users. At the end of the day, we don't care if your features work in staging, we care if they work in production.

Transcript

1. Testing in Production Talia Nassi
2. @talia_nassi talia.nassi@split.io Hi
3. “I love my staging environment” ~No one ever
4. DATA MISMATCH Configuration drift Test results do not match
5. WHAT’S THE POINT?
6. Staging = slow
8. What I Learned Staging = norm Multiple staging envs Companies are testing in prod
9. Testing in production means testing your features in the environment that your features will live in.
10. All tests e2e Flows Production Tests
11. Feature flags
12. Code deployment Feature Release
13. PRODUCTION Feature Flag Target Users: Devs Testers Product Design Only these people see the changes These people (end users) do not see any changes
14. PRODUCTION Feature Flag Target Users: Devs Testers Product Design These people see the bugs and fix them These people (end users) do not see any bugs
15. PRODUCTION Feature Flag Target Users: Devs Testers Product Design Everyone sees the new feature
16. That’s all wonderful... ...but how do you automate it?
17. Set the state of the flag for the duration of the test OVERRIDE YOUR FEATURE FLAGS
18. Test user Real Object Test Object PRODUCTION is_test = true is_test = true is_test = false
19. Email confirmations, third parties. etc. There are exceptions
20. Automation Framework Job Scheduler Alerting Tool
21. Recommended Tools Feature Flagging Automation Framework Job Scheduler Alerting
23. FEAR
24. Feature flags before launch Canary Release after launch Start with an AA test Start small Risk Mitigation
25. Increased confidence in releases Increased developer velocity Release Faster Increased team happiness Outcome of Testing in Prod
26. Think of the last feature your team deployed Is it working? Right now? In Production? How do you know?
27. Testing in production is the only way to know your features are working right now in production
28. Use examples from your past Start using feature flags for free! split.io Culture Shift Is your staging environment unreliable?
29. No one cares if your feature is working in staging, we care if it works in production The only way to know if it’s working in production is to test it in production
30. Backend Engineer Data Infrastructure Engineer Frontend Engineer Infrastructure Engineer SDK Engineer Sr. Backend Engineer We’re hiring! split.io/careers
31. @talia_nassi talia.nassi@split.io Hi