[NGINX Meetup] Practicing Thoughtful Controlled Chaos Engineering

Why practice Chaos Engineering?
How to practice thoughtful controlled Chaos Engineering?
Also sharing a few findings from GameDays with customers.
#ChaosEngineering

Transcript

1. Practicing Thoughtful Controlled Chaos Engineering Ho Ming Li || NGINX meetup
2. Ho Ming Li @HoReaL @GremlinInc
3. Chaos Engineering Thoughtful, planned experiments designed to reveal the weakness in our systems.
4. Like a vaccine, we inject harm to build immunity.
5. Dedicated time for teams to collaboratively run Chaos Experiments to reveal weaknesses in your systems GameDay
6. 1. Why? 2. How?
10. “Computers aren’t the thing. They’re the thing that gets us to the thing.” - Halt and Catch Fire Chaos Engineering isn’t the thing. It’s the thing that gets us to Resilience.
11. Prime Down Amazon’s sale day turns into fail day TechCrunch Delta Outage Computer malfunction results in nationwide ground stop NBC Slack Outage Connectivity issues hit workplaces WSJ
12. $$$$$$$$$$$$$$$$$$$$ Reputation CX Employee Burnout
14. Better Fire Fighting ≠ More Resilient
18. Start Small Start Simple
20. Start in Staging Mature to Production
21. GameDay Anatomy of a GameDay Experiment #1 Experiment #2 Attack (Inject Failure) Attack Attack Attack ... ... Experiment #3 Attack Attack ...
22. Stop Guessing. Observe it!
23. Observability Get rid of the Fog of War so you can clearly see the map and strategize accordingly. Gain Deep Insight with: - Metrics - Logging - Request Tracing
24. Win/Lose Pass/Fail
25. Hypothesis Results Next Step Resilient Resilient Automate Fail Fail Improve Resilient Fail Dig Deeper Fail Resilient Dig Deeper
26. GameDay Findings Teaser
27. Sort of Expected Attack - Can’t connect to DynamoDB. Expectation - Frontend gets a 5XX Error from Backend.
28. Magnified Wait Attack - Inject small amount of latency between app and database Expectation - Users experience delay roughly same as injected latency
29. I can see this, but I can’t see that Attack - Consumer cannot connect to Database Expectation - Consumer can no longer process messages
30. Loosely coupled... … or Not Attack - Container dies Expectation - Orchestrator will spawn new container
31. “An Application” “Edge” DNS, CDN “Front End” LB, API “Back End” App/Web Server Queue, RDB, KV DB Search Index “Infrastructure”: Container Kubernetes Virtual Machine Physical Server Storage Network Data Center Geography
32. Don’t Forget the Human Last Updated: 04/01/2013 Last Validated: 02/01/2019
33. Reliably Yours Break Things on Purpose tinyurl.com/chaoseng meetup.com/pro/chaos
34. Hard Disk (Storage) NIC/Cables (Network) Power Supply Bugs in Apps Unpredictable Load Etc.
35. before
36. • A simple exercise or “box to check” • an opportunity to maliciously expose faulty services • A one time event • A high-risk endeavor What a GameDay isn’t:
37. What a GameDay is and can be • A dedicated time to come together to gain insights • The execution of one or more experiments • The proof or disproof of a hypothesis • A time to test, sometimes destructively, the resilience of your application and architecture