[GOTOBER] GameDays: Practice Thoughtful Chaos Engineering

GameDay is a dedicated time to intentionally create failure scenarios in a safe environment. Regularly running GameDays is an effective Chaos Engineering practice to test the resiliency of your services; to validate the technical intricacies, and to also surface conversations around observability and incident management. GameDays can also expose you to blind spots when systems are operating under suboptimal conditions. In this talk, Ho Ming will be sharing what it takes to run successful GameDays.

Transcript

1. GameDays Practice Thoughtful Chaos Engineering (Gamer Edition) Ho Ming Li Lead Solutions Architect @ Gremlin
2. Lead Solutions Architect @ Gremlin Ex - AWS, NetApp, IBM Ran 10+ GameDays With 5+ Companies a retired gamer Ho Ming Li @HoReaL
3. Rainbows Unicorns
4. Reality Fire Fighting
5. Own Your User’s Experience Because Everything Fails
6. Chaos Engineering Thoughtful, planned, experiments designed to reveal weaknesses in your systems
7. Chaos Engineering Engineer Chaos… ...on your own terms
8. Like a vaccine, we inject harm to build immunity.
9. GameDay Dedicated time for teams to collaboratively focus on using Chaos Engineering practices to reveal weaknesses in your systems
10. GameDay Let’s Run some Chaos Experiments Together Day
11. “Why we do what we do?”
13. “Computers aren’t the thing. They’re the thing that gets us to the thing.” - Halt and Catch Fire Chaos Engineering isn’t the thing. It’s the thing that gets us to Resilience.
14. Know your Objective
15. Why resilience matters - Motivations Business Case Avoid very costly downtime. Engineering Case Improve quality and increase agility. On-Call Case Avoid constant fire fighting and pager fatigue.
19. Level 1 Level 80+
20. Challenge to Scale.
21. Inevitably die a horrible death.
22. Let’s do… Active-Active Multi-Region AI-Enabled Detection with Auto-magic Recovery System Using Request Tokens on Globally Decentralized Blockchain But… Can you withstand a critical host going down right now?
23. Start at your level. Make progress.
24. Start Small Start Simple
25. Now, this is You Done your homework Ready to go Go take on the world alone?
26. Form your Party Everyone brings something different to the table. Together Everyone Achieves More
27. CTO / VP Engineering Budget / Objectives Organizer Invitations / Coordination Engineering Director / Manager Prioritization / Engineer Availability Engineers / Subject Matter Expert Architecture / Experiments New Hires / Interns Learning / New Perspectives Other Stakeholders / Observers Understand Impact on their Functions Chaos Party
28. It’s more Fun with Friends
29. “Where do we Start?”
32. Start in Staging Mature to Production
33. Anatomy of a GameDay GameDay Experiment #1 Experiment #2 Attack (Inject Failure) Attack Attack Attack ... ... Experiment #3 Attack Attack ...
35. What could go wrong? What went wrong?
37. What is the scope of the attack?
40. DON’T START HERE
41. START HERE
45. “But I know already what will happen.”
46. World Cup 2018 Guess which team won? If they play again, do you know which team will win? vs vs vs
47. World Cup 2018 Guess which team won? If they play again, do you know which team will win? 1(3) - 1(4) 0 - 2 0 - 1
49. Stop Guessing Prove it!
50. Scenarios Terminate Hosts Inject Latency Consume CPU Fill up Disk Bad Hosts Going Offline Service is Slow Workload runs Hot Logs not Rotating Attacks
51. What to use? Bare Hand Knife Machine Gun Sniper Rifle Shotgun
52. Equip Yourself Chaos Monkey Gremlin Pumba Toxiproxy Chaos Toolkit PowerfulSeal
53. “What am I looking at?”
54. Observibility Get rid of the Fog of War so you can clearly see the map and strategize accordingly. Gain Deep Insight with: - Metrics - Logging - Request Tracing
55. Win/Lose Pass/Fail
56. Level Up Manual Runs - Exploratory - GameDays Automated Runs - Scheduled Runs - Include in Pipeline
57. GameDay #1 .. N GameDay is not just a one time event Think about the next GameDay Track and Measure Success over time
58. Practice Practice Practice
59. “I don’t need Chaos, I do Serverless”
60. Application Level User Experience
61. Chaos All the Things Very Loosely Break down of “an application” “Edge”: DNS, CDN, Cloudflare “Front End”: LB, API, Nginx “Back End”: MySQL, Kafka, ES “Infrastructure”: Kubernetes, Container, Virtual Machine, Physical Server, Data Center "AWS Lambda protects you against some infrastructure failures, but you still need to defend against weakness in your own code." -- Yan Cui, Principal Engineer at DAZN
62. Don’t Forget the Human
63. GameDay Findings Teaser
64. Sort of Expected Attack: Disconnect from DynamoDB. Hypothesis: Frontend gets a 5XX Error from Backend.
65. Magnified Wait App Server to Database slowed Attack: Inject small amount of latency between services Hypothesis: Users experience slowness roughly equates to the injected delay
66. I can see this, but I can’t see that Requiring Database to process messages from Queue Attack: Consumer cannot connect to Database. Hypothesis: Consumer can no longer process messages.
67. Loosely coupled... … or Not Orchestrating Containers in Microservices Attack: Container dies. Hypothesis: Orchestrator will spawn new container.
68. Game On! Start your Journey tinyurl.com/chaoseng meetup.com/pro/chaos