Optimizing Monitoring with Chaos Engineering

How do you monitor a distributed system with emergent properties? Thoughtful Chaos Engineering experiments help us build confidence that our monitoring and dashboards will give us the data we need when it matters most.

Transcript

1. Optimizing Monitoring with Chaos Engineering Austin Gunter Gremlin @austingunter
2. How Do you monitor a large, emergent system? 2
3. Chaos Engineering • A way to validate our monitoring and alerting by testing an explicit hypothesis • A method to build confidence that our systems will give us useful information when we need it most
5. Let’s test our monitoring!
6. Gremlin’s Monitoring Gameday 02 03 API Server Shutdown SNS Unreachable from API Server Slow response from DynamoDB Experiments To Simulate -- -- -- What happens when a server crashes or is taken out of service? An outage with the Amazon Simple notification Service (SNS). An unpredictable latency spike with the Amazon DynamoDB service. 01
7. GameDay Dashboard
8. Experiment 1: API Server Shutdown Target: Tag service:gremlin-api, random 1 Gremlin: State > Shutdown Parameters: Delay for 0m, Reboot (one with, one without) Hypothesis: Shutdown API servers should be detected, and terminated. A replacement should launch within 5 minutes.
9. Results Observations: Failed hypothesis. AutoScalingGroup and load balancer were too slow to pick up on shutdown. Follow up Actions: Tune health checks so instances are marked unhealthy sooner. Experiment 1: API Server Shutdown
10. Experiment 2: SNS Unreachable from API Server Target: Tag service:gremlin-api, [zone:us-west-1a], Random 100% Gremlin: Network > Blackhole Parameters: Length 60s (1m) → 300s (5m), Hostname sns.us-west-1.amazonaws.com Hypothesis: Unreachable SNS should delay or fail Tier 2/Tier 3 services (Slack integration, Mixpanel, Salesforce, etc).
11. Experiment 2: SNS Unreachable from API Server Results Observations: Passed the first experiment (60s). Didn’t pass the second (300s). Follow up Actions: Improve SNS monitoring. Hard to know which attack is which because there is no GUID displayed with the Slack integration.
12. Experiment 3: Slow Response from Database Target: Tag service:gremlin-api, Random 50% →100% Gremlin: Network > Latency Parameters: Length 60s (1m) → 300s (5m), Milliseconds 100ms → 200ms, Hostname dynamodb.us-west-1.amazonaws.com Hypothesis: DB calls time out after 500ms, or if the total request time exceeds 1.5s. API timeout should return a 500 error.
13. Experiment 3: Slow Response from Database Results Observations • UI Failures, completed attacks aren’t showing up • 4 database failures and 4 API failures... 2% error rate • With the second attack (100%) team reports won’t load even after attack is over Follow up Actions: Identify ways to improve time to recovery (MTTR) after a latency injection on DynamoDB.
14. 14 How to Chaos Engineer?