Resilience and Chaos Engineering in the Cloud

Deck presented at GIDS Cloud 2020: https://wurreka.com/ict/virtual-conference/cloud/session/resilience-and-chaos-engineering-in-the-cloud

Transcript

1. www.developersummit.com Resilience & Chaos Engineering in the Cloud Nikos Katirtzis, Daniel Albuquerque
2. Expedia Group
3. Facts (2019) • $100+ billion in gross bookings • $12+ billion revenue • 200+ travel booking sites Every incident impacts our customers
4. INCIDENTS & POSTMORTEMS Why do bad things keep happening?
5. Having checklists ü Unit tests coverage ü Monitoring ü Alerting ü Etc.
6. Having checklists Just ü Unit tests coverage ü Monitoring ü Alerting ü Etc.
7. KPIs can be weapons Test coverage Test confidence
8. Incident 1 Vague title Unrelated metrics No units?
9. Having well-defined processes ü Runbooks ü Escalation Policies ü Etc.
10. Having well-defined processes Just ü Runbooks ü Escalation Policies ü Etc.
11. Expectation
12. Incident 2 Alert Paged on call developer Problem diagnosed Problem fixed
13. Operator Atrophy
14. Fire drills Image: Creative Commons https://commons.wikimedia.org/wiki/File:Fire_drill_in_kvofd.jpg
15. Improvement Loop Problem Solution
16. HOW CAN WE BECOME MORE RESILIENT Common Practices
17. Common Practices - Traditional Retries Application Timeouts Application Circuit Breaking Application Redundancy Multiple Instances Data Replication Distributed Caching Fallbacks Application Load Balancing Round-robin Rate Limiting Application
18. Common Practices – Cloud Providers Redundancy Multi-AZ Region 1 AZ1 Redundancy Multi-region Data Replication Cross-region Replication AZ2 AZ3 Region 1 AZ1 AZ2 AZ3 Region 2 AZ1 AZ2 AZ3 Region 1 Region 2
19. Common Practices – Cloud Providers AWS EC2 Autoscaling Serverless Aurora VPC Failover Load Balancing AWS VPC Peering AWS Elastic Load Balancing
20. Common Practices – Kubernetes Readiness/Liveness Probes Multiple Replicas Graceful Shutdowns Horizontal Autoscaling Pods spread across AZs Pod Disruption Budget AZ1 AZ2
21. Common Practices – Service Mesh Retries Mesh Deadline Propagation Mesh Circuit Breaking Mesh Fault Injection Load Balancing Latency-aware Rate Limiting Mesh
22. Common Practices – Release Safety Canaries Blue-Green Deploys Feature Flags A/B Testing Auto-Remediation Adaptive Alerting
23. Common Practices – Chaos Engineering The discipline of experimenting on a software system in production in order to build confidence in the system's capability to withstand turbulent and unexpected conditions
24. CHAOS ENGINEERING Why we need it Practical Examples in the Cloud and on Kubernetes
25. Why Chaos Engineering Chaos experiments are not expected to have catastrophic consequences But many times things go wrong J. Paul Reed - Senior Applied Resilience Engineer @Netflix Chernobyl
26. Chaos Experiments vs Resilience Tests Testcontainers Toxiproxy
27. Chaos Experiments vs Resilience Tests Testcontainers Toxiproxy
28. Chaos Experiments vs Resilience Tests Chaos Experiments UNKNOWNS Resilience Tests KNOWNS
29. Anatomy of a Chaos Experiment Steady State Hypothesis Run Experiment Verify
30. Demo - Architecture Reviews Service Amazon RDS Amazon ElastiCache
31. Demo – Reviews on the Website
32. Demo - Hypothesis 1. When the cache becomes unavailable the service falls back to fetching the reviews from the DB 2. Given (1), when the DB is slow, but latency < circuit breaker's threshold, the service is still healthy 3. Given (1), when the DB is very slow and latency > circuit breaker's threshold, the circuit breaker opens and we don't see reviews.
33. Demo - Action!
34. STATE OF THE ART What's next?
35. State of the Art Progressive Delivery Service Mesh Failure-as-a-Service Tools GitOps
36. Continuous Verification Source: Casey Rosenthal & Nora Jones – Chaos Engineering, System Resiliency in Practice Proactive Experimentation Tool Verification System Behaviors Reactive Testing Methodology Validation Known Properties Continuous Verification Continuous Verification Continuous Verification Continuous Verification Continuous Verification Disaster Recovery Incident Response Alerting Property-based Testing Load Testing Squeeze Testing Unit Testing Functional Testing Integration Testing Version Control Continuous Integration Continuous Delivery Agile DevOps SRE Code Review Static Analysis Logging Distributed Tracing Observability Monitoring Alerting Synthetic Monitoring
37. www.developersummit.com