Chaos Engineering and design principles for building Highly Available services on the cloud

Complex Distributed Systems are hard to operate and has very complex failure modes. In this talk, we are going to discuss how we can build confidence in large scale distributed systems by introducing random but controlled failures in them in production and understand how services de-generate and work towards healing and recovering from failures automatically. We will also discuss patterns and various techniques for designing highly available and resilient distributed systems.

Transcript

1. HASHICORP Chaos Engineering and design principles for building 
 Highly Available services on the cloud Diptanu Gon Choudhury @diptanu RootConf 2016
4. HASHICORP Correlation of failures with scale and change Rate of Change Scale
5. HASHICORP Evolution of hardware Commodity hardware and network are the new normal Processors are not getting faster, we are running more of them
6. HASHICORP Evolution of application architecture SOA and Micro Services are replacing monoliths Distributed Systems are the new normal
7. HASHICORP Fanout in Micro Services
8. HASHICORP A common architecture of the modern web
9. HASHICORP Steady state of the service
10. HASHICORP Steady state of the service
11. HASHICORP Steady state of the service
12. HASHICORP Failure in the caching sub-system
13. HASHICORP Failure in the caching sub-system
14. HASHICORP Failures cascade all the way to the edge
15. HASHICORP Steady state of a SOA based application
16. HASHICORP Failure in a single service
17. HASHICORP Failure cascades to the edge
18. HASHICORP Drift into Failure - Sydney Dekkar We can model and understand in isolation. But, when released into competitive, nominally regulated societies, their connections proliferate, their interactions and interdependencies multiply, 
 their complexities mushroom. 
 And we are caught short
19. WE MUST 
 DESIGN FOR FAILURE
20. HASHICORP Resilience By Design Feature Complete is only 30% of the journey
 Implications of dependencies failing Implications of surge in traffic
 How quickly can the system recover
 Which failures can be mitigated and how
 How does the system grow
 Implications of Data Centers failing
21. THE BEST WAY 
 TO AVOID FAILURES
 IS TO FAIL CONSTANTLY
22. HASHICORP Chaos Engineering Chaos Engineering is the discipline of experimenting on a distributed system 
 in order to build confidence in the system’s capability to withstand turbulent 
 conditions in production.
23. HASHICORP Failure Injection Faults across network boundaries are blurry in distributed systems
 It’s hard to reason why a service across the network has increased latency
 Introducing failures by disrupting hardware uncovers a lot of issues Introduce failures in dependencies in a slow but gradual manner
 Use auditing and monitoring while failures being injected in a system
24. HASHICORP Simian Army from Netflix
25. HASHICORP Simian Army Chaos Monkey
 Chaos Gorilla
 Chaos Kong Latency Monkey
 Monkey Commander
26. HASHICORP Simian Army Chaos Monkey
 Chaos Gorilla
 Chaos Kong Latency Monkey
 Monkey Commander
27. HASHICORP Simian Army Chaos Monkey
 Chaos Gorilla
 Chaos Kong Latency Monkey
 Monkey Commander
28. HASHICORP Simian Army Chaos Monkey
 Chaos Gorilla
 Chaos Kong Latency Monkey
 Monkey Commander
29. HASHICORP Simian Army Chaos Monkey
 Chaos Gorilla
 Chaos Kong Latency Monkey
 Monkey Commander
30. HASHICORP Failures revisited Node Failures
 Switch Failures
 Datacenter interconnect Failures
 Region Failures
 DNS Failures Deploy clusters not nodes
 State should be at a service level
 Unleash chaos monkey
31. HASHICORP Failures revisited Node Failures
 Switch Failures
 Datacenter interconnect Failures
 Region Failures
 DNS Failures Spread clusters across rack
 Use smart load balancers
 Rely on service discovery
 Unleash latency monkey
32. HASHICORP Failures revisited Node Failures
 Switch Failures
 Datacenter interconnect Failures
 Region Failures
 DNS Failures Spread clusters across data centers
 Loadbalancer to fall back on healthy DC
 Unleash the Chaos Gorilla
33. HASHICORP Failures revisited Node Failures
 Switch Failures
 Datacenter interconnect Failures
 Region Failures
 DNS Failures Run services Active-Active
 Use Geo-DNS to divide traffic
 Proxies at the edge to relieve pressure
 Unleash the Chaos Kong
34. HASHICORP Failures revisited Node Failures
 Switch Failures
 Datacenter interconnect Failures
 Region Failures
 DNS Failures Be prepared to know how to escalate
 Increase TTL
35. HASHICORP Chaos Engineering applied to Human Resources Be prepared to not have the full team to deal with outages
 Send your team members on vacation
 Prepare runbooks and dashboards to triage and avoid tribal knowledge
36. HASHICORP Tools for building resilient systems Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure
37. HASHICORP Reactive Load Balancers Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Score nodes based on latencies
 Parallel requests Request cancellation
 Shuffle and randomize balancing
 Stream oriented protocols
38. HASHICORP Circuit Breakers Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Guard all network calls
 Return fallbacks 
 Bulkhead requests
 Aggressive timeouts
39. HASHICORP Cluster Schedulers Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Restarts services when the fail
 Faster deployments
 Higher utilization
 Quality of Service
40. HASHICORP Dynamic Service Discovery Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Registry of the data center
 Remove and add services quickly
 Be prone to failures
41. HASHICORP Reactive Scaling Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Scale out in reaction to latencies and QPS
 Throttle when hit by thundering storms

42. HASHICORP Immutable Infrastructure Reactive Load Balancers Circuit Breakers
 Dynamic Cluster Schedulers
 Dynamic Service Discovery
 Reactive scaling
 Immutable infrastructure Cattle and not pets
 Infrastructure should be considered disposable
43. HOPE IS NOT A STRATEGY
44. HASHICORP Thanks! Drift into Failure: From hunting broken systems to Understanding Complex Systems
 - Sidney Dekkar How Complex Systems Fail
 - Richard Cook Notes on Distributed Systems for the young blood
 - Jeff Hodges