Self-healing Software

Services that Handle Failure

Transcript

1. Self-healing Software Services That Never Fail
2. Self-healing Software Services That N̶e̶v̶e̶r̶ ̶F̶a̶i̶l̶ Handle Failure
3. A Toy System
4. A Toy System Mobile App API Browser API Search API Data API Search Index Data Store
5. Let’s Break It
6. THE API IS DOWN Mobile App API Browser API Search API Data API Search Index Data Store
7. Redundant Processes • Run multiple processes in case of failure • Load balance across the healthy processes • Processes should be physically separated • Example ◦ Heroku: Multiple Dynos ◦ Digital Ocean: Droplets in different datacenters ◦ AWS: Instances in different availability zones
8. The API Recovered Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
9. THE API IS DOWN AGAIN Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
10. Platform Automation • Platform infrastructure monitors and replaces failed processes • Examples ◦ Heroku ◦ Kubernetes ◦ AWS Auto Scale Groups ◦ AWS Beanstalk
11. The API Recovered Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
12. Oh Hai Internet
13. THAT’S A LOT OF USERS! Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
14. Platform Automation • Platform infrastructure monitors load and adds more instances • Examples ◦ Heroku ◦ Kubernetes ◦ AWS Auto Scale Groups ◦ AWS Beanstalk
15. Moar Instances Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API Browser API Browser API
16. OH NO l33t Hax0r! Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
17. Request Throttling • Throttle requests, typically by IP address or user credential • Protects the service from malicious attack but also bugs • Examples ◦ dryruby/rack-throttle ◦ jhurliman/node-rate-limiter
18. Go Home Hax0r Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
19. Let’s Get Unstable
20. THE NETWORK IS FLAKEY Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
21. Retries • Retry operations in the face of transient errors • Exponential backoff + jitter reduces load on the flakey component • Give up after a given number of retries or time retrying • Examples ◦ zeit/async-retry
22. If At First You Don’t Succeed ... Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Search API Search API Data API Data API
23. Let’s Slooow It Down
24. SEARCH IS SLOW Search API Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Data API Data API Search API Search API
25. Timeouts • Add timeouts to all operations that can block • Often combined with retries and backoff • Prevents the whole system from falling over
26. Graceful Degradation Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Data API Data API Search API Search API Search API
27. SEARCH IS GETTING WORSE Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Data API Data API Search API Search API Search API
28. Circuit Breakers • Code that wraps an operation which can fail or hang • Tracks error rates and request latency • “Opens” the circuit under high latency or error rates • When available, a fallback response can be returned • Allows some request through to test if it should “Close” • Often incorporates timeouts and is combined with retries
29. Circuit Breakers • Prevents exhausting resources ◦ Client requests fail fast and don’t tie up resources waiting • Allows the system to recover ◦ Removes load from dependencies so they can recover • Examples ◦ Shopify/semian ◦ nodeshift/opossum
30. Search API Search Recovers Data API Search Index Data Store Mobile App API Browser API Load Balancer Load Balancer Data API Data API Search API Search API
31. Go Build Resilient Systems!
32. Questions