Resilience patterns (Server edition)

Talk about server-based resilience patterns

Transcript

1. Resilience patterns (server edition) @slok69 https://slok.dev github.com/slok
2. HELLO! I am Xabier Larrakoetxea Let me be your guide in the resilience world
3. Agenda ✘ Introduction ✘ Problem/solution ✘ Experiments ✘ Conclusions
4. Resilience It is the ability to absorb or avoid damage without suffering complete failure. Wikipedia
5. Queueing theory Is the mathematical study of waiting line Wikipedia
8. R R R R R R R R R Concurrency Queue
9. Little’s law Is a theorem by John Little which states that the long-term average number L of customers in a stationary system is equal to the long-term average effective arrival rate λ multiplied by the average time W that a customer spends in the system Wikipedia
10. Little’s law Is a theorem by John Little which states that the long-term average number L of customers in a stationary system is equal to the long-term average effective arrival rate λ multiplied by the average time W that a customer spends in the system Wikipedia
11. L = λ W Arrival rate (avg) Inflight Processing duration (avg)
12. R R R R R R R R R Concurrency Queue L W λ L = λ W
13. Our server can handle ~ 100 concurrent requests (due to any of these: CPU, Memory, fixed concurrency, threads, file descriptors….) Example
14. 50 = 100RPS * 0.5s
15. 100 = 100RPS * 1s
16. 140 = 700RPS * 200ms
17. Queueing Processing Queueing is... good, is... bad Optimal usage
18. Problems of bad queueing ✘ Latency increase ✘ Collapse ✘ Cascading failure ✘ Crash ✘ ...
19. You can’t control your clients, protect your server
20. Solutions ✘ Autoscaling ✘ Rate limits ✘ Caches ✘ Load balancing ✘ Load shedding ✘ ...
21. Load shedding Is a technique used in information systems, especially web services, to avoid overloading the system and making it unavailable for all users. The idea is to ignore some requests rather than crashing a system and making it fail to serve any request. Wikipedia
23. Resilience patterns
24. The tests ✘ Handler with random latency (250-600ms) ✘ 15-25RPS is the regular capacity ✘ Test1: 15RPS for 3m (with 50RPS 1m middle spike) ✘ Test2: 60RPS for 15m ✘ Limits to 0.1 CPU and 50mb on memory ✘ Goresilience library (github.com/slok/goresilience) The tests demo can be found at https://github.com/slok/resilience-demo
25. Naked Server without protection.
26. Exp1: Naked server ✘ Accepts everything. ✘ No protection against spikes/bursts. ✘ The least recommended one. ✘ The most used one.
27. ✘ Success: 61% ✘ P95: 15.8s ✘ P99: 24.7s 15 RPS with middle spike (50RPS) Unresponsive on spike
28. Max latencies 503 OOM ✘ Success: 9.62% ✘ P95: 24s ✘ P99: 29s (Max) 60 RPS (15m)
29. Naked 15RPS Success: 61% P95: 15.8s P99: 24.7s 60RPS Success: 9.62% P95: 24s P99: 29s
30. Good ✘ Simple Bad ✘ Cascading failure (latency) ✘ Crash (OOM)
31. Bulkhead Bulkhead for concurrency control
33. App (without bulkhead) Bulkhead pattern Isolate elements of an application into pools so that if one fails, the others will continue to function. Service A Service B App (with bulkheads) Service A Service B
35. Exp2: Bulkhead ✘ Bulkhead will limit the concurrent handlings. ✘ Queue timeout will clean long time queued requests. ✘ Needs to be configured . ✘ Will protect us from bursts/spikes.
36. ✘ Success: 83% ✘ P95: 6.5s ✘ P99: 8.5s 15 RPS with middle spike (50RPS)
37. P90 < 10s Controlled failures (429) ✘ Success: 38.75% ✘ P95: 9.3s ✘ P99: 12.2s 60 RPS (15m)
38. Naked Bulkhead 15RPS Success: 61% P95: 15.8s P99: 24.7s Success: 83% P95: 6.5s P99: 8.5s 60RPS Success: 9.62% P95: 24s P99: 29s Success: 38.75% P95: 9.3s P99: 12.2s
39. Good ✘ Simple ✘ Load shedding Bad ✘ Static configuration (requires load tests) ✗ Timeouts ✗ Number of workers (concurrency) ✘ Wrong configuration could waste capacity or overload the server
40. Bulkhead + circuit breaker Bulkhead for concurrency control and circuit breaker to control the constant overload
41. Circuit breaker pattern It is used to detect failures and encapsulates the logic of preventing a failure from constantly recurring, during maintenance, temporary external system failure or unexpected system difficulties. Closed Open Half open Error limit exceeded Timeout Tests failed Tests succeeded Regular flow Fail fast Regular flow
42. Exp3: Bulkhead + circuit breaker ✘ Bulkhead will limit the concurrent handlings. ✘ Circuit breaker will release the load (fast). ✘ Hystrix style pattern. ✘ Needs to be configured . ✘ Will protect us from bursts/spikes. ✘ Circuit breaker wraps bulkhead.
43. ✘ Success: 76.96% ✘ P95: 5.2s ✘ P99: 7.5s 15 RPS with middle spike (50RPS)
44. Less latency than plain bulkhead Circuit breaker opened (fail in spikes) ✘ Success: 30% ✘ P95: 6.5s ✘ P99: 8s 60 RPS (15m)
45. Naked Bulkhead Bk + CB 15RPS Success: 61% P95: 15.8s P99: 24.7s Success: 83% P95: 6.5s P99: 8.5s Success: 76.96% P95: 5.2s P99: 7.5s 60RPS Success: 9.62% P95: 24s P99: 29s Success: 38.75% P95: 9.3s P99: 12.2s Success: 30% P95: 6.5s P99: 8s
46. Good ✘ Load shedding ✘ Recover faster than bulkhead Bad ✘ Static configuration (requires load tests) ✘ Fail requests although server is ok
47. Adaptive resilience patterns
49. LIFO + CoDel (Facebook) Unfair request handling and aggressive timeouts on congestion Explanation and algorithm: https://queue.acm.org/detail.cfm?id=2839461 Original CoDel: https://queue.acm.org/detail.cfm?id=2209336 Airbnb uses also: https://medium.com/airbnb-engineering/building-services-at-airbnb-part-3-ac6d4972fc2d
50. CoDel (Controlled delay) We have 2 kinds of in queue timeouts: ✘ Regular (Interval): 100ms by default ✘ Aggressive (target): 5ms by default By default requests will have the interval timeout on queue. Measure when the queue was empty for the last time. If the duration since last time is greater than interval duration, congestion detected. If congested the requests will have the target timeout on queue.
51. CoDel (Controlled delay) // Enqueue request. if (queue.timeSinceEmpty() > interval) { // Congestion. timeout = target } else { timeout = interval } queue.enqueue(req, timeout)
52. Adaptive LIFO We have 2 kinds of in queue priorities: ✘ FIFO: On regular mode first in first out. ✘ LIFO: On congestion mode last in first out (unfair). When CoDel detects congestion it will change queue dequeue priority and the last requests will be served first (CoDel will clean old queued requests). The algorithm assumes that delayed queued requests are gone already and the new ones have more probability of being served. Dropbox proxy uses adaptive LIFO : https://blogs.dropbox.com/tech/2018/03/meet-bandaid-the-dropbox-service-proxy
53. Adaptive LIFO
54. Adaptive LIFO Low load High load First in First out Last in First out CoDel will drain if persists
55. R R R R R R R R R Concurrency Queue No congestion In
56. R R R R R R R R R Concurrency Queue Congestion In
57. Exp4: Adaptive LIFO + CoDel ✘ Based on TCP congestion CoDel algorithm and adapted by Facebook. ✘ Dynamic timeout and queue priority (Will adapt and change policies on congestion) . ✘ No configuration required (almost, safe defaults). ✘ Very aggressive timeouts on congestion.
58. R R R R R R R R R Concurrency Queue
59. ✘ Success: 78% ✘ P95: 3.1s ✘ P99: 5.4s 15 RPS with middle spike (50RPS) Triggered congestion
60. ✘ Success: 30% ✘ P95: 6s ✘ P99: 8s 60 RPS (15m)
61. Naked Bulkhead Bk + CB CoDel 15RPS Success: 61% P95: 15.8s P99: 24.7s Success: 83% P95: 6.5s P99: 8.5s Success: 76.96% P95: 5.2s P99: 7.5s Success: 78% P95: 3.1s P99: 5.4s 60RPS Success: 9.62% P95: 24s P99: 29s Success: 38.75% P95: 9.3s P99: 12.2s Success: 30% P95: 6.5s P99: 8s Success: 30% P95: 6s P99: 8s
62. Good ✘ Load shedding. ✘ Recover faster than circuit breaker. ✘ Unfair serving (serve new ones, leave old ones). ✘ No configuration Bad ✘ Static concurrency config (doesn’t affect much).
63. Concurrency limits (Netflix) Adapt concurrency Explanation and algorithms: https://medium.com/@NetflixTechBlog/performance-under-load-3e6fa9a60581 Concurrecy-limits: https://github.com/Netflix/concurrency-limits
64. Concurrency limits Implements and integrates concepts from TCP congestion control to auto-detect concurrency limits for services in order to achieve optimal throughput with optimal latency. Concurrent requests Time Initial limit Discovered limit Real limit
65. Exp5: Adaptive concurrency (limits) ✘ Different algorithms (in this example AIMD but there are more like Vegas, Gradient...). ✘ Adaptive concurrency. ✘ Static queue timeout and priority. ✘ No configuration required. ✘ Adapts based on execution results (errors and latency).
66. R R R R R R R R R Concurrency Queue
67. ✘ Success: 82% ✘ P95: 4s ✘ P99: 6s 15 RPS with middle spike (50RPS)
68. ✘ Success: 36% ✘ P95: 4.8s ✘ P99: 6.5s 60 RPS (15m)
69. 60 RPS (15m)
70. Naked Bulkhead Bk + CB CoDel CL 15RPS Success: 61% P95: 15.8s P99: 24.7s Success: 83% P95: 6.5s P99: 8.5s Success: 76.96% P95: 5.2s P99: 7.5s Success: 78% P95: 3.1s P99: 5.4s Success: 82% P95: 4s P99: 6s 60RPS Success: 9.62% P95: 24s P99: 29s Success: 38.75% P95: 9.3s P99: 12.2s Success: 30% P95: 6.5s P99: 8s Success: 30% P95: 6s P99: 8s Success: 36% P95: 4.8s P99: 6.5s
71. Good ✘ Load shedding. ✘ Recover faster. ✘ No configuration. ✘ Adapts to any environment (Hardware, autoscaling, noisy neighbor...) Bad ✘ Depending on load and algorithm can be slow to adapt (or not adapt).
72. Conclusions
73. Conclusions ✘ There is no winner, depends on the app and context. ✘ There is a loser: no protection (naked server). ✘ Adaptive algorithms add complexity (use libraries like goresilience) but are better for dynamic envs like cloud native. ✘ A bulkhead or circuit breaker can be enough. ✘ You can use a front proxy (or use sidecar pattern) ✘ Don’t trust your clients, protect yourself.
74. THANKS! Any questions? You can find me at ✘ @slok69 ✘ https://slok.dev ✘ github.com/slok http://bit.ly/resilience-form