Building Resilient Services in Clojure

By employing strategies like bulkheads, circuit breakers, load shedding, how we can make services written in Clojure battle ready for production systems. And how to go beyond the now in terms of constant evolution of the resiliency strategies. This was presented at re:Clojure 2020, https://reclojure.org.

Transcript

1. Building Resilient Services in Clojure
2. (hello re-clojure) I am Mourjo Sen! ◎ Software Engineer at Helpshift ◎ 5 years with Clojure ◎ @mourjo_sen 2
3. Stability of Systems 3
4. Stability of Systems 4
5. Stability of Systems 5
6. Stability of Systems Resiliency is the ability of a system to gracefully handle and recover from failures. 6
7. 7 Business Logic
8. 8 Business Logic
9. 9 Business Logic
10. 10 Business Logic https://twitter.com/cfiesler/status/1330542730469052418
11. 11 Business Logic https://twitter.com/cfiesler/status/1330542730469052418
12. Journey to Resilience 12 Business logic User-centric System-centric
13. User-centric Resilience Systems we depend on will fail. Design for the optimal user experience. 13
14. Exception Handling 14 Business logic Exception Handling User-centric System-centric
15. Exception Handling 15 Business logic Exception Handling User-centric System-centric
16. Exception Handling 16 Business logic Exception Handling User-centric System-centric
17. Fallbacks 17 Business logic Exception Handling Fallbacks User-centric System-centric
18. Fallbacks 18 Business logic Exception Handling Fallbacks User-centric System-centric
19. Fallbacks 19 Business logic Exception Handling Fallbacks User-centric System-centric
20. Timeouts 20 Business logic Exception Handling Fallbacks Timeouts User-centric System-centric
21. Timeouts 21 Business logic Exception Handling Fallbacks Timeouts User-centric System-centric
22. Retries
23. Retries Transient failure
24. Retries
25. Retries
26. Retries 26
27. Retries 27
28. Circuit Breakers 28 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers User-centric System-centric
29. Circuit Breakers 29 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers User-centric System-centric History of failures
30. Circuit Breakers 30 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers User-centric System-centric Circuit open
31. Circuit Breakers 31 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers User-centric System-centric Circuit closed
32. Circuit Breakers 32 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers User-centric System-centric Retry five times and then break the circuit
33. Circuit Breakers 33
34. Circuit Breakers 34
35. ◎ Time since last failure ◎ Number or % of failures in last W seconds ◎ Number of slow ops Circuit Breaker Strategy 35
36. Health Checks 36 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks User-centric System-centric
37. System-centric resilience Design to prevent failures as far as possible. 37
38. System Configuration ◎ JVM options like Heap, GC ◎ Machine config like t2.micro, r3.2xlarge 38 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks System Configs User-centric System-centric
39. Instrumentation ◎ Monitoring / Alerting ◎ Clojure specific instrumentation ◦ https://github.com/metrics-clojure/metrics-clojure 39 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks System Configs Instrumentation User-centric System-centric
40. Share System Resources by Pooling ◎ Resources are finite ◎ System components should respect finiteness 40 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks System Configs Instrumentation Resource Pooling User-centric System-centric
41. Share System Resources by Pooling 41
42. Share System Resources by Pooling 42 Incident timeline JVM threads
43. Share System Resources by Pooling 43 Database driver pool HTTP connection pool Background task pool Web server pool
44. Bulkheads 44 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks System Configs Instrumentation Resource Pooling Bulkheads User-centric System-centric
45. Bulkheads 45
46. Load Shedding: Prevent Cascading Failures 46 Business logic Exception Handling Fallbacks Timeouts Retries Circuit Breakers Health Checks System Configs Instrumentation Resource Pooling Bulkheads Load shedding User-centric System-centric
47. Load Shedding: Prevent Cascading Failures 47
48. Load Shedding: Prevent Cascading Failures 48
49. Load Shedding: Prevent Cascading Failures 49
50. Load Shedding: Prevent Cascading Failures 50
51. When to Shed Load? ◎ Response Time = Queueing Time + Service Time 51
52. When to Shed Load? ◎ Response Time = Queueing Time + Service Time ◎ Use queueing time as a signal for load 52
53. Load Shedding Ring Middleware 53
54. Load Shedding Ring Middleware 54
55. Load Shedding Ring Middleware 55
56. Load Shedding vs Rate Limiting ◎ More dynamic 56
57. Load Shedding vs Rate Limiting ◎ More dynamic 57
58. Load Shedding vs Rate Limiting ◎ More dynamic 58
59. The Journey Thus Far 59 Business logic Exception handling Fallbacks Timeouts Retries Circuit breakers Health checks System Configs Instrumentation Resource pooling Bulkheads Load shedding ? User-centric System-centric
60. ◎ Constant uphill battle Feedback in Resilience Engineering 60
61. ◎ Constant uphill battle ◎ Incidents are lessons for the future Feedback in Resilience Engineering 61
62. ◎ Constant uphill battle ◎ Incidents are lessons for the future Feedback in Resilience Engineering 62 Product Engineering Incident Failure Discovery Knowledge Dissemination Incident Analysis
63. Beyond Incidental Resilience ◎ How to ensure our resilience patterns are reliable? 63
64. Beyond Incidental Resilience ◎ How to ensure our resilience patterns are reliable? ◎ Few opportunities to learn failure patterns 64
65. Beyond Incidental Resilience ◎ Chaos Engineering: Ingest failures deliberately ◦ Confirmation of resilience ◦ Discovery of new failure patterns https://netflix.github.io/chaosmonkey/ 65
66. Beyond Incidental Resilience 66 Product Engineering Incident Failure Discovery Chaos Engineering Knowledge Dissemination Incident Analysis
67. Conclusion: Tokyo was not Built in a Day 67
68. Conclusion: Tokyo was not Built in a Day Earthquake-prone zones are home to the safest buildings in the world. 68 https://www.bbc.com/future/gallery/20190114-how-japans-skyscrapers-are-built-to-survive-earthquakes https://en.wikipedia.org/wiki/Tokyo
69. Thanks! Any questions? @mourjo_sen 69 Business logic Exception handling Fallbacks Timeouts Retries Circuit breakers Health checks System Configs Instrumentation Resource pooling Bulkheads Load shedding ? User-centric System-centric
70. References ◎ https://github.com/resilience4j/resilience4j ◎ https://github.com/ylgrgyq/resilience-for-clojure ◎ https://docs.microsoft.com/en-us/azure/architecture/patterns/category/resiliency ◎ https://netflix.github.io/chaosmonkey/ ◎ https://github.com/mourjo/procrustes ◎ https://github.com/dakrone/clj-http ◎ https://github.com/swaldman/c3p0 ◎ https://github.com/seancorfield/next-jdbc ◎ https://github.com/ring-clojure/ring ◎ https://github.com/TheClimateCorporation/claypoole ◎ https://medium.com/helpshift-engineering/achieving-graceful-restarts-of-clojure-serv ices-b3a3b9c1d60d ◎ https://medium.com/helpshift-engineering/load-shedding-in-clojure-d4857ce11588 ◎ https://sre.google/sre-book/table-of-contents/ 70