Designing for failure

Failure has been neglected for quite some time in our industry, but how can we design for failure? And especially, how Gocan help us be prepared for the unexpected?

In recent years we’ve been talking more and more about failure and resiliency. This is something that mission-critical systems always had to think of first but many of our systems (until today) neglect the importance of this. In this talk I would like to talk about how to design a system for failure, what are the pitfalls and gotchas that we have to be prepared in a microservice environment? And most importantly, how go can help us be prepared for the unexpected?

Transcript

1. Designing for Failure @italolelis
2. Italo Vietro Brazilian living in Berlin and working @Lykon Worked @HelloFresh and @N26
3. Designing systems unexpected for the
5. 39,000,000 flights in 2019
6. 8 deadly crashes https://aviation-safety.net/statistics/period/barometer3.php fatalities 233
7. ELMS (Electrical Load Management System)
8. Essentials
9. – Liang Guo “Resilience is a Requirement, Not a Feature.”
10. Latency Control Load Shedding Isolation Self Healing Traffic Control
11. Latency Control Load Shedding Isolation Self Healing Traffic Control
12. Timeouts
13. Client Timeouts Server Timeouts
17. Client Timeouts Server Timeouts
19. Circuit Breakers
22. Retries
24. Latency Control Load Shedding Isolation Self Healing Traffic Control
25. Load Balancing
26. Service A Service B Service B Service B
27. Service A Service B Service B Service B
28. Service A Service B Service B Service B
29. Outlier Server Host Detection
30. Service A Service B Service B Service B
31. Service A Service B Service B Service B
32. Service A Service B Service B
33. Service A Service B Service B
34. Latency Control Load Shedding Isolation Self Healing Traffic Control
35. Bulkhead
38. Latency Control Load Shedding Isolation Self Healing Traffic Control
39. Health Checks
41. If everything is OK you get...
43. If things are not good but your app still can work...
45. Otherwise...
47. Outbox Pattern
48. Service A Kafka Consumer A Consumer B
49. Service A Consumer A Consumer B
50. Service A Sender Kafka Consumer A Consumer B
52. https://github.com/italolelis/outboxer
53. Latency Control Load Shedding Isolation Self Healing Traffic Control
54. Rate Limiters
56. Latency Control Load Shedding Isolation Self Healing Traffic Control
57. Latency Control Load Shedding Isolation Self Healing Traffic Control
58. Latency Control Load Shedding Isolation Self Healing Traffic Control
59. Latency Control Load Shedding Isolation Self Healing Traffic Control
60. Latency Control Load Shedding Isolation Self Healing Traffic Control
61. Latency Control Load Shedding Isolation Self Healing Traffic Control
62. Too much…
63. 1 Service 10 100 1000 Services
64. Service Mesh
66. Show me some code…
67. Observability
68. Metrics
71. Distributed Tracing
73. Open Census Open Tracing
74. Coming soon…
75. https://github.com/italolelis/talks Slides https://github.com/italolelis/designing-for-failure https://github.com/italolelis/coffee-shop
76. Grazie! @italolelis Questions?