Expect the un-expected: How to handle errors gracefully

Even though you tested your application perfectly errors and bugs will still happen in production, especially if other services or databases go down or are under high load. Thus it is very important to see errors happing and to be able to react to them quickly. In this talk I’ll introduce you to efficient ways for monitoring and logging for errors and show how you can handle them if they happen, covering deployment strategies, using intelligent circuit breakers and gracefully reducing functionality.

Transcript

1. Expect the un-expected How to handle errors gracefully @BastianHofmann
4. 12 million users
5. 193 countries
6. ~1800 request/s
7. lots of data
8. ~ 140 components
9. ~ 80 engineers
10. Growing Traffic
11. Active development
12. Continuous Delivery
13. Bugs
14. Performance regressions
15. Database overloads
16. Hardware failures
17. Microservices
18. Distributed Systems
19. Multiple technology stacks
20. $complexity++
21. Errors happen
22. How to detect them
23. How to handle them
24. Detecting if something breaks
25. Logging
26. Error Logs
27. Log additional info
28. Log in a structured way
29. JSON http://www.ietf.org/rfc/rfc4627.txt
30. Microservices
31. web server http service http service http service http service user request log log log log log
32. Correlation / Tracing ID
33. web server http service http service http service http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log
34. Getting to the logs
36. Put the logs in a central place
37. Full-text search
38. Aggregation
39. Logstash + Elasticsearch + Kibana
40. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
41. Kibana http://www.elasticsearch.org/overview/kibana/
43. Monitoring
44. Latency
45. Availability
46. Throughput
47. Graphite http://graphite.wikidot.com/
49. https://www.librato.com
50. http://www.soasta.com/
51. Dashboards
55. Alerts
57. Errors happen
58. Handling Exceptions gracefully
60. Component based fronted
66. Degrading Functionality
67. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution
68. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION Institution
69. Profile Publications Publication Publication Publication LeftColumn Image Menu Institution
70. Deployments
71. Errors happen
72. Safe deployments
73. Release !== Deployment
74. Feature Flags/ Toggles
75. public function hasAccess( $accountId, array $roleNames ) { return featureFlag()->isActive( FeatureFlag::TEST_ONE ); }
77. Partial rollout
78. Oftentimes it’s not only one monolith
80. Service A Service B 200 OK
81. Distributed Systems are hard
82. Errors happen
83. Safe Service-to Service Communication
84. Failures
85. Service A Service B 5xx
86. Timeouts
87. Service A Service B Timeout
88. Low timeout settings
89. Service A Service B Only wait for max. 50ms
90. Circuit Breakers
91. Service A Service B 200 OK Circuit Breaker Status: closed Error rate: 0
92. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold
93. Service A Service B Circuit Breaker Status: -> open Error rate: > threshold
94. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold Test if still failing
95. Service A Service B 200 OK Circuit Breaker Status: -> close Error rate: 0 Test if still failing
96. How do I handle traffic spikes?
97. Service A Service B 200 OK Circuit Breaker
98. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
99. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
100. Throttling
101. Service A Service B Circuit Breaker Service C Circuit Breaker Only allow xx% of calls
102. Priority
103. Service A Service B Circuit Breaker Service C Circuit Breaker 100% of calls 10% of calls
104. https://github.com/Netflix/Hystrix
106. Service A Service B Service C Circuit Breaker LinkerD
107. Test it
108. http://techblog.netflix.com/2014/09/introducing-chaos-engineering.html
109. Summing it up
110. Central Log Management
111. Monitor and Measure everything
112. Alerts
113. Graceful Exception Handling
114. Feature Flags
115. Partial Rollouts
116. Circuit Breakers
117. http://speakerdeck.com/u/bastianhofmann
118. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de