Expect the un-expected: How to handle errors gracefully

Even though you tested your application perfectly, errors and bugs will still happen in production, especially if other services or databases go down or are under high load. Thus it is very important to see errors happening and to be able to react to them quickly. In this session we’ll introduce you to efficient ways for monitoring and logging for errors and show how you can handle them if they happen, covering deployment strategies, using intelligent circuit breakers, and gracefully reducing functionality. The session will give examples and recommendations so that you can quickly get started with implementing these.

Transcript

1. Expect the un-expected How to handle errors gracefully @BastianHofmann
4. 14 million users
5. 193 countries
6. ~1800 request/s
7. >100 million publications
8. ~ 140 components
9. ~ 400 repositories
10. ~ 80 engineers
11. Growing Traffic
12. Active development
13. Continuous Delivery
14. Constant change
15. Things will go wrong
16. Bugs
17. Performance regressions
18. Database overloads
19. Hardware failures
20. Distributed Systems
21. Microservices
22. Network
23. Multiple technology stacks
24. $complexity++
25. Errors happen
26. How to detect them
27. How to handle them
28. … in a way that the user does not notice them
29. Detecting if something breaks
30. Logging
31. You don’t want to show a stack trace to your user
34. Error Logs
35. Log additional info
37. Request Information
38. Log in a structured way
39. JSON
40. Let’s talk about…
41. Microservices
42. Logs from many services
43. gateway service http service http service http service http service user request log log log log log
44. Correlation / Tracing ID
45. http service http service http service http service create unique trace_id for request user request trace_id trace_id trace_id trace_id log log log log log gateway service
46. Getting to the logs
50. Put the logs in a central place
51. Make them easily full-text searchable
52. Make them aggregate-able
53. Elasticsearch + Logstash + Kibana
54. Full text search
55. Structured Messages
56. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
57. logstash http://logstash.net/
58. input filter output
59. Very rich plugin system
60. Logstash elasticsearch webserver webserver webserver AMQP log log log logstash logstash logstash
61. Filebeat https://www.elastic.co/products/beats/filebeat
62. Logstash elasticsearch webserver webserver webserver log log log filebeat filebeat filebeat
63. Kibana http://www.elasticsearch.org/overview/kibana/
65. Always Log to file
66. But there are not only exceptions
67. Performance regressions
68. Conversion regressions
69. Traffic regressions
70. Monitoring
71. Latency
72. Availability
73. Throughput
74. Overall
75. Granular
76. Graphite http://graphite.wikidot.com/
78. StatsD https://github.com/etsy/statsd/
79. webserver webserver webserver statsd statsd statsd graphite
80. Other solutions
81. Prometheus
82. https://www.librato.com
83. http://www.soasta.com/
84. Dashboards
86. Alerts
88. But…
89. Errors happen
90. Handling Exceptions gracefully
92. Component based fronted
100. Degrading Functionality
101. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu Institution
102. Profile Publications Publication Publication Publication AboutMe LeftColumn Image Menu EXCEPTION Institution
103. Profile Publications Publication Publication Publication LeftColumn Image Menu Institution
105. Big cause for regressions
106. Deployments
107. Errors happen
108. Safe deployments
109. Feature Flags/ Toggles
110. Release !== Deployment
111. public function hasAccess() { return featureFlag()->isActive( FeatureFlag::YOUR_FEATURE ); }
114. Partial rollout
115. Fast rollbacks
116. Microservices
117. Safe Service-to Service Communication
118. Service A Service B 200 OK
119. Distributed Systems are hard
120. Errors happen
121. Failures
122. Service A Service B 5xx
123. Very slow responses
124. Service A Service B 2 seconds later
125. One incoming request leads to multiple requests to services
126. The incoming request can be slowed down a lot
127. Low timeout settings
128. Service A Service B Only wait for max. 50ms
129. High availability
130. Use a Load Balancer
131. Service A Service B Service C Service C Load balancer Choose one running instance
132. Health checks
133. Service A Service B Service C Service C Load balancer Are you still alive?
134. Service A Service B Service C Service C Load balancer Are you still alive?
135. Service A Service B Service C Service C Load balancer
136. Even better
137. Circuit Breakers
138. Service A Service B 200 OK Circuit Breaker Status: closed Error rate: 0
139. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold
140. Service A Service B Circuit Breaker Status: -> open Error rate: > threshold
141. Service A Service B Error Circuit Breaker Status: -> open Error rate: > threshold Test if still failing
142. Service A Service B 200 OK Circuit Breaker Status: -> close Error rate: 0 Test if still failing
143. Handling traffic spikes
144. Service A Service B 200 OK Circuit Breaker
145. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
146. Service A Service B Circuit Breaker Service C Circuit Breaker Timeouts
147. Throttling
148. Service A Service B Circuit Breaker Service C Circuit Breaker Only allow xx% of calls
150. Priority
151. Service A Service B Circuit Breaker Service C Circuit Breaker 100% of calls 10% of calls
152. https://github.com/Netflix/Hystrix
153. Or if you don’t want to change all your services…
155. Service A Service B Service C Circuit Breaker LinkerD
156. Summing it up
157. Central Log Management
158. Monitor and Measure everything
159. Alerts
160. Graceful Exception Handling
161. Feature Flags
162. Partial Rollouts
163. Circuit Breakers
164. http://speakerdeck.com/u/bastianhofmann
165. http://twitter.com/BastianHofmann http://lanyrd.com/people/BastianHofmann http://speakerdeck.com/u/bastianhofmann mail@bastianhofmann.de