Crash Only Software

Transcript

1. Happiness through Crash-Only Software @AntoineGrondin
2. Who am I
10. crash-only
11. crash-only Catchy term that describes a way of coding and organizing your infrastructure.
12. crash-only sounds like a crazy idea
13. crash-only it’s not =]
14. crash-only Failure doesn’t throw system into chaos.
15. crash-only Equilibrium toward progress and consistency.
16. litmus test
17. litmus test kill -9 every 10s
18. kill -9 Is progress still happening?
19. kill -9 Will you need to urgently wake up?
20. kill -9 Can you say
 “wtv, it’ll fix itself”
21. kill -9 Will you keep all important data?
22. kill -9 Will your system remain consistent?
23. litmus test Answered no to any of the previous?
29. preventing failure is a lost battle
30. it’s a lost battle Can only push failure into more and more corners.
31. it’s a lost battle Failure minimization code adds complexity.
32. it’s a lost battle Added complexity induces more failures.
33. it’s a lost battle Attempts to fight failures result in more failures.
36. only way to win is to choose not to fight
37. choose not to fight Failures will happen no matter what you do.
38. choose not to fight Be pessimistic, assume it will happen all the time.
39. choose not to fight How to design a pessimistic system while minimizing complexity?
41. write less code
42. crash-only
43. crash-only: write less code collapse code paths
51. write less code
56. write less code do not gracefully shut things down
57. write less code do not gracefully shut things down false illusion of safety
58. write less code do not gracefully shut things down counterproductive
59. write less code do not gracefully shut things down adds complexity -> adds failure modes
60. beyond a single component
61. crash-only architecture made of crash-only components respect a contract
62. #1 servers try to process requests or crash
63. #2 clients send requests until success or TTL
64. #3 most components are stateless
65. #4 non-volatile state pushed outside of applications
66. distinguish between types of data #5
67. pick proper type of datastore based on volatility #6
68. communicate in a specific manner #7
69. communications
70. requests must be self-describing communications
71. requests must be self-describing time to live (TTL) communications
72. requests must be self-describing is_idempotent flag communications
74. on failure: retry_after gives a chance to crash-only component to restart communications
75. exponential randomized backoffs communications
76. every resource acquisition is leased communications
77. communications
90. communication sidebar: exactly-once delivery
91. when you send exactly 1 message to exactly 1 server exactly-once delivery
92. hint: it’s impossible exactly-once delivery
93. only 2 alternatives exactly-once delivery
94. only 2 alternatives: at-least-once delivery at-most-once delivery exactly-once delivery
95. IF message IS idempotent upon failure, retry_after until TTL exactly at-least-once delivery
96. IF message IS NOT idempotent upon failure, rollback exactly at-most-once delivery
101. delivery tradeoff I can see… tradeoffs… everywhere…
102. request have deadline request have is_idempotent flag servers try to process or crash clients retry until deadline crash-only communication
103. state
104. volatile state
105. non-volatile state
106. crash-only datastores
107. RDBMS: postgres crash-only datastores
108. KV: BerkeleyDB crash-only datastores
109. append-log based datastores crash-only datastores
110. real-story
111. Disk Image Converter at DigitalOcean real-story
112. requirements real-story
113. Need to convert millions of images when we change format. (say qcow to raw) real-story - requirements
114. Want to migrate ASAP to avoid running legacy code. real-story - requirements
115. Don’t want to watch the thing. real-story - requirements
116. design real-story
117. Acceptable: at-least-once delivery real-story - design
118. at-least-once delivery If an image converted >1 time, we don’t really care. Waste some time, better than losing customer image. real-story - design
119. non-volatile data: stored in crash-only DB real-story - design
120. volatile data: stored in Redis (not crash-only) real-story - design
121. fancy schema real-story - design
122. all jobs in a queue real-story - design
123. job leases stored in hash with TTL real-story - design
124. workers look for jobs without leases real-story - design
125. Step 1: make a lease Step 2: refresh lease while working Step 3: delete job+lease once done real-story - design
126. Step 1: make a lease Step 2: refresh lease while working Step 3: delete job+lease once done real-story - design
127. Step 1: make a lease Step 2: refresh lease while working Step 3: delete job+lease once done real-story - design
128. if worker dies? real-story - design
129. if worker dies: its leases expire jobs will get picked up again real-story - design
130. if worker dies: its leases expire jobs will get picked up again real-story - design
131. if worker fails to delete job once done? real-story - design
132. job will be performed again that’s ok real-story - design
133. if redis crashes? real-story - design
134. can reconstruct the queue from primary data source real-story - design
135. outcome real-story
136. when issue arose real-story - outcome
137. get paged by operator real-story - outcome
138. tell operator to let it go real-story - outcome
139. tell operator to let it go … the process manager restarts components real-story - outcome
140. tell operator to let it go … if issues, system converges toward progress real-story - outcome
141. never had to fix anything (touches wood) real-story - outcome
142. still in use real-story - outcome
143. and we’re happy real-story - outcome
144. key points real-story
145. non-volatile state
 stored in crash-only DB real-story - key points
146. volatile state
 stored in crash-unsafe DB (more like a message bus) real-story - key points
147. resources are leased real-story - key points
148. resources are self-describing real-story - key points
149. at-least-once vs. at-most-once tradeoff deliberately thought out real-story - key points
150. crash-only
151. is it a panacea? crash-only
152. no crash-only
153. other important considerations remain crash-only
154. caveats
155. Things That Are Still Important caveats
156. Things That Are Still Important: Circuit Breakers caveats
157. Things That Are Still Important: Fallbacks caveats
158. Things That Are Still Important: Error Recovery caveats
159. Things That Are Still Important: Degraded Modes caveats
160. Things That Are Still Important: Cattle vs Pets caveats
161. Things That Are Still Important: Error Tracking/Reporting caveats
162. Things That Are Still Important: Debugging Crashed Components caveats
163. Goes in hand with those strategies caveats
164. It’s not a Free Lunch caveats
165. It’s not a Free Lunch (it’s a good step toward one) caveats
166. conclusion
167. Crash-Only software is great! conclusion
168. Don’t need to wake up! Crash-Only Software Is Great
169. Stuff just works. Crash-Only Software Is Great
170. Failures are fun to look at. Crash-Only Software Is Great
171. It feels good. Crash-Only Software Is Great
172. Easy to fix when it goes wrong. Crash-Only Software Is Great
173. Reduces complexity in code. Crash-Only Software Is Great
174. Don’t need to wake up! Crash-Only Software Is Great
175. References Recursive Restartability Candea & Fox, 2001 Crash-Only software Candea & Fox, 2003 Crash-Only software, More than meets the eye LWN.net, https://lwn.net/Articles/191059/ A Crash Course In Failure NPlus1.org, http://web.archive.org/web/20090430014122/http://nplus1.org/articles/a-crash- course-in-failure/
176. La Fin comments, ideas: tweet me @AntoineGrondin