Failing With Grace - Sean O'Connor

One of the biggest challenges of building distributed systems is dealing with failure. In this talk we'll cover a number of approaches and tools to help you build systems that deal with failure as gracefully as possible.

Transcript

1. Failing with Grace by @theSeanOC
2. shortlinks?
4. flickr.com/photos/lobo235/75085378
8. How hard can it be? 270,000,000 redirects 26,000,000 shortens 4,000,000 page crawls 400servers 2data centers each day we handle and we do it with 20+deploys 20engineers EVERY DAY
9. http://gph.is/15Jp0hq
10. Distributed Systems
12. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
13. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
14. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
15. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
16. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
17. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
18. A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages. The components interact with each other in order to achieve a common goal. Three significant characteristics of distributed systems are: concurrency of components, lack of a global clock, and independent failure of components. - Distributed Systems: Concepts and Design
 via seanoc.co/ds-cite
20. seanoc.co/1tSEw7d
21. seanoc.co/1tSGpAB
22. seanoc.co/1tSHxo7
23. seanoc.co/1tSHZCC
24. http://gph.is/165vbfC
25. http://gph.is/1aljymT
26. seanoc.co/1iMl7QC
27. A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable. - Leslie Lamport
29. seanoc.co/1mSI0Rb
30. seanoc.co/1mSIN4y
31. gph.is/1e7QIuO
32. seanoc.co/1mSKUVR
33. seanoc.co/1mSKEX6
34. seanoc.co/1mSK9MB
35. seanoc.co/1BlJ7SX
36. gph.is/1cXtHam
37. seanoc.co/1AmLeVg
38. gph.is/175t90L
39. gph.is/146J6jt
40. Async Queues
41. Async > Sync except when it isn’t
42. seanoc.co/1vT7RSo
43. seanoc.co/1vT8coj
44. seanoc.co/1pn5LVQ
45. gph.is/1jskbwn
46. gph.is/12HPWO8
48. nsq.io
49. seanoc.co/1pn8Ox7
50. github.com/bitly/pynsq
51. Timeouts
52. seanoc.co/1vT7JCo
53. gph.is/1cAMr0S
54. Retries
55. seanoc.co/1lat4Tu
56. seanoc.co/1latkli
57. seanoc.co/1lauddR
58. seanoc.co/1AmM8kz
59. github.com/jehiah/host_pool
61. Immutable Data
62. seanoc.co/1lauCwH
63. seanoc.co/1kogw9x
64. seanoc.co/1kohCSx
65. Monitoring
68. seanoc.co/1AmNo7q
70. Distributed Systems
71. seanoc.co/1mSIN4y
72. seanoc.co/1mSK9MB
73. seanoc.co/1vT7RSo
74. seanoc.co/1vT7JCo
75. seanoc.co/1lat4Tu
76. seanoc.co/1lauCwH
78. Questions?