Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Servers are doomed to fail

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
May 17, 2019
Technology
3
1.2k

Servers are doomed to fail

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

May 17, 2019
Tweet

More Decks by JBD

See All by JBD
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
1.3k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
1k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
1.5k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
5
1.3k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
8
1.8k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
140
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
170
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
910
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
3.8k

Other Decks in Technology

See All in Technology
A61adad507d7bc0ee5b52ebe333abed7?s=48 hoto17296
0
260
7cb78d6108cc74bec0f2f001127f662a?s=48 hiroki_hasegawa
0
180
119659c28d16f22d01eb48a6f3ee1391?s=48 iwatatomoya
0
770
06986333d704c400f7a2053994058e6e?s=48 cmtamai
0
420
3318abf64ba11ee70d4f5dee62051801?s=48 tagucch
0
180
76df086a2bef4b357cf29acb07959cce?s=48 m_coder
4
4.4k
Dafc4723e9a1c067796c0fec6f509774?s=48 lemiorhan
2
1.5k
4c8dc0add99b3ba830fc28453a895ab7?s=48 suzukij1028
0
290
A19876f5694283f26a061746ba82c3b0?s=48 germsvel
1
180
928b1395afedebb5ee48d44ab917c5f1?s=48 ryusa
1
280
13d936e697fe0f4fa96f926d0a712f6c?s=48 sansanbuildersbox
PRO
0
150
Fad0d157244126a95094c26f2857d164?s=48 jukiya330
0
230

Featured

See All Featured
Aff5641764408271f7bc398f2097edd0?s=48 denniskardys
223
120k
F68cb25ae3e5c2106997454b13b7737d?s=48 brad_frost
162
6.8k
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
85
8.2k
7cbd3f5f922d798cc312eaf596de7ae6?s=48 iamctodd
32
3k
719435d98d452de7ac367c828266cf01?s=48 erikaheidi
29
5.1k
2bb923c57a1fdc3a2eb484bb8d565fd2?s=48 ufuk
61
7.2k
C4de88ea47538e4594750e3861a341c8?s=48 brettharned
96
3.3k
F29327647a9cff5c69618bae420792ea?s=48 tenderlove
58
3.8k
433acaea1012b25d97ae66da9b998dad?s=48 malarkey
193
9k
1b75d89772b7eea1f6c89fbf362607d9?s=48 moore
129
22k
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
11
1.1k
8081b26e05bb4354f7d65ffc34cbbd67?s=48 chriscoyier
147
20k

Transcript

  1. Servers are doomed to fail Jaana B. Dogan jbd@google.com @rakyll

  2. Serverless is also doomed to fail Jaana B. Dogan jbd@google.com

    @rakyll

  3. Systems are doomed to fail Jaana B. Dogan jbd@google.com @rakyll

  4. Is failure OK? Is failure an unexpected case?

  5. Failure is not an exception. Systems change all the time.

  6. “I haven’t touched the code for a century, it should

    just work.” Said no one ever.

  7. Failure is expected. Yes, it is.

  8. None

  9. @rakyll monitoring debugging postmortem

  10. Monitoring is about saying if something is broken.

  11. “99.99% of the requests should return in 100ms.”

  12. @rakyll

  13. @rakyll

  14. Debugging

  15. Debugging is collaborative.

  16. Debugging comes in flavors. Logs Traces Metrics ...

  17. Postmortems

  18. Postmortems

  19. Postmortems

  20. Blameless? Focus on identifying problems.

  21. Collaboration Design for collaboration.

  22. Design for failure Set SLOs, plan for instrumentation, plan for

    debugging.

  23. Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google

  24. Correlation Jump from monitoring/debugging data to data.

  25. On-call debugging Jump from distributed tracing data to on-call information.

    who to page?

  26. Dynamic collection Capability to enable more collection in production when

    needed.

  27. Continuous collection Continuously collect signals, generate fleet-wide analysis reports.

  28. Introspection Introspection pages provided from the services.

  29. @rakyll monitoring debugging postmortem

  30. Thank you Jaana B. Dogan Google jbd@google.com