Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Servers are doomed to fail

E7526ec3e801f8ba99f6746498a154a6?s=47 JBD
May 17, 2019
Technology
2
1.1k

Servers are doomed to fail

E7526ec3e801f8ba99f6746498a154a6?s=128

JBD

May 17, 2019
Tweet

More Decks by JBD

See All by JBD
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
5
1.2k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
7
1.5k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
120
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
110
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
870
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
3.6k
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
1
190
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
0
100
E7526ec3e801f8ba99f6746498a154a6?s=48 rakyll
2
1.5k

Other Decks in Technology

See All in Technology
Bd82857e5cd002562af37c7b6b9bc561?s=48 yokatsuki
1
190
93a5ee4559a212954cc17f072ae16748?s=48 yukitodate
2
320
Bf7fe621f4fe1615c228ef8a79b87282?s=48 shirayanagiryuji
1
400
567600e04dbcb14d6bd8f120e6625a27?s=48 tsuyo
0
180
3487ed46845ad0c09d36c3c1c0533ea6?s=48 masakick
0
120
7cd783377515bdf8207062840b7b2f4e?s=48 soracom
0
260
A8f0494d61f79b35ff8a4902c7decb73?s=48 iqbocchi
0
520
A4d5b3aae2e67f31b513c88c726514c2?s=48 smzksts
0
210
2ee165e506f59e3fe6f463bf7f150ccb?s=48 minamizaki
0
560
F3c727195d975cf7a4b300e4f5c2c0d1?s=48 am7cinnamon
2
2.7k
706ff501573a736401aa4de5adc88e05?s=48 nihonbuson
2
1.6k
E783e2d26dc6388730b73ced1348c638?s=48 vkbaba
0
110

Featured

See All Featured
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
310
21k
96270e4c3e5e9806cf7245475c00b275?s=48 addyosmani
494
110k
7653c7c449918ff6542880a8c284f5e7?s=48 jponch
103
4.9k
E837d2996f52008ace055a08fbfff992?s=48 frogandcode
127
20k
9952dfcd5d338f8a8e7175c8a8f65fb5?s=48 wjessup
338
16k
F68cb25ae3e5c2106997454b13b7737d?s=48 brad_frost
156
6.4k
9cde37f47e4a800ea081ea42de8d749a?s=48 dougneiner
55
5.4k
1b8ad785acdd1ce1c99914b1c2a4e10e?s=48 sugarenia
233
830k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
437
28k
5b6a2bd86ef643786a381a212e0ef2f1?s=48 geoffreycrofte
18
770
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
11
740
7edec06b5435e0dac07a353b2cc26253?s=48 geeforr
332
29k

Transcript

  1. Servers are doomed to fail Jaana B. Dogan jbd@google.com @rakyll

  2. Serverless is also doomed to fail Jaana B. Dogan jbd@google.com

    @rakyll

  3. Systems are doomed to fail Jaana B. Dogan jbd@google.com @rakyll

  4. Is failure OK? Is failure an unexpected case?

  5. Failure is not an exception. Systems change all the time.

  6. “I haven’t touched the code for a century, it should

    just work.” Said no one ever.

  7. Failure is expected. Yes, it is.

  8. None

  9. @rakyll monitoring debugging postmortem

  10. Monitoring is about saying if something is broken.

  11. “99.99% of the requests should return in 100ms.”

  12. @rakyll

  13. @rakyll

  14. Debugging

  15. Debugging is collaborative.

  16. Debugging comes in flavors. Logs Traces Metrics ...

  17. Postmortems

  18. Postmortems

  19. Postmortems

  20. Blameless? Focus on identifying problems.

  21. Collaboration Design for collaboration.

  22. Design for failure Set SLOs, plan for instrumentation, plan for

    debugging.

  23. Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google

  24. Correlation Jump from monitoring/debugging data to data.

  25. On-call debugging Jump from distributed tracing data to on-call information.

    who to page?

  26. Dynamic collection Capability to enable more collection in production when

    needed.

  27. Continuous collection Continuously collect signals, generate fleet-wide analysis reports.

  28. Introspection Introspection pages provided from the services.

  29. @rakyll monitoring debugging postmortem

  30. Thank you Jaana B. Dogan Google jbd@google.com