Upgrade to Pro — share decks privately, control downloads, hide ads and more …

production: an owner's manual

A4b95be2145cc46f891707b6db9dd82d?s=47 Igor Wiedler
April 23, 2018
Programming
0
79

production: an owner's manual

from exec(ut) 2018

A4b95be2145cc46f891707b6db9dd82d?s=128

Igor Wiedler

April 23, 2018
Tweet

More Decks by Igor Wiedler

See All by Igor Wiedler
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
3
830
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
76
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
150
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
150
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
190
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
150
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
170
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
680
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
7
1.1k

Other Decks in Programming

See All in Programming
C6860c38adec703e995505b9afa12c83?s=48 yumemi
1
110
5afbf65e9e32cef9ab74e5b5347a5bce?s=48 korosuke613
2
260
74253b7e85d0a315a23d74db50abd8e4?s=48 coa00
2
160
Dcbf2269af2483cabf43128ad1718c11?s=48 grapecity_dev
0
180
428a9c6e5b151e618d4db288105bc5f3?s=48 rarous
0
170
B1196c22fd3ca181eec43e1b67823d3f?s=48 aftiopk
0
130
36e748c47de34549cfa3ac0950c5351d?s=48 ybrliiu
0
100
471605f6e94a32caa4b0101524f80429?s=48 kimyan
3
480
Fb469a72f1b12726aea391f479dc1b6d?s=48 gernotstarke
0
390
B8403d102456248570005ee7fb2ba0f7?s=48 tooppoo
0
200
A7c0e931e8189d7a709eaebb44ed69d4?s=48 rince
3
250
Dcbf2269af2483cabf43128ad1718c11?s=48 grapecity_dev
0
180

Featured

See All Featured
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
200
20k
F68cb25ae3e5c2106997454b13b7737d?s=48 brad_frost
157
6.4k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
437
28k
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
225
8.5k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
104
6.2k
9128d500301ae51524e887bb680f471d?s=48 caitiem20
308
17k
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
112
10k
Bfd6b681ec6e8c9bef82ff0521c364f7?s=48 kastner
54
1.9k
E190023b66e2b8aa73a842b106920c93?s=48 zenorocha
296
40k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
61
9.4k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
47k
B3d6434763caa0ef5dc4b792662c49f7?s=48 smashingmag
230
18k

Transcript

  1. production: an owner's manual

  2. hello!

  3. broken computers

  4. None

  5. getting sidetracked now so sorry* * not sorry

  6. None
  7. None
  8. None

  9. back to serious business

  10. !

  11. None

  12. a production system is a system that serves real users

  13. the goal of operations is to ensure services are reliable

  14. in order to provide a good user experience

  15. None

  16. failure

  17. app

  18. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  19. app linux kernel the cloud

  20. None

  21. • cosmic rays • disk failure • power outages •

    software bugs • ...

  22. entropy

  23. None

  24. capacity

  25. None
  26. None
  27. None

  28. cascading failure

  29. None

  30. system design

  31. redundancy

  32. "

  33. scale

  34. None

  35. "

  36. p1 m3 c1 m2 m1 p2 c2

  37. data storage

  38. "

  39. "

  40. protocols

  41. None

  42. monitoring

  43. many components many req/s

  44. None

  45. measure all the things?

  46. ✅ ⏱

  47. golden signals • latency • traffic • errors • saturation

  48. golden signals • latency • traffic • errors • saturation

  49. golden signals • latency • traffic • errors • saturation

  50. golden signals • latency • traffic • errors • saturation

  51. golden signals • latency • traffic • errors • saturation

    0 - 50 [1620]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (74.55%) 50 - 100 [ 447]: ∎∎∎∎∎∎∎∎∎∎ (20.57%) 100 - 150 [ 49]: ∎ (2.25%) 150 - 200 [ 15]: (0.69%) 200 - 250 [ 15]: (0.69%) 250 - 300 [ 10]: (0.46%) 300 - 350 [ 6]: (0.28%) 350 - 400 [ 1]: (0.05%) 400 - 450 [ 0]: (0.00%) 450 - 500 [ 4]: (0.18%)

  52. golden signals • latency • traffic • errors • saturation

  53. saturation traffic latency errors

  54. None

  55. humans

  56. None

  57. oops, deleted the database

  58. bad human!

  59. why does this button even exist?

  60. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  61. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans

  62. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans h u m a n s

  63. epic failure is almost always systemic

  64. failure

  65. recap

  66. • a production system serves real users • users like

    things that work and are fast • epic failure is almost always systemic

  67. thx @igorwhilefalse

  68. None