Upgrade to Pro — share decks privately, control downloads, hide ads and more …

production: an owner's manual

A4b95be2145cc46f891707b6db9dd82d?s=47 Igor Wiedler
April 23, 2018
Programming
0
91

production: an owner's manual

from exec(ut) 2018

A4b95be2145cc46f891707b6db9dd82d?s=128

Igor Wiedler

April 23, 2018
Tweet

More Decks by Igor Wiedler

See All by Igor Wiedler
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
3
840
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
79
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
160
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
170
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
190
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
160
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
170
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
730
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
7
1.1k

Other Decks in Programming

See All in Programming
3c9c5f9a91cac73073db8bb2903bd968?s=48 mamy1326
12
3.1k
7867fe52a9be4257508a516d4df61578?s=48 tkmnzm
0
150
F21d69109b1c03921abf7d12f0fb6654?s=48 dscs
0
130
Fe1782174fa3f0a2d07011df414a1267?s=48 evils
0
230
6aaaf03c4f5c681d89c3078bd6070a6f?s=48 rumi_yamaguchi
0
1.4k
2c0947c6a28e7f771ebd9859ecf54e5c?s=48 shinyorke
0
240
15934fa2aa7b2ce21f091e9b7cffa856?s=48 manfredsteyer
PRO
0
150
80a3a3857a55f154d23acb705eff72cc?s=48 star_zero
2
4.1k
E0e036f89c14b3e59640318eedf9670b?s=48 bkuhlmann
0
130
6c8b509fe5422470d148c2c4cf2eb4b0?s=48 jrodbx
4
280
6e704ad00f166b420eba3d174713c82b?s=48 urushiyama
0
390
0c2ae3d2aa77423465e1e533ff0d030a?s=48 mski_iksm
5
1.9k

Featured

See All Featured
Eb8975af8e49e19e3dd6b6b84a542e26?s=48 qrush
288
19k
3d6ace9554821d552146413bcdf874f6?s=48 trishagee
37
4.9k
0c6fd6a08d0f898159cda7cafcacf07f?s=48 afnizarnur
180
14k
5f50f94346fbcb23706f0707169317a4?s=48 aarron
261
36k
E9221b172e78e888355fa2fe4541b595?s=48 mclloyd
PRO
6
7.5k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
35
3.8k
C86443373fbfe92996aa882d0d7a59f8?s=48 imathis
484
150k
C7393b7ba7ec9c8890dd77d209fbb3c9?s=48 maltzj
504
37k
20bfe76b3d6105641f879fe45cfc9272?s=48 bkeepers
PRO
324
54k
95d9f37115fbb0d13135d0f77132f856?s=48 morganepeng
31
1.8k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
372
43k
Ecdea9b9714877b86cee08458f085481?s=48 trallard
25
1.2k

Transcript

  1. production: an owner's manual

  2. hello!

  3. broken computers

  4. None

  5. getting sidetracked now so sorry* * not sorry

  6. None
  7. None
  8. None

  9. back to serious business

  10. !

  11. None

  12. a production system is a system that serves real users

  13. the goal of operations is to ensure services are reliable

  14. in order to provide a good user experience

  15. None

  16. failure

  17. app

  18. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  19. app linux kernel the cloud

  20. None

  21. • cosmic rays • disk failure • power outages •

    software bugs • ...

  22. entropy

  23. None

  24. capacity

  25. None
  26. None
  27. None

  28. cascading failure

  29. None

  30. system design

  31. redundancy

  32. "

  33. scale

  34. None

  35. "

  36. p1 m3 c1 m2 m1 p2 c2

  37. data storage

  38. "

  39. "

  40. protocols

  41. None

  42. monitoring

  43. many components many req/s

  44. None

  45. measure all the things?

  46. ✅ ⏱

  47. golden signals • latency • traffic • errors • saturation

  48. golden signals • latency • traffic • errors • saturation

  49. golden signals • latency • traffic • errors • saturation

  50. golden signals • latency • traffic • errors • saturation

  51. golden signals • latency • traffic • errors • saturation

    0 - 50 [1620]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (74.55%) 50 - 100 [ 447]: ∎∎∎∎∎∎∎∎∎∎ (20.57%) 100 - 150 [ 49]: ∎ (2.25%) 150 - 200 [ 15]: (0.69%) 200 - 250 [ 15]: (0.69%) 250 - 300 [ 10]: (0.46%) 300 - 350 [ 6]: (0.28%) 350 - 400 [ 1]: (0.05%) 400 - 450 [ 0]: (0.00%) 450 - 500 [ 4]: (0.18%)

  52. golden signals • latency • traffic • errors • saturation

  53. saturation traffic latency errors

  54. None

  55. humans

  56. None

  57. oops, deleted the database

  58. bad human!

  59. why does this button even exist?

  60. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  61. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans

  62. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans h u m a n s

  63. epic failure is almost always systemic

  64. failure

  65. recap

  66. • a production system serves real users • users like

    things that work and are fast • epic failure is almost always systemic

  67. thx @igorwhilefalse

  68. None