production: an owner's manual

A4b95be2145cc46f891707b6db9dd82d?s=47 Igor Wiedler
April 23, 2018
Programming
0
77

production: an owner's manual

from exec(ut) 2018

A4b95be2145cc46f891707b6db9dd82d?s=128

Igor Wiedler

April 23, 2018
Tweet

More Decks by Igor Wiedler

See All by Igor Wiedler
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
3
820
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
72
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
150
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
140
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
190
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
140
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
170
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
1
670
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
7
1.1k

Other Decks in Programming

See All in Programming
70ae2d149767d4198aac83453481b8dd?s=48 amaotone
0
140
F0493f5bcb4cc9fdccc54ae3e8ab6bd0?s=48 matsu7874
0
270
40575ebf9c2f4a6e3bcb68630f446a3b?s=48 tomzoh
4
120
9e9eb825c69d719f2d3c32bdd3bc971e?s=48 atp
3
620
D2151968789d3a241fe9ce2c19936661?s=48 hisakakoji
0
420
1fe64ebb176498be5f73ab51986c6b7b?s=48 davidfowl
9
3.6k
A030c34a2bc0d0610b4d87f8c56a2ab7?s=48 masatomix
0
150
3931098ae486b6df619050c63e24f6cc?s=48 osyo
1
280
44f37c1fe3a6a56376bb41a338741355?s=48 ahiru
3
320
Bb49e68eb46070029845fb3022d9026f?s=48 yukilabo
1
200
473b12ebcfb79adfaef97796e94fc25c?s=48 hironytic
0
130
9f33218af7ae5c04e102fcc3076f2f5c?s=48 yhkaplan
1
370

Featured

See All Featured
8ec1383b240b5ba15ffb9743fceb3c0e?s=48 philnash
4
290
282d599b414022eba5096510f675b6e2?s=48 chrislema
171
14k
C157b16234f1e75e8eac3698c1d4414a?s=48 ddemaree
273
31k
C9b18d9dff88a9dd2393364c2b3b21bd?s=48 colly
185
14k
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
19
1k
465724d73fe3a92c0879fdfb43a3a6f3?s=48 philhawksworth
190
17k
5b9fe87ec1faa67a4599782930f45ec9?s=48 sstephenson
144
12k
D36d2c1821af9249b69ff7f5ed60529b?s=48 tammielis
236
23k
79acae287842acf29084005533a7fb3b?s=48 hursman
104
9.1k
7c8469ee8c9e594c65c59b919626c08d?s=48 scottboms
250
11k
56c4053438af8e8b90d6f53cbb7573be?s=48 jakevdp
770
190k
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
236
9.8k

Transcript

  1. production: an owner's manual

  2. hello!

  3. broken computers

  4. None

  5. getting sidetracked now so sorry* * not sorry

  6. None
  7. None
  8. None

  9. back to serious business

  10. !

  11. None

  12. a production system is a system that serves real users

  13. the goal of operations is to ensure services are reliable

  14. in order to provide a good user experience

  15. None

  16. failure

  17. app

  18. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  19. app linux kernel the cloud

  20. None

  21. • cosmic rays • disk failure • power outages •

    software bugs • ...

  22. entropy

  23. None

  24. capacity

  25. None
  26. None
  27. None

  28. cascading failure

  29. None

  30. system design

  31. redundancy

  32. "

  33. scale

  34. None

  35. "

  36. p1 m3 c1 m2 m1 p2 c2

  37. data storage

  38. "

  39. "

  40. protocols

  41. None

  42. monitoring

  43. many components many req/s

  44. None

  45. measure all the things?

  46. ✅ ⏱

  47. golden signals • latency • traffic • errors • saturation

  48. golden signals • latency • traffic • errors • saturation

  49. golden signals • latency • traffic • errors • saturation

  50. golden signals • latency • traffic • errors • saturation

  51. golden signals • latency • traffic • errors • saturation

    0 - 50 [1620]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (74.55%) 50 - 100 [ 447]: ∎∎∎∎∎∎∎∎∎∎ (20.57%) 100 - 150 [ 49]: ∎ (2.25%) 150 - 200 [ 15]: (0.69%) 200 - 250 [ 15]: (0.69%) 250 - 300 [ 10]: (0.46%) 300 - 350 [ 6]: (0.28%) 350 - 400 [ 1]: (0.05%) 400 - 450 [ 0]: (0.00%) 450 - 500 [ 4]: (0.18%)

  52. golden signals • latency • traffic • errors • saturation

  53. saturation traffic latency errors

  54. None

  55. humans

  56. None

  57. oops, deleted the database

  58. bad human!

  59. why does this button even exist?

  60. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber

  61. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans

  62. app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans h u m a n s

  63. epic failure is almost always systemic

  64. failure

  65. recap

  66. • a production system serves real users • users like

    things that work and are fast • epic failure is almost always systemic

  67. thx @igorwhilefalse

  68. None