production: an owner's manual

A4b95be2145cc46f891707b6db9dd82d?s=47 Igor Wiedler
April 23, 2018
Programming
0
64

production: an owner's manual

from exec(ut) 2018

A4b95be2145cc46f891707b6db9dd82d?s=128

Igor Wiedler

April 23, 2018
Tweet

Want more?

A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
3
790
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
48
A4b95be2145cc46f891707b6db9dd82d?s=48 igorw
0
150
3ab1249be442027903e1180025340b3f?s=48 reverentgeek
13
760
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
4
410
766b9746b59e6f09e280cd33cf4ed419?s=48 skipperchong
0
260
61857dafbd287b3027c4dcea9008ad3c?s=48 carmenhchung
8
550
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
9
300
C0390e8b11079f8761c0cd3274ed61cb?s=48 productmarketing
3
350
C35e01544a0dd94a2cf1619ee8a42ebb?s=48 clairelew
4
800
6bb82b13cda86d3b821fa44100335ddf?s=48 thoeni
1
210
B3ace07412a5178ea778e7eec161fc0a?s=48 jcasabona
4
280

Transcript

  1. 1.

    production: an owner's manual

  2. 2.

    hello!

  3. 3.

    broken computers

  4. 4.
    None
  5. 5.

    getting sidetracked now so sorry* * not sorry

  6. 6.
    None
  7. 7.
    None
  8. 8.
    None
  9. 9.

    back to serious business

  10. 10.

    !

  11. 11.
    None
  12. 12.

    a production system is a system that serves real users

  13. 13.

    the goal of operations is to ensure services are reliable

  14. 14.

    in order to provide a good user experience

  15. 15.
    None
  16. 16.

    failure

  17. 17.

    app

  18. 18.

    app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber
  19. 19.

    app linux kernel the cloud

  20. 20.
    None
  21. 21.

    • cosmic rays • disk failure • power outages •

    software bugs • ...
  22. 22.

    entropy

  23. 23.
    None
  24. 24.

    capacity

  25. 25.
    None
  26. 26.
    None
  27. 27.
    None
  28. 28.

    cascading failure

  29. 29.
    None
  30. 30.

    system design

  31. 31.

    redundancy

  32. 32.

    "

  33. 33.

    scale

  34. 34.
    None
  35. 35.

    "

  36. 36.

    p1 m3 c1 m2 m1 p2 c2

  37. 37.

    data storage

  38. 38.

    "

  39. 39.

    "

  40. 40.

    protocols

  41. 41.
    None
  42. 42.

    monitoring

  43. 43.

    many components many req/s

  44. 44.
    None
  45. 45.

    measure all the things?

  46. 46.

    ✅ ⏱

  47. 47.

    golden signals • latency • traffic • errors • saturation

  48. 48.

    golden signals • latency • traffic • errors • saturation

  49. 49.

    golden signals • latency • traffic • errors • saturation

  50. 50.

    golden signals • latency • traffic • errors • saturation

  51. 51.

    golden signals • latency • traffic • errors • saturation

    0 - 50 [1620]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (74.55%) 50 - 100 [ 447]: ∎∎∎∎∎∎∎∎∎∎ (20.57%) 100 - 150 [ 49]: ∎ (2.25%) 150 - 200 [ 15]: (0.69%) 200 - 250 [ 15]: (0.69%) 250 - 300 [ 10]: (0.46%) 300 - 350 [ 6]: (0.28%) 350 - 400 [ 1]: (0.05%) 400 - 450 [ 0]: (0.00%) 450 - 500 [ 4]: (0.18%)
  52. 52.

    golden signals • latency • traffic • errors • saturation

  53. 53.

    saturation traffic latency errors

  54. 54.
    None
  55. 55.

    humans

  56. 56.
    None
  57. 57.

    oops, deleted the database

  58. 58.

    bad human!

  59. 59.

    why does this button even exist?

  60. 60.

    app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber
  61. 61.

    app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans
  62. 62.

    app linux kernel cpu dram disk network power supply switches

    load balancer dns submarine cables routers fiber humans h u m a n s
  63. 63.

    epic failure is almost always systemic

  64. 64.

    failure

  65. 65.

    recap

  66. 66.

    • a production system serves real users • users like

    things that work and are fast • epic failure is almost always systemic
  67. 67.

    thx @igorwhilefalse

  68. 68.
    None