From 100, to 1000+ deployments a day [TechoramaNL editie]

C1d39cd3809e83af7958c6f45fa3c4f2?s=47 Pat Hermens
October 03, 2018

From 100, to 1000+ deployments a day [TechoramaNL editie]

C1d39cd3809e83af7958c6f45fa3c4f2?s=128

Pat Hermens

October 03, 2018
Tweet

Transcript

  1. None
  2. From 100 to 1,000+ deployments a day Pat Hermens @phermens

  3. From 100 to 1,000+ deployments a day Pat Hermens @phermens

  4. Pat Hermens Development Manager Coding for 15+ years Father &

    husband Rotterdam
  5. None
  6. None
  7. Why 1,000+ deployments/day?

  8. 8 @phermens ¯\_(ツ)_/¯

  9. 9 @phermens “It depends”

  10. 10 @phermens YAGNI & KISS

  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. None
  18. 100 / 1,000 / 10,000; so what?

  19. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf

  20. … Apparently we are too much trained to disregard differences

    in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! ...
  21. ... A one-year old child will crawl on all fours

    with a speed of, say, one mile per hour. But a speed of a thousand miles per hour is that of a supersonic jet. Considered as objects with moving ability the child and the jet are incomparable, for whatever one can do the other cannot and vice versa. — Edsger Dijkstra, 1969
  22. None
  23. Prerequisites...

  24. None
  25. 25 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  26. 26 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  27. 27 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist ☑
  28. 28 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  29. None
  30. Story time!

  31. Responsibility Autonomy Failure Ownership

  32. Responsibility Autonomy Failure Ownership

  33. Responsibility

  34. 34 @phermens Hosting & Deployment “DevOps”

  35. 35 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team B Team C
  36. 36 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  37. 37 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  38. 38 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  39. 39 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  40. 40 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  41. 41 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  42. 42 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  43. 43 @phermens Team H Team A Team C Team G

    Team E Team J Team F Team B Team I Team C Team D Team L Hosting & Deploy- ment
  44. 44 @phermens

  45. 45 @phermens

  46. 46 @phermens

  47. Responsibility Autonomy Failure Ownership

  48. Responsibility Autonomy Failure Ownership

  49. Autonomy

  50. Autonomy

  51. 51 @phermens Serilog + Splunk + DataDog

  52. None
  53. None
  54. None
  55. None
  56. None
  57. Autonomy

  58. None
  59. 59 @phermens

  60. 60 @phermens

  61. 61 @phermens

  62. 62 @phermens $ $ $ $ $

  63. 63 @phermens $ $ $ $ $

  64. 64 @phermens

  65. 65 @phermens UDP

  66. 66 @phermens UDP

  67. 67 @phermens UDP

  68. 68 @phermens UDP

  69. 69 @phermens UDP

  70. 70 @phermens

  71. 71 @phermens TCP

  72. 72 @phermens TCP

  73. 73 @phermens

  74. 74 @phermens

  75. 75 @phermens API Gateway Lambda CloudWatch

  76. 76 @phermens API Gateway Kinesis Lambda CloudWatch

  77. 77 @phermens AWS Lambda stats for Vanessa Optima Prime (as

    of 2018-09-28) Last 30 days Last 7 days
  78. 78 @phermens AWS Lambda stats for Vanessa Optima Prime (as

    of 2018-09-28) Last 30 days Last 7 days
  79. 79 @phermens

  80. 80 @phermens

  81. Responsibility Autonomy Failure Ownership

  82. Responsibility Autonomy Failure Ownership

  83. 83 @phermens Ownership

  84. 84 @phermens Build System

  85. 85 @phermens

  86. 86 @phermens

  87. 87 @phermens Development team tooling choices

  88. 88 @phermens Infrastructure team tooling choices

  89. 89 @phermens

  90. 90 @phermens What’s next?

  91. 91 @phermens What’s next? ?

  92. 92 @phermens ¯\_(ツ)_/¯

  93. 93 @phermens

  94. Responsibility Autonomy Failure Ownership

  95. Responsibility Autonomy Failure Ownership

  96. 96 @phermens Fail early, fail often, but always fail forward.

  97. 97 @phermens Failure

  98. 98 @phermens

  99. 99 @phermens

  100. None
  101. None
  102. None
  103. None
  104. 104 @phermens

  105. None
  106. NOTE: 20 cubic centimeters

  107. None
  108. NOTE: Now 50cc!

  109. None
  110. NOTE: 49 cubic centimeters

  111. None
  112. None
  113. NOTE: More than 49cc’s!

  114. None
  115. None
  116. None
  117. None
  118. NOTE: Now 45cc!

  119. None
  120. None
  121. 121 @phermens

  122. 122 @phermens

  123. None
  124. 124 @phermens #war-room

  125. 125 @phermens

  126. Responsibility Autonomy Failure Ownership

  127. Responsibility Autonomy Failure Ownership

  128. 128 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can

    I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  129. Questions?

  130. Checklists!

  131. 131 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  132. 132 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can

    I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  133. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf

  134. 134 @phermens Thanks https://hermens.com.au @phermens

  135. @phermens pat@coolblue.nl Pat Hermens @phermens Thanks!