From 100 to 1000+ deployments a day [CodeMotion Edition]

From 100 to 1000+ deployments a day [CodeMotion Edition]

The slides from this mornings session about what it takes to TRULY go faster as a team/department/group of developers.

C1d39cd3809e83af7958c6f45fa3c4f2?s=128

Pat Hermens

April 02, 2019
Tweet

Transcript

  1. From 100 to 1000+ deployments a day Pat Hermens Amsterdam

    | April 2-3, 2019
  2. From 100 to 1,000+ deployments What does it take to

    get to the next level?
  3. Pat Hermens Development Manager Coding for 15+ years Father &

    husband Rotterdam, Netherlands @phermens pat@coolblue.nl
  4. None
  5. Why 1,000+ deployments/day?

  6. 6 @phermens ¯\_(ツ)_/¯

  7. 7 @phermens “It depends”

  8. None
  9. None
  10. None
  11. None
  12. None
  13. None
  14. None
  15. 100 / 1,000 / 10,000; so what?

  16. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf

  17. … Apparently we are too much trained to disregard differences

    in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! — Edsger Dijkstra, 1969
  18. None
  19. Prerequisites...

  20. None
  21. 21 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  22. 22 @phermens Developing ☑ Coding guidelines ☑ Code reviews ⬜

    Code analysis ☑ Short-lived branches ⬜ Feature toggles Monitoring ☑ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ☑ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code The “faster to master” checklist
  23. 23 @phermens Developing ☑ Coding guidelines ☑ Code reviews ⬜

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ⬜ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ⬜ Infrastructure as Code The “faster to master” checklist
  24. 24 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  25. Story time!

  26. Responsibility Autonomy Failure Ownership

  27. Responsibility Autonomy Failure Ownership

  28. Responsibility

  29. 29 @phermens Hosting & Deployment “DevOps”

  30. 30 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team B Team C
  31. 31 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  32. 32 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  33. 33 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  34. 34 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  35. 35 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  36. 36 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  37. 37 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  38. 38 @phermens Team H Team A Team C Team G

    Team E Team J Team F Team B Team I Team C Team D Team L Deploy- ment
  39. 39 @phermens

  40. Responsibility Autonomy Failure Ownership

  41. Responsibility Autonomy Failure Ownership

  42. Autonomy

  43. Autonomy

  44. 44 @phermens Serilog + Splunk + DataDog

  45. None
  46. None
  47. None
  48. None
  49. None
  50. Autonomy

  51. None
  52. 52 @phermens

  53. 53 @phermens

  54. 54 @phermens

  55. 55 @phermens

  56. 56 @phermens

  57. 57 @phermens

  58. 58 @phermens UDP

  59. 59 @phermens UDP

  60. 60 @phermens UDP

  61. 61 @phermens UDP

  62. 62 @phermens UDP

  63. 63 @phermens

  64. 64 @phermens TCP

  65. 65 @phermens TCP

  66. 66 @phermens

  67. 67 @phermens

  68. 68 @phermens

  69. 69 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  70. 70 @phermens

  71. 71 @phermens

  72. 72 @phermens

  73. 73 @phermens

  74. 74 @phermens

  75. Responsibility Autonomy Failure Ownership

  76. Responsibility Autonomy Failure Ownership

  77. 77 @phermens Ownership

  78. 78 @phermens Build System

  79. 79 @phermens

  80. 80 @phermens

  81. 81 @phermens Development team tooling choices

  82. 82 @phermens Infrastructure team tooling choices

  83. 83 @phermens

  84. 84 @phermens What’s next?

  85. 85 @phermens ? What’s next?

  86. 86 @phermens ¯\_(ツ)_/¯

  87. 87 @phermens

  88. Responsibility Autonomy Failure Ownership

  89. Responsibility Autonomy Failure Ownership

  90. 90 @phermens Fail early, fail often, but always fail forward.

  91. 91 @phermens Failure

  92. None
  93. None
  94. None
  95. None
  96. None
  97. None
  98. 98 @phermens [REDACTED] [REDACTED]

  99. None
  100. None
  101. None
  102. None
  103. None
  104. None
  105. None
  106. None
  107. None
  108. None
  109. None
  110. None
  111. None
  112. None
  113. None
  114. None
  115. 115 @phermens

  116. 116 @phermens

  117. None
  118. 118 @phermens #war-room

  119. Responsibility Autonomy Failure Ownership

  120. Responsibility Autonomy Failure Ownership

  121. 121 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  122. 122 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can

    I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces (or contracts?) ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  123. Questions?

  124. Checklists!

  125. 125 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  126. 126 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can

    I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  127. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf

  128. 128 @phermens Thanks! @phermens hermens.com.au pat@coolblue.nl careersatcoolblue.com