From 100, to 1000+ deployments a day [TechoramaNL editie]

Transcript

2. From 100 to 1,000+ deployments a day Pat Hermens @phermens
3. From 100 to 1,000+ deployments a day Pat Hermens @phermens
4. Pat Hermens Development Manager Coding for 15+ years Father & husband Rotterdam
7. Why 1,000+ deployments/day?
8. 8 @phermens ¯\_(ツ)_/¯
9. 9 @phermens “It depends”
10. 10 @phermens YAGNI & KISS
18. 100 / 1,000 / 10,000; so what?
19. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf
20. … Apparently we are too much trained to disregard differences in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! ...
21. ... A one-year old child will crawl on all fours with a speed of, say, one mile per hour. But a speed of a thousand miles per hour is that of a supersonic jet. Considered as objects with moving ability the child and the jet are incomparable, for whatever one can do the other cannot and vice versa. — Edsger Dijkstra, 1969
23. Prerequisites...
25. 25 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
26. 26 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
27. 27 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist ☑
28. 28 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
30. Story time!
31. Responsibility Autonomy Failure Ownership
32. Responsibility Autonomy Failure Ownership
33. Responsibility
34. 34 @phermens Hosting & Deployment “DevOps”
35. 35 @phermens Hosting & Deployment “DevOps” Team A Team D Team B Team C
36. 36 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
37. 37 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
38. 38 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
39. 39 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
40. 40 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
41. 41 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
42. 42 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
43. 43 @phermens Team H Team A Team C Team G Team E Team J Team F Team B Team I Team C Team D Team L Hosting & Deploy- ment
44. 44 @phermens
45. 45 @phermens
46. 46 @phermens
47. Responsibility Autonomy Failure Ownership
48. Responsibility Autonomy Failure Ownership
49. Autonomy
50. Autonomy
51. 51 @phermens Serilog + Splunk + DataDog
57. Autonomy
59. 59 @phermens
60. 60 @phermens
61. 61 @phermens
62. 62 @phermens $ $ $ $ $
63. 63 @phermens $ $ $ $ $
64. 64 @phermens
65. 65 @phermens UDP
66. 66 @phermens UDP
67. 67 @phermens UDP
68. 68 @phermens UDP
69. 69 @phermens UDP
70. 70 @phermens
71. 71 @phermens TCP
72. 72 @phermens TCP
73. 73 @phermens
74. 74 @phermens
75. 75 @phermens API Gateway Lambda CloudWatch
76. 76 @phermens API Gateway Kinesis Lambda CloudWatch
77. 77 @phermens AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28) Last 30 days Last 7 days
78. 78 @phermens AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28) Last 30 days Last 7 days
79. 79 @phermens
80. 80 @phermens
81. Responsibility Autonomy Failure Ownership
82. Responsibility Autonomy Failure Ownership
83. 83 @phermens Ownership
84. 84 @phermens Build System
85. 85 @phermens
86. 86 @phermens
87. 87 @phermens Development team tooling choices
88. 88 @phermens Infrastructure team tooling choices
89. 89 @phermens
90. 90 @phermens What’s next?
91. 91 @phermens What’s next? ?
92. 92 @phermens ¯\_(ツ)_/¯
93. 93 @phermens
94. Responsibility Autonomy Failure Ownership
95. Responsibility Autonomy Failure Ownership
96. 96 @phermens Fail early, fail often, but always fail forward.
97. 97 @phermens Failure
98. 98 @phermens
99. 99 @phermens
104. 104 @phermens
106. NOTE: 20 cubic centimeters
108. NOTE: Now 50cc!
110. NOTE: 49 cubic centimeters
113. NOTE: More than 49cc’s!
118. NOTE: Now 45cc!
121. 121 @phermens
122. 122 @phermens
124. 124 @phermens #war-room
125. 125 @phermens
126. Responsibility Autonomy Failure Ownership
127. Responsibility Autonomy Failure Ownership
128. 128 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
129. Questions?
130. Checklists!
131. 131 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
132. 132 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
133. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf
134. 134 @phermens Thanks https://hermens.com.au @phermens
135. @phermens pat@coolblue.nl Pat Hermens @phermens Thanks!