A Coolblue CI/CD Story

Similar to the presentation that I gave at Techorama, this tells the story about what we've needed to change in our organisation to make our CI/CD processes better, primarily autonomy, responsibility, ownership and (accepting, while learning from) failure.

Transcript

2. From 100 to 1,000+ deployments What does it take to get to the next level?
3. Pat Hermens Currently Development Manager Coding for 15+ years Father & husband Rotterdam
6. Why 1,000+ deployments/day?
7. 7 @phermens ¯\_(ツ)_/¯
8. 8 @phermens “It depends”
9. 9 @phermens YAGNI & KISS
17. 100 / 1,000 / 10,000; so what?
18. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf
19. … Apparently we are too much trained to disregard differences in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! ...
20. ... A one-year old child will crawl on all fours with a speed of, say, one mile per hour. But a speed of a thousand miles per hour is that of a supersonic jet. Considered as objects with moving ability the child and the jet are incomparable, for whatever one can do the other cannot and vice versa. — Edsger Dijkstra, 1969
22. Prerequisites...
24. 24 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
25. 25 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
26. 26 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist ☑
27. 27 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑ Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
29. Story time!
30. Responsibility Autonomy Failure Ownership
31. Responsibility Autonomy Failure Ownership
32. Responsibility
33. 33 @phermens Hosting & Deployment “DevOps”
34. 34 @phermens Hosting & Deployment “DevOps” Team A Team D Team B Team C
35. 35 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
36. 36 @phermens Hosting & Deployment “DevOps” Team A Team D Team F Team B Team C Team E
37. 37 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
38. 38 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
39. 39 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
40. 40 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
41. 41 @phermens Hosting & Deployment “DevOps” Team H Team A Team D Team G Team E Team J Team F Team B Team I Team C
42. 42 @phermens Team H Team A Team C Team G Team E Team J Team F Team B Team I Team C Team D Team L Team K
43. 43 @phermens
44. 44 @phermens
45. Responsibility Autonomy Failure Ownership
46. Responsibility Autonomy Failure Ownership
47. Autonomy
48. 48 @phermens
49. 49 @phermens Log4Net + RDP
50. 50 @phermens Serilog + Splunk + DataDog
55. Responsibility Autonomy Failure Ownership
56. Responsibility Autonomy Failure Ownership
57. 57 @phermens Ownership
58. 58 @phermens Build System
59. 59 @phermens
60. 60 @phermens
61. 61 @phermens Development team tooling choices
62. 62 @phermens Infrastructure team tooling choices
63. 63 @phermens
64. 64 @phermens What’s next?
65. 65 @phermens ? What’s next?
66. 66 @phermens ¯\_(ツ)_/¯
67. 67 @phermens
68. Responsibility Autonomy Failure Ownership
69. Responsibility Autonomy Failure Ownership
70. 70 @phermens Fail early, fail often, but always fail forward.
71. 71 @phermens Failure
72. 72 @phermens
77. 77 @phermens
79. 79 @phermens
80. 80 @phermens
82. 82 @phermens #war-room
83. 83 @phermens
84. Responsibility Autonomy Failure Ownership
85. Responsibility Autonomy Failure Ownership
86. Questions? If not, drinks follow this talk!
87. Checklists!
88. 88 @phermens The “faster to master” checklist Developing ⬜ Coding guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
89. 89 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
90. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf
91. 91 @phermens Thanks https://hermens.com.au @phermens