A Coolblue CI/CD Story

A Coolblue CI/CD Story

Similar to the presentation that I gave at Techorama, this tells the story about what we've needed to change in our organisation to make our CI/CD processes better, primarily autonomy, responsibility, ownership and (accepting, while learning from) failure.

C1d39cd3809e83af7958c6f45fa3c4f2?s=128

Pat Hermens

June 15, 2018
Tweet

Transcript

  1. None
  2. From 100 to 1,000+ deployments What does it take to

    get to the next level?
  3. Pat Hermens Currently Development Manager Coding for 15+ years Father

    & husband Rotterdam
  4. None
  5. None
  6. Why 1,000+ deployments/day?

  7. 7 @phermens ¯\_(ツ)_/¯

  8. 8 @phermens “It depends”

  9. 9 @phermens YAGNI & KISS

  10. None
  11. None
  12. None
  13. None
  14. None
  15. None
  16. None
  17. 100 / 1,000 / 10,000; so what?

  18. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf

  19. … Apparently we are too much trained to disregard differences

    in scale, to treat them as “gradual differences that are not essential.” We tell ourselves that what we can do once, we can also do twice and by induction, we fool ourselves into believing that we can do it as many times as needed, but this is just not true! ...
  20. ... A one-year old child will crawl on all fours

    with a speed of, say, one mile per hour. But a speed of a thousand miles per hour is that of a supersonic jet. Considered as objects with moving ability the child and the jet are incomparable, for whatever one can do the other cannot and vice versa. — Edsger Dijkstra, 1969
  21. None
  22. Prerequisites...

  23. None
  24. 24 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  25. 25 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  26. 26 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist ☑
  27. 27 @phermens Developing ☑ Coding guidelines ☑ Code reviews ☑

    Code analysis ☑ Short-lived branches ☑ Feature toggles Monitoring ☑ Defined key metrics ☑ Measure – create dashboards ☑ Proactivity – add alerting (& act!) Testing ☑ Unit tests ☑ Module tests ☑ Integration tests ☑ End-to-end tests ☑ Test automation Deploying ☑ Continuous Integration ☑ Continuous Delivery ☑ Infrastructure as Code The “faster to master” checklist
  28. None
  29. Story time!

  30. Responsibility Autonomy Failure Ownership

  31. Responsibility Autonomy Failure Ownership

  32. Responsibility

  33. 33 @phermens Hosting & Deployment “DevOps”

  34. 34 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team B Team C
  35. 35 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  36. 36 @phermens Hosting & Deployment “DevOps” Team A Team D

    Team F Team B Team C Team E
  37. 37 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  38. 38 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  39. 39 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  40. 40 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  41. 41 @phermens Hosting & Deployment “DevOps” Team H Team A

    Team D Team G Team E Team J Team F Team B Team I Team C
  42. 42 @phermens Team H Team A Team C Team G

    Team E Team J Team F Team B Team I Team C Team D Team L Team K
  43. 43 @phermens

  44. 44 @phermens

  45. Responsibility Autonomy Failure Ownership

  46. Responsibility Autonomy Failure Ownership

  47. Autonomy

  48. 48 @phermens

  49. 49 @phermens Log4Net + RDP

  50. 50 @phermens Serilog + Splunk + DataDog

  51. None
  52. None
  53. None
  54. None
  55. Responsibility Autonomy Failure Ownership

  56. Responsibility Autonomy Failure Ownership

  57. 57 @phermens Ownership

  58. 58 @phermens Build System

  59. 59 @phermens

  60. 60 @phermens

  61. 61 @phermens Development team tooling choices

  62. 62 @phermens Infrastructure team tooling choices

  63. 63 @phermens

  64. 64 @phermens What’s next?

  65. 65 @phermens ? What’s next?

  66. 66 @phermens ¯\_(ツ)_/¯

  67. 67 @phermens

  68. Responsibility Autonomy Failure Ownership

  69. Responsibility Autonomy Failure Ownership

  70. 70 @phermens Fail early, fail often, but always fail forward.

  71. 71 @phermens Failure

  72. 72 @phermens

  73. None
  74. None
  75. None
  76. None
  77. 77 @phermens

  78. None
  79. 79 @phermens

  80. 80 @phermens

  81. None
  82. 82 @phermens #war-room

  83. 83 @phermens

  84. Responsibility Autonomy Failure Ownership

  85. Responsibility Autonomy Failure Ownership

  86. Questions? If not, drinks follow this talk!

  87. Checklists!

  88. 88 @phermens The “faster to master” checklist Developing ⬜ Coding

    guidelines ⬜ Code reviews ⬜ Code analysis ⬜ Short-lived branches ⬜ Feature toggles Monitoring ⬜ Defined key metrics ⬜ Measure – create dashboards ⬜ Proactivity – add alerting (& act!) Testing ⬜ Unit tests ⬜ Module tests ⬜ Integration tests ⬜ End-to-end tests ⬜ Test automation Deploying ⬜ Continuous Integration ⬜ Continuous Delivery ⬜ Infrastructure as Code
  89. 89 @phermens The “100 to 1,000” checklist Responsibility ⬜ Can

    I control my deployments? ⬜ Have I removed all bottlenecks? ⬜ Do I have good support systems? ⬜ Am I collaborating with my peers? Autonomy ⬜ Freedom to innovate/improve? ⬜ Am I in control and independent? ⬜ Can I find out what went wrong? ⬜ Do I have the skills to fix it? Ownership ⬜ Agreed interfaces or contracts? ⬜ Out-of-the-box solutions? ⬜ Tooling is 100% automated? ⬜ Local build == “Production” build? Failure ⬜ Monitoring = Alerting = Action ⬜ Take pride in improving ⬜ Small deliverables, quickly ⬜ Non-blocking deployments
  90. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf

  91. 91 @phermens Thanks https://hermens.com.au @phermens