A Coolblue CI/CD Story

Similar to the presentation that I gave at Techorama, this tells the story about what we've needed to change in our organisation to make our CI/CD processes better, primarily autonomy, responsibility, ownership and (accepting, while learning from) failure.

Transcript

1. View Slide

2. From 100 to 1,000+ deployments
   What does it take to get to the next level?
   

   View Slide

3. Pat Hermens
   Currently Development Manager
   Coding for 15+ years
   Father & husband
   Rotterdam
   

   View Slide

4. View Slide

5. View Slide

6. Why 1,000+ deployments/day?
   

   View Slide

7. 7
   @phermens
   ¯\_(ツ)_/¯
   

   View Slide

8. 8
   @phermens
   “It depends”
   

   View Slide

9. 9
   @phermens
   YAGNI
   & KISS
   

   View Slide

10. View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. 100 / 1,000 / 10,000; so what?
    

    View Slide

18. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf
    

    View Slide

19. … Apparently we are too much trained to disregard
    differences in scale, to treat them as “gradual differences that
    are not essential.”
    We tell ourselves that what we can do once, we can also do
    twice and by induction, we fool ourselves into believing that
    we can do it as many times as needed, but this is just not true!
    ...
    

    View Slide

20. ... A one-year old child will crawl on all fours with a speed of,
    say, one mile per hour. But a speed of a thousand miles per
    hour is that of a supersonic jet.
    Considered as objects with moving ability the child and the jet
    are incomparable, for whatever one can do the other cannot
    and vice versa.
    — Edsger Dijkstra, 1969
    

    View Slide

21. View Slide

22. Prerequisites...
    

    View Slide

23. View Slide

24. 24
    @phermens
    The “faster to master” checklist
    Developing
    ⬜ Coding guidelines
    ⬜ Code reviews
    ⬜ Code analysis
    ⬜ Short-lived branches
    ⬜ Feature toggles
    Monitoring
    ⬜ Defined key metrics
    ⬜ Measure – create dashboards
    ⬜ Proactivity – add alerting (& act!)
    Testing
    ⬜ Unit tests
    ⬜ Module tests
    ⬜ Integration tests
    ⬜ End-to-end tests
    ⬜ Test automation
    Deploying
    ⬜ Continuous Integration
    ⬜ Continuous Delivery
    ⬜ Infrastructure as Code
    

    View Slide

25. 25
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    

    View Slide

26. 26
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    ☑
    

    View Slide

27. 27
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    

    View Slide

28. View Slide

29. Story time!
    

    View Slide

30. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

31. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

32. Responsibility
    

    View Slide

33. 33
    @phermens
    Hosting &
    Deployment
    “DevOps”
    

    View Slide

34. 34
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    B
    Team
    C
    

    View Slide

35. 35
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E
    

    View Slide

36. 36
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E
    

    View Slide

37. 37
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

38. 38
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

39. 39
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

40. 40
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

41. 41
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

42. 42
    @phermens
    Team
    H
    Team
    A
    Team
    C
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    Team
    D
    Team
    L
    Team
    K
    

    View Slide

43. 43
    @phermens
    

    View Slide

44. 44
    @phermens
    

    View Slide

45. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

46. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

47. Autonomy
    

    View Slide

48. 48
    @phermens
    

    View Slide

49. 49
    @phermens
    Log4Net + RDP
    

    View Slide

50. 50
    @phermens
    Serilog + Splunk + DataDog
    

    View Slide

51. View Slide

52. View Slide

53. View Slide

54. View Slide

55. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

56. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

57. 57
    @phermens
    Ownership
    

    View Slide

58. 58
    @phermens
    Build System
    

    View Slide

59. 59
    @phermens
    

    View Slide

60. 60
    @phermens
    

    View Slide

61. 61
    @phermens
    Development team tooling choices
    

    View Slide

62. 62
    @phermens
    Infrastructure team tooling choices
    

    View Slide

63. 63
    @phermens
    

    View Slide

64. 64
    @phermens
    What’s next?
    

    View Slide

65. 65
    @phermens
    ?
    What’s next?
    

    View Slide

66. 66
    @phermens
    ¯\_(ツ)_/¯
    

    View Slide

67. 67
    @phermens
    

    View Slide

68. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

69. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

70. 70
    @phermens
    Fail early, fail often,
    but always fail forward.
    

    View Slide

71. 71
    @phermens
    Failure
    

    View Slide

72. 72
    @phermens
    

    View Slide

73. View Slide

74. View Slide

75. View Slide

76. View Slide

77. 77
    @phermens
    

    View Slide

78. View Slide

79. 79
    @phermens
    

    View Slide

80. 80
    @phermens
    

    View Slide

81. View Slide

82. 82
    @phermens
    #war-room
    

    View Slide

83. 83
    @phermens
    

    View Slide

84. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

85. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

86. Questions?
    If not, drinks follow this talk!
    

    View Slide

87. Checklists!
    

    View Slide

88. 88
    @phermens
    The “faster to master” checklist
    Developing
    ⬜ Coding guidelines
    ⬜ Code reviews
    ⬜ Code analysis
    ⬜ Short-lived branches
    ⬜ Feature toggles
    Monitoring
    ⬜ Defined key metrics
    ⬜ Measure – create dashboards
    ⬜ Proactivity – add alerting (& act!)
    Testing
    ⬜ Unit tests
    ⬜ Module tests
    ⬜ Integration tests
    ⬜ End-to-end tests
    ⬜ Test automation
    Deploying
    ⬜ Continuous Integration
    ⬜ Continuous Delivery
    ⬜ Infrastructure as Code
    

    View Slide

89. 89
    @phermens
    The “100 to 1,000” checklist
    Responsibility
    ⬜ Can I control my deployments?
    ⬜ Have I removed all bottlenecks?
    ⬜ Do I have good support systems?
    ⬜ Am I collaborating with my peers?
    Autonomy
    ⬜ Freedom to innovate/improve?
    ⬜ Am I in control and independent?
    ⬜ Can I find out what went wrong?
    ⬜ Do I have the skills to fix it?
    Ownership
    ⬜ Agreed interfaces or contracts?
    ⬜ Out-of-the-box solutions?
    ⬜ Tooling is 100% automated?
    ⬜ Local build == “Production” build?
    Failure
    ⬜ Monitoring = Alerting = Action
    ⬜ Take pride in improving
    ⬜ Small deliverables, quickly
    ⬜ Non-blocking deployments
    

    View Slide

90. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf
    

    View Slide

91. 91
    @phermens
    Thanks
    https://hermens.com.au
    @phermens
    

    View Slide