From 100, to 1000+ deployments a day [TechoramaNL editie]

Transcript

1. View Slide

2. From 100 to 1,000+ deployments a day
   Pat Hermens
   @phermens
   

   View Slide

3. From 100 to 1,000+
   deployments a day
   Pat Hermens
   @phermens
   

   View Slide

4. Pat Hermens
   Development Manager
   Coding for 15+ years
   Father & husband
   Rotterdam
   

   View Slide

5. View Slide

6. View Slide

7. Why 1,000+ deployments/day?
   

   View Slide

8. 8
   @phermens
   ¯\_(ツ)_/¯
   

   View Slide

9. 9
   @phermens
   “It depends”
   

   View Slide

10. 10
    @phermens
    YAGNI
    & KISS
    

    View Slide

11. View Slide

12. View Slide

13. View Slide

14. View Slide

15. View Slide

16. View Slide

17. View Slide

18. 100 / 1,000 / 10,000; so what?
    

    View Slide

19. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf
    

    View Slide

20. … Apparently we are too much trained to disregard
    differences in scale, to treat them as “gradual differences that
    are not essential.”
    We tell ourselves that what we can do once, we can also do
    twice and by induction, we fool ourselves into believing that
    we can do it as many times as needed, but this is just not true!
    ...
    

    View Slide

21. ... A one-year old child will crawl on all fours with a speed of,
    say, one mile per hour. But a speed of a thousand miles per
    hour is that of a supersonic jet.
    Considered as objects with moving ability the child and the jet
    are incomparable, for whatever one can do the other cannot
    and vice versa.
    — Edsger Dijkstra, 1969
    

    View Slide

22. View Slide

23. Prerequisites...
    

    View Slide

24. View Slide

25. 25
    @phermens
    The “faster to master” checklist
    Developing
    ⬜ Coding guidelines
    ⬜ Code reviews
    ⬜ Code analysis
    ⬜ Short-lived branches
    ⬜ Feature toggles
    Monitoring
    ⬜ Defined key metrics
    ⬜ Measure – create dashboards
    ⬜ Proactivity – add alerting (& act!)
    Testing
    ⬜ Unit tests
    ⬜ Module tests
    ⬜ Integration tests
    ⬜ End-to-end tests
    ⬜ Test automation
    Deploying
    ⬜ Continuous Integration
    ⬜ Continuous Delivery
    ⬜ Infrastructure as Code
    

    View Slide

26. 26
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    

    View Slide

27. 27
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    ☑
    

    View Slide

28. 28
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist
    

    View Slide

29. View Slide

30. Story time!
    

    View Slide

31. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

32. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

33. Responsibility
    

    View Slide

34. 34
    @phermens
    Hosting &
    Deployment
    “DevOps”
    

    View Slide

35. 35
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    B
    Team
    C
    

    View Slide

36. 36
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E
    

    View Slide

37. 37
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E
    

    View Slide

38. 38
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

39. 39
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

40. 40
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

41. 41
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

42. 42
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    

    View Slide

43. 43
    @phermens
    Team
    H
    Team
    A
    Team
    C
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    Team
    D
    Team
    L
    Hosting
    &
    Deploy-
    ment
    

    View Slide

44. 44
    @phermens
    

    View Slide

45. 45
    @phermens
    

    View Slide

46. 46
    @phermens
    

    View Slide

47. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

48. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

49. Autonomy
    

    View Slide

50. Autonomy
    

    View Slide

51. 51
    @phermens
    Serilog + Splunk + DataDog
    

    View Slide

52. View Slide

53. View Slide

54. View Slide

55. View Slide

56. View Slide

57. Autonomy
    

    View Slide

58. View Slide

59. 59
    @phermens
    

    View Slide

60. 60
    @phermens
    

    View Slide

61. 61
    @phermens
    

    View Slide

62. 62
    @phermens
    $ $ $
    $ $
    

    View Slide

63. 63
    @phermens
    $ $ $
    $ $
    

    View Slide

64. 64
    @phermens
    

    View Slide

65. 65
    @phermens
    UDP
    

    View Slide

66. 66
    @phermens
    UDP
    

    View Slide

67. 67
    @phermens
    UDP
    

    View Slide

68. 68
    @phermens
    UDP
    

    View Slide

69. 69
    @phermens
    UDP
    

    View Slide

70. 70
    @phermens
    

    View Slide

71. 71
    @phermens
    TCP
    

    View Slide

72. 72
    @phermens
    TCP
    

    View Slide

73. 73
    @phermens
    

    View Slide

74. 74
    @phermens
    

    View Slide

75. 75
    @phermens
    API Gateway Lambda CloudWatch
    

    View Slide

76. 76
    @phermens
    API Gateway Kinesis Lambda CloudWatch
    

    View Slide

77. 77
    @phermens
    AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28)
    Last 30 days
    Last 7 days
    

    View Slide

78. 78
    @phermens
    AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28)
    Last 30 days
    Last 7 days
    

    View Slide

79. 79
    @phermens
    

    View Slide

80. 80
    @phermens
    

    View Slide

81. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

82. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

83. 83
    @phermens
    Ownership
    

    View Slide

84. 84
    @phermens
    Build System
    

    View Slide

85. 85
    @phermens
    

    View Slide

86. 86
    @phermens
    

    View Slide

87. 87
    @phermens
    Development team tooling choices
    

    View Slide

88. 88
    @phermens
    Infrastructure team tooling choices
    

    View Slide

89. 89
    @phermens
    

    View Slide

90. 90
    @phermens
    What’s next?
    

    View Slide

91. 91
    @phermens
    What’s next?
    ?
    

    View Slide

92. 92
    @phermens
    ¯\_(ツ)_/¯
    

    View Slide

93. 93
    @phermens
    

    View Slide

94. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

95. Responsibility Autonomy
    Failure
    Ownership
    

    View Slide

96. 96
    @phermens
    Fail early, fail often,
    but always fail forward.
    

    View Slide

97. 97
    @phermens
    Failure
    

    View Slide

98. 98
    @phermens
    

    View Slide

99. 99
    @phermens
    

    View Slide

100. View Slide

101. View Slide

102. View Slide

103. View Slide

104. 104
     @phermens
     

     View Slide

105. View Slide

106. NOTE:
     20 cubic
     centimeters
     

     View Slide

107. View Slide

108. NOTE:
     Now 50cc!
     

     View Slide

109. View Slide

110. NOTE:
     49 cubic
     centimeters
     

     View Slide

111. View Slide

112. View Slide

113. NOTE:
     More than
     49cc’s!
     

     View Slide

114. View Slide

115. View Slide

116. View Slide

117. View Slide

118. NOTE:
     Now 45cc!
     

     View Slide

119. View Slide

120. View Slide

121. 121
     @phermens
     

     View Slide

122. 122
     @phermens
     

     View Slide

123. View Slide

124. 124
     @phermens
     #war-room
     

     View Slide

125. 125
     @phermens
     

     View Slide

126. Responsibility Autonomy
     Failure
     Ownership
     

     View Slide

127. Responsibility Autonomy
     Failure
     Ownership
     

     View Slide

128. 128
     @phermens
     The “100 to 1,000” checklist
     Responsibility
     ⬜ Can I control my deployments?
     ⬜ Have I removed all bottlenecks?
     ⬜ Do I have good support systems?
     ⬜ Am I collaborating with my peers?
     Autonomy
     ⬜ Freedom to innovate/improve?
     ⬜ Am I in control and independent?
     ⬜ Can I find out what went wrong?
     ⬜ Do I have the skills to fix it?
     Ownership
     ⬜ Agreed interfaces or contracts?
     ⬜ Out-of-the-box solutions?
     ⬜ Tooling is 100% automated?
     ⬜ Local build == “Production” build?
     Failure
     ⬜ Monitoring = Alerting = Action
     ⬜ Take pride in improving
     ⬜ Small deliverables, quickly
     ⬜ Non-blocking deployments
     

     View Slide

129. Questions?
     

     View Slide

130. Checklists!
     

     View Slide

131. 131
     @phermens
     The “faster to master” checklist
     Developing
     ⬜ Coding guidelines
     ⬜ Code reviews
     ⬜ Code analysis
     ⬜ Short-lived branches
     ⬜ Feature toggles
     Monitoring
     ⬜ Defined key metrics
     ⬜ Measure – create dashboards
     ⬜ Proactivity – add alerting (& act!)
     Testing
     ⬜ Unit tests
     ⬜ Module tests
     ⬜ Integration tests
     ⬜ End-to-end tests
     ⬜ Test automation
     Deploying
     ⬜ Continuous Integration
     ⬜ Continuous Delivery
     ⬜ Infrastructure as Code
     

     View Slide

132. 132
     @phermens
     The “100 to 1,000” checklist
     Responsibility
     ⬜ Can I control my deployments?
     ⬜ Have I removed all bottlenecks?
     ⬜ Do I have good support systems?
     ⬜ Am I collaborating with my peers?
     Autonomy
     ⬜ Freedom to innovate/improve?
     ⬜ Am I in control and independent?
     ⬜ Can I find out what went wrong?
     ⬜ Do I have the skills to fix it?
     Ownership
     ⬜ Agreed interfaces or contracts?
     ⬜ Out-of-the-box solutions?
     ⬜ Tooling is 100% automated?
     ⬜ Local build == “Production” build?
     Failure
     ⬜ Monitoring = Alerting = Action
     ⬜ Take pride in improving
     ⬜ Small deliverables, quickly
     ⬜ Non-blocking deployments
     

     View Slide

133. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf
     

     View Slide

134. 134
     @phermens
     Thanks
     https://hermens.com.au
     @phermens
     

     View Slide

135. @phermens
     pat@coolblue.nl
     Pat Hermens
     @phermens
     Thanks!
     

     View Slide