Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From 100, to 1000+ deployments a day [TechoramaNL editie]

Pat Hermens
October 03, 2018

From 100, to 1000+ deployments a day [TechoramaNL editie]

Pat Hermens

October 03, 2018
Tweet

More Decks by Pat Hermens

Other Decks in Technology

Transcript

  1. View Slide

  2. From 100 to 1,000+ deployments a day
    Pat Hermens
    @phermens

    View Slide

  3. From 100 to 1,000+
    deployments a day
    Pat Hermens
    @phermens

    View Slide

  4. Pat Hermens
    Development Manager
    Coding for 15+ years
    Father & husband
    Rotterdam

    View Slide

  5. View Slide

  6. View Slide

  7. Why 1,000+ deployments/day?

    View Slide

  8. 8
    @phermens
    ¯\_(ツ)_/¯

    View Slide

  9. 9
    @phermens
    “It depends”

    View Slide

  10. 10
    @phermens
    YAGNI
    & KISS

    View Slide

  11. View Slide

  12. View Slide

  13. View Slide

  14. View Slide

  15. View Slide

  16. View Slide

  17. View Slide

  18. 100 / 1,000 / 10,000; so what?

    View Slide

  19. http://www.cs.utexas.edu/users/ewd/ewd02xx/ewd249.pdf

    View Slide

  20. … Apparently we are too much trained to disregard
    differences in scale, to treat them as “gradual differences that
    are not essential.”
    We tell ourselves that what we can do once, we can also do
    twice and by induction, we fool ourselves into believing that
    we can do it as many times as needed, but this is just not true!
    ...

    View Slide

  21. ... A one-year old child will crawl on all fours with a speed of,
    say, one mile per hour. But a speed of a thousand miles per
    hour is that of a supersonic jet.
    Considered as objects with moving ability the child and the jet
    are incomparable, for whatever one can do the other cannot
    and vice versa.
    — Edsger Dijkstra, 1969

    View Slide

  22. View Slide

  23. Prerequisites...

    View Slide

  24. View Slide

  25. 25
    @phermens
    The “faster to master” checklist
    Developing
    ⬜ Coding guidelines
    ⬜ Code reviews
    ⬜ Code analysis
    ⬜ Short-lived branches
    ⬜ Feature toggles
    Monitoring
    ⬜ Defined key metrics
    ⬜ Measure – create dashboards
    ⬜ Proactivity – add alerting (& act!)
    Testing
    ⬜ Unit tests
    ⬜ Module tests
    ⬜ Integration tests
    ⬜ End-to-end tests
    ⬜ Test automation
    Deploying
    ⬜ Continuous Integration
    ⬜ Continuous Delivery
    ⬜ Infrastructure as Code

    View Slide

  26. 26
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist

    View Slide

  27. 27
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist

    View Slide

  28. 28
    @phermens
    Developing
    ☑ Coding guidelines
    ☑ Code reviews
    ☑ Code analysis
    ☑ Short-lived branches
    ☑ Feature toggles
    Monitoring
    ☑ Defined key metrics
    ☑ Measure – create dashboards
    ☑ Proactivity – add alerting (& act!)
    Testing
    ☑ Unit tests
    ☑ Module tests
    ☑ Integration tests
    ☑ End-to-end tests
    ☑ Test automation
    Deploying
    ☑ Continuous Integration
    ☑ Continuous Delivery
    ☑ Infrastructure as Code
    The “faster to master” checklist

    View Slide

  29. View Slide

  30. Story time!

    View Slide

  31. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  32. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  33. Responsibility

    View Slide

  34. 34
    @phermens
    Hosting &
    Deployment
    “DevOps”

    View Slide

  35. 35
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    B
    Team
    C

    View Slide

  36. 36
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E

    View Slide

  37. 37
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    A
    Team
    D
    Team
    F
    Team
    B
    Team
    C
    Team
    E

    View Slide

  38. 38
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C

    View Slide

  39. 39
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C

    View Slide

  40. 40
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C

    View Slide

  41. 41
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C

    View Slide

  42. 42
    @phermens
    Hosting &
    Deployment
    “DevOps”
    Team
    H
    Team
    A
    Team
    D
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C

    View Slide

  43. 43
    @phermens
    Team
    H
    Team
    A
    Team
    C
    Team
    G
    Team
    E
    Team
    J
    Team
    F
    Team
    B
    Team
    I
    Team
    C
    Team
    D
    Team
    L
    Hosting
    &
    Deploy-
    ment

    View Slide

  44. 44
    @phermens

    View Slide

  45. 45
    @phermens

    View Slide

  46. 46
    @phermens

    View Slide

  47. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  48. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  49. Autonomy

    View Slide

  50. Autonomy

    View Slide

  51. 51
    @phermens
    Serilog + Splunk + DataDog

    View Slide

  52. View Slide

  53. View Slide

  54. View Slide

  55. View Slide

  56. View Slide

  57. Autonomy

    View Slide

  58. View Slide

  59. 59
    @phermens

    View Slide

  60. 60
    @phermens

    View Slide

  61. 61
    @phermens

    View Slide

  62. 62
    @phermens
    $ $ $
    $ $

    View Slide

  63. 63
    @phermens
    $ $ $
    $ $

    View Slide

  64. 64
    @phermens

    View Slide

  65. 65
    @phermens
    UDP

    View Slide

  66. 66
    @phermens
    UDP

    View Slide

  67. 67
    @phermens
    UDP

    View Slide

  68. 68
    @phermens
    UDP

    View Slide

  69. 69
    @phermens
    UDP

    View Slide

  70. 70
    @phermens

    View Slide

  71. 71
    @phermens
    TCP

    View Slide

  72. 72
    @phermens
    TCP

    View Slide

  73. 73
    @phermens

    View Slide

  74. 74
    @phermens

    View Slide

  75. 75
    @phermens
    API Gateway Lambda CloudWatch

    View Slide

  76. 76
    @phermens
    API Gateway Kinesis Lambda CloudWatch

    View Slide

  77. 77
    @phermens
    AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28)
    Last 30 days
    Last 7 days

    View Slide

  78. 78
    @phermens
    AWS Lambda stats for Vanessa Optima Prime (as of 2018-09-28)
    Last 30 days
    Last 7 days

    View Slide

  79. 79
    @phermens

    View Slide

  80. 80
    @phermens

    View Slide

  81. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  82. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  83. 83
    @phermens
    Ownership

    View Slide

  84. 84
    @phermens
    Build System

    View Slide

  85. 85
    @phermens

    View Slide

  86. 86
    @phermens

    View Slide

  87. 87
    @phermens
    Development team tooling choices

    View Slide

  88. 88
    @phermens
    Infrastructure team tooling choices

    View Slide

  89. 89
    @phermens

    View Slide

  90. 90
    @phermens
    What’s next?

    View Slide

  91. 91
    @phermens
    What’s next?
    ?

    View Slide

  92. 92
    @phermens
    ¯\_(ツ)_/¯

    View Slide

  93. 93
    @phermens

    View Slide

  94. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  95. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  96. 96
    @phermens
    Fail early, fail often,
    but always fail forward.

    View Slide

  97. 97
    @phermens
    Failure

    View Slide

  98. 98
    @phermens

    View Slide

  99. 99
    @phermens

    View Slide

  100. View Slide

  101. View Slide

  102. View Slide

  103. View Slide

  104. 104
    @phermens

    View Slide

  105. View Slide

  106. NOTE:
    20 cubic
    centimeters

    View Slide

  107. View Slide

  108. NOTE:
    Now 50cc!

    View Slide

  109. View Slide

  110. NOTE:
    49 cubic
    centimeters

    View Slide

  111. View Slide

  112. View Slide

  113. NOTE:
    More than
    49cc’s!

    View Slide

  114. View Slide

  115. View Slide

  116. View Slide

  117. View Slide

  118. NOTE:
    Now 45cc!

    View Slide

  119. View Slide

  120. View Slide

  121. 121
    @phermens

    View Slide

  122. 122
    @phermens

    View Slide

  123. View Slide

  124. 124
    @phermens
    #war-room

    View Slide

  125. 125
    @phermens

    View Slide

  126. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  127. Responsibility Autonomy
    Failure
    Ownership

    View Slide

  128. 128
    @phermens
    The “100 to 1,000” checklist
    Responsibility
    ⬜ Can I control my deployments?
    ⬜ Have I removed all bottlenecks?
    ⬜ Do I have good support systems?
    ⬜ Am I collaborating with my peers?
    Autonomy
    ⬜ Freedom to innovate/improve?
    ⬜ Am I in control and independent?
    ⬜ Can I find out what went wrong?
    ⬜ Do I have the skills to fix it?
    Ownership
    ⬜ Agreed interfaces or contracts?
    ⬜ Out-of-the-box solutions?
    ⬜ Tooling is 100% automated?
    ⬜ Local build == “Production” build?
    Failure
    ⬜ Monitoring = Alerting = Action
    ⬜ Take pride in improving
    ⬜ Small deliverables, quickly
    ⬜ Non-blocking deployments

    View Slide

  129. Questions?

    View Slide

  130. Checklists!

    View Slide

  131. 131
    @phermens
    The “faster to master” checklist
    Developing
    ⬜ Coding guidelines
    ⬜ Code reviews
    ⬜ Code analysis
    ⬜ Short-lived branches
    ⬜ Feature toggles
    Monitoring
    ⬜ Defined key metrics
    ⬜ Measure – create dashboards
    ⬜ Proactivity – add alerting (& act!)
    Testing
    ⬜ Unit tests
    ⬜ Module tests
    ⬜ Integration tests
    ⬜ End-to-end tests
    ⬜ Test automation
    Deploying
    ⬜ Continuous Integration
    ⬜ Continuous Delivery
    ⬜ Infrastructure as Code

    View Slide

  132. 132
    @phermens
    The “100 to 1,000” checklist
    Responsibility
    ⬜ Can I control my deployments?
    ⬜ Have I removed all bottlenecks?
    ⬜ Do I have good support systems?
    ⬜ Am I collaborating with my peers?
    Autonomy
    ⬜ Freedom to innovate/improve?
    ⬜ Am I in control and independent?
    ⬜ Can I find out what went wrong?
    ⬜ Do I have the skills to fix it?
    Ownership
    ⬜ Agreed interfaces or contracts?
    ⬜ Out-of-the-box solutions?
    ⬜ Tooling is 100% automated?
    ⬜ Local build == “Production” build?
    Failure
    ⬜ Monitoring = Alerting = Action
    ⬜ Take pride in improving
    ⬜ Small deliverables, quickly
    ⬜ Non-blocking deployments

    View Slide

  133. https://sddconf.com/brands/sdd/library/DOLLARD-Debugging.pdf

    View Slide

  134. 134
    @phermens
    Thanks
    https://hermens.com.au
    @phermens

    View Slide

  135. @phermens
    [email protected]
    Pat Hermens
    @phermens
    Thanks!

    View Slide