Beyond Continuous Delivery: learn, adapt, improve (JAX DevOps 2019)

Beyond Continuous Delivery: learn, adapt, improve (JAX DevOps 2019)

This talk tells the story of a recent real life project I did together with four development teams. Learn how we adopted the textbook practices, patterns and principles of Continuous Delivery and applied them to an existing application.

In this talk, I’ll discuss what worked well, what didn’t, and how we dealt with the messy reality of failing pipelines, flaky tests, and brittle infrastructure. Find out how we transitioned from mostly manual, bi-weekly, error-prone releases to 30 automated deployments per day, while significantly reducing overhead and increasing quality.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

May 15, 2019
Tweet

Transcript

  1. BEYOND CD LEARN, ADAPT & IMPROVE MICHIEL ROOK
 @MICHIELTCS

  2. BACKGROUND

  3. @michieltcs LARGE SEMI-GOV
 IN NL

  4. @michieltcs MYSQL ELASTIC INTERNET

  5. @michieltcs TOPIC TEAMS

  6. @michieltcs ON-CALL

  7. @michieltcs FIRE TEAM

  8. @michieltcs PACKAGE/DEP UPDATES

  9. @michieltcs BUG TRIAGE

  10. @michieltcs OFFICE HOURS INCIDENT RESPONSE

  11. @michieltcs RELEASE ROLLOUT

  12. @michieltcs A TYPICAL RELEASE...

  13. @michieltcs RELEASE CHECKLIST

  14. @michieltcs @michieltcs Create release Test release Deploy release every
 2

    weeks Thursday
 (sprint end) Monday (sprint start) Tuesday

  15. @michieltcs 1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL

    TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD
  16. @michieltcs 2-3 DAYS
 MANUAL WORK

  17. @michieltcs ... HOT FIXES?

  18. @michieltcs CURRENT STATUS

  19. @michieltcs PLENTY OF MANUAL WORK

  20. @michieltcs PLENTY OF WASTE

  21. @michieltcs RELEASE TIMING

  22. @michieltcs SYNCED TO TECHNICAL CADENCE (2 WKS)

  23. @michieltcs GOALS

  24. @michieltcs FAST FEEDBACK

  25. @michieltcs REDUCE TOIL

  26. @michieltcs REDUCE TOIL the kind of work tied to running

    a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows
  27. @michieltcs RESPONSIVE ORGANIZATION

  28. @michieltcs EXPERIMENTATION / A/B TESTING

  29. @michieltcs PUT PO/BIZ BACK IN DRIVER'S SEAT

  30. @michieltcs ABOUT THIS
 CD THING

  31. @michieltcs IF IT HURTS
 DO IT MORE OFTEN

  32. @michieltcs BIG STEPS

  33. @michieltcs FAIL BIG

  34. @michieltcs SMALL STEPS

  35. @michieltcs FAIL SMALL

  36. @michieltcs REDUCE RISK

  37. @michieltcs $ = REALIZED VALUE
 CREDITS TO @FGOULDING

  38. @michieltcs @michieltcs HIGH VS. LOW PERFORMING TEAMS Source: Accelerate -

    The Science Of DevOps
  39. APPROACH

  40. @michieltcs 4 PHASES

  41. @michieltcs DEFER SPLITTING THE MONOLITH

  42. @michieltcs INCREASE TEMPO

  43. @michieltcs AUTOMATION

  44. @michieltcs PIPELINE

  45. @michieltcs TESTING

  46. @michieltcs INFRASTRUCTURE

  47. PHASE 1: INCREASING RELEASE CADENCE

  48. @michieltcs WEEKLY RELEASE

  49. @michieltcs AUTOMATION

  50. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  51. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  52. @michieltcs pipeline {
 stages {
 stage('Run tests') {
 sh "gradle

    check"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 }
 PIPELINE AS CODE image: registry.local/runner:latest
 
 stages:
 - test
 - images
 - deploy
 
 phpunit:
 stage: test
 script:
 - bin/phpunit
 
 images:
 stage: images
 script:
 - make images publish
 only:
 - master
 
 deploy:
 stage: deploy
 script:
 - make deploy
 only:
 - master
  53. None
  54. None
  55. @michieltcs WAY OF WORKING

  56. @michieltcs T-SHAPED

  57. @michieltcs PAIR
 PROGRAMMING

  58. BOY SCOUT RULE

  59. PHASE 2: SINGLE PIPELINE

  60. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  61. @michieltcs CONTINUOUS DEPLOYMENT

  62. @michieltcs ROLLING DEPLOYS

  63. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 @michieltcs
  64. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  65. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.0 @michieltcs
  66. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  67. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  68. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.1 @michieltcs
  69. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  70. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 @michieltcs
  71. @michieltcs @michieltcs

  72. @michieltcs @michieltcs

  73. @michieltcs PIPELINE FAILURES

  74. @michieltcs FLAKY TESTS

  75. @michieltcs EXTERNAL DEPS IN TESTS

  76. @michieltcs NETWORK STABILITY ISSUES

  77. @michieltcs BRITTLE INFRASTRUCTURE

  78. @michieltcs EXTREME FEEDBACK

  79. STOP THE LINE

  80. @michieltcs @michieltcs

  81. PHASE 3: IMPROVEMENTS & TESTING

  82. @michieltcs PIPELINE SPEED

  83. @michieltcs CLEVER PIPELINE IMPROVEMENTS: PARALLELIZE

  84. @michieltcs TESTING IMPROVEMENTS

  85. @michieltcs @michieltcs UNIT TESTS E2E /
 VISUAL TESTS Exploratory
 testing

    & user
 feedback Monitoring
 & alerting INTEGRATION / CONTRACT
 TESTS COST SPEED
  86. @michieltcs @michieltcs CONTRACT TESTING

  87. @michieltcs @michieltcs E2E TESTING

  88. @michieltcs @michieltcs VISUAL TESTING

  89. FEATURE TOGGLES

  90. @michieltcs DECOUPLE
 DEPLOYMENTS
 FROM
 RELEASES

  91. @michieltcs @michieltcs

  92. @michieltcs @michieltcs

  93. @michieltcs CAUTION:
 FEATURE TOGGLE DEBT

  94. PHASE 4: SERVICES & INFRASTRUCTURE

  95. @michieltcs ARCHITECTURE PRINCIPLES

  96. @michieltcs API DESIGN AND VERSIONING

  97. @michieltcs SERVICE ORIENTED

  98. @michieltcs INDEPENDENTLY DEPLOYABLE UNITS

  99. @michieltcs INFRASTRUCTURE IMPROVEMENTS

  100. @michieltcs NETWORKING

  101. @michieltcs KUBERNETES

  102. @michieltcs SLAYING THE MONOLITH

  103. @michieltcs @michieltcs

  104. @michieltcs VERTICAL SPLITS

  105. @michieltcs @michieltcs Frontend Backend Database Elasticsearch ... Frontend Backend Database

    Elasticsearch ... Frontend Backend Database Elasticsearch ... Team 1 Team 2 Team 3
  106. @michieltcs TECH ALIGNMENT BETWEEN TEAMS & SERVICES

  107. RESULTS & TAKEAWAYS

  108. @michieltcs LEARNINGS

  109. @michieltcs VISUAL TESTING TOOLS

  110. @michieltcs (AUTOMATED) SECURITY TESTING

  111. @michieltcs TECHNICAL ALIGNMENT BETWEEN TEAMS

  112. @michieltcs AUTOMATION

  113. @michieltcs PIPELINE SPEED & STABILITY

  114. @michieltcs INFRA OPERABILITY
 & STABILITY

  115. @michieltcs WAY OF WORKING & MINDSET

  116. @michieltcs SOME STATS

  117. @michieltcs ~4000 TESTS PER DEPLOYMENT

  118. @michieltcs ~10 MINUTES PER
 PIPELINE RUN

  119. @michieltcs ~1.5 HOURS FROM DEV TO PROD

  120. @michieltcs IN 6 MONTHS:
 MORE THAN 5000 DEPLOYMENTS

  121. @michieltcs @michieltcs THANK YOU FOR LISTENING! @michieltcs / michiel@michielrook.nl www.michielrook.nl