Beyond Continuous Delivery: Learn, Adapt, Improve (JAX London 2019)

Beyond Continuous Delivery: Learn, Adapt, Improve (JAX London 2019)

This talk tells the story of a recent real life project I did together with four development teams. Learn how we adopted the textbook practices, patterns and principles of Continuous Delivery and applied them to an existing monolithic application. I’ll discuss what worked well and what didn’t, and how we dealt with the messy reality of failing pipelines, flaky tests and brittle infrastructure. Find out how we transitioned from mostly manual, bi-weekly, error-prone releases to 30 automated deployments per day, while significantly reducing overhead and increasing quality.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

October 08, 2019
Tweet

Transcript

  1. BEYOND CD LEARN, ADAPT & IMPROVE MICHIEL ROOK
 @MICHIELTCS

  2. BACKGROUND

  3. @michieltcs LARGE SEMI-GOV
 IN NL

  4. @michieltcs MYSQL ELASTIC INTERNET

  5. @michieltcs TOPIC TEAMS

  6. @michieltcs ON-CALL

  7. @michieltcs FIRE TEAM

  8. @michieltcs PACKAGE/DEP UPDATES

  9. @michieltcs BUG TRIAGE

  10. @michieltcs OFFICE HOURS INCIDENT RESPONSE

  11. @michieltcs RELEASE ROLLOUT

  12. @michieltcs A TYPICAL RELEASE...

  13. @michieltcs RELEASE CHECKLIST

  14. @michieltcs @michieltcs Create release Test release Deploy release every
 2

    weeks Thursday
 (sprint end) Monday (sprint start) Tuesday

  15. @michieltcs 1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL

    TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD
  16. @michieltcs 2-3 DAYS
 MANUAL WORK

  17. @michieltcs ... HOT FIXES?

  18. @michieltcs GOALS

  19. @michieltcs REDUCE COST

  20. @michieltcs FAST FEEDBACK

  21. @michieltcs REDUCE TOIL

  22. @michieltcs REDUCE TOIL the kind of work tied to running

    a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows
  23. @michieltcs PUT PO/BIZ BACK IN DRIVER'S SEAT

  24. @michieltcs RESPONSIVE ORGANIZATION

  25. @michieltcs WHY CD THEN?

  26. @michieltcs REDUCE RISK

  27. @michieltcs REDUCE WASTE

  28. @michieltcs REDUCE COST

  29. @michieltcs IF IT HURTS
 DO IT MORE OFTEN

  30. @michieltcs BIG STEPS

  31. @michieltcs FAIL BIG

  32. @michieltcs SMALL STEPS

  33. @michieltcs FAIL SMALL

  34. @michieltcs $ = REALIZED VALUE
 CREDITS TO @FGOULDING

  35. @michieltcs @michieltcs HIGH VS. LOW PERFORMING TEAMS Source: Accelerate -

    The Science Of DevOps
  36. APPROACH

  37. @michieltcs 3 PHASES

  38. @michieltcs DEFER SPLITTING THE MONOLITH

  39. @michieltcs INCREASE TEMPO & AUTOMATION

  40. @michieltcs SINGLE PIPELINE

  41. @michieltcs IMPROVEMENTS & TESTING

  42. PHASE 1: INCREASING RELEASE CADENCE

  43. @michieltcs WEEKLY RELEASE

  44. @michieltcs AUTOMATION

  45. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  46. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  47. @michieltcs pipeline {
 stages {
 stage('Run tests') {
 sh "gradle

    check"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 }
 PIPELINE AS CODE image: registry.local/runner:latest
 
 stages:
 - test
 - images
 - deploy
 
 phpunit:
 stage: test
 script:
 - bin/phpunit
 
 images:
 stage: images
 script:
 - make images publish
 only:
 - master
 
 deploy:
 stage: deploy
 script:
 - make deploy
 only:
 - master
  48. None
  49. None
  50. @michieltcs WAY OF WORKING

  51. @michieltcs PAIR
 PROGRAMMING

  52. BOY SCOUT RULE

  53. @michieltcs NO MORE RELEASE BRANCHES

  54. PHASE 2: SINGLE PIPELINE

  55. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  56. @michieltcs CONTINUOUS DEPLOYMENT

  57. @michieltcs ROLLING DEPLOYS

  58. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 @michieltcs
  59. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  60. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.0 @michieltcs
  61. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  62. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  63. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.1 @michieltcs
  64. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  65. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 @michieltcs
  66. @michieltcs @michieltcs

  67. @michieltcs @michieltcs

  68. @michieltcs @michieltcs

  69. @michieltcs PIPELINE FAILURES

  70. @michieltcs FLAKY TESTS

  71. @michieltcs TIMEOUTS

  72. @michieltcs EXTERNAL DEPS IN TESTS

  73. @michieltcs NETWORK STABILITY ISSUES

  74. @michieltcs BRITTLE INFRASTRUCTURE

  75. @michieltcs IF YOU CAN'T TRUST IT - REMOVE IT

  76. @michieltcs EXTREME FEEDBACK

  77. STOP THE LINE

  78. @michieltcs @michieltcs

  79. PHASE 3: IMPROVEMENTS & TESTING

  80. @michieltcs PIPELINE SPEED

  81. @michieltcs BUY A BIGGER BOX

  82. @michieltcs CLEVER PIPELINE IMPROVEMENTS: PARALLELIZE

  83. @michieltcs TESTING IMPROVEMENTS

  84. @michieltcs @michieltcs UNIT
 TESTS E2E /
 VISUAL TESTS INTEGRATION
 TESTS

    LOTS OF MANUAL TESTING E2E TESTS
  85. @michieltcs AUTOMATION

  86. @michieltcs @michieltcs CONTRACT TESTING

  87. @michieltcs @michieltcs UI TESTING

  88. @michieltcs @michieltcs VISUAL TESTING

  89. @michieltcs @michieltcs UNIT TESTS UI /
 VISUAL TESTS Exploratory
 testing

    & user
 feedback Monitoring
 & alerting INTEGRATION / CONTRACT
 TESTS COST SPEED
  90. @michieltcs RELEASE
 APPROVAL?

  91. FEATURE TOGGLES

  92. @michieltcs DECOUPLE
 DEPLOYMENTS
 FROM
 RELEASES

  93. @michieltcs @michieltcs

  94. @michieltcs @michieltcs

  95. @michieltcs CAUTION:
 FEATURE TOGGLE DEBT

  96. RESULTS & TAKEAWAYS

  97. @michieltcs LEARNINGS

  98. @michieltcs PIPELINE SPEED & STABILITY

  99. @michieltcs WAY OF WORKING & MINDSET

  100. @michieltcs VISUAL TESTING TOOLS

  101. @michieltcs (AUTOMATED) SECURITY TESTING

  102. @michieltcs SOME STATS

  103. @michieltcs ~4000 TESTS PER DEPLOYMENT

  104. @michieltcs ~10 MINUTES PER
 PIPELINE RUN

  105. @michieltcs ~1.5 HOURS FROM DEV TO PROD

  106. @michieltcs IN 6 MONTHS:
 MORE THAN 5000 DEPLOYMENTS

  107. @michieltcs @michieltcs THANK YOU FOR LISTENING! @michieltcs / michiel@michielrook.nl www.michielrook.nl