Beyond continuous delivery: learn, adapt, improve (Lean Agile Scotland 2019)

Beyond continuous delivery: learn, adapt, improve (Lean Agile Scotland 2019)

Discover how 4 development teams adopted the principles of continuous delivery on an existing application. Learn what worked well and what didn’t, and the reality of failing pipelines, flaky tests and brittle infrastructure: transitioning from manual bi-weekly releases to 30 deployments per day.

2f4800411154a8c66dde489448a044d2?s=128

Michiel Rook

October 09, 2019
Tweet

Transcript

  1. BEYOND CD LEARN, ADAPT & IMPROVE MICHIEL ROOK
 @MICHIELTCS

  2. LARGE SEMI-GOV
 IN NL

  3. @michieltcs MYSQL ELASTIC INTERNET

  4. @michieltcs TOPIC TEAMS

  5. @michieltcs FIRE TEAM

  6. @michieltcs PACKAGE/DEP UPDATES

  7. @michieltcs BUG TRIAGE

  8. @michieltcs RELEASE ROLLOUT

  9. @michieltcs A TYPICAL RELEASE...

  10. @michieltcs RELEASE CHECKLIST

  11. @michieltcs @michieltcs Create release Test release Deploy release every
 2

    weeks Thursday
 (sprint end) Monday (sprint start) Tuesday

  12. @michieltcs 1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL

    TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD
  13. @michieltcs 2-3 DAYS
 MANUAL WORK

  14. @michieltcs ... HOT FIXES?

  15. @michieltcs GOALS

  16. @michieltcs REDUCE COST

  17. @michieltcs FAST FEEDBACK

  18. @michieltcs REDUCE TOIL

  19. @michieltcs REDUCE TOIL the kind of work tied to running

    a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows
  20. @michieltcs PUT PO/BIZ BACK IN DRIVER'S SEAT

  21. @michieltcs WHY CD THEN?

  22. @michieltcs MAKE THINGS
 SMALL

  23. @michieltcs BIG STEPS

  24. @michieltcs FAIL BIG

  25. @michieltcs SMALL STEPS

  26. @michieltcs FAIL SMALL

  27. @michieltcs IF IT HURTS
 DO IT MORE OFTEN

  28. @michieltcs $ = REALIZED VALUE
 CREDITS TO @FGOULDING

  29. @michieltcs @michieltcs HIGH VS. LOW PERFORMING TEAMS Source: Accelerate -

    The Science Of DevOps
  30. APPROACH

  31. @michieltcs 3 PHASES

  32. @michieltcs DEFER SPLITTING THE MONOLITH

  33. @michieltcs INCREASE TEMPO & AUTOMATION

  34. @michieltcs SINGLE PIPELINE

  35. @michieltcs IMPROVEMENTS & TESTING

  36. PHASE 1: INCREASING RELEASE CADENCE

  37. @michieltcs WEEKLY RELEASE

  38. @michieltcs AUTOMATION

  39. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  40. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  41. @michieltcs pipeline {
 stages {
 stage('Run tests') {
 sh "gradle

    check"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 }
 PIPELINE AS CODE image: registry.local/runner:latest
 
 stages:
 - test
 - images
 - deploy
 
 phpunit:
 stage: test
 script:
 - bin/phpunit
 
 images:
 stage: images
 script:
 - make images publish
 only:
 - master
 
 deploy:
 stage: deploy
 script:
 - make deploy
 only:
 - master
  42. None
  43. None
  44. @michieltcs WAY OF WORKING

  45. @michieltcs PAIR
 PROGRAMMING

  46. BOY SCOUT RULE

  47. @michieltcs NO MORE RELEASE BRANCHES

  48. @michieltcs @michieltcs

  49. @michieltcs @michieltcs

  50. @michieltcs @michieltcs

  51. PHASE 2: SINGLE PIPELINE

  52. @michieltcs @michieltcs Build / Package Automated
 Tests Acceptance
 Environment Production


    Environment
  53. @michieltcs CONTINUOUS DEPLOYMENT

  54. @michieltcs ZERO DOWNTIME
 DEPLOYS

  55. @michieltcs ROLLING
 DEPLOYS

  56. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 @michieltcs
  57. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  58. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.0 @michieltcs
  59. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.0 SERVICE V1.1 @michieltcs
  60. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  61. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.1 @michieltcs
  62. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 SERVICE V1.0 @michieltcs
  63. @michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.1 SERVICE V1.1

    SERVICE V1.1 @michieltcs
  64. @michieltcs PIPELINE FAILURES

  65. @michieltcs FLAKY TESTS

  66. @michieltcs TIMEOUTS

  67. @michieltcs NETWORK STABILITY ISSUES

  68. @michieltcs EXTERNAL DEPS IN TESTS

  69. @michieltcs BRITTLE INFRASTRUCTURE

  70. @michieltcs IF YOU CAN'T TRUST IT - REMOVE IT

  71. STOP THE LINE

  72. @michieltcs EXTREME FEEDBACK

  73. @michieltcs @michieltcs

  74. @michieltcs RELEASE
 TIMING?

  75. FEATURE TOGGLES

  76. @michieltcs DECOUPLE
 DEPLOYMENTS
 FROM
 RELEASES

  77. @michieltcs @michieltcs

  78. @michieltcs @michieltcs

  79. @michieltcs CAUTION:
 FEATURE TOGGLE DEBT

  80. PHASE 3: IMPROVEMENTS & TESTING

  81. @michieltcs PIPELINE SPEED

  82. @michieltcs BUY A BIGGER BOX

  83. @michieltcs CLEVER PIPELINE IMPROVEMENTS: PARALLELIZE

  84. @michieltcs TESTING IMPROVEMENTS

  85. @michieltcs @michieltcs UNIT
 TESTS E2E /
 VISUAL TESTS INTEGRATION
 TESTS

    LOTS OF MANUAL TESTING E2E TESTS
  86. @michieltcs QUALITY ENGINEERING

  87. @michieltcs AUTOMATION

  88. @michieltcs @michieltcs CONTRACT TESTING

  89. @michieltcs @michieltcs UI TESTING

  90. @michieltcs @michieltcs VISUAL TESTING

  91. @michieltcs @michieltcs DEPENDENCY SCANNING

  92. @michieltcs @michieltcs UNIT TESTS UI /
 VISUAL TESTS Exploratory
 testing

    & user
 feedback Monitoring
 & alerting INTEGRATION / CONTRACT
 TESTS COST SPEED
  93. RESULTS & TAKEAWAYS

  94. @michieltcs LEARNINGS

  95. @michieltcs PIPELINE SPEED & STABILITY

  96. @michieltcs TEST THE PIPELINE

  97. @michieltcs WAY OF WORKING & MINDSET

  98. @michieltcs VISUAL TESTING TOOLS

  99. @michieltcs SOME STATS

  100. @michieltcs ~4000 TESTS PER DEPLOYMENT

  101. @michieltcs ~10 MINUTES PER
 PIPELINE RUN

  102. @michieltcs ~1.5 HOURS FROM DEV TO PROD

  103. @michieltcs IN 6 MONTHS:
 MORE THAN 5000 DEPLOYMENTS

  104. @michieltcs @michieltcs THANK YOU FOR LISTENING! @michieltcs / michiel@michielrook.nl www.michielrook.nl