$30 off During Our Annual Pro Sale. View Details »

Real-world Continuous Delivery: Learn, Adapt, Improve (ConFoo 2020)

Real-world Continuous Delivery: Learn, Adapt, Improve (ConFoo 2020)

Discover how 4 development teams adopted the principles of Continuous Delivery on an existing application. Learn what worked well & what didn’t, and the reality of failing pipelines, flaky tests and brittle infrastructure: transitioning from manual bi-weekly releases to 30 deployments per day.

Michiel Rook

February 27, 2020
Tweet

More Decks by Michiel Rook

Other Decks in Programming

Transcript

  1. REAL WORLD CD
    LEARN, ADAPT & IMPROVE
    MICHIEL ROOK
    @MICHIELTCS

    View Slide

  2. LARGE SEMI-GOV
    IN NL

    View Slide

  3. @michieltcs
    MYSQL ELASTIC
    INTERNET

    View Slide

  4. @michieltcs
    TOPIC TEAMS

    View Slide

  5. @michieltcs
    FIRE TEAM

    View Slide

  6. @michieltcs
    PACKAGE/DEP
    UPDATES

    View Slide

  7. @michieltcs
    BUG TRIAGE

    View Slide

  8. @michieltcs
    RELEASE ROLLOUT

    View Slide

  9. @michieltcs
    A TYPICAL RELEASE...

    View Slide

  10. @michieltcs
    RELEASE
    CHECKLIST

    View Slide

  11. @michieltcs
    @michieltcs
    Create
    release
    Test
    release
    Deploy
    release
    every
    2 weeks
    Thursday
    (sprint end)
    Monday
    (sprint start)
    Tuesday

    View Slide

  12. @michieltcs
    1. BRANCH RC
    2. DEPLOY TO ACC
    3. SIGNAL TESTER(S)
    4. SIGNAL PO
    5. FIX ON BRANCH
    6. GET APPROVAL
    7. DEPLOY TO PROD
    8. VERIFY PROD

    View Slide

  13. @michieltcs
    2-3 DAYS
    MANUAL WORK

    View Slide

  14. @michieltcs
    ... HOT FIXES?

    View Slide

  15. @michieltcs
    GOALS

    View Slide

  16. @michieltcs
    REDUCE COST

    View Slide

  17. @michieltcs
    FAST FEEDBACK

    View Slide

  18. @michieltcs
    REDUCE TOIL

    View Slide

  19. @michieltcs
    REDUCE TOIL
    the kind of work tied to running a production service that
    tends to be manual, repetitive, automatable, tactical,
    devoid of enduring value, and that scales linearly as a
    service grows

    View Slide

  20. @michieltcs
    PUT PO/BIZ BACK IN
    DRIVER'S SEAT

    View Slide

  21. @michieltcs
    WHY CD THEN?

    View Slide

  22. @michieltcs
    MAKE THINGS
    SMALL

    View Slide

  23. @michieltcs
    BIG STEPS

    View Slide

  24. @michieltcs
    FAIL BIG

    View Slide

  25. @michieltcs
    SMALL STEPS

    View Slide

  26. @michieltcs
    FAIL SMALL

    View Slide

  27. @michieltcs
    IF IT HURTS
    DO IT MORE OFTEN

    View Slide

  28. @michieltcs
    $ = REALIZED VALUE
    CREDITS TO @FGOULDING

    View Slide

  29. @michieltcs
    @michieltcs
    HIGH VS. LOW PERFORMING TEAMS
    Source: Accelerate - The Science Of DevOps

    View Slide

  30. APPROACH

    View Slide

  31. @michieltcs
    3 PHASES

    View Slide

  32. @michieltcs
    DEFER SPLITTING
    THE MONOLITH

    View Slide

  33. @michieltcs
    INCREASE TEMPO &
    AUTOMATION

    View Slide

  34. @michieltcs
    SINGLE PIPELINE

    View Slide

  35. @michieltcs
    IMPROVEMENTS &
    TESTING

    View Slide

  36. PHASE 1:
    INCREASING
    RELEASE CADENCE

    View Slide

  37. @michieltcs
    WEEKLY RELEASE

    View Slide

  38. @michieltcs
    AUTOMATION

    View Slide

  39. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment

    View Slide

  40. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment

    View Slide

  41. @michieltcs
    pipeline {

    stages {

    stage('Run tests') {

    sh "gradle check"

    }


    stage('Build docker image') {

    sh "docker build -t jobservice:${env.BUILD_NUMBER} ."

    sh "docker push jobservice:${env.BUILD_NUMBER}"

    }


    stage('Deploy staging') {

    sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}

    -i staging deploy.yml"

    }


    stage('Deploy production') {

    sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}

    -i prod deploy.yml"

    }

    }

    PIPELINE AS CODE
    image: registry.local/runner:latest


    stages:

    - test

    - images

    - deploy


    phpunit:

    stage: test

    script:

    - bin/phpunit


    images:

    stage: images

    script:

    - make images publish

    only:

    - master


    deploy:

    stage: deploy

    script:

    - make deploy

    only:

    - master

    View Slide

  42. View Slide

  43. View Slide

  44. @michieltcs
    WAY OF WORKING

    View Slide

  45. @michieltcs
    PAIR
    PROGRAMMING

    View Slide

  46. BOY SCOUT RULE

    View Slide

  47. @michieltcs
    NO MORE RELEASE
    BRANCHES

    View Slide

  48. @michieltcs
    @michieltcs

    View Slide

  49. @michieltcs
    @michieltcs

    View Slide

  50. @michieltcs
    @michieltcs

    View Slide

  51. PHASE 2:
    SINGLE PIPELINE

    View Slide

  52. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment

    View Slide

  53. @michieltcs
    CONTINUOUS
    DEPLOYMENT

    View Slide

  54. @michieltcs
    ZERO DOWNTIME
    DEPLOYS

    View Slide

  55. @michieltcs
    ROLLING
    DEPLOYS

    View Slide

  56. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.0
    SERVICE V1.0
    @michieltcs

    View Slide

  57. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.0
    SERVICE V1.0
    SERVICE V1.1
    @michieltcs

    View Slide

  58. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.0
    SERVICE V1.0
    @michieltcs

    View Slide

  59. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.0
    SERVICE V1.1
    @michieltcs

    View Slide

  60. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.0
    @michieltcs

    View Slide

  61. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.1
    @michieltcs

    View Slide

  62. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.1 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.0
    @michieltcs

    View Slide

  63. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.1 SERVICE V1.1
    SERVICE V1.1
    @michieltcs

    View Slide

  64. @michieltcs
    PIPELINE FAILURES

    View Slide

  65. @michieltcs
    FLAKY TESTS

    View Slide

  66. @michieltcs
    TIMEOUTS

    View Slide

  67. @michieltcs
    NETWORK STABILITY
    ISSUES

    View Slide

  68. @michieltcs
    EXTERNAL DEPS IN
    TESTS

    View Slide

  69. @michieltcs
    BRITTLE
    INFRASTRUCTURE

    View Slide

  70. @michieltcs
    IF YOU CAN'T TRUST
    IT - REMOVE IT

    View Slide

  71. STOP THE LINE

    View Slide

  72. @michieltcs
    EXTREME FEEDBACK

    View Slide

  73. @michieltcs
    @michieltcs

    View Slide

  74. @michieltcs
    RELEASE
    TIMING?

    View Slide

  75. FEATURE TOGGLES

    View Slide

  76. @michieltcs
    DECOUPLE
    DEPLOYMENTS
    FROM
    RELEASES

    View Slide

  77. @michieltcs
    @michieltcs

    View Slide

  78. @michieltcs
    @michieltcs

    View Slide

  79. @michieltcs
    CAUTION:
    FEATURE TOGGLE
    DEBT

    View Slide

  80. PHASE 3:
    IMPROVEMENTS &
    TESTING

    View Slide

  81. @michieltcs
    PIPELINE SPEED

    View Slide

  82. @michieltcs
    BUY A BIGGER BOX

    View Slide

  83. @michieltcs
    CLEVER PIPELINE
    IMPROVEMENTS:
    PARALLELIZE

    View Slide

  84. @michieltcs
    TESTING
    IMPROVEMENTS

    View Slide

  85. @michieltcs
    @michieltcs
    UNIT
    TESTS
    E2E /
    VISUAL TESTS
    INTEGRATION
    TESTS
    LOTS OF MANUAL TESTING
    E2E TESTS

    View Slide

  86. @michieltcs
    QUALITY
    ENGINEERING

    View Slide

  87. @michieltcs
    AUTOMATION

    View Slide

  88. @michieltcs
    @michieltcs
    CONTRACT TESTING

    View Slide

  89. @michieltcs
    @michieltcs
    UI TESTING

    View Slide

  90. @michieltcs
    @michieltcs
    VISUAL TESTING

    View Slide

  91. @michieltcs
    @michieltcs
    DEPENDENCY SCANNING

    View Slide

  92. @michieltcs
    @michieltcs
    UNIT TESTS
    UI /
    VISUAL TESTS
    Exploratory
    testing & user
    feedback
    Monitoring
    & alerting
    INTEGRATION / CONTRACT
    TESTS
    COST SPEED

    View Slide

  93. RESULTS &
    TAKEAWAYS

    View Slide

  94. @michieltcs
    LEARNINGS

    View Slide

  95. @michieltcs
    PIPELINE SPEED &
    STABILITY

    View Slide

  96. @michieltcs
    TEST THE PIPELINE

    View Slide

  97. @michieltcs
    WAY OF WORKING &
    MINDSET

    View Slide

  98. @michieltcs
    VISUAL TESTING
    TOOLS

    View Slide

  99. @michieltcs
    SOME STATS

    View Slide

  100. @michieltcs
    ~4000 TESTS PER
    DEPLOYMENT

    View Slide

  101. @michieltcs
    ~10 MINUTES PER
    PIPELINE RUN

    View Slide

  102. @michieltcs
    ~1.5 HOURS FROM
    DEV TO PROD

    View Slide

  103. @michieltcs
    IN 6 MONTHS:
    MORE THAN 5000
    DEPLOYMENTS

    View Slide

  104. @michieltcs
    @michieltcs
    THANK YOU FOR
    LISTENING!
    @michieltcs / [email protected]
    www.michielrook.nl

    View Slide