Real-world Continuous Delivery: Learn, Adapt, Improve (ConFoo 2020)

Discover how 4 development teams adopted the principles of Continuous Delivery on an existing application. Learn what worked well & what didn’t, and the reality of failing pipelines, flaky tests and brittle infrastructure: transitioning from manual bi-weekly releases to 30 deployments per day.

Transcript

1. REAL WORLD CD
   LEARN, ADAPT & IMPROVE
   MICHIEL ROOK
   @MICHIELTCS
   

   View Slide

2. LARGE SEMI-GOV
   IN NL
   

   View Slide

3. @michieltcs
   MYSQL ELASTIC
   INTERNET
   

   View Slide

4. @michieltcs
   TOPIC TEAMS
   

   View Slide

5. @michieltcs
   FIRE TEAM
   

   View Slide

6. @michieltcs
   PACKAGE/DEP
   UPDATES
   

   View Slide

7. @michieltcs
   BUG TRIAGE
   

   View Slide

8. @michieltcs
   RELEASE ROLLOUT
   

   View Slide

9. @michieltcs
   A TYPICAL RELEASE...
   

   View Slide

10. @michieltcs
    RELEASE
    CHECKLIST
    

    View Slide

11. @michieltcs
    @michieltcs
    Create
    release
    Test
    release
    Deploy
    release
    every
    2 weeks
    Thursday
    (sprint end)
    Monday
    (sprint start)
    Tuesday
    

    View Slide

12. @michieltcs
    1. BRANCH RC
    2. DEPLOY TO ACC
    3. SIGNAL TESTER(S)
    4. SIGNAL PO
    5. FIX ON BRANCH
    6. GET APPROVAL
    7. DEPLOY TO PROD
    8. VERIFY PROD
    

    View Slide

13. @michieltcs
    2-3 DAYS
    MANUAL WORK
    

    View Slide

14. @michieltcs
    ... HOT FIXES?
    

    View Slide

15. @michieltcs
    GOALS
    

    View Slide

16. @michieltcs
    REDUCE COST
    

    View Slide

17. @michieltcs
    FAST FEEDBACK
    

    View Slide

18. @michieltcs
    REDUCE TOIL
    

    View Slide

19. @michieltcs
    REDUCE TOIL
    the kind of work tied to running a production service that
    tends to be manual, repetitive, automatable, tactical,
    devoid of enduring value, and that scales linearly as a
    service grows
    

    View Slide

20. @michieltcs
    PUT PO/BIZ BACK IN
    DRIVER'S SEAT
    

    View Slide

21. @michieltcs
    WHY CD THEN?
    

    View Slide

22. @michieltcs
    MAKE THINGS
    SMALL
    

    View Slide

23. @michieltcs
    BIG STEPS
    

    View Slide

24. @michieltcs
    FAIL BIG
    

    View Slide

25. @michieltcs
    SMALL STEPS
    

    View Slide

26. @michieltcs
    FAIL SMALL
    

    View Slide

27. @michieltcs
    IF IT HURTS
    DO IT MORE OFTEN
    

    View Slide

28. @michieltcs
    $ = REALIZED VALUE
    CREDITS TO @FGOULDING
    

    View Slide

29. @michieltcs
    @michieltcs
    HIGH VS. LOW PERFORMING TEAMS
    Source: Accelerate - The Science Of DevOps
    

    View Slide

30. APPROACH
    

    View Slide

31. @michieltcs
    3 PHASES
    

    View Slide

32. @michieltcs
    DEFER SPLITTING
    THE MONOLITH
    

    View Slide

33. @michieltcs
    INCREASE TEMPO &
    AUTOMATION
    

    View Slide

34. @michieltcs
    SINGLE PIPELINE
    

    View Slide

35. @michieltcs
    IMPROVEMENTS &
    TESTING
    

    View Slide

36. PHASE 1:
    INCREASING
    RELEASE CADENCE
    

    View Slide

37. @michieltcs
    WEEKLY RELEASE
    

    View Slide

38. @michieltcs
    AUTOMATION
    

    View Slide

39. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment
    

    View Slide

40. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment
    

    View Slide

41. @michieltcs
    pipeline {

    stages {

    stage('Run tests') {

    sh "gradle check"

    }

    

    stage('Build docker image') {

    sh "docker build -t jobservice:${env.BUILD_NUMBER} ."

    sh "docker push jobservice:${env.BUILD_NUMBER}"

    }

    

    stage('Deploy staging') {

    sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}

    -i staging deploy.yml"

    }

    

    stage('Deploy production') {

    sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}

    -i prod deploy.yml"

    }

    }

    PIPELINE AS CODE
    image: registry.local/runner:latest

    

    stages:

    - test

    - images

    - deploy

    

    phpunit:

    stage: test

    script:

    - bin/phpunit

    

    images:

    stage: images

    script:

    - make images publish

    only:

    - master

    

    deploy:

    stage: deploy

    script:

    - make deploy

    only:

    - master
    

    View Slide

42. View Slide

43. View Slide

44. @michieltcs
    WAY OF WORKING
    

    View Slide

45. @michieltcs
    PAIR
    PROGRAMMING
    

    View Slide

46. BOY SCOUT RULE
    

    View Slide

47. @michieltcs
    NO MORE RELEASE
    BRANCHES
    

    View Slide

48. @michieltcs
    @michieltcs
    

    View Slide

49. @michieltcs
    @michieltcs
    

    View Slide

50. @michieltcs
    @michieltcs
    

    View Slide

51. PHASE 2:
    SINGLE PIPELINE
    

    View Slide

52. @michieltcs
    @michieltcs
    Build /
    Package
    Automated
    Tests
    Acceptance
    Environment
    Production
    Environment
    

    View Slide

53. @michieltcs
    CONTINUOUS
    DEPLOYMENT
    

    View Slide

54. @michieltcs
    ZERO DOWNTIME
    DEPLOYS
    

    View Slide

55. @michieltcs
    ROLLING
    DEPLOYS
    

    View Slide

56. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.0
    SERVICE V1.0
    @michieltcs
    

    View Slide

57. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.0
    SERVICE V1.0
    SERVICE V1.1
    @michieltcs
    

    View Slide

58. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.0
    SERVICE V1.0
    @michieltcs
    

    View Slide

59. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.0
    SERVICE V1.1
    @michieltcs
    

    View Slide

60. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.0
    @michieltcs
    

    View Slide

61. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.0 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.1
    @michieltcs
    

    View Slide

62. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.1 SERVICE V1.1
    SERVICE V1.1
    SERVICE V1.0
    @michieltcs
    

    View Slide

63. @michieltcs
    @michieltcs
    LOAD BALANCER
    ROLLING UPDATE
    SERVICE V1.1 SERVICE V1.1
    SERVICE V1.1
    @michieltcs
    

    View Slide

64. @michieltcs
    PIPELINE FAILURES
    

    View Slide

65. @michieltcs
    FLAKY TESTS
    

    View Slide

66. @michieltcs
    TIMEOUTS
    

    View Slide

67. @michieltcs
    NETWORK STABILITY
    ISSUES
    

    View Slide

68. @michieltcs
    EXTERNAL DEPS IN
    TESTS
    

    View Slide

69. @michieltcs
    BRITTLE
    INFRASTRUCTURE
    

    View Slide

70. @michieltcs
    IF YOU CAN'T TRUST
    IT - REMOVE IT
    

    View Slide

71. STOP THE LINE
    

    View Slide

72. @michieltcs
    EXTREME FEEDBACK
    

    View Slide

73. @michieltcs
    @michieltcs
    

    View Slide

74. @michieltcs
    RELEASE
    TIMING?
    

    View Slide

75. FEATURE TOGGLES
    

    View Slide

76. @michieltcs
    DECOUPLE
    DEPLOYMENTS
    FROM
    RELEASES
    

    View Slide

77. @michieltcs
    @michieltcs
    

    View Slide

78. @michieltcs
    @michieltcs
    

    View Slide

79. @michieltcs
    CAUTION:
    FEATURE TOGGLE
    DEBT
    

    View Slide

80. PHASE 3:
    IMPROVEMENTS &
    TESTING
    

    View Slide

81. @michieltcs
    PIPELINE SPEED
    

    View Slide

82. @michieltcs
    BUY A BIGGER BOX
    

    View Slide

83. @michieltcs
    CLEVER PIPELINE
    IMPROVEMENTS:
    PARALLELIZE
    

    View Slide

84. @michieltcs
    TESTING
    IMPROVEMENTS
    

    View Slide

85. @michieltcs
    @michieltcs
    UNIT
    TESTS
    E2E /
    VISUAL TESTS
    INTEGRATION
    TESTS
    LOTS OF MANUAL TESTING
    E2E TESTS
    

    View Slide

86. @michieltcs
    QUALITY
    ENGINEERING
    

    View Slide

87. @michieltcs
    AUTOMATION
    

    View Slide

88. @michieltcs
    @michieltcs
    CONTRACT TESTING
    

    View Slide

89. @michieltcs
    @michieltcs
    UI TESTING
    

    View Slide

90. @michieltcs
    @michieltcs
    VISUAL TESTING
    

    View Slide

91. @michieltcs
    @michieltcs
    DEPENDENCY SCANNING
    

    View Slide

92. @michieltcs
    @michieltcs
    UNIT TESTS
    UI /
    VISUAL TESTS
    Exploratory
    testing & user
    feedback
    Monitoring
    & alerting
    INTEGRATION / CONTRACT
    TESTS
    COST SPEED
    

    View Slide

93. RESULTS &
    TAKEAWAYS
    

    View Slide

94. @michieltcs
    LEARNINGS
    

    View Slide

95. @michieltcs
    PIPELINE SPEED &
    STABILITY
    

    View Slide

96. @michieltcs
    TEST THE PIPELINE
    

    View Slide

97. @michieltcs
    WAY OF WORKING &
    MINDSET
    

    View Slide

98. @michieltcs
    VISUAL TESTING
    TOOLS
    

    View Slide

99. @michieltcs
    SOME STATS
    

    View Slide

100. @michieltcs
     ~4000 TESTS PER
     DEPLOYMENT
     

     View Slide

101. @michieltcs
     ~10 MINUTES PER
     PIPELINE RUN
     

     View Slide

102. @michieltcs
     ~1.5 HOURS FROM
     DEV TO PROD
     

     View Slide

103. @michieltcs
     IN 6 MONTHS:
     MORE THAN 5000
     DEPLOYMENTS
     

     View Slide

104. @michieltcs
     @michieltcs
     THANK YOU FOR
     LISTENING!
     @michieltcs / michiel@michielrook.nl
     www.michielrook.nl
     

     View Slide