DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

Talk delivered by Manuel Pais

Transcript

1. Continuous Delivery Patterns
   for Boring Releases
   

   View Slide

2. About me
   Manuel Pais
   MS Software Eng
   @manupaisable
   manuelpais.net
   me@manuelpais.net
   DevOps and Delivery Consultant
   Focused on teams and flow
   2
   @manupaisable | manuelpais.net
   

   View Slide

3. @manupaisable | manuelpais.net 3
   Today
   1. Intro to boring releases
   2. Patterns for safer releases
   3. Patterns for faster releases
   4. Patterns for sustainable delivery
   

   View Slide

4. @manupaisable | manuelpais.net 4
   Today
   1. Intro to boring releases
   2. Patterns for safer releases
   3. Patterns for faster releases
   4. Patterns for sustainable delivery
   

   View Slide

5. 5
   @manupaisable | manuelpais.net
   

   View Slide

6. 6
   App down,
   refuses to restart
   @manupaisable | manuelpais.net
   

   View Slide

7. 7
   App down,
   refuses to restart
   Data migration
   completed… partially
   @manupaisable | manuelpais.net
   

   View Slide

8. 8
   App down,
   refuses to restart…
   Missing
   libraries…
   Data migration
   completed… partially
   @manupaisable | manuelpais.net
   

   View Slide

9. 9
   App down,
   refuses to restart…
   The password master…
   Data migration
   completed… partially
   Missing
   libraries…
   @manupaisable | manuelpais.net
   

   View Slide

10. 10
    @manupaisable | manuelpais.net
    

    View Slide

11. 11
    @manupaisable | manuelpais.net
    

    View Slide

12. @manupaisable | manuelpais.net 12
    

    View Slide

13. @manupaisable | manuelpais.net 13
    

    View Slide

14. @manupaisable | manuelpais.net 14
    Build Test Deploy
    

    View Slide

15. @manupaisable | manuelpais.net 15
    Build Test Deploy
    Build Test Deploy
    

    View Slide

16. @manupaisable | manuelpais.net 16
    Build Test Deploy
    Build Test Deploy
    Build Test Deploy
    

    View Slide

17. @manupaisable | manuelpais.net 17
    Build Test Deploy
    Build Test Deploy
    Build Test Deploy Monitor
    

    View Slide

18. 18
    Outcomes
    @manupaisable | manuelpais.net
    

    View Slide

19. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 19
    

    View Slide

20. @manupaisable | manuelpais.net 20
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery
    

    View Slide

21. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 21
    

    View Slide

22. STEP ONE
    map all release activities in a
    deployment pipeline
    22
    @manupaisable | manuelpais.net
    

    View Slide

23. 23
    @manupaisable | manuelpais.net
    

    View Slide

24. Gains from
    improving visibility to everyone,
    and increasing repeatability and
    traceability are enormous
    24
    @manupaisable | manuelpais.net
    

    View Slide

25. Value Stream Mapping
    @manupaisable | manuelpais.net 25
    

    View Slide

26. Value Stream Mapping
    is a pen and paper tool that
    highlights bottlenecks in a matter of
    hours and raises awareness of what
    difficulties other teams face
    @manupaisable | manuelpais.net 26
    

    View Slide

27. STEP TWO
    measure key metrics on:
    -speed (e.g. cycle time)
    -quality (e.g. defect rate)
    -operability (e.g. MTTR)
    27
    @manupaisable | manuelpais.net
    

    View Slide

28. 28
    @manupaisable | manuelpais.net
    

    View Slide

29. STEP THREE
    PUT IN THE WORK !!!
    29
    @manupaisable | manuelpais.net
    

    View Slide

30. Which work?
    Automated builds
    in clean environments
    30
    @manupaisable | manuelpais.net
    

    View Slide

31. Which work?
    You can provision a
    mini-replica of production
    environnment
    31
    @manupaisable | manuelpais.net
    

    View Slide

32. Which work?
    You have application health checks
    and fast smoke tests
    for sanity checking
    32
    @manupaisable | manuelpais.net
    

    View Slide

33. Which work?
    You have acceptance tests
    with reasonable coverage,
    including failure scenarios
    33
    @manupaisable | manuelpais.net
    

    View Slide

34. Which work?
    Single source of truth.
    Single binary.
    Single path to production.
    34
    @manupaisable | manuelpais.net
    

    View Slide

35. STEP THREE
    PUT IN THE WORK !!!
    (but in iterative fashion…
    walking skeletons FTW!)
    35
    @manupaisable | manuelpais.net
    

    View Slide

36. 36
    @manupaisable | manuelpais.net
    

    View Slide

37. 37
    practices first,
    tools second
    @manupaisable | manuelpais.net
    

    View Slide

38. Practices for boring releases
    •Automated build (and unit tests)
    •Provision prod replica in pipeline
    •Automated acceptance tests (BDD)
    •No “invisible” activities
    •One source of truth
    •One path to production
    @manupaisable | manuelpais.net 38
    

    View Slide

39. @manupaisable | manuelpais.net 39
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery
    

    View Slide

40. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 40
    

    View Slide

41. @manupaisable | manuelpais.net 41
    can’t auto-
    scale people
    How to cope with
    ever increasing
    cognitive load on
    teams to build and
    run applications?
    

    View Slide

42. @manupaisable | manuelpais.net 42
    CI
    Peer
    review
    Infra Security Comply
    

    View Slide

43. @manupaisable | manuelpais.net 43
    CI
    Peer
    review
    Infra Security Comply
    Database Accept UX Deploy
    

    View Slide

44. @manupaisable | manuelpais.net 44
    Fix
    Monitor
    Run
    CI
    Peer
    review
    Infra Security Comply
    Database Accept UX Deploy
    

    View Slide

45. Multiple approaches needed
    1. Smarter pipelines
    2. Team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 45
    

    View Slide

46. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 46
    

    View Slide

47. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    4. Resilient delivery system
    @manupaisable | manuelpais.net 47
    

    View Slide

48. Multiple approaches needed
    1. Smarter pipelines
    2. Well-thought team structures
    3. Self-service platforms
    @manupaisable | manuelpais.net 49
    

    View Slide

49. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    50
    @manupaisable | manuelpais.net
    1. Smarter pipelines
    

    View Slide

50. Waiting for pipeline
    @manupaisable | manuelpais.net 51
    

    View Slide

51. Waiting for pipeline
    @manupaisable | manuelpais.net 52
    

    View Slide

52. Reduce wait for pipeline
    @manupaisable | manuelpais.net 53
    

    View Slide

53. Scalable CI/CD Infrastructure
    •Solved problem in cloud systems
    •Starts with infrastructure-as-code
    •Agent farm (auto scaling if possible)
    •Pipelines need to evolve as number of
    teams grows
    @manupaisable | manuelpais.net 54
    

    View Slide

54. Waiting for dependencies
    @manupaisable | manuelpais.net 55
    

    View Slide

55. Waiting for dependencies
    @manupaisable | manuelpais.net 56
    

    View Slide

56. Flow efficiency =
    “teams who aren’t paying attention to this concept generally
    have flow efficiencies around the 15% mark - that means that
    work normally spends 85% of its lifecycle waiting on something.”
    http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using
    @manupaisable | manuelpais.net 57
    

    View Slide

57. issue is not how long it takes to do
    something, it's how long we're
    waiting for it to get done
    @manupaisable | manuelpais.net 58
    

    View Slide

58. @manupaisable | manuelpais.net 59
    

    View Slide

59. Initial state: serial pipeline
    @manupaisable | manuelpais.net 60
    

    View Slide

60. Intermediate state: CAB in path to prod
    @manupaisable | manuelpais.net 61
    

    View Slide

61. End state: short and wide pipeline
    @manupaisable | manuelpais.net 62
    

    View Slide

62. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    63
    @manupaisable | manuelpais.net
    1. Smarter pipelines
    

    View Slide

63. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    64
    @manupaisable | manuelpais.net
    1. Smarter pipelines
    

    View Slide

64. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    65
    @manupaisable | manuelpais.net
    1. Smarter pipelines
    

    View Slide

65. Reduce wait times
    Minimum path to production
    Risk-based activities
    Continuous pruning
    66
    @manupaisable | manuelpais.net
    1. Smarter pipelines
    

    View Slide

66. @manupaisable | manuelpais.net 70
    Today
    1. Intro to boring releases
    2. Patterns for safer releases
    3. Patterns for faster releases
    4. Patterns for sustainable delivery
    

    View Slide

67. “ability to get changes of all types,
    into production, or into the hands of
    users, safely and quickly in a
    sustainable way”
    –Dave Farley & Jez Humble
    continuousdelivery.com
    @manupaisable | manuelpais.net 71
    

    View Slide

68. @manupaisable | manuelpais.net 72
    

    View Slide

69. Delivery system itself should be boring, not just the pipeline.
    73
    @manupaisable | manuelpais.net
    

    View Slide

70. Resilient delivery system
    that enables
    fast(er) feedback loops
    74
    @manupaisable | manuelpais.net
    

    View Slide

71. Delivery system
    •CI tool
    •Pipeline orchestration tool
    •Orchestration plugins / 3rd party tools
    •Pipeline definitions
    •Source repos
    •CI + CD infrastructure
    •And ?
    @manupaisable | manuelpais.net 75
    

    View Slide

72. Delivery system
    @manupaisable | manuelpais.net 76
    

    View Slide

73. Delivery system
    CI/CD Toolchain
    @manupaisable | manuelpais.net 77
    

    View Slide

74. Delivery system
    CI/CD Toolchain
    App 1
    App 2
    @manupaisable | manuelpais.net 78
    

    View Slide

75. Delivery system
    CI/CD Toolchain
    Pipelines
    App 1
    App 2
    @manupaisable | manuelpais.net 79
    

    View Slide

76. Delivery system
    CI/CD Toolchain
    Pipelines
    Infra
    App 1
    App 2
    @manupaisable | manuelpais.net 80
    

    View Slide

77. How resilient delivery looks like
    •Tooling and configuration changes do
    not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 81
    

    View Slide

78. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 82
    

    View Slide

79. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, roll backed if needed
    •Disaster recovery at a click of a button
    @manupaisable | manuelpais.net 83
    

    View Slide

80. How resilient delivery looks like
    •Changes (plugins, configuration, jobs,
    etc) do not impact regular delivery
    •Gracefully handles peak load of
    pipeline runs
    •Issues with underlying infra/tools
    handled swiftly, rollback if needed
    •Disaster recovery at a click of a button
    (almost)
    @manupaisable | manuelpais.net 84
    

    View Slide

81. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 85
    

    View Slide

82. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 86
    

    View Slide

83. @manupaisable | manuelpais.net 87
    

    View Slide

84. Bonus Points
    pipeline-as-code allows
    designing future states of the
    value stream and draw evolution
    @manupaisable | manuelpais.net 88
    

    View Slide

85. Practices for resiliency (1/3)
    •Pipeline infrastructure-as-code
    •Pipeline configuration-as-code
    •Build and release from zero to live
    @manupaisable | manuelpais.net 89
    

    View Slide

86. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 90
    

    View Slide

87. The glitch is believed to have been caused by a power supply issue and there is
    no evidence of a cyber-attack, the airline said.
    @manupaisable | manuelpais.net 91
    

    View Slide

88. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 92
    

    View Slide

89. Team D
    Team B
    Team A
    Team C
    @manupaisable | manuelpais.net 93
    

    View Slide

90. Team A
    @manupaisable | manuelpais.net 94
    Team D
    Team B
    Team C
    

    View Slide

91. Team A
    Team C
    @manupaisable | manuelpais.net 95
    Team D
    Team B
    

    View Slide

92. Team A Team B
    Team C
    @manupaisable | manuelpais.net 96
    Team D
    

    View Slide

93. Team A Team B
    Team C Team D
    @manupaisable | manuelpais.net 97
    

    View Slide

94. Borat’s Law / Caveat
    @manupaisable | manuelpais.net 98
    

    View Slide

95. Practices for resiliency (2/3)
    •Pipeline immutable infrastructure
    •Blue-green pipeline deployments
    @manupaisable | manuelpais.net 99
    

    View Slide

96. Practices for resiliency (2/3)
    •Pipeline immutable infrastructure
    •Blue-green pipeline deployments
    @manupaisable | manuelpais.net 100
    

    View Slide

97. @manupaisable | manuelpais.net 101
    

    View Slide

98. @manupaisable | manuelpais.net 102
    

    View Slide

99. Practices for resiliency (3/3)
    •Aggregated logging
    •Monitoring & alerting
    •Auto-scaling
    @manupaisable | manuelpais.net 103
    

    View Slide

100. Books
     104
     @manupaisable | manuelpais.net
     

     View Slide

101. releasabilitybook.com
     Book sample out now!
     Team Guide to
     Software Releasability
     by Chris O’Dell & Manuel Pais
     105
     @manupaisable | manuelpais.net
     

     View Slide

102. @manupaisable | manuelpais.net 106
     manuelpais.net
     

     View Slide

103. thank you
     Manuel Pais
     MS Software Eng
     @manupaisable
     manuelpais.net
     me@manuelpais.net
     DevOps and Delivery Consultant
     Focused on teams and flow
     107
     @manupaisable | manuelpais.net
     

     View Slide