DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

Talk delivered by Manuel Pais

A2c14a1c4e16aa337c7d36abe7d1cf8f?s=128

DevOpsPorto

July 12, 2018
Tweet

Transcript

  1. Continuous Delivery Patterns for Boring Releases

  2. About me Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net

    DevOps and Delivery Consultant Focused on teams and flow 2 @manupaisable | manuelpais.net
  3. @manupaisable | manuelpais.net 3 Today 1. Intro to boring releases

    2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  4. @manupaisable | manuelpais.net 4 Today 1. Intro to boring releases

    2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  5. 5 @manupaisable | manuelpais.net

  6. 6 App down, refuses to restart @manupaisable | manuelpais.net

  7. 7 App down, refuses to restart Data migration completed… partially

    @manupaisable | manuelpais.net
  8. 8 App down, refuses to restart… Missing libraries… Data migration

    completed… partially @manupaisable | manuelpais.net
  9. 9 App down, refuses to restart… The password master… Data

    migration completed… partially Missing libraries… @manupaisable | manuelpais.net
  10. 10 @manupaisable | manuelpais.net

  11. 11 @manupaisable | manuelpais.net

  12. @manupaisable | manuelpais.net 12

  13. @manupaisable | manuelpais.net 13

  14. @manupaisable | manuelpais.net 14 Build Test Deploy

  15. @manupaisable | manuelpais.net 15 Build Test Deploy Build Test Deploy

  16. @manupaisable | manuelpais.net 16 Build Test Deploy Build Test Deploy

    Build Test Deploy
  17. @manupaisable | manuelpais.net 17 Build Test Deploy Build Test Deploy

    Build Test Deploy Monitor
  18. 18 Outcomes @manupaisable | manuelpais.net

  19. “ability to get changes of all types, into production, or

    into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 19
  20. @manupaisable | manuelpais.net 20 Today 1. Intro to boring releases

    2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  21. “ability to get changes of all types, into production, or

    into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 21
  22. STEP ONE map all release activities in a deployment pipeline

    22 @manupaisable | manuelpais.net
  23. 23 @manupaisable | manuelpais.net

  24. Gains from improving visibility to everyone, and increasing repeatability and

    traceability are enormous 24 @manupaisable | manuelpais.net
  25. Value Stream Mapping @manupaisable | manuelpais.net 25

  26. Value Stream Mapping is a pen and paper tool that

    highlights bottlenecks in a matter of hours and raises awareness of what difficulties other teams face @manupaisable | manuelpais.net 26
  27. STEP TWO measure key metrics on: -speed (e.g. cycle time)

    -quality (e.g. defect rate) -operability (e.g. MTTR) 27 @manupaisable | manuelpais.net
  28. 28 @manupaisable | manuelpais.net

  29. STEP THREE PUT IN THE WORK !!! 29 @manupaisable |

    manuelpais.net
  30. Which work? Automated builds in clean environments 30 @manupaisable |

    manuelpais.net
  31. Which work? You can provision a mini-replica of production environnment

    31 @manupaisable | manuelpais.net
  32. Which work? You have application health checks and fast smoke

    tests for sanity checking 32 @manupaisable | manuelpais.net
  33. Which work? You have acceptance tests with reasonable coverage, including

    failure scenarios 33 @manupaisable | manuelpais.net
  34. Which work? Single source of truth. Single binary. Single path

    to production. 34 @manupaisable | manuelpais.net
  35. STEP THREE PUT IN THE WORK !!! (but in iterative

    fashion… walking skeletons FTW!) 35 @manupaisable | manuelpais.net
  36. 36 @manupaisable | manuelpais.net

  37. 37 practices first, tools second @manupaisable | manuelpais.net

  38. Practices for boring releases •Automated build (and unit tests) •Provision

    prod replica in pipeline •Automated acceptance tests (BDD) •No “invisible” activities •One source of truth •One path to production @manupaisable | manuelpais.net 38
  39. @manupaisable | manuelpais.net 39 Today 1. Intro to boring releases

    2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  40. “ability to get changes of all types, into production, or

    into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 40
  41. @manupaisable | manuelpais.net 41 can’t auto- scale people How to

    cope with ever increasing cognitive load on teams to build and run applications?
  42. @manupaisable | manuelpais.net 42 CI Peer review Infra Security Comply

  43. @manupaisable | manuelpais.net 43 CI Peer review Infra Security Comply

    Database Accept UX Deploy
  44. @manupaisable | manuelpais.net 44 Fix Monitor Run CI Peer review

    Infra Security Comply Database Accept UX Deploy
  45. Multiple approaches needed 1. Smarter pipelines 2. Team structures 3.

    Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 45
  46. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures

    3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 46
  47. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures

    3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 47
  48. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures

    3. Self-service platforms @manupaisable | manuelpais.net 49
  49. Reduce wait times Minimum path to production Risk-based activities Continuous

    pruning 50 @manupaisable | manuelpais.net 1. Smarter pipelines
  50. Waiting for pipeline @manupaisable | manuelpais.net 51

  51. Waiting for pipeline @manupaisable | manuelpais.net 52

  52. Reduce wait for pipeline @manupaisable | manuelpais.net 53

  53. Scalable CI/CD Infrastructure •Solved problem in cloud systems •Starts with

    infrastructure-as-code •Agent farm (auto scaling if possible) •Pipelines need to evolve as number of teams grows @manupaisable | manuelpais.net 54
  54. Waiting for dependencies @manupaisable | manuelpais.net 55

  55. Waiting for dependencies @manupaisable | manuelpais.net 56

  56. Flow efficiency = “teams who aren’t paying attention to this

    concept generally have flow efficiencies around the 15% mark - that means that work normally spends 85% of its lifecycle waiting on something.” http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using @manupaisable | manuelpais.net 57
  57. issue is not how long it takes to do something,

    it's how long we're waiting for it to get done @manupaisable | manuelpais.net 58
  58. @manupaisable | manuelpais.net 59

  59. Initial state: serial pipeline @manupaisable | manuelpais.net 60

  60. Intermediate state: CAB in path to prod @manupaisable | manuelpais.net

    61
  61. End state: short and wide pipeline @manupaisable | manuelpais.net 62

  62. Reduce wait times Minimum path to production Risk-based activities Continuous

    pruning 63 @manupaisable | manuelpais.net 1. Smarter pipelines
  63. Reduce wait times Minimum path to production Risk-based activities Continuous

    pruning 64 @manupaisable | manuelpais.net 1. Smarter pipelines
  64. Reduce wait times Minimum path to production Risk-based activities Continuous

    pruning 65 @manupaisable | manuelpais.net 1. Smarter pipelines
  65. Reduce wait times Minimum path to production Risk-based activities Continuous

    pruning 66 @manupaisable | manuelpais.net 1. Smarter pipelines
  66. @manupaisable | manuelpais.net 70 Today 1. Intro to boring releases

    2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
  67. “ability to get changes of all types, into production, or

    into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 71
  68. @manupaisable | manuelpais.net 72

  69. Delivery system itself should be boring, not just the pipeline.

    73 @manupaisable | manuelpais.net
  70. Resilient delivery system that enables fast(er) feedback loops 74 @manupaisable

    | manuelpais.net
  71. Delivery system •CI tool •Pipeline orchestration tool •Orchestration plugins /

    3rd party tools •Pipeline definitions •Source repos •CI + CD infrastructure •And ? @manupaisable | manuelpais.net 75
  72. Delivery system @manupaisable | manuelpais.net 76

  73. Delivery system CI/CD Toolchain @manupaisable | manuelpais.net 77

  74. Delivery system CI/CD Toolchain App 1 App 2 @manupaisable |

    manuelpais.net 78
  75. Delivery system CI/CD Toolchain Pipelines App 1 App 2 @manupaisable

    | manuelpais.net 79
  76. Delivery system CI/CD Toolchain Pipelines Infra App 1 App 2

    @manupaisable | manuelpais.net 80
  77. How resilient delivery looks like •Tooling and configuration changes do

    not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 81
  78. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc)

    do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 82
  79. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc)

    do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, roll backed if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 83
  80. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc)

    do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button (almost) @manupaisable | manuelpais.net 84
  81. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and

    release from zero to live @manupaisable | manuelpais.net 85
  82. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and

    release from zero to live @manupaisable | manuelpais.net 86
  83. @manupaisable | manuelpais.net 87

  84. Bonus Points pipeline-as-code allows designing future states of the value

    stream and draw evolution @manupaisable | manuelpais.net 88
  85. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and

    release from zero to live @manupaisable | manuelpais.net 89
  86. Team A Team B Team C Team D @manupaisable |

    manuelpais.net 90
  87. The glitch is believed to have been caused by a

    power supply issue and there is no evidence of a cyber-attack, the airline said. @manupaisable | manuelpais.net 91
  88. Team A Team B Team C Team D @manupaisable |

    manuelpais.net 92
  89. Team D Team B Team A Team C @manupaisable |

    manuelpais.net 93
  90. Team A @manupaisable | manuelpais.net 94 Team D Team B

    Team C
  91. Team A Team C @manupaisable | manuelpais.net 95 Team D

    Team B
  92. Team A Team B Team C @manupaisable | manuelpais.net 96

    Team D
  93. Team A Team B Team C Team D @manupaisable |

    manuelpais.net 97
  94. Borat’s Law / Caveat @manupaisable | manuelpais.net 98

  95. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments

    @manupaisable | manuelpais.net 99
  96. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments

    @manupaisable | manuelpais.net 100
  97. @manupaisable | manuelpais.net 101

  98. @manupaisable | manuelpais.net 102

  99. Practices for resiliency (3/3) •Aggregated logging •Monitoring & alerting •Auto-scaling

    @manupaisable | manuelpais.net 103
  100. Books 104 @manupaisable | manuelpais.net

  101. releasabilitybook.com Book sample out now! Team Guide to Software Releasability

    by Chris O’Dell & Manuel Pais 105 @manupaisable | manuelpais.net
  102. @manupaisable | manuelpais.net 106 manuelpais.net

  103. thank you Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net

    DevOps and Delivery Consultant Focused on teams and flow 107 @manupaisable | manuelpais.net