DevOpsPorto Meetup18: Continuous Delivery Patterns for Boring Releases by Manuel Pais

Talk delivered by Manuel Pais

Transcript

1. Continuous Delivery Patterns for Boring Releases
2. About me Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net DevOps and Delivery Consultant Focused on teams and flow 2 @manupaisable | manuelpais.net
3. @manupaisable | manuelpais.net 3 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
4. @manupaisable | manuelpais.net 4 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
5. 5 @manupaisable | manuelpais.net
6. 6 App down, refuses to restart @manupaisable | manuelpais.net
7. 7 App down, refuses to restart Data migration completed… partially @manupaisable | manuelpais.net
8. 8 App down, refuses to restart… Missing libraries… Data migration completed… partially @manupaisable | manuelpais.net
9. 9 App down, refuses to restart… The password master… Data migration completed… partially Missing libraries… @manupaisable | manuelpais.net
10. 10 @manupaisable | manuelpais.net
11. 11 @manupaisable | manuelpais.net
12. @manupaisable | manuelpais.net 12
13. @manupaisable | manuelpais.net 13
14. @manupaisable | manuelpais.net 14 Build Test Deploy
15. @manupaisable | manuelpais.net 15 Build Test Deploy Build Test Deploy
16. @manupaisable | manuelpais.net 16 Build Test Deploy Build Test Deploy Build Test Deploy
17. @manupaisable | manuelpais.net 17 Build Test Deploy Build Test Deploy Build Test Deploy Monitor
18. 18 Outcomes @manupaisable | manuelpais.net
19. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 19
20. @manupaisable | manuelpais.net 20 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
21. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 21
22. STEP ONE map all release activities in a deployment pipeline 22 @manupaisable | manuelpais.net
23. 23 @manupaisable | manuelpais.net
24. Gains from improving visibility to everyone, and increasing repeatability and traceability are enormous 24 @manupaisable | manuelpais.net
25. Value Stream Mapping @manupaisable | manuelpais.net 25
26. Value Stream Mapping is a pen and paper tool that highlights bottlenecks in a matter of hours and raises awareness of what difficulties other teams face @manupaisable | manuelpais.net 26
27. STEP TWO measure key metrics on: -speed (e.g. cycle time) -quality (e.g. defect rate) -operability (e.g. MTTR) 27 @manupaisable | manuelpais.net
28. 28 @manupaisable | manuelpais.net
29. STEP THREE PUT IN THE WORK !!! 29 @manupaisable | manuelpais.net
30. Which work? Automated builds in clean environments 30 @manupaisable | manuelpais.net
31. Which work? You can provision a mini-replica of production environnment 31 @manupaisable | manuelpais.net
32. Which work? You have application health checks and fast smoke tests for sanity checking 32 @manupaisable | manuelpais.net
33. Which work? You have acceptance tests with reasonable coverage, including failure scenarios 33 @manupaisable | manuelpais.net
34. Which work? Single source of truth. Single binary. Single path to production. 34 @manupaisable | manuelpais.net
35. STEP THREE PUT IN THE WORK !!! (but in iterative fashion… walking skeletons FTW!) 35 @manupaisable | manuelpais.net
36. 36 @manupaisable | manuelpais.net
37. 37 practices first, tools second @manupaisable | manuelpais.net
38. Practices for boring releases •Automated build (and unit tests) •Provision prod replica in pipeline •Automated acceptance tests (BDD) •No “invisible” activities •One source of truth •One path to production @manupaisable | manuelpais.net 38
39. @manupaisable | manuelpais.net 39 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
40. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 40
41. @manupaisable | manuelpais.net 41 can’t auto- scale people How to cope with ever increasing cognitive load on teams to build and run applications?
42. @manupaisable | manuelpais.net 42 CI Peer review Infra Security Comply
43. @manupaisable | manuelpais.net 43 CI Peer review Infra Security Comply Database Accept UX Deploy
44. @manupaisable | manuelpais.net 44 Fix Monitor Run CI Peer review Infra Security Comply Database Accept UX Deploy
45. Multiple approaches needed 1. Smarter pipelines 2. Team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 45
46. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 46
47. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms 4. Resilient delivery system @manupaisable | manuelpais.net 47
48. Multiple approaches needed 1. Smarter pipelines 2. Well-thought team structures 3. Self-service platforms @manupaisable | manuelpais.net 49
49. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 50 @manupaisable | manuelpais.net 1. Smarter pipelines
50. Waiting for pipeline @manupaisable | manuelpais.net 51
51. Waiting for pipeline @manupaisable | manuelpais.net 52
52. Reduce wait for pipeline @manupaisable | manuelpais.net 53
53. Scalable CI/CD Infrastructure •Solved problem in cloud systems •Starts with infrastructure-as-code •Agent farm (auto scaling if possible) •Pipelines need to evolve as number of teams grows @manupaisable | manuelpais.net 54
54. Waiting for dependencies @manupaisable | manuelpais.net 55
55. Waiting for dependencies @manupaisable | manuelpais.net 56
56. Flow efficiency = “teams who aren’t paying attention to this concept generally have flow efficiencies around the 15% mark - that means that work normally spends 85% of its lifecycle waiting on something.” http://leankanban.com/flow-efficiency-a-great-metric-you-probably-arent-using @manupaisable | manuelpais.net 57
57. issue is not how long it takes to do something, it's how long we're waiting for it to get done @manupaisable | manuelpais.net 58
58. @manupaisable | manuelpais.net 59
59. Initial state: serial pipeline @manupaisable | manuelpais.net 60
60. Intermediate state: CAB in path to prod @manupaisable | manuelpais.net 61
61. End state: short and wide pipeline @manupaisable | manuelpais.net 62
62. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 63 @manupaisable | manuelpais.net 1. Smarter pipelines
63. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 64 @manupaisable | manuelpais.net 1. Smarter pipelines
64. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 65 @manupaisable | manuelpais.net 1. Smarter pipelines
65. Reduce wait times Minimum path to production Risk-based activities Continuous pruning 66 @manupaisable | manuelpais.net 1. Smarter pipelines
66. @manupaisable | manuelpais.net 70 Today 1. Intro to boring releases 2. Patterns for safer releases 3. Patterns for faster releases 4. Patterns for sustainable delivery
67. “ability to get changes of all types, into production, or into the hands of users, safely and quickly in a sustainable way” –Dave Farley & Jez Humble continuousdelivery.com @manupaisable | manuelpais.net 71
68. @manupaisable | manuelpais.net 72
69. Delivery system itself should be boring, not just the pipeline. 73 @manupaisable | manuelpais.net
70. Resilient delivery system that enables fast(er) feedback loops 74 @manupaisable | manuelpais.net
71. Delivery system •CI tool •Pipeline orchestration tool •Orchestration plugins / 3rd party tools •Pipeline definitions •Source repos •CI + CD infrastructure •And ? @manupaisable | manuelpais.net 75
72. Delivery system @manupaisable | manuelpais.net 76
73. Delivery system CI/CD Toolchain @manupaisable | manuelpais.net 77
74. Delivery system CI/CD Toolchain App 1 App 2 @manupaisable | manuelpais.net 78
75. Delivery system CI/CD Toolchain Pipelines App 1 App 2 @manupaisable | manuelpais.net 79
76. Delivery system CI/CD Toolchain Pipelines Infra App 1 App 2 @manupaisable | manuelpais.net 80
77. How resilient delivery looks like •Tooling and configuration changes do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 81
78. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 82
79. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, roll backed if needed •Disaster recovery at a click of a button @manupaisable | manuelpais.net 83
80. How resilient delivery looks like •Changes (plugins, configuration, jobs, etc) do not impact regular delivery •Gracefully handles peak load of pipeline runs •Issues with underlying infra/tools handled swiftly, rollback if needed •Disaster recovery at a click of a button (almost) @manupaisable | manuelpais.net 84
81. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 85
82. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 86
83. @manupaisable | manuelpais.net 87
84. Bonus Points pipeline-as-code allows designing future states of the value stream and draw evolution @manupaisable | manuelpais.net 88
85. Practices for resiliency (1/3) •Pipeline infrastructure-as-code •Pipeline configuration-as-code •Build and release from zero to live @manupaisable | manuelpais.net 89
86. Team A Team B Team C Team D @manupaisable | manuelpais.net 90
87. The glitch is believed to have been caused by a power supply issue and there is no evidence of a cyber-attack, the airline said. @manupaisable | manuelpais.net 91
88. Team A Team B Team C Team D @manupaisable | manuelpais.net 92
89. Team D Team B Team A Team C @manupaisable | manuelpais.net 93
90. Team A @manupaisable | manuelpais.net 94 Team D Team B Team C
91. Team A Team C @manupaisable | manuelpais.net 95 Team D Team B
92. Team A Team B Team C @manupaisable | manuelpais.net 96 Team D
93. Team A Team B Team C Team D @manupaisable | manuelpais.net 97
94. Borat’s Law / Caveat @manupaisable | manuelpais.net 98
95. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments @manupaisable | manuelpais.net 99
96. Practices for resiliency (2/3) •Pipeline immutable infrastructure •Blue-green pipeline deployments @manupaisable | manuelpais.net 100
97. @manupaisable | manuelpais.net 101
98. @manupaisable | manuelpais.net 102
99. Practices for resiliency (3/3) •Aggregated logging •Monitoring & alerting •Auto-scaling @manupaisable | manuelpais.net 103
100. Books 104 @manupaisable | manuelpais.net
101. releasabilitybook.com Book sample out now! Team Guide to Software Releasability by Chris O’Dell & Manuel Pais 105 @manupaisable | manuelpais.net
102. @manupaisable | manuelpais.net 106 manuelpais.net
103. thank you Manuel Pais MS Software Eng @manupaisable manuelpais.net me@manuelpais.net DevOps and Delivery Consultant Focused on teams and flow 107 @manupaisable | manuelpais.net