Continuous Delivery: More Than a Pipeline

Continuous Delivery is a lot more than just a pipeline. This talks discusses reasons why software should be deployed more frequently and the barriers that keep teams from doing so.

1. Eberhard Wolff Fellow INNOQ @ewolff http://ewolff.com
2. http://continuous-delivery-buch.de/ http://continuous-delivery-book.com/
3. http://microservices-buch.de/ http://microservices-book.com/
4. http://microservices-buch.de/ ueberblick.html http://microservices-book.com/ primer.html FREE!!!!
5. http://microservices-praxisbuch.de http://practical-microservices.com/
6. http://microservices-praxisbuch.de/ rezepte.html http://practical-microservices.com/ recipes.html FREE!!!!
7. Why Care About Continuous Delivery?
8. Continuous Delivery is just about time-to- market!
9. Continuous Delivery is just about time-to- market!
10. https://puppet.com/resources/whitepaper/state-of-devops-report
11. Deployment Frequency: Results • Elite Performers vs. Low Performers • Multiple times per day vs. once per week / month • 2.555x better lead time for change • 2.604x better time to restore service • 7x better change failure rate • 50% vs 30% time spent on new work
12. Higher Deployment Frequency Improves Lead Time Reliability New Work % dramatically
13. MAERSK COST OF DELAY POWER LAW CURVE 200K 10 30 60 70 80 20 50 40 Number of Requirements 1.2MM 2.2MM 400K 1.4MM 2.4MM 600K 1.6MM 2.6MM 1MM 2MM 1.8MM 2.8MM Cost of Delay/Week Source: State of DevOps Report, blackswarmfarming.com
14. MAERSK COST OF DELAY POWER LAW CURVE 200K 10 30 60 70 80 20 50 40 Number of Requirements 1.2MM 2.2MM 400K 1.4MM 2.4MM 600K 1.6MM 2.6MM 1MM 2MM 1.8MM 2.8MM Cost of Delay/Week Source: State of DevOps Report, blackswarmfarming.com Cost of delay might be much higher than cost reductions in operations!
15. No Silver Bullet Essence and Accident in Software Engineering Frederick P. Brooks, Jr. There is no single development... which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.
16. No Silver Bullet
17. No Silver Bullet except more frequent deployments?
18. Continuous Delivery: Goals • Deploy often • More than once a week / month …because that is still low performer • Multiple times a day! • How???
19. There many barriers to deploy multiple times a day!
20. Deployment Testing Architecture Release Branching Multiple Deploy- ments per Day Elite
21. Choose your speed - no need to break every barrier!
22. There is much room for improvement!
23. Deployment Testing Architecture Release Branching ? ? The next step is important!
24. Deployment
25. Manual Deployment • Manually deploy software to servers using scripts • Somehow check correctness • Takes hours or days
26. Problem: Slow deployment Max. Frequency: Week
27. Automated Deployment • Fully automated scripts …might be Dockerfiles • Faster • Automated deployment to deploy every few months? • Deployment: often focus but not enough
28. Customer Scenario • Quarterly releases • 10 weeks of testing • Release two days over the weekend • Goal: Several deployments a day • Automated deployment not enough Development Testing Release Testing + Release + Development Time
29. Testing
30. Separate Dev and QA • Hand over code • Hand over bug reports
38. Separate Dev and QA • Problem: Hand over of code • Hand over of bug reports • Takes time • Lean: Hand-over is waste • Does not enforce collaboration
39. Development Testing Release Time
40. Problem: Hand-over Max. Frequency: Month
41. Cross-functional Teams
42. Cross-functional Teams
43. Cross-functional Teams • Hand-over in team quicker • Might work very closely …pairing
44. Development Testing Release Time Development Testing Release Development Testing Release
45. Problem: Manual work Max. Frequency: Weeks
46. Only Automated Tests For Deployment • Fully automated test suite • …including capacity tests and acceptance tests • No manual sign-off • Really: no manual sign-off!
47. Commit Stage Automated Acceptance Testing Automated Capacity Testing Manual Explorative Testing Release Continuous Delivery Pipeline
48. Development Testing Release Time Development Testing Release Development Testing Release
49. Problem: Time for tests Max. Frequency: Days / hours
50. Going faster • Reduce risk not just with tests • …but with releasing • Smaller deployment unit • Full automation might not be feasible for large deployment units
51. Architecture
52. Monolith • Deployment monolith • Deploy the whole system at once • Internal structure: Might be great or not so great… • A valid architecture
53. Monolith: Deployment • Deploy all at once • It is hard to deploy a large system • Tests even harder • Can take loooooong – days, weeks, months • So large infrequent deployments • Risky
54. Problem: Deployment size Max. Frequency: Months / weeks
55. Microservices: Deployment • Deploy individually • Requires automated deployment • Commodity with Docker, Kubernetes… • Requires decoupled components & tests • Enables faster deployments & tests • Enables frequent deployments
56. Development Testing Release Time Development Testing Release Development Testing Release
57. Releases
58. Release Train • Release multiple dependent systems together • Extreme: Release all systems in the IT together • The reason why developers at one point in the sixties or so stopped deploying continuously?
59. Release Train • In total: a day or a few days Release System 1 Release dependent System 2 Release dependent System 3 …
60. Problem: Coordinate many changes, risk Max. Frequency: Months
61. Release per System • Much easier to coordinate • Backward compatible interfaces • Speed depends on deployment size …see architecture
62. Problem: Risk Max. Frequency: Weeks
63. Blue / Green Deployment Server Server Server Server Database Server Server Server Server Database
64. Blue / Green Deployment • Needs lots of resources • Might be unrealistic: another production environment? • Migrate database • ...and keep consistent for fallback • Easier with small microservices
65. Canary Release Server Server Server Server Database Server Server
66. Canary Release Server Server Server Server Database Server Server Server Server
67. Canary Releases • Less resources • Needs backwards compatibility of other components / database • Easier with small microservices
68. Problem: Risk (depends on size of deployment unit) Max. Frequency: Days / hours
69. Dark Launching • Install new version • Let it handle production traffic • ...but don’t let it change data. • Requires monitoring • Enables less focus on tests
70. Dark Launching Server Server Server Server Database Server
71. Branching
72. Feature Branches • Create a new branch to implement a feature • Integrate into master when done • Easy to parallelize development • Easy to add features to master at a specific point in time
73. Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit Branch Commit Commit Commit Merge Merge Release with Features
74. Feature Branches • Feature branches’ goal: Delay integration • Might still compile all changes from all branches together to find issues early • True integration into master only after go / no go for features
75. Feature Branch Master Feature Registration Feature Prediction Commit Branch Commit Branch Commit Commit Commit Merge Merge Release with Features Integration Integration
76. Feature Branches vs. Continuous Integration • Continuous Integration: Continuously integrate all changes • Feature branches’ goal: Delay integration • Both approaches might work • See https://www.heise.de/ developer/artikel/Continuous-Integration- widerspricht-Feature-Branches-2736487.html • …or https://www.innoq.com/en/blog/continuous- integration-contradicts-feature-branches/
77. Problem: Integration takes time Max. Frequency: Days?
78. Trunk-based Development • Each commit goes to master • No more integration
79. Trunk-based Development • Could deploy any time • But: User must not use unfinished features Commit Registration Commit Prediction Commit Registration Commit Prediction Commit Prediction
80. Feature Toggles • Deployment should not activate new features • Disable feature in production • Problem: Must be cleaned up eventually • Problem: Harder to test
82. Humble et al: Lean Enterprise, source: https://paulhammant.com/2013/03/13/facebook-tbd-take-2/
83. Conclusion • More frequent deployments have many advantages • Choose a desired speed • ...and break the next barrier! • It’s not just microservices or deployment automation!
84. But there are sooo many problems to solve! L
85. Actually there are sooo many solutions! J
86. Identify barriers and eliminate them!