From bi-annual to fortnightly releases in 4 months for 15 teams and a single monolith

15 teams, 1 shared monolith, 1 release every 6 months, and product demand for 1 release every 2 weeks. How do you know where to start with Continuous Delivery, when you’re surrounded by technology and organisational challenges?

This is the journey of 15 teams and their 1 shared monolith, at a federal Belgian agency. They increased their throughput from bi-annual releases to fortnightly releases in under 4 months, achieving a state of Continuous Delivery.

We used the Improvement Kata as a continuous improvement framework to execute and measure organisational change, Value Stream Mapping to analyse the current delivery process, and the Theory Of Constraints to choose which changes to apply first and start off the organisational changes we needed to improve quality and drive down lead times.

Six months after achieving Continuous Delivery, I realised it was not the plan that helped the organisation. Instead, it was Fear Conversations that helped the organisation. They allowed us to uncover, locate and understand the stakeholders’ fears in order to mitigate these fears and navigate the difficult conversations we had to have.

If you thought Continuous Delivery was just for the happy few having trendy microservices, think again!

Transcript

1. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io From bi-annual to fortnightly releases in 4 months for 15 teams and a single monolith Tenderhearted Thierry Korenlei & Graslei Ghent, Belgium
2. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Being vulnerable … I’m shy and introverted
3. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io A Belgian federal agency 15 teams (~ 150 people) and 1 monolith
4. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io 3-4 planned major releases per year 2017: 3 major releases 2018: 2 major releases deployment lead time of 28 days consuming 334 person-days
5. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io September 5th 2018: Can you help us go from bi-annual releases to fortnightly releases by the end of December 2018?
6. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io “In preparing for battle I have always found that plans are useless, but planning is indispensable.” -- Dwight Eisenhower, 1957
7. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io What not to do? Maturity Models They are fundamentally flawed.
8. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io • Improvement Kata • Value Stream Mapping • Measuring Continuous Delivery • Theory of Constraints
9. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Establish a core team representing all roles to lead the Continuous Delivery adoption.
10. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Investing in Continuous Delivery is very valuable. But it isn’t easy nor cheap !
11. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Technological changes Version Control Test Driven Development Trunk Based Development Acceptance Test Driven Development Deployment Pipeline Contract Tests Continuous Integration Exploratory Testing Automated Configuration Smoke Tests Automated Infrastructure Test Data Management Evolutionary Architecture Monitoring & Alerting Database Migrations Incremental Releases source: Steve Smith
12. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Organisational changes Small Batch Sizes Everyone does On-Call Empowered Product Teams Continuous Change Review Cross-Functional Teams Traceability of Changes Shared Incentives Upskilling and Empowering Employees Blameless Post-Mortems Conway's Law Alignment You build it, You run it Continuous Improvement source: Steve Smith
13. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Improvement Kata
14. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Agency's Improvement Kata
15. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Agency's Improvement Kata
16. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Major release ~ 6 months lead time 5 months: accumulation of features 3 weeks: code freeze 1 week: production deployment Champagne 🍾🎉 🥂 334 person-days
17. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Patch releases - every fortnight
18. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Theory of Constraints Every system has a bottleneck. Spending time optimising anything other than the bottleneck is an illusion.
19. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Where is the bottleneck? activity lead time activity failure rate not here
20. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Where is the bottleneck? activity lead time activity failure rate VCS Branches but here !!!
21. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io First experiment: the deployment pipeline and its metrics
22. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Agency's Improvement Kata
23. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Agency's Improvement Kata
24. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io The Agency's Improvement Kata Set up a Deployment Pipeline ✔ Daily evaluation of failing tests Dedicated environment for automated acceptance tests Recreate database before running acceptance tests to clean up test data Stub 3rd party services Collect lead time and failure rate metrics from the deployment pipeline
25. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io “... but plans are worthless when the fighting is once begun, and all depends on the inspiration of the moment.” -- a war correspondent for the British newspaper “The Daily News”, 1877 ➔ Fear conversations
26. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Fear 1: Fear of Complexity
27. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Complex branching strategy
28. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Proposed branching strategy roadmap
29. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Fear 2: Fear of Missing Key Steps
30. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Meanwhile … 18/10: LT of 8h, 21/3529 failing tests 29/10: LT of 4h, 545/5390 failing tests 05/11: no pipeline run for 5 days due to a Jenkins upgrade 23/11: first green pipeline, all tests are passing 🎉 After which the pipeline went red for another 2 weeks 😒
31. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Fear 3: Fear of Deadlines
32. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Introduce a manual overrule step “Sunlight is said to be the best disinfectants.” -- Louis Brandeis
33. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Fear 4: Fear of Bugs
34. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Introduce quarantined tests
35. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io “They did not know it was impossible, so they did it.” -- Mark Twain
36. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io But, but, but … are fortnightly releases considered to be Continuous Delivery? Yes!
37. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io It is possible to achieve Continuous Delivery without first reaching Continuous Integration. But we will not be able to sustain Continuous Delivery in the long run.
38. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io WTF am I doing here? And yet, it happened. In the most unexpected way
39. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Acknowledgments: Els, the one I love. Steve Smith for reviewing this presentation, always making sure I do not hide myself. Jeffrey Fredrick and Douglas Squirrel for including this case in “Agile Conversations” https://www.conversationaltransformation.com/ Johanna Rothman for reviewing and helping to improve the abstract. https://www.jrothman.com/books/write-a-conference-proposal-the-conference-wants-and-accepts/ Slidedeck: https://thinkinglabs.io/talks/from-bi-annual-to-fortnightly-releases Hello, I am Thierry de Pauw fancies dark chocolate, black coffee, peated whisky
40. @tdpauw@mastodon.social @tdpauw.bsky.social thinkinglabs.io Bibliography Lean Enterprise (ch 6 Deploy Continuous Improvement), Jez Humble, Joanne Molesky, Barry O'Reilly Supplemental chapter to Accelerate: How to transform, Jez Humble The Improvement Kata, Mike Rother Measuring Continuous Delivery, Steve Smith The Goal, Eliyahu Goldratt A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware, Gary Gruver, Mike Young, Pat Fulghum Dual Value Streams, Steve Smith Got a wicked problem? First, tell me how you make toast, Tom Wujec, TEDGlobal 2013 Agile Conversations, Jeffrey Fredericks and Douglas Squirrel