Slide 1
Slide 1 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
From bi-annual to fortnightly releases
in 4 months
for 15 teams and a single monolith
Tenderhearted Thierry
Korenlei & Graslei
Ghent, Belgium
Slide 2
Slide 2 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Being vulnerable …
I’m shy and introverted
Slide 3
Slide 3 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
A Belgian federal agency
15 teams (~ 150 people) and 1 monolith
Slide 4
Slide 4 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
3-4 planned major releases per year
2017: 3 major releases
2018: 2 major releases
deployment lead time of 28 days
consuming 334 person-days
Slide 5
Slide 5 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
September 5th 2018:
Can you help us go from
bi-annual releases
to fortnightly releases
by the end of December 2018?
Slide 6
Slide 6 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
“In preparing for battle I have always found that plans are useless,
but planning is indispensable.”
-- Dwight Eisenhower, 1957
Slide 7
Slide 7 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
What not to do? Maturity Models
They are fundamentally flawed.
Slide 8
Slide 8 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
● Improvement Kata
● Value Stream Mapping
● Measuring Continuous Delivery
● Theory of Constraints
Slide 9
Slide 9 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Establish a core team
representing all roles
to lead the Continuous Delivery adoption.
Slide 10
Slide 10 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Investing in Continuous Delivery
is very valuable.
But it isn’t easy nor cheap !
Slide 11
Slide 11 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Technological changes
Version Control Test Driven Development
Trunk Based Development Acceptance Test Driven Development
Deployment Pipeline Contract Tests
Continuous Integration Exploratory Testing
Automated Configuration Smoke Tests
Automated Infrastructure Test Data Management
Evolutionary Architecture Monitoring & Alerting
Database Migrations Incremental Releases
source: Steve Smith
Slide 12
Slide 12 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Organisational changes
Small Batch Sizes Everyone does On-Call
Empowered Product Teams Continuous Change Review
Cross-Functional Teams Traceability of Changes
Shared Incentives Upskilling and Empowering Employees
Blameless Post-Mortems Conway's Law Alignment
You build it, You run it Continuous Improvement
source: Steve Smith
Slide 14
Slide 14 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
The Agency's Improvement Kata
Slide 15
Slide 15 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
The Agency's Improvement Kata
Slide 16
Slide 16 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Major release ~ 6 months lead time
5 months: accumulation of features 3 weeks: code freeze 1 week:
production
deployment
Champagne 🍾🎉
🥂
334 person-days
Slide 17
Slide 17 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Patch releases - every fortnight
Slide 18
Slide 18 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Theory of Constraints
Every system has a bottleneck.
Spending time optimising anything other than the
bottleneck is an illusion.
Slide 19
Slide 19 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Where is the bottleneck?
activity lead time
activity failure rate
not here
Slide 20
Slide 20 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Where is the bottleneck?
activity lead time
activity failure rate
VCS Branches
but here !!!
Slide 21
Slide 21 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
First experiment: the deployment pipeline and its metrics
Slide 22
Slide 22 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
The Agency's Improvement Kata
Slide 23
Slide 23 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
The Agency's Improvement Kata
Slide 24
Slide 24 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
The Agency's Improvement Kata
Set up a Deployment Pipeline ✔
Daily evaluation of failing tests
Dedicated environment for automated acceptance tests
Recreate database before running acceptance tests to clean up test data
Stub 3rd party services
Collect lead time and failure rate metrics from the deployment pipeline
Slide 25
Slide 25 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
“... but plans are worthless when the fighting is once begun, and all
depends on the inspiration of the moment.”
-- a war correspondent for the British newspaper “The Daily News”, 1877
➔ Fear conversations
Slide 26
Slide 26 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Fear 1: Fear of Complexity
Slide 27
Slide 27 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Complex branching strategy
Slide 28
Slide 28 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Proposed branching strategy roadmap
Slide 29
Slide 29 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Fear 2: Fear of Missing Key Steps
Slide 30
Slide 30 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Meanwhile …
18/10: LT of 8h, 21/3529 failing tests
29/10: LT of 4h, 545/5390 failing tests
05/11: no pipeline run for 5 days due to a Jenkins upgrade
23/11: first green pipeline, all tests are passing 🎉
After which the pipeline went red for another 2 weeks 😒
Slide 31
Slide 31 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Fear 3: Fear of Deadlines
Slide 32
Slide 32 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Introduce a manual overrule step
“Sunlight is said to be the best disinfectants.”
-- Louis Brandeis
Slide 34
Slide 34 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Introduce quarantined tests
Slide 35
Slide 35 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
“They did not know it was impossible, so they did it.”
-- Mark Twain
Slide 36
Slide 36 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
But, but, but …
are fortnightly releases considered to be
Continuous Delivery? Yes!
Slide 37
Slide 37 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
It is possible to achieve Continuous Delivery
without first reaching Continuous Integration.
But we will not be able to sustain
Continuous Delivery in the long run.
Slide 38
Slide 38 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
WTF am I doing here?
And yet, it happened.
In the most unexpected way
Slide 39
Slide 39 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Acknowledgments:
Els, the one I love.
Steve Smith for reviewing this presentation, always making sure I do not hide myself.
Jeffrey Fredrick and Douglas Squirrel for including this case in “Agile Conversations”
https://www.conversationaltransformation.com/
Johanna Rothman for reviewing and helping to improve the abstract.
https://www.jrothman.com/books/write-a-conference-proposal-the-conference-wants-and-accepts/
Slidedeck:
https://thinkinglabs.io/talks/from-bi-annual-to-fortnightly-releases
Hello, I am Thierry de Pauw
fancies dark chocolate, black coffee, peated whisky
Slide 40
Slide 40 text
@[email protected]
@tdpauw.bsky.social thinkinglabs.io
Bibliography
Lean Enterprise (ch 6 Deploy Continuous Improvement), Jez Humble, Joanne Molesky, Barry O'Reilly
Supplemental chapter to Accelerate: How to transform, Jez Humble
The Improvement Kata, Mike Rother
Measuring Continuous Delivery, Steve Smith
The Goal, Eliyahu Goldratt
A Practical Approach to Large-Scale Agile Development: How HP Transformed LaserJet FutureSmart Firmware,
Gary Gruver, Mike Young, Pat Fulghum
Dual Value Streams, Steve Smith
Got a wicked problem? First, tell me how you make toast, Tom Wujec, TEDGlobal 2013
Agile Conversations, Jeffrey Fredericks and Douglas Squirrel