Learning in production (or why Apollo 11 nearly failed) (ConFoo 2025)

LEARNING IN PRODUCTION or why the Apollo 11 landing nearly
failed Michiel Rook @michieltcs

@michieltcs #StandWithUkraine IT ALMOST DIDN'T HAPPEN

@michieltcs #StandWithUkraine

@michieltcs #StandWithUkraine "SpaceX provided audio recordings from the Crew Dragon’s
fi rst orbital test fl ight to help prepare Hurley and Behnken for the ride during launch and re-entry."

@michieltcs #StandWithUkraine KEY TAKEAWAYS:

@michieltcs #StandWithUkraine TESTING

@michieltcs #StandWithUkraine EXPERIMENTATION

@michieltcs #StandWithUkraine SIMULATION

@michieltcs #StandWithUkraine ADAPTATION

@michieltcs #StandWithUkraine "BUT THAT IS ROCKET SCIENCE!"

@michieltcs #StandWithUkraine "THAT WOULDN'T WORK HERE"

@michieltcs #StandWithUkraine "WE DON'T HAVE THE BUDGET"

@michieltcs #StandWithUkraine "WE DON'T HAVE THE PEOPLE"

@michieltcs #StandWithUkraine "WE DON'T HAVE THE TIME"

@michieltcs #StandWithUkraine WHAT CAN WE LEARN FROM SPACE?

@michieltcs #StandWithUkraine WE ARE BUILDING

@michieltcs #StandWithUkraine WE ARE PART OF

@michieltcs #StandWithUkraine COMPLEX (DISTRIBUTED) SYSTEMS

@michieltcs #StandWithUkraine NOT LINEAR NOT NEWTONIAN NOT CARTESIAN

@michieltcs #StandWithUkraine CHANGES GENERALLY NOT REVERSIBLE

@michieltcs #StandWithUkraine MODEL DOES NOT MATCH REALITY

@michieltcs #StandWithUkraine SURPRISING FAILURE MODES

@michieltcs #StandWithUkraine "OKAY, BUT WE CAN BUILD SIMPLE THINGS"

@michieltcs #StandWithUkraine "WE SHOULD JUST PLAN BETTER"

@michieltcs #StandWithUkraine "WE SHOULD JUST BE MORE CAREFUL"

@michieltcs #StandWithUkraine "WE SHOULD JUST NOT MAKE MISTAKES"

@michieltcs #StandWithUkraine DAV I D WO O D S H
T T P S : // YO U T U. B E /G N V X FG C - 5 J W

@michieltcs #StandWithUkraine "human failure is the biggest threat to security,
therefore we periodically train our employees" (found on linkedin)

@michieltcs #StandWithUkraine "OKAY, BUT WHAT IF WE JUST TEST MORE"

@michieltcs @michieltcs #StandWithUkraine UNIT TESTS UI / E2E / VISUAL
TESTS INTEGRATION / CONTRACT TESTS COST SPEED

@michieltcs #StandWithUkraine "Testing shows the presence, not absence, of bugs."
E D S G E R W. D I J K S T RA

@michieltcs @michieltcs #StandWithUkraine

@michieltcs #StandWithUkraine "OKAY, BUT WHAT IF WE STOP CHANGE"

@michieltcs #StandWithUkraine "... incidents resulting from change is one of
the most e ff ective metrics .... It isn’t a measure of system failures; it’s a measure of departmental failures"

@michieltcs #StandWithUkraine "Every week of delay between having an idea
and launching it to customers can mean millions of dollars lost in opportunity costs. IT matters." S T E V E S M I T H

@michieltcs #StandWithUkraine CHANGE IS INEVITABLE

@michieltcs #StandWithUkraine CHANGE IS NECESSARY

@michieltcs #StandWithUkraine "IF YOU’VE GOT TO BLAME SOMETHING, BLAME IMPERMANENCE."

@michieltcs #StandWithUkraine DEALING WITH THE UNKNOWN

@michieltcs #StandWithUkraine BUILD YOUR ADAPTIVE CAPACITY

@michieltcs #StandWithUkraine "We don’t rise to the level of our
expectations, we fall to the level of our training." A R C H I LO C H U S , G R E E K S O L D I E R , P O E T, C . 6 5 0 B C

@michieltcs #StandWithUkraine WHAT DO YOU NEED?

@michieltcs #StandWithUkraine TO BUILD AN ADAPTIVE ORGANISATION?

@michieltcs #StandWithUkraine ‣ An adaptive architecture ‣ Incremental deployments ‣
Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews auto scaling, circuit breakers, health checks H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews small & frequent deploys, blue/green, canary, rolling, rollbacks H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews terraform, ansible, packer, etc. H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews observability and operability H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews controlled experiments on systems H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews you OWN it you run it H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N / blameless postmortems, knowledge sharing, learning

Automated provisioning ‣ Ubiquitous telemetry ‣ Chaos Engineering ‣ You Build It You Run It ‣ Post-incident reviews H T T P S : // W W W. S T E V ES M I T H .T EC H / B LO G / B U I L D - O P E RA B I L I T Y- I N /

@michieltcs #StandWithUkraine INTEGRATE EARLY

@michieltcs #StandWithUkraine INTEGRATE OFTEN

@michieltcs #StandWithUkraine MAKE THINGS SMALL

@michieltcs #StandWithUkraine BIG STEPS

@michieltcs #StandWithUkraine FAIL BIG

@michieltcs #StandWithUkraine SMALL STEPS

@michieltcs #StandWithUkraine FAIL SMALL

@michieltcs #StandWithUkraine $ = REALIZED VALUE C R E D
I T S TO @ FG O U L D I N G

21 Accelerate: State of DevOps 2019 | How Do We
Compare? ELITE PERFORMERS Comparing the elite group against the low performers, we find that elite performers have… frequent code deployments 208 TIMES MORE time to recover from incidents 2,604 TIMES FASTER lead time from commit to deploy 106 TIMES FASTER change failure rate (changes are 1/7 as likely to fail) 7 TIMES LOWER Throughput Stability Source: 2019 State Of DevOps report

@michieltcs #StandWithUkraine OBSERVABILITY AND OPERABILITY

@michieltcs #StandWithUkraine "a measure of how well internal states of
a system can be inferred from knowledge of its external outputs." H T T P S : // E N .W I K I P E D I A .O R G / W I K I / O B S E RVA B I L I T Y

@michieltcs @michieltcs #StandWithUkraine source: laredoute.io

@michieltcs #StandWithUkraine "the properties of a system which make it
work well in production " H T T P S : //C O N F LU X D I G I TA L . N E T/ W H AT- I S - O P E RA B I L I T Y

@michieltcs #StandWithUkraine "the properties of a system which make it
work well in production " H T T P S : //C O N F LU X D I G I TA L . N E T/ W H AT- I S - O P E RA B I L I T Y the operator experience

@michieltcs #StandWithUkraine EASY TO DEPLOY

@michieltcs #StandWithUkraine EASY TO TEST

@michieltcs #StandWithUkraine EASY TO INSPECT

@michieltcs #StandWithUkraine FEEDBACK LOOPS

@michieltcs #StandWithUkraine

@michieltcs #StandWithUkraine CHAOS ENGINEERING

@michieltcs #StandWithUkraine "the facilitation of experiments to uncover systemic weaknesses"

@michieltcs #StandWithUkraine "the discipline of experimenting on a distributed system
in order to build con fi dence in the system’s capability to withstand turbulent conditions in production*"

@michieltcs #StandWithUkraine NOT (JUST) ABOUT BREAKING THINGS

@michieltcs #StandWithUkraine NOT (JUST) ABOUT BREAKING PROD

@michieltcs #StandWithUkraine START SMALL

@michieltcs #StandWithUkraine TEST ACC PROD

@michieltcs #StandWithUkraine H T T P S : // W
W W.YO U T U B E .C O M / WATC H ? V = N O O G K N BW0 G K

@michieltcs #StandWithUkraine INCIDENT ANALYSIS

@michieltcs #StandWithUkraine "Here's the secret: Incident analysis is not actually
about the incident." N O RA J O N ES

EVERY INCIDENT IS AN ICEBERG https:/ /www.hdwallpapers.net/nature/the-invisible-part-of-the-iceberg-wallpaper-746.htm

@michieltcs #StandWithUkraine ROOT CAUSE ANALYSIS?

@michieltcs #StandWithUkraine "What you call 'root cause' is simply the
place where you stop looking any further." S I D N E Y D E K K E R

@michieltcs #StandWithUkraine “What is the cause of the accident? This
question is just as bizarre as asking what the cause is of not having an accident.” S I D N E Y D E K K E R

@michieltcs #StandWithUkraine “... the more I realized that this accident
occurred not because something extraordinary had happened, but rather just the opposite” S C OT T A . S N O O K

@michieltcs #StandWithUkraine WHAT THEN?

@michieltcs #StandWithUkraine “"Regardless of what we discover, we understand and
truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand."” H T T P S : // R E T R O S P EC T I V E W I K I .O R G / I N D E X . P H P ? T I T L E = T H E _ P R I M E _ D I R EC T I V E

@michieltcs #StandWithUkraine BLAMELESS POSTMORTEMS

@michieltcs #StandWithUkraine BLAME AWARE POSTMORTEMS

@michieltcs #StandWithUkraine WHAT INFORMATION WAS AVAILABLE?

@michieltcs #StandWithUkraine WHAT INFORMATION MADE SENSE?

@michieltcs #StandWithUkraine WHAT FACTORS CONTRIBUTED TO THE EVENT?

@michieltcs #StandWithUkraine HOW DID THIS DIFFER FROM "NORMAL" OPS?

@michieltcs #StandWithUkraine WEAK VS STRONG SIGNALS

@michieltcs #StandWithUkraine HOW DID PEOPLE ADAPT?

@michieltcs #StandWithUkraine JUST CULTURE

@michieltcs #StandWithUkraine DO YOU ENCOURAGE SHARING AND LEARNING?

@michieltcs #StandWithUkraine PROTECT PEOPLE THAT SHARE BAD NEWS?

@michieltcs #StandWithUkraine CELEBRATE PEOPLE THAT SHARE BAD NEWS?

@michieltcs #StandWithUkraine IN SUMMARY

@michieltcs #StandWithUkraine YOU CAN'T TEST EVERYTHING

@michieltcs #StandWithUkraine YOU CAN'T PREPARE FOR EVERYTHING

@michieltcs #StandWithUkraine YOU CAN LEARN

@michieltcs #StandWithUkraine TO BE PREPARED

@michieltcs #StandWithUkraine TO DEAL WITH ANYTHING

@michieltcs @michieltcs #StandWithUkraine THANK YOU FOR LISTENING! @michieltcs / @michielrook.bsky.social
email: [email protected] www.michielrook.nl

@michieltcs @michieltcs #StandWithUkraine https://www.michielrook.nl/2024/08/literature-tips/

Learning in production (or why Apollo 11 nearly...

Learning in production (or why Apollo 11 nearly failed) (ConFoo 2025)

More Decks by Michiel Rook

Other Decks in Technology

Featured

Transcript