Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Distributed Failure: Learning Lessons From Avia...
Search
Andrew Godwin
April 24, 2018
Programming
2
450
Distributed Failure: Learning Lessons From Aviation
A talk I first gave at Code Europe Warsaw, spring 2018.
Andrew Godwin
April 24, 2018
Tweet
Share
More Decks by Andrew Godwin
See All by Andrew Godwin
Reconciling Everything
andrewgodwin
1
250
Django Through The Years
andrewgodwin
0
150
Writing Maintainable Software At Scale
andrewgodwin
0
380
A Newcomer's Guide To Airflow's Architecture
andrewgodwin
0
300
Async, Python, and the Future
andrewgodwin
2
590
How To Break Django: With Async
andrewgodwin
1
650
Taking Django's ORM Async
andrewgodwin
0
660
The Long Road To Asynchrony
andrewgodwin
0
580
The Scientist & The Engineer
andrewgodwin
1
680
Other Decks in Programming
See All in Programming
A Journey of Contribution and Collaboration in Open Source
ivargrimstad
0
870
CSC509 Lecture 12
javiergs
PRO
0
160
Jakarta EE meets AI
ivargrimstad
0
580
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
9
3.3k
Nurturing OpenJDK distribution: Eclipse Temurin Success History and plan
ivargrimstad
0
870
詳細解説! ArrayListの仕組みと実装
yujisoftware
0
580
どうして僕の作ったクラスが手続き型と言われなきゃいけないんですか
akikogoto
1
120
WebフロントエンドにおけるGraphQL(あるいはバックエンドのAPI)との向き合い方 / #241106_plk_frontend
izumin5210
4
1.4k
AWS Lambdaから始まった Serverlessの「熱」とキャリアパス / It started with AWS Lambda Serverless “fever” and career path
seike460
PRO
1
250
ペアーズにおけるAmazon Bedrockを⽤いた障害対応⽀援 ⽣成AIツールの導⼊事例 @ 20241115配信AWSウェビナー登壇
fukubaka0825
6
1.8k
とにかくAWS GameDay!AWSは世界の共通言語! / Anyway, AWS GameDay! AWS is the world's lingua franca!
seike460
PRO
1
860
シェーダーで魅せるMapLibreの動的ラスタータイル
satoshi7190
1
480
Featured
See All Featured
Adopting Sorbet at Scale
ufuk
73
9.1k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
GraphQLの誤解/rethinking-graphql
sonatard
67
10k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
109
49k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
126
18k
Docker and Python
trallard
40
3.1k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
6.9k
Put a Button on it: Removing Barriers to Going Fast.
kastner
59
3.5k
Documentation Writing (for coders)
carmenintech
65
4.4k
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.3k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
740
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Transcript
DISTRIBUTED FAILURE Andrew Godwin @andrewgodwin Learning lessons from aviation
Hi, I’m Andrew Godwin
Content Warning Aviation accidents Road accidents Discussion of death
Software is difficult.
Distributed is even harder.
None
Not unique to distributed systems
None
Who's solved this? Aviation.
A Boeing 747 has six million parts
A Boeing 747 has six million parts
Airplane Car Walking Train 220 130 30.8 Deaths per billion
hours (UK 1990-2000) 30
People matter as much as machines
Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other
16% Mechanical
Let's look at some aviation principles
Principle #1 Hard Failure
If something is wrong it turns itself off
This only works if you have redundancy
None
These are great ways to ensure you never fix something.
No accident or outage has a single cause. Stop your
code getting into odd states.
None
Single points of failure can be good
None
Principle #2 Good Alerting
Cockpits are incredibly selective about what sets off an audio
alarm
Alert fatigue is real. Avoid at all costs.
Never, ever, put all errors in the same place
Critical Normal Background
Critical Normal Background Wakes someone up. Actionable.
Critical Normal Background Wakes someone up. Actionable. Fixed over the
next week.
Critical Normal Background Wakes someone up. Actionable. Fixed over the
next week. Metrics, not errors.
Have you been ignoring an error for weeks? Then turn
off its error reporting.
Principle #3 Find your limits
Everything will fail. You should know when.
Copyright Boeing
What's your Minimum Equipment List?
REQUIRED OPTIONAL
Did you load test? Did you fuzz test?
You don't have to perfectly scale.
Risk is fine when you're informed!
Principle #4 Build for failure
No single thing in an aircraft can fail and take
it down.
We all want this for our code, but the way
to do it is to build for failure.
Kill your application randomly Practice server network failures Develop on
unreliable connections
The majority of pilot training is handling emergencies.
None
Use checklists. Don't rely on memory.
If you practice failure, you'll be ready when the inevitable
happens.
Pilot 76% Aviation Accident Causes (2005 Nall report) 9% Other
16% Mechanical
Principle #5 Communicate well
Distributed software means separate teams.
As you grow, communication becomes exponentially harder.
None
None
None
Clear communication is vital.
Write everything down.
Have a clear chain of command.
Make decisions.
Principle #6 No blame culture
How do I know all these aviation stats?
Every incident is reported and investigated.
There is never a single cause of a problem.
Make it very difficult to do again.
None
None
Encourage reporting.
Reward maintenance as well as firefighting
None
In aviation, every rule is written in blood.
Software is not yet there. But we are getting closer.
Margaret Hamilton Her error detection code saved Apollo 11
Therac-25 Killed 3, severely injured at least 3 more
None
None
Hard failure Good alerting Find your limits Build for failure
Communicate well No blame culture
Thanks.