Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Engineering Large Systems When You're Not Googl...
Search
Charity Majors
April 30, 2018
Technology
20
5.6k
Engineering Large Systems When You're Not Google Or Facebook (test in prod)
lightning talk at Clever, 4/30/18
Charity Majors
April 30, 2018
Tweet
Share
More Decks by Charity Majors
See All by Charity Majors
In Praise of "Normal" Engineers (LDX3)
charity
4
2.6k
In Praise of "Normal" Engineers (with full speaker notes)
charity
1
190
AIOps: Prove It! (An Open Letter to Vendors Selling AI for SREs)
charity
1
29
SRECon 2024 Keynote: Is It Already Time To Version Observability? (Signs Point To Yes)
charity
3
480
CTO Craft Con Keynote: Observability is due for a version change: are you ready for it?
charity
4
1.3k
Case Studies: Modern Development Practices In Highly Regulated Environments
charity
6
4.1k
Compliance & Regulatory Standards Are NOT Incompatible With Modern Development Best Practices
charity
7
6.1k
Perils, Pitfalls and Pratfalls of Platform Engineering (QCon NYC, 2023)
charity
1
390
The Death of DevOps Has Been Greatly Exaggerated, but Platform Engineering Is Here To Stay
charity
2
580
Other Decks in Technology
See All in Technology
「技術負債にならない・間違えない」 権限管理の設計と実装
naro143
34
10k
動画データのポテンシャルを引き出す! Databricks と AI活用への奮闘記(現在進行形)
databricksjapan
0
130
データエンジニアがこの先生きのこるには...?
10xinc
0
420
後進育成のしくじり〜任せるスキルとリーダーシップの両立〜
matsu0228
1
930
#普通の文系サラリーマンチャレンジ 自分でアプリ開発と電子工作を続けたら人生が変わった
tatsuya1970
0
890
「AI駆動PO」を考えてみる - 作る速さから価値のスループットへ:検査・適応で未来を開発 / AI-driven product owner. scrummat2025
yosuke_nagai
0
300
[2025-09-30] Databricks Genie を利用した分析基盤とデータモデリングの IVRy の現在地
wxyzzz
0
430
バイブコーディングと継続的デプロイメント
nwiizo
2
380
Trust as Infrastructure
bcantrill
0
270
BtoBプロダクト開発の深層
16bitidol
0
150
Go Conference 2025: GoのinterfaceとGenericsの内部構造と進化 / Go type system internals
ryokotmng
3
570
“2件同時配達”の開発舞台裏 〜出前館PMが挑んだダブルピック実現に向けた体験設計〜
demaecan
0
170
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Embracing the Ebb and Flow
colly
88
4.8k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
229
22k
YesSQL, Process and Tooling at Scale
rocio
173
14k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
840
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Bash Introduction
62gerente
615
210k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Producing Creativity
orderedlist
PRO
347
40k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
How to Ace a Technical Interview
jacobian
280
23k
Transcript
Engineering Large Systems When You’re Not Google Or Facebook Some
Advice By Charity Majors
None
I blame this guy: Testing in production has gotten a
bad rap.
None
how they think we are how we really are
but *why*?
monitoring => observability known unknowns => unknown unknowns LAMP stack
=> distributed systems
“Complexity is increasing” - Science
Many catastrophic states exist at any given time. Your system
is never entirely ‘up’
We are all distributed systems engineers now the unknowns outstrip
the knowns why does this matter more and more?
Distributed systems are particularly hostile to being cloned or imitated
(or monitored). (clients, concurrency, chaotic traffic patterns, edge cases …)
Distributed systems have an infinitely long list of almost-impossible failure
scenarios that make staging environments particularly worthless. this is a black hole for engineering time
unit tests integration tests functional tests basic failover test before
prod: … the basics. the simple stuff. known-unknowns
behavioral tests experiments load tests (!!) edge cases canaries rolling
deploys multi-region test in prod: unknown-unknowns
test in staging? meh
unit tests integration tests functional tests “What happens when …”
(you know the answer) “What happens when …” (you don’t) behavioral tests experiments load tests (!!) edge cases canaries rolling deploys multi-region test before prod: test in prod:
Only production is production. You can ONLY verify the deploy
for any env by deploying to that env
1. Every deploy is a *unique* exercise of your process+
code+system 2. Deploy scripts are production code. If you’re using fabric or capistrano, this means you have fab/cap in production.
Staging is not production.
Why do people sink so much time into staging, when
they can’t even tell if their own production environment is healthy or not?
That energy is better used elsewhere: Production. You can catch
80% of the bugs with 20% of the effort. And you should. @caitie’s PWL talk: https://youtu.be/-3tw2MYYT0Q
feature flags (launch darkly) high cardinality tooling (honeycomb) canary canary
canaries, shadow systems (goturbine, linkerd) capture/replay for databases (apiary, percona) also build or use: plz dont build your own ffs
Failure is not rare Practice shipping and fixing lots of
small problems And practice on your users!!
Failure: it’s “when”, not “if” (lots and lots and lots
of “when’s”)
Does everyone … know what normal looks like? know how
to deploy? know how to roll back? know how to canary? know how to debug in production? Practice!!~
None
None
None
• Charity Majors @mipsytipsy