Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Engineering Large Systems When You're Not Googl...
Search
Charity Majors
April 30, 2018
Technology
20
5.5k
Engineering Large Systems When You're Not Google Or Facebook (test in prod)
lightning talk at Clever, 4/30/18
Charity Majors
April 30, 2018
Tweet
Share
More Decks by Charity Majors
See All by Charity Majors
SRECon 2024 Keynote: Is It Already Time To Version Observability? (Signs Point To Yes)
charity
3
380
CTO Craft Con Keynote: Observability is due for a version change: are you ready for it?
charity
4
1.1k
Case Studies: Modern Development Practices In Highly Regulated Environments
charity
5
3.9k
Compliance & Regulatory Standards Are NOT Incompatible With Modern Development Best Practices
charity
7
5.8k
Perils, Pitfalls and Pratfalls of Platform Engineering (QCon NYC, 2023)
charity
1
330
The Death of DevOps Has Been Greatly Exaggerated, but Platform Engineering Is Here To Stay
charity
2
460
The Future of Ops Jobs (PlatformCon 2023)
charity
3
180
Observability and the Glorious Future (with Liz Fong-Jones)
charity
0
310
The Engineer/Manager Pendulum (QCon SF 2022)
charity
1
880
Other Decks in Technology
See All in Technology
分解して理解する Aspire
nenonaninu
2
730
ビジネスモデリング道場 目的と背景
masuda220
PRO
9
690
「正しく」失敗できる チームの作り方 〜リアルな事例から紐解く失敗を恐れない組織とは〜 / A team that can fail correctly
i35_267
2
710
Apache Iceberg Case Study in LY Corporation
lycorptech_jp
PRO
0
260
Raycast Favorites × Script Command で実現するお手軽情報チェック
smasato
1
120
【詳説】コンテンツ配信 システムの複数機能 基盤への拡張
hatena
0
190
RayでPHPのデバッグをちょっと快適にする
muno92
PRO
0
120
次世代KYC活動報告 / 20250219-BizDay17-KYC-nextgen
oidfj
0
460
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
200
脳波を用いた嗜好マッチングシステム
hokkey621
0
270
Visualize, Visualize, Visualize and rclone
tomoaki0705
9
75k
JEDAI Meetup! Databricks AI/BI概要
databricksjapan
0
300
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.7k
How STYLIGHT went responsive
nonsquared
98
5.4k
Fireside Chat
paigeccino
34
3.2k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
50k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
Raft: Consensus for Rubyists
vanstee
137
6.8k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.8k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
It's Worth the Effort
3n
184
28k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
4
360
Transcript
Engineering Large Systems When You’re Not Google Or Facebook Some
Advice By Charity Majors
None
I blame this guy: Testing in production has gotten a
bad rap.
None
how they think we are how we really are
but *why*?
monitoring => observability known unknowns => unknown unknowns LAMP stack
=> distributed systems
“Complexity is increasing” - Science
Many catastrophic states exist at any given time. Your system
is never entirely ‘up’
We are all distributed systems engineers now the unknowns outstrip
the knowns why does this matter more and more?
Distributed systems are particularly hostile to being cloned or imitated
(or monitored). (clients, concurrency, chaotic traffic patterns, edge cases …)
Distributed systems have an infinitely long list of almost-impossible failure
scenarios that make staging environments particularly worthless. this is a black hole for engineering time
unit tests integration tests functional tests basic failover test before
prod: … the basics. the simple stuff. known-unknowns
behavioral tests experiments load tests (!!) edge cases canaries rolling
deploys multi-region test in prod: unknown-unknowns
test in staging? meh
unit tests integration tests functional tests “What happens when …”
(you know the answer) “What happens when …” (you don’t) behavioral tests experiments load tests (!!) edge cases canaries rolling deploys multi-region test before prod: test in prod:
Only production is production. You can ONLY verify the deploy
for any env by deploying to that env
1. Every deploy is a *unique* exercise of your process+
code+system 2. Deploy scripts are production code. If you’re using fabric or capistrano, this means you have fab/cap in production.
Staging is not production.
Why do people sink so much time into staging, when
they can’t even tell if their own production environment is healthy or not?
That energy is better used elsewhere: Production. You can catch
80% of the bugs with 20% of the effort. And you should. @caitie’s PWL talk: https://youtu.be/-3tw2MYYT0Q
feature flags (launch darkly) high cardinality tooling (honeycomb) canary canary
canaries, shadow systems (goturbine, linkerd) capture/replay for databases (apiary, percona) also build or use: plz dont build your own ffs
Failure is not rare Practice shipping and fixing lots of
small problems And practice on your users!!
Failure: it’s “when”, not “if” (lots and lots and lots
of “when’s”)
Does everyone … know what normal looks like? know how
to deploy? know how to roll back? know how to canary? know how to debug in production? Practice!!~
None
None
None
• Charity Majors @mipsytipsy