Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Engineering Large Systems When You're Not Googl...
Search
Charity Majors
April 30, 2018
Technology
20
5.6k
Engineering Large Systems When You're Not Google Or Facebook (test in prod)
lightning talk at Clever, 4/30/18
Charity Majors
April 30, 2018
Tweet
Share
More Decks by Charity Majors
See All by Charity Majors
In Praise of "Normal" Engineers (LDX3)
charity
4
2.6k
In Praise of "Normal" Engineers (with full speaker notes)
charity
1
190
AIOps: Prove It! (An Open Letter to Vendors Selling AI for SREs)
charity
1
31
SRECon 2024 Keynote: Is It Already Time To Version Observability? (Signs Point To Yes)
charity
3
480
CTO Craft Con Keynote: Observability is due for a version change: are you ready for it?
charity
4
1.4k
Case Studies: Modern Development Practices In Highly Regulated Environments
charity
6
4.2k
Compliance & Regulatory Standards Are NOT Incompatible With Modern Development Best Practices
charity
7
6.1k
Perils, Pitfalls and Pratfalls of Platform Engineering (QCon NYC, 2023)
charity
1
390
The Death of DevOps Has Been Greatly Exaggerated, but Platform Engineering Is Here To Stay
charity
2
580
Other Decks in Technology
See All in Technology
いまからでも遅くない!SSL/TLS証明書超入門(It's not too late to start! SSL/TLS Certificates: The Absolute Beginner's Guide)
norimuraz
0
240
オープンソースでどこまでできる?フォーマル検証チャレンジ
msyksphinz
0
140
Vibe Coding Year in Review. From Karpathy to Real-World Agents by Niels Rolland, CEO Paatch
vcoisne
0
140
E2Eテスト設計_自動化のリアル___Playwrightでの実践とMCPの試み__AIによるテスト観点作成_.pdf
findy_eventslides
2
620
Wasmのエコシステムを使った ツール作成方法
askua
0
170
Data Hubグループ 紹介資料
sansan33
PRO
0
2.2k
フレームワークを意識させないワークショップづくり
keigosuda
0
180
防災デジタル分野での官民共創の取り組み (2)DIT/CCとD-CERTについて
ditccsugii
0
290
【Kaigi on Rails 事後勉強会LT】MeはどうしてGirlsに? 私とRubyを繋いだRail(s)
joyfrommasara
0
260
能登半島地震において デジタルができたこと・できなかったこと
ditccsugii
0
200
リセラー企業のテクサポ担当が考える、生成 AI 時代のトラブルシュート 2025
kazzpapa3
1
330
20251010_HCCJP_AdaptiveCloudUpdates
sdosamut
0
120
Featured
See All Featured
Docker and Python
trallard
46
3.6k
The Cost Of JavaScript in 2023
addyosmani
55
9k
Learning to Love Humans: Emotional Interface Design
aarron
274
41k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Java REST API Framework Comparison - PWX 2021
mraible
34
8.9k
GitHub's CSS Performance
jonrohan
1032
470k
Done Done
chrislema
185
16k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
9
590
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
115
20k
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
Transcript
Engineering Large Systems When You’re Not Google Or Facebook Some
Advice By Charity Majors
None
I blame this guy: Testing in production has gotten a
bad rap.
None
how they think we are how we really are
but *why*?
monitoring => observability known unknowns => unknown unknowns LAMP stack
=> distributed systems
“Complexity is increasing” - Science
Many catastrophic states exist at any given time. Your system
is never entirely ‘up’
We are all distributed systems engineers now the unknowns outstrip
the knowns why does this matter more and more?
Distributed systems are particularly hostile to being cloned or imitated
(or monitored). (clients, concurrency, chaotic traffic patterns, edge cases …)
Distributed systems have an infinitely long list of almost-impossible failure
scenarios that make staging environments particularly worthless. this is a black hole for engineering time
unit tests integration tests functional tests basic failover test before
prod: … the basics. the simple stuff. known-unknowns
behavioral tests experiments load tests (!!) edge cases canaries rolling
deploys multi-region test in prod: unknown-unknowns
test in staging? meh
unit tests integration tests functional tests “What happens when …”
(you know the answer) “What happens when …” (you don’t) behavioral tests experiments load tests (!!) edge cases canaries rolling deploys multi-region test before prod: test in prod:
Only production is production. You can ONLY verify the deploy
for any env by deploying to that env
1. Every deploy is a *unique* exercise of your process+
code+system 2. Deploy scripts are production code. If you’re using fabric or capistrano, this means you have fab/cap in production.
Staging is not production.
Why do people sink so much time into staging, when
they can’t even tell if their own production environment is healthy or not?
That energy is better used elsewhere: Production. You can catch
80% of the bugs with 20% of the effort. And you should. @caitie’s PWL talk: https://youtu.be/-3tw2MYYT0Q
feature flags (launch darkly) high cardinality tooling (honeycomb) canary canary
canaries, shadow systems (goturbine, linkerd) capture/replay for databases (apiary, percona) also build or use: plz dont build your own ffs
Failure is not rare Practice shipping and fixing lots of
small problems And practice on your users!!
Failure: it’s “when”, not “if” (lots and lots and lots
of “when’s”)
Does everyone … know what normal looks like? know how
to deploy? know how to roll back? know how to canary? know how to debug in production? Practice!!~
None
None
None
• Charity Majors @mipsytipsy