Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Engineering Large Systems When You're Not Googl...
Search
Charity Majors
April 30, 2018
Technology
20
5.6k
Engineering Large Systems When You're Not Google Or Facebook (test in prod)
lightning talk at Clever, 4/30/18
Charity Majors
April 30, 2018
Tweet
Share
More Decks by Charity Majors
See All by Charity Majors
In Praise of "Normal" Engineers (LDX3)
charity
4
2.4k
In Praise of "Normal" Engineers (with full speaker notes)
charity
1
180
AIOps: Prove It! (An Open Letter to Vendors Selling AI for SREs)
charity
1
21
SRECon 2024 Keynote: Is It Already Time To Version Observability? (Signs Point To Yes)
charity
3
470
CTO Craft Con Keynote: Observability is due for a version change: are you ready for it?
charity
4
1.3k
Case Studies: Modern Development Practices In Highly Regulated Environments
charity
6
4.1k
Compliance & Regulatory Standards Are NOT Incompatible With Modern Development Best Practices
charity
7
6k
Perils, Pitfalls and Pratfalls of Platform Engineering (QCon NYC, 2023)
charity
1
380
The Death of DevOps Has Been Greatly Exaggerated, but Platform Engineering Is Here To Stay
charity
2
560
Other Decks in Technology
See All in Technology
Datasets for Critical Operations by Dataform
kimujun
0
120
KCD Lima: eBee in Peru!
lizrice
0
110
生成AIを活用した野球データ分析 - メジャーリーグ編 / Baseball Analytics for Gen AI
shinyorke
PRO
1
240
Kiro Hookを Terraformで検証
ao_inoue
0
140
生成AIによる情報システムへのインパクト
taka_aki
1
200
M365アカウント侵害時の初動対応
lhazy
7
5.1k
MCPと認可まわりの話 / mcp_and_authorization
convto
2
300
少人数でも回る! DevinとPlaybookで支える運用改善
ishikawa_pro
4
1.8k
robocopy の怖い話/scary-story-about-robocopy
emiki
0
410
Microsoft Learn MCP/Fabric データエージェント/Fabric MCP/Copilot Studio-簡単・便利なAIエージェント作ってみた -"Building Simple and Powerful AI Agents with Microsoft Learn MCP, Fabric Data Agent, Fabric MCP, and Copilot Studio"-
reireireijinjin6
1
160
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
2
260
今日からあなたもGeminiを好きになる
subaruhello
1
660
Featured
See All Featured
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.3k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
5.9k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.9k
How STYLIGHT went responsive
nonsquared
100
5.7k
Speed Design
sergeychernyshev
32
1k
It's Worth the Effort
3n
185
28k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
The World Runs on Bad Software
bkeepers
PRO
70
11k
We Have a Design System, Now What?
morganepeng
53
7.7k
Balancing Empowerment & Direction
lara
1
510
Transcript
Engineering Large Systems When You’re Not Google Or Facebook Some
Advice By Charity Majors
None
I blame this guy: Testing in production has gotten a
bad rap.
None
how they think we are how we really are
but *why*?
monitoring => observability known unknowns => unknown unknowns LAMP stack
=> distributed systems
“Complexity is increasing” - Science
Many catastrophic states exist at any given time. Your system
is never entirely ‘up’
We are all distributed systems engineers now the unknowns outstrip
the knowns why does this matter more and more?
Distributed systems are particularly hostile to being cloned or imitated
(or monitored). (clients, concurrency, chaotic traffic patterns, edge cases …)
Distributed systems have an infinitely long list of almost-impossible failure
scenarios that make staging environments particularly worthless. this is a black hole for engineering time
unit tests integration tests functional tests basic failover test before
prod: … the basics. the simple stuff. known-unknowns
behavioral tests experiments load tests (!!) edge cases canaries rolling
deploys multi-region test in prod: unknown-unknowns
test in staging? meh
unit tests integration tests functional tests “What happens when …”
(you know the answer) “What happens when …” (you don’t) behavioral tests experiments load tests (!!) edge cases canaries rolling deploys multi-region test before prod: test in prod:
Only production is production. You can ONLY verify the deploy
for any env by deploying to that env
1. Every deploy is a *unique* exercise of your process+
code+system 2. Deploy scripts are production code. If you’re using fabric or capistrano, this means you have fab/cap in production.
Staging is not production.
Why do people sink so much time into staging, when
they can’t even tell if their own production environment is healthy or not?
That energy is better used elsewhere: Production. You can catch
80% of the bugs with 20% of the effort. And you should. @caitie’s PWL talk: https://youtu.be/-3tw2MYYT0Q
feature flags (launch darkly) high cardinality tooling (honeycomb) canary canary
canaries, shadow systems (goturbine, linkerd) capture/replay for databases (apiary, percona) also build or use: plz dont build your own ffs
Failure is not rare Practice shipping and fixing lots of
small problems And practice on your users!!
Failure: it’s “when”, not “if” (lots and lots and lots
of “when’s”)
Does everyone … know what normal looks like? know how
to deploy? know how to roll back? know how to canary? know how to debug in production? Practice!!~
None
None
None
• Charity Majors @mipsytipsy