Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Servers are doomed to fail
Search
JBD
May 17, 2019
Technology
3
1.5k
Servers are doomed to fail
JBD
May 17, 2019
Tweet
Share
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2.1k
eBPF in Microservices Observability
rakyll
1
1.7k
OpenTelemetry at AWS
rakyll
1
1.8k
Debugging Code Generation in Go
rakyll
5
1.5k
Are you ready for production?
rakyll
8
2.7k
Serverless Containers
rakyll
1
240
Critical Path Analysis
rakyll
0
530
Monitoring and Debugging Containers
rakyll
2
1.1k
CPDD
rakyll
0
4.1k
Other Decks in Technology
See All in Technology
AWS re:Invent 2024で発表された コードを書く開発者向け機能について
maruto
0
180
生成AIのガバナンスの全体像と現実解
fnifni
1
180
非機能品質を作り込むための実践アーキテクチャ
knih
3
840
NW-JAWS #14 re:Invent 2024(予選落ち含)で 発表された推しアップデートについて
nagisa53
0
250
CustomCopを使ってMongoidのコーディングルールを整えてみた
jinoketani
0
220
NilAway による静的解析で「10 億ドル」を節約する #kyotogo / Kyoto Go 56th
ytaka23
3
370
AIのコンプラは何故しんどい?
shujisado
1
190
Wvlet: A New Flow-Style Query Language For Functional Data Modeling and Interactive Data Analysis - Trino Summit 2024
xerial
1
110
1等無人航空機操縦士一発試験 合格までの道のり ドローンミートアップ@大阪 2024/12/18
excdinc
0
150
プロダクト開発を加速させるためのQA文化の築き方 / How to build QA culture to accelerate product development
mii3king
1
260
バクラクのドキュメント解析技術と実データにおける課題 / layerx-ccc-winter-2024
shimacos
2
1k
LINE Developersプロダクト(LIFF/LINE Login)におけるフロントエンド開発
lycorptech_jp
PRO
0
120
Featured
See All Featured
Producing Creativity
orderedlist
PRO
341
39k
Thoughts on Productivity
jonyablonski
67
4.4k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
17
2.3k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
6
510
Rebuilding a faster, lazier Slack
samanthasiow
79
8.7k
Rails Girls Zürich Keynote
gr2m
94
13k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
Building an army of robots
kneath
302
44k
Designing for Performance
lara
604
68k
Six Lessons from altMBA
skipperchong
27
3.5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
Designing Experiences People Love
moore
138
23k
Transcript
Servers are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Serverless is also doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Systems are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Is failure OK? Is failure an unexpected case?
Failure is not an exception. Systems change all the time.
“I haven’t touched the code for a century, it should
just work.” Said no one ever.
Failure is expected. Yes, it is.
None
@rakyll monitoring debugging postmortem
Monitoring is about saying if something is broken.
“99.99% of the requests should return in 100ms.”
@rakyll
@rakyll
Debugging
Debugging is collaborative.
Debugging comes in flavors. Logs Traces Metrics ...
Postmortems
Postmortems
Postmortems
Blameless? Focus on identifying problems.
Collaboration Design for collaboration.
Design for failure Set SLOs, plan for instrumentation, plan for
debugging.
Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google
Correlation Jump from monitoring/debugging data to data.
On-call debugging Jump from distributed tracing data to on-call information.
who to page?
Dynamic collection Capability to enable more collection in production when
needed.
Continuous collection Continuously collect signals, generate fleet-wide analysis reports.
Introspection Introspection pages provided from the services.
@rakyll monitoring debugging postmortem
Thank you Jaana B. Dogan Google
[email protected]