Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Servers are doomed to fail
Search
JBD
May 17, 2019
Technology
3
1.5k
Servers are doomed to fail
JBD
May 17, 2019
Tweet
Share
More Decks by JBD
See All by JBD
eBPF in Microservices Observability at eBPF Day
rakyll
1
2k
eBPF in Microservices Observability
rakyll
1
1.6k
OpenTelemetry at AWS
rakyll
1
1.8k
Debugging Code Generation in Go
rakyll
5
1.5k
Are you ready for production?
rakyll
8
2.6k
Serverless Containers
rakyll
1
240
Critical Path Analysis
rakyll
0
500
Monitoring and Debugging Containers
rakyll
2
1.1k
CPDD
rakyll
0
4.1k
Other Decks in Technology
See All in Technology
オープニング / MIERUNE JCT - Tokyo 2024
mierune
PRO
1
340
C# 13 / .NET 9 の新機能 (RC 1 時点)
nenonaninu
1
1.1k
BLEA v3.0.0の新しいベストプラクティスを取り入れた効率的なAWS CDK開発/jawsug_cdk16
mhrtech
3
340
【ログ分析勉強会#5】Elasticsearch/Kibana のパイプ型クエリー言語ES|QLの紹介 ~ Pandasと一緒にログ分析してみた
nobuhikosekiya
0
160
Oracle GoldenGate 23ai 導入Tips
oracle4engineer
PRO
1
200
小さな勉強会の始め方、広げ方、あるいは友達の作り方 / How to Start, Grow, and Build Connections with Small Study Groups
ar_tama
1
270
【shownet.conf_】ShowNet x 宇宙ネットワーク
shownet
PRO
0
290
いまからでも遅くない! コンテナでWebアプリケーションを 動かしてみよう(2-1)WebAPI座学
nomu
0
140
トークナイザー入門
payanotty
2
270
Rubyはなぜ「たのしい」のか? / Why is Ruby a programmers' best friend? #tqrk15
expajp
4
1.7k
Dual level of task scheduling for VM workloads
ennael
PRO
0
180
VS CodeでF1〜12キーつかってますか? / Do you use the F1-12 keys in VS Code?
74th
1
250
Featured
See All Featured
Testing 201, or: Great Expectations
jmmastey
38
7k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.1k
Designing with Data
zakiwarfel
98
5.1k
Happy Clients
brianwarren
97
6.6k
The MySQL Ecosystem @ GitHub 2015
samlambert
250
12k
How to Think Like a Performance Engineer
csswizardry
16
1k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
279
13k
A Philosophy of Restraint
colly
202
16k
Bootstrapping a Software Product
garrettdimon
PRO
304
110k
Typedesign – Prime Four
hannesfritz
39
2.3k
The Mythical Team-Month
searls
218
43k
VelocityConf: Rendering Performance Case Studies
addyosmani
324
23k
Transcript
Servers are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Serverless is also doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Systems are doomed to fail Jaana B. Dogan
[email protected]
@rakyll
Is failure OK? Is failure an unexpected case?
Failure is not an exception. Systems change all the time.
“I haven’t touched the code for a century, it should
just work.” Said no one ever.
Failure is expected. Yes, it is.
None
@rakyll monitoring debugging postmortem
Monitoring is about saying if something is broken.
“99.99% of the requests should return in 100ms.”
@rakyll
@rakyll
Debugging
Debugging is collaborative.
Debugging comes in flavors. Logs Traces Metrics ...
Postmortems
Postmortems
Postmortems
Blameless? Focus on identifying problems.
Collaboration Design for collaboration.
Design for failure Set SLOs, plan for instrumentation, plan for
debugging.
Cross-stack debugging Accountability across stack with high cardinality data. speakerdeck.com/rakyll/rpc-metrics-at-google
Correlation Jump from monitoring/debugging data to data.
On-call debugging Jump from distributed tracing data to on-call information.
who to page?
Dynamic collection Capability to enable more collection in production when
needed.
Continuous collection Continuously collect signals, generate fleet-wide analysis reports.
Introspection Introspection pages provided from the services.
@rakyll monitoring debugging postmortem
Thank you Jaana B. Dogan Google
[email protected]