Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critica...
Search
Caitie McCaffrey
June 14, 2016
Technology
8
470
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
350
Argus Papers We Love
caitiem20
14
1.2k
The Verification of a Distributed System
caitiem20
22
2.3k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
830
The Verification of a Distributed System
caitiem20
12
800
The Verification of a Distributed System
caitiem20
6
800
A Brief History of Distributed Programming: RPC
caitiem20
31
6.7k
Building Scalable Stateful Services
caitiem20
12
1.7k
Other Decks in Technology
See All in Technology
SoccerNet GSRの紹介と技術応用:選手視点映像を提供するサッカー作戦盤ツール
mixi_engineers
PRO
1
190
自動テストのコストと向き合ってみた
qa
0
200
GA technologiesでのAI-Readyの取り組み@DataOps Night
yuto16
0
280
小学4年生夏休みの自由研究「ぼくと Copilot エージェント」
taichinakamura
0
500
多様な事業ドメインのクリエイターへ 価値を届けるための営みについて
massyuu
1
400
Large Vision Language Modelを用いた 文書画像データ化作業自動化の検証、運用 / shibuya_AI
sansan_randd
0
110
How to achieve interoperable digital identity across Asian countries
fujie
0
120
Exadata Database Service on Dedicated Infrastructure(ExaDB-D) UI スクリーン・キャプチャ集
oracle4engineer
PRO
2
5.5k
JAZUG 15周年記念 × JAT「AI Agent開発者必見:"今"のOracle技術で拡張するAzure × OCIの共存アーキテクチャ」
shisyu_gaku
0
130
空間を設計する力を考える / 20251004 Naoki Takahashi
shift_evolve
PRO
3
400
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
社内お問い合わせBotの仕組みと学び
nish01
0
460
Featured
See All Featured
How to train your dragon (web standard)
notwaldorf
96
6.3k
The Art of Programming - Codeland 2020
erikaheidi
56
14k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6.1k
Mobile First: as difficult as doing things right
swwweet
224
10k
Build your cross-platform service in a week with App Engine
jlugia
232
18k
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.7k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
32
2.2k
Designing Experiences People Love
moore
142
24k
Context Engineering - Making Every Token Count
addyosmani
5
200
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie