Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critical Failures
Search
Caitie McCaffrey
June 14, 2016
Technology
8
400
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
323
20k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
210
Argus Papers We Love
caitiem20
13
1.1k
The Verification of a Distributed System
caitiem20
22
2.1k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
710
The Verification of a Distributed System
caitiem20
12
670
The Verification of a Distributed System
caitiem20
6
680
A Brief History of Distributed Programming: RPC
caitiem20
31
6.3k
Building Scalable Stateful Services
caitiem20
12
1.3k
Other Decks in Technology
See All in Technology
個人のAWSアカウントをマルチ運用してみた
miura55
2
270
リテール金融(キャッシュレス・ネット銀行・ネット証券)の競争環境と経済圏
8maki
0
1.9k
自らを知り外と繋がる、日経のエンジニア採用とDevRel活動/devreljp92
nishiuma
2
200
AWSやJAWS-UGとの出会いを振り返る
yoyoyopg
1
190
How to do well in consulting–Balkan Ruby 2024
irinanazarova
0
180
実例で紹介するRAG導入時の知見と精度向上の勘所
yamahiro
7
2.5k
Babylon.js JAPAN活動紹介 (2024/4)
limes2018
1
130
生成AIの変革の時代に、直近1年で直面した課題とその解決策
ktc_wada
1
800
中年男性がメインフレームから クラウドへキャリアシフトしてみた
uechishingo
1
430
TiDBにおけるテーブル設計と最適化の事例
cygames
0
460
データ基盤を支える技術
chanyou0311
4
2.5k
今日からできる!簡単 .NET 高速化 Tips -2024 edition-
xin9le
8
4.9k
Featured
See All Featured
Mobile First: as difficult as doing things right
swwweet
217
8.6k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
501
140k
The Invisible Side of Design
smashingmag
294
49k
Practical Orchestrator
shlominoach
183
9.7k
Designing for humans not robots
tammielis
247
25k
Building Your Own Lightsaber
phodgson
100
5.7k
Put a Button on it: Removing Barriers to Going Fast.
kastner
58
3.1k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
123
39k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
242
1.2M
Debugging Ruby Performance
tmm1
70
11k
A designer walks into a library…
pauljervisheath
201
23k
Product Roadmaps are Hard
iamctodd
45
9.8k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie