Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
PWL NY: Simple Testing Can Prevent Most Critica...
Search
Caitie McCaffrey
June 14, 2016
Technology
8
470
PWL NY: Simple Testing Can Prevent Most Critical Failures
Caitie McCaffrey
June 14, 2016
Tweet
Share
More Decks by Caitie McCaffrey
See All by Caitie McCaffrey
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
The Path Towards Simplifying Consistency in Distributed Systems
caitiem20
1
360
Argus Papers We Love
caitiem20
14
1.2k
The Verification of a Distributed System
caitiem20
22
2.3k
We Hear You Like Papers: Eventual Consistency
caitiem20
14
840
The Verification of a Distributed System
caitiem20
12
800
The Verification of a Distributed System
caitiem20
6
800
A Brief History of Distributed Programming: RPC
caitiem20
31
6.7k
Building Scalable Stateful Services
caitiem20
12
1.7k
Other Decks in Technology
See All in Technology
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing Hiroshima 2025 Edition
tomzoh
0
140
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
20251010_HCCJP_AdaptiveCloudUpdates
sdosamut
0
140
RDS の負荷が高い場合に AWS で取りうる具体策 N 連発/a-series-of-specific-countermeasures-available-on-aws-when-rds-is-under-high-load
emiki
5
3.7k
Dylib Hijacking on macOS: Dead or Alive?
patrickwardle
0
220
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
lehupa
0
170
AI Agent Dojo #2 watsonx Orchestrateフローの作成
oniak3ibm
PRO
0
130
AI時代こそ求められる設計力- AWSクラウドデザインパターン3選で信頼性と拡張性を高める-
kenichirokimura
3
340
能登半島地震において デジタルができたこと・できなかったこと
ditccsugii
0
250
セキュアな認可付きリモートMCPサーバーをAWSマネージドサービスでつくろう! / Let's build an OAuth protected remote MCP server based on AWS managed services
kaminashi
3
350
20251007: What happens when multi-agent systems become larger? (CyberAgent, Inc)
ornew
1
310
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
12
80k
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
4k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
230
22k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.5k
Speed Design
sergeychernyshev
32
1.2k
KATA
mclloyd
PRO
32
15k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
37
2.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
The Language of Interfaces
destraynor
162
25k
Faster Mobile Websites
deanohume
310
31k
Context Engineering - Making Every Token Count
addyosmani
7
260
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.5k
A Tale of Four Properties
chriscoyier
161
23k
Transcript
Simple Testing Can Prevent Most Critical Failures: An Analysis of
Production Failures in Distributed Data-Intensive Systems Papers We Love New York - June 2016
Caitie McCaffrey @caitie Distributed Systems Engineer CaitieM.com
None
None
Analyzed Failures in Real World Systems
“A majority (77%) of failures require more than one input
event to manifest, but most of the failures (90%) require no more than 3” Complexity of Failures
“The specific order of events is important in 88% of
the failures that require multiple events Complexity of Failures
“3 Nodes or less can reproduce 98% of Failures” Complexity
of Failures
Unit Tests “A majority of production failures (77%) can be
reproduced by a unit test”
Top Down Fault Injection & State Space Exploration is Expensive
Logging • 76% of the failures print explicit failure- related
error messages • For 84% of the failures, all of the triggering events are logged • Logs are noisy: each failure prints 824 log messages (median)
Catastrophic Failures
Error Handling • 92% of failures were the result of
incorrect handling of non-fatal errors • 58% of faults could have been detected via simple testing • 35% of failures caused by bad practices in error handling code
• Error Handling Code is simply empty or only contains
a Log statement • Error Handler aborts cluster on an overly general exception • Error Handler contains comments like FIXME or TODO Bad Practices
Aspirator Performs static analysis of Java bytecode to detect: •
error handler is empty • error handler over-catches exceptions and aborts • error handler contains phrases like “TODO” or “FIXME”
• 500 New Bugs & Bad Practices • 115 Fasle
Positives • 171 bugs reported • 143 bugs confirmed or fixed Aspirator Results
-developer “I fail to see the reason to handle every
exception” Developer Reactions
“It is often much harder to reason about the correctness
of a system’s abnormal path than its normal execution path ”
Moving Forward • Use a tool like Aspirator that is
capable of identifying trivial bugs • Enforce code reviews of error handling code • High code coverage on error handling code
Questions @caitie