Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Walking Dead - A Survival Guide to Resilien...
Search
Michael Nitschinger
May 12, 2015
Programming
0
210
The Walking Dead - A Survival Guide to Resilient Reactive Applications
I gave this talk at GeeCon 2015 in Krakow. Recording will be available through the GeeCon channels.
Michael Nitschinger
May 12, 2015
Tweet
Share
More Decks by Michael Nitschinger
See All by Michael Nitschinger
High Performance JVM Networking with Netty
daschl
5
1.2k
Reactive Data Access with RxJava... and N1QL!
daschl
0
180
Spark with Couchbase
daschl
0
150
Reactive Data Access with RxJava ... and N1QL!
daschl
0
170
State of the Art JVM Networking with Netty
daschl
2
440
The Walking Dead - A Survival Guide to Resilient Reactive Applications
daschl
0
360
The Walking Dead - A Survival Guide to Resilient Reactive Applications
daschl
1
430
The Walking Dead - A Survival Guide to Resilient Applications
daschl
0
1.3k
Building a Reactive Database Driver on the JVM
daschl
2
950
Other Decks in Programming
See All in Programming
overlayPreferenceValue で実現する ピュア SwiftUI な AdMob ネイティブ広告
uhucream
0
110
株式会社 Sun terras カンパニーデック
sunterras
0
220
GraphQL×Railsアプリのデータベース負荷分散 - 月間3,000万人利用サービスを無停止で
koxya
1
1.1k
Go Conference 2025: Goで体感するMultipath TCP ― Go 1.24 時代の MPTCP Listener を理解する
takehaya
7
1.6k
Go言語の特性を活かした公式MCP SDKの設計
hond0413
1
170
NetworkXとGNNで学ぶグラフデータ分析入門〜複雑な関係性を解き明かすPythonの力〜
mhrtech
3
990
明日から始めるリファクタリング
ryounasso
0
110
LLMとPlaywright/reg-suitを活用した jQueryリファクタリングの実際
kinocoboy2
4
660
高度なUI/UXこそHotwireで作ろう Kaigi on Rails 2025
naofumi
4
3.4k
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
3
370
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
4
1.3k
Web技術を最大限活用してRAW画像を現像する / Developing RAW Images on the Web
ssssota
2
1.2k
Featured
See All Featured
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.2k
The Cult of Friendly URLs
andyhume
79
6.6k
The Pragmatic Product Professional
lauravandoore
36
6.9k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.6k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
How GitHub (no longer) Works
holman
315
140k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
850
Building Adaptive Systems
keathley
43
2.8k
How to train your dragon (web standard)
notwaldorf
96
6.3k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Designing for humans not robots
tammielis
254
25k
Transcript
The Walking Dead A Survival Guide to Resilient Reactive Applications
Michael Nitschinger @daschl
the right Mindset 2
– U.S. Marine Corps “The more you sweat in peace,
the less you bleed in war.” 3
4
5
Not so fast, mister fancy tests! 6
What can go wrong? Always ask yourself 7
Fault Tolerance 101 8
Fault Error Failure A fault is a latent defect that
can cause an error when activated. 9
Fault Error Failure Errors are the manifestations of faults. 10
Fault Error Failure Failure occurs when the service no longer
complies with its specifications. 11
Fault Error Failure Errors are inevitable. We need to detect,
recover and mitigate them before they become failures. 12
Reliability is the probability that a system will perform failure
free for a given amount of time. MTTF Mean Time To Failure MTTR Mean Time To Repair 13
Availability is the percentage of time the system is able
to perform its function. availability = MTTF MTTF + MTTR 14
Expression Downtime/Year Three 9s 99.9% 525.6 min Four 9s 99.99%
52.56 min Four 9s and a 5 99.995% 26.28 min Five 9s 99.999% 5.256 min Six 9s 99.9999% 0.5256 min 100% 0 15
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability ??? ??? ??? 16
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability 99.99% 17 99.99% 99.99%
Pop Quiz! Edge Service User Service Session Store Data Warehouse
Wanted: 99.99% Availability ~99.999% ~99.999% ~99.999% 18
Fault Tolerant Architecture 19
Units of Mitigation are the basic units of error containment
and recovery. 20
Escalation is used when recovery or mitigation is not possible
inside the unit. 21
Escalation 22 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 23 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 24 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Escalation 25 Cluster Node Node Service Service Service Service Service
Endpoint Endpoint Endpoint Endpoint Endpoint
Redundancy Cost Active/Active Active/Standby N+M Active/Passive Cost Time To Recover
26
The Fault Observer receives system and error events and can
guide and orchestrate detection and recovery Unit Unit Observer Listener Listener Unit Unit 27
28
29
Detecting Errors 30
A silent system is a dead system. 31
A System Monitor helps to study behaviour and to make
sure it is operating as specified. http://upload.wikimedia.org/wikipedia/commons/3/3b/Mission_control_center.jpg 32
https://github.com/Netflix/Turbine 33
Periodic Checking Heartbeats monitor tasks or remote services and initiate
recovery Routine Exercises prevent idle unit starvation and surface malfunctions 34
35 Encoder( Encoder( Ne*y( Writes( Ne*y( Reads( Decoder( Decoder( Event
on Idle No Traffic Endpoint
Riding over Transients is used to defer error recovery if
the error is temporary. “‘Patience is a virtue’ to allow the true signature of an error to show itself.” - Robert S. Hanmer 36
37
And more! • Complete Parameter Checking • Watchdogs • Voting
• Checksums • Routine Audits 38
Recovery and Mitigation of Errors 39
Timeout to not wait forever and keep holding up the
resource. 40 X
Failover to a redundant unit when the error has been
detected and isolated. Cost Active/Active Active/Standby N+M Cost Time To Recover Redundancy Reminder 41
Intelligent Retries Time between Retries Number of Attempts Fixed Linear
Exponential 42
Restart can be used as a last resort with the
trade-off to lose state and time. 43
Fail Fast to shed load and give a partial great
service than a complete bad one. Boundary 44
Backpressure & Batching! 45
Case Study: Hystrix https://raw.githubusercontent.com/wiki/Netflix/Hystrix/images/hystrix-flow-chart-original.png 46
And more! • Rollback • Roll-Forward • Checkpoints • Data
Reset Recovery Mitigation • Bounded Queuing • Expansive Controls • Marking Data • Error Correcting Codes 47
And more! • Rollback • Roll-Forward • Checkpoints • Data
Reset Recovery Mitigation • Bounded Queuing • Expansive Controls • Marking Data • Error Correcting Codes 48
Recommended Reading 49
Patterns for Fault-Tolerant Software by Robert S. Hanmer 50
Release It! by Michael T. Nygard 51
Any Questions? 52
twitter @daschl email
[email protected]
Thank you! 53