Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Design for Retry (Oneshot Budapest)
Search
Aria Stewart
November 21, 2014
Programming
0
65
Design for Retry (Oneshot Budapest)
Aria Stewart
November 21, 2014
Tweet
Share
More Decks by Aria Stewart
See All by Aria Stewart
Nuts and Bolts of Internationalization
aredridel
0
190
Design for Retry (Nodevember)
aredridel
0
55
Other Decks in Programming
See All in Programming
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
280
私はどうやって技術力を上げたのか
yusukebe
43
18k
スマホから Youtube Shortsを見られないようにする
lemolatoon
27
31k
PHPに関数型の魂を宿す〜PHP 8.5 で実現する堅牢なコードとは〜 #phpcon_hiroshima / phpcon-hiroshima-2025
shogogg
1
200
Go Conference 2025: Goで体感するMultipath TCP ― Go 1.24 時代の MPTCP Listener を理解する
takehaya
9
1.7k
組込みだけじゃない!TinyGo で始める無料クラウド開発入門
otakakot
0
240
overlayPreferenceValue で実現する ピュア SwiftUI な AdMob ネイティブ広告
uhucream
0
180
Catch Up: Go Style Guide Update
andpad
0
220
明日から始めるリファクタリング
ryounasso
0
140
Writing Better Go: Lessons from 10 Code Reviews
konradreiche
0
1.1k
Pull-Requestの内容を1クリックで動作確認可能にするワークフロー
natmark
2
500
[Kaigi on Rais 2025] 全問正解率3%: RubyKaigiで出題したやりがちな危険コード5選
power3812
0
120
Featured
See All Featured
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Java REST API Framework Comparison - PWX 2021
mraible
33
8.9k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
We Have a Design System, Now What?
morganepeng
53
7.8k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Git: the NoSQL Database
bkeepers
PRO
431
66k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
A designer walks into a library…
pauljervisheath
209
24k
RailsConf 2023
tenderlove
30
1.2k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
37
2.6k
Leading Effective Engineering Teams in the AI Era
addyosmani
2
320
Building Adaptive Systems
keathley
43
2.8k
Transcript
Design for Retry: Microservices, REST, and why Idempotency is the
only way to scale I'm Aria Stewart, that's @aredridel just about everywhere. I'm here thanks to PayPal. I work on the open source Kraken.js framework.
I'm going to talk about errors. It's going to be
okay.
if (err) { alert(err.message); } else { doMyThing(); }
We all know HTTP
2xx OK 3xx Go elsewhere 4xx Tell user what they
did wrong 5xx Bail out and log an error I'd call this Error avoidance
You can't avoid errors
Here's the secret Handle errors instead
4xx Tell the user what they did wrong 5xx Save
that request and do something with it later.
Retry it 5xx are errors the requestor can handle
But you can't just do things twice? We must make
operations idempotent
Idempotency Repeated actions have no effect, give the same result
This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
Causes! —database down —bug in a service —Deploy in progress
—power failure —kicked a cable —Network congestion —Capacity exceeded —Microbursts
—Tree fell on the data center —earthquake —tornado —birds, snakes
and aeroplanes —Black Friday —Slashdot effect —Interns —QA tests —DoS attack
You need a queue
Lots of ways to do it Database on each node.
Maybe LevelDB? Log file Queue server
gearman Queues built in There are many alternatives, but gearmand
is very simple. The memcache of job queues.
Three statuses: —OK (Like 200) —FAIL (Like 400) —ERROR (Like
500)
design so ERROR can be retried.
gearmand automatically tries a job ERROR again. And again. And
again.
If it isn't sure it worked? Tries it again.
You cannot know if an error is a failure.
Error handling gets simpler —Exception? ERROR. —Database down? ERROR. —Downstream
service timeout? ERROR. Maybe you retry right away.
How many of you have used a job queue?
You have used a job queue
Let me tell you about one TRILLIONS of messages MILLIONS
of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
EMAIL. 250 OK 4xx RETRY 5xx Fail
Responsibility for messages 250 - accept responsibility 4xx - reject
responsibility 5xx - return responsibility
reject responsibility. If there's an error? Fail fast. The requester
can retry.
Fail fast. Queue work you can't reject. Reject everything you
can if there is an error.
You need a smart client. Keeps outstanding requests. Resubmit. Try
a different server! Try a second queue service. Maybe have a fallback plan.
Smart Clients on the device Toto, we're not in AWS
anymore.
Ever lose an email because you've been logged out?
Latency + Mutable state = Distributed system CAP Theorem Applies!
C = Consistency If there's state that one part knows
of that another doesn't? That's inconsistency.
Job queues are controlled inconsistency.
Ever try to write email on the web while not
on the Internet? It's cloud easy!
This is really good for offline-first design! Being offline is
the ultimate retriable error.
Some ideas
Use your queue as a place to measure for system
sizing
Queue things in localStorage
Use third-party storage
Integrate third-party services with this approach.
Use different strategies for available resources vs contended
Thank you! I hope you have lots of ideas queued
up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!