Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Design for Retry (Nodevember)
Search
Aria Stewart
November 15, 2014
Programming
0
50
Design for Retry (Nodevember)
Aria Stewart
November 15, 2014
Tweet
Share
More Decks by Aria Stewart
See All by Aria Stewart
Nuts and Bolts of Internationalization
aredridel
0
160
Design for Retry (Oneshot Budapest)
aredridel
0
56
Other Decks in Programming
See All in Programming
PHPの次期バージョンはこの時期どうなっているのか - Internalsの開発体制について - PHPカンファレンス小田原
youkidearitai
PRO
1
180
PHP8.3の機能を振り返る / Review of PHP 8.3 features
seike460
PRO
1
110
Folding Cheat Sheet #2
philipschwarz
PRO
0
110
Tailwind CSSを本気でカスタマイズする方法
fsubal
2
260
単体テストを書かない技術 #phpcon_odawara
o0h
PRO
25
7.8k
StreamlitとTerraformでデータカタログを作った話
gussan0223
0
300
SwiftUI Performance 不要なViewの再描画と更新を抑える
bigamitiongit
1
160
二郎系ラーメンのコールで学ぶ AST 解析
memory1994
PRO
7
1.7k
pixivアプリでマルチモジュールを実現するまで
gatosyocora
1
130
Build with AI 2024 Seoul - 제로부터 시작하는 Flutter with Gemini 생활 - 박제창
itsmedreamwalker
0
200
AWS Application Composerで始める、 サーバーレスなデータ基盤構築 / 20240406-jawsug-hokuriku-shinkansen
kasacchiful
1
250
educure_カリキュラム生操作マニュアル.pdf
linew_official
0
470
Featured
See All Featured
Designing the Hi-DPI Web
ddemaree
276
33k
YesSQL, Process and Tooling at Scale
rocio
163
13k
Scaling GitHub
holman
457
140k
Design by the Numbers
sachag
274
18k
The Cult of Friendly URLs
andyhume
74
5.7k
Building an army of robots
kneath
300
41k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
How to train your dragon (web standard)
notwaldorf
72
5.1k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
273
13k
How GitHub (no longer) Works
holman
304
140k
Designing with Data
zakiwarfel
95
4.8k
Transcript
None
Hi! I'm Aria Stewart, that's @aredridel on just about every
service out there. Right now I'm an engineer at PayPal, working on the open source Kraken.js framework.
I'm going to talk about errors. It's going to be
okay.
We all know HTTP
if (err) { alert(err.message); } else { doMyThing(); }
2xx OK 3xx Go elsewhere 4xx Tell user what they
did wrong 5xx Bail out and log an error I'd call this Error avoidance
You can't avoid errors
Here's the secret Handle errors instead
4xx Tell the user what they did wrong 5xx Save
that request and do something with it later.
Retry it 5xx are errors the requestor can handle
But you can't just do things twice? We must make
operations idempotent
Idempotency Repeated actions have no effect, give the same result
This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
Causes! • database down • bug in a service •
Deploy in progress • power failure • kicked a cable • Network congestion • Capacity exceeded • Microbursts
• Tree fell on the data center • earthquake •
tornado • birds, snakes and aeroplanes • Black Friday • Slashdot effect • Interns • QA tests • DoS attack
You need a queue
Lots of ways to do it Database on the nodes
Log file Queue server
gearman Queues built in There are many alternatives, but gearmand
is very simple. The memcache of job queues.
Three statuses: • OK (Like 200) • FAIL (Like 400)
• ERROR (Like 500)
design so ERROR can be retried.
gearmand automatically tries a job ERROR again. And again. And
again.
If it isn't sure it worked? Tries it again.
You cannot know if an error is a failure.
Error handling gets simpler • Exception? ERROR. • Database down?
ERROR. • Downstream service timeout? ERROR. Maybe you retry right away.
How many of you have used a job queue?
You have used a job queue
Let me tell you about one TRILLIONS of messages MILLIONS
of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
EMAIL. 250 OK 4xx RETRY 5xx Fail
Responsibility for messages 250 - accept responsibility 4xx - reject
responsibility 5xx - return responsibility
reject responsibility. If there's an error? Fail fast. The requester
can retry.
Fail fast. Queue work you can't reject. Reject everything you
can if there is an error.
You need a smart client. Keeps outstanding requests. Resubmit. Try
a different server! Try a second queue service. Maybe have a fallback plan.
Smart Clients on the device Toto, we're not in AWS
anymore.
Ever lose an email because you've been logged out?
Latency + Mutable state = Distributed system CAP Theorem Applies!
C = Consistency If there's state that one part knows
of that another doesn't? That's inconsistency.
Job queues are controlled inconsistency.
Ever try to write email on the web while not
on the Internet? It's cloud easy!
This is really good for offline- first design! Being offline
is the ultimate retriable error.
Some ideas
Queue things in localStorage
Use third-party storage
Integrate third-party services with this approach.
Use different strategies for available resources vs contended
Thank you! I hope you have lots of ideas queued
up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!