Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Design for Retry (Nodevember)
Search
Aria Stewart
November 15, 2014
Programming
63
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Design for Retry (Nodevember)
Aria Stewart
November 15, 2014
Transcript
44 slides
None
Hi! I'm Aria Stewart, that's @aredridel on just about every
service out there. Right now I'm an engineer at PayPal, working on the open source Kraken.js framework.
I'm going to talk about errors. It's going to be
okay.
We all know HTTP
if (err) { alert(err.message); } else { doMyThing(); }
2xx OK 3xx Go elsewhere 4xx Tell user what they
did wrong 5xx Bail out and log an error I'd call this Error avoidance
You can't avoid errors
Here's the secret Handle errors instead
4xx Tell the user what they did wrong 5xx Save
that request and do something with it later.
Retry it 5xx are errors the requestor can handle
But you can't just do things twice? We must make
operations idempotent
Idempotency Repeated actions have no effect, give the same result
This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
Causes! • database down • bug in a service •
Deploy in progress • power failure • kicked a cable • Network congestion • Capacity exceeded • Microbursts
• Tree fell on the data center • earthquake •
tornado • birds, snakes and aeroplanes • Black Friday • Slashdot effect • Interns • QA tests • DoS attack
You need a queue
Lots of ways to do it Database on the nodes
Log file Queue server
gearman Queues built in There are many alternatives, but gearmand
is very simple. The memcache of job queues.
Three statuses: • OK (Like 200) • FAIL (Like 400)
• ERROR (Like 500)
design so ERROR can be retried.
gearmand automatically tries a job ERROR again. And again. And
again.
If it isn't sure it worked? Tries it again.
You cannot know if an error is a failure.
Error handling gets simpler • Exception? ERROR. • Database down?
ERROR. • Downstream service timeout? ERROR. Maybe you retry right away.
How many of you have used a job queue?
You have used a job queue
Let me tell you about one TRILLIONS of messages MILLIONS
of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
EMAIL. 250 OK 4xx RETRY 5xx Fail
Responsibility for messages 250 - accept responsibility 4xx - reject
responsibility 5xx - return responsibility
reject responsibility. If there's an error? Fail fast. The requester
can retry.
Fail fast. Queue work you can't reject. Reject everything you
can if there is an error.
You need a smart client. Keeps outstanding requests. Resubmit. Try
a different server! Try a second queue service. Maybe have a fallback plan.
Smart Clients on the device Toto, we're not in AWS
anymore.
Ever lose an email because you've been logged out?
Latency + Mutable state = Distributed system CAP Theorem Applies!
C = Consistency If there's state that one part knows
of that another doesn't? That's inconsistency.
Job queues are controlled inconsistency.
Ever try to write email on the web while not
on the Internet? It's cloud easy!
This is really good for offline- first design! Being offline
is the ultimate retriable error.
Some ideas
Queue things in localStorage
Use third-party storage
Integrate third-party services with this approach.
Use different strategies for available resources vs contended
Thank you! I hope you have lots of ideas queued
up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!
More Decks by Aria Stewart
See All by Aria Stewart
Nuts and Bolts of Internationalization
aredridel
0
250
Design for Retry (Oneshot Budapest)
aredridel
0
76
Other Decks in Programming
See All in Programming
protojson における文字化けの扱い
kazuki1023
0
460
AIが生成したコードは読むべきか?抽象化設計とコードの品質保証 / abstraction-design-and-quality
minodriven
17
9.5k
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
3
210
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
2
400
baseline으로 마이그레이션, 리팩토링 안전하게 맡기기
fornewid
1
1k
Augmenting AI with the Power of Jakarta EE
ivargrimstad
0
400
Crash Auto Pipelines
veronikapj
1
440
Proxmox x Talos Linux x Terraform で実現するなんかいい感じの Kubernetes クラスタ / A Sleek Kubernetes Cluster Powered by Proxmox, Talos Linux, and Terraform
nnaka2992
0
300
地方こそサーバーレス — 10年続けた、顔が見えるチームで確かな基盤を / Serverless Matters Most in Regional Areas — 10 Years of Building Reliable Foundations with Close-Knit Teams
seike460
PRO
2
500
祈るテストから、探索するテストへ 〜Lincheckではじめる並行処理テスト〜
yyh_gl
0
420
テストだけで見つかる? Rubyでつくるメソッド発掘器
izawa
0
160
Web3Dライブラリを作ってVue Custom Rendererの仕組みを理解しよう / lets learn vue custom renderer by making 3d library
drumath2237
1
140
Featured
See All Featured
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
katarinadahlin
PRO
3
3.9k
Testing 201, or: Great Expectations
jmmastey
46
8.4k
Visualization
eitanlees
153
17k
Making the Leap to Tech Lead
cromwellryan
135
10k
Beyond borders and beyond the search box: How to win the global "messy middle" with AI-driven SEO
davidcarrasco
3
310
A Modern Web Designer's Workflow
chriscoyier
699
190k
Designing for Timeless Needs
cassininazir
1
570
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
254
23k
Digital Projects Gone Horribly Wrong (And the UX Pros Who Still Save the Day) - Dean Schuster
uxyall
2
4k
Documentation Writing (for coders)
carmenintech
77
5.6k
We Have a Design System, Now What?
morganepeng
55
8.4k
Building Flexible Design Systems
yeseniaperezcruz
330
41k