Design for Retry (Nodevember)

Transcript

2. Hi! I'm Aria Stewart, that's @aredridel on just about every service out there. Right now I'm an engineer at PayPal, working on the open source Kraken.js framework.
3. I'm going to talk about errors. It's going to be okay.
4. We all know HTTP
5. if (err) { alert(err.message); } else { doMyThing(); }
6. 2xx OK 3xx Go elsewhere 4xx Tell user what they did wrong 5xx Bail out and log an error I'd call this Error avoidance
7. You can't avoid errors
8. Here's the secret Handle errors instead
9. 4xx Tell the user what they did wrong 5xx Save that request and do something with it later.
10. Retry it 5xx are errors the requestor can handle
11. But you can't just do things twice? We must make operations idempotent
12. Idempotency Repeated actions have no effect, give the same result This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
13. Causes! • database down • bug in a service • Deploy in progress • power failure • kicked a cable • Network congestion • Capacity exceeded • Microbursts
14. • Tree fell on the data center • earthquake • tornado • birds, snakes and aeroplanes • Black Friday • Slashdot effect • Interns • QA tests • DoS attack
15. You need a queue
16. Lots of ways to do it Database on the nodes Log file Queue server
17. gearman Queues built in There are many alternatives, but gearmand is very simple. The memcache of job queues.
18. Three statuses: • OK (Like 200) • FAIL (Like 400) • ERROR (Like 500)
19. design so ERROR can be retried.
20. gearmand automatically tries a job ERROR again. And again. And again.
21. If it isn't sure it worked? Tries it again.
22. You cannot know if an error is a failure.
23. Error handling gets simpler • Exception? ERROR. • Database down? ERROR. • Downstream service timeout? ERROR. Maybe you retry right away.
24. How many of you have used a job queue?
25. You have used a job queue
26. Let me tell you about one TRILLIONS of messages MILLIONS of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
27. EMAIL. 250 OK 4xx RETRY 5xx Fail
28. Responsibility for messages 250 - accept responsibility 4xx - reject responsibility 5xx - return responsibility
29. reject responsibility. If there's an error? Fail fast. The requester can retry.
30. Fail fast. Queue work you can't reject. Reject everything you can if there is an error.
31. You need a smart client. Keeps outstanding requests. Resubmit. Try a different server! Try a second queue service. Maybe have a fallback plan.
32. Smart Clients on the device Toto, we're not in AWS anymore.
33. Ever lose an email because you've been logged out?
34. Latency + Mutable state = Distributed system CAP Theorem Applies!
35. C = Consistency If there's state that one part knows of that another doesn't? That's inconsistency.
36. Job queues are controlled inconsistency.
37. Ever try to write email on the web while not on the Internet? It's cloud easy!
38. This is really good for offline- first design! Being offline is the ultimate retriable error.
39. Some ideas
40. Queue things in localStorage
41. Use third-party storage
42. Integrate third-party services with this approach.
43. Use different strategies for available resources vs contended
44. Thank you! I hope you have lots of ideas queued up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!