Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Design for Retry (Nodevember)

Design for Retry (Nodevember)

Aria Stewart

November 15, 2014

More Decks by Aria Stewart

Other Decks in Programming


  1. Hi! I'm Aria Stewart, that's @aredridel on just about every

    service out there. Right now I'm an engineer at PayPal, working on the open source Kraken.js framework.
  2. 2xx OK 3xx Go elsewhere 4xx Tell user what they

    did wrong 5xx Bail out and log an error I'd call this Error avoidance
  3. 4xx Tell the user what they did wrong 5xx Save

    that request and do something with it later.
  4. Idempotency Repeated actions have no effect, give the same result

    This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
  5. Causes! • database down • bug in a service •

    Deploy in progress • power failure • kicked a cable • Network congestion • Capacity exceeded • Microbursts
  6. • Tree fell on the data center • earthquake •

    tornado • birds, snakes and aeroplanes • Black Friday • Slashdot effect • Interns • QA tests • DoS attack
  7. gearman Queues built in There are many alternatives, but gearmand

    is very simple. The memcache of job queues.
  8. Error handling gets simpler • Exception? ERROR. • Database down?

    ERROR. • Downstream service timeout? ERROR. Maybe you retry right away.
  9. Let me tell you about one TRILLIONS of messages MILLIONS

    of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
  10. You need a smart client. Keeps outstanding requests. Resubmit. Try

    a different server! Try a second queue service. Maybe have a fallback plan.
  11. C = Consistency If there's state that one part knows

    of that another doesn't? That's inconsistency.
  12. Ever try to write email on the web while not

    on the Internet? It's cloud easy!
  13. Thank you! I hope you have lots of ideas queued

    up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!