Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Design for Retry (Oneshot Budapest)

Design for Retry (Oneshot Budapest)

Aria Stewart

November 21, 2014

More Decks by Aria Stewart

Other Decks in Programming


  1. Design for Retry: Microservices, REST, and why Idempotency is the

    only way to scale I'm Aria Stewart, that's @aredridel just about everywhere. I'm here thanks to PayPal. I work on the open source Kraken.js framework.
  2. 2xx OK 3xx Go elsewhere 4xx Tell user what they

    did wrong 5xx Bail out and log an error I'd call this Error avoidance
  3. 4xx Tell the user what they did wrong 5xx Save

    that request and do something with it later.
  4. Idempotency Repeated actions have no effect, give the same result

    This means being smart about IDs. Don't recycle! Check if things are already done. They are? Just give the same answer again.
  5. Causes! —database down —bug in a service —Deploy in progress

    —power failure —kicked a cable —Network congestion —Capacity exceeded —Microbursts
  6. —Tree fell on the data center —earthquake —tornado —birds, snakes

    and aeroplanes —Black Friday —Slashdot effect —Interns —QA tests —DoS attack
  7. Lots of ways to do it Database on each node.

    Maybe LevelDB? Log file Queue server
  8. gearman Queues built in There are many alternatives, but gearmand

    is very simple. The memcache of job queues.
  9. Let me tell you about one TRILLIONS of messages MILLIONS

    of nodes 100% availability (at least partial) for years. 32 years. Resilient to MILLIONS of bad actors. It is attached to the most malicious network.
  10. You need a smart client. Keeps outstanding requests. Resubmit. Try

    a different server! Try a second queue service. Maybe have a fallback plan.
  11. C = Consistency If there's state that one part knows

    of that another doesn't? That's inconsistency.
  12. Ever try to write email on the web while not

    on the Internet? It's cloud easy!
  13. Thank you! I hope you have lots of ideas queued

    up. Save your ideas and unspool them onto Twitter when you get home. Let me know if this changed how you think about designing applications!