Distributed Systems: Your Only Guarantee Is Inconsistency

2cabbed1afd95c74a7a4d175e8225a88?s=47 Anthony
September 29, 2017

Distributed Systems: Your Only Guarantee Is Inconsistency

2cabbed1afd95c74a7a4d175e8225a88?s=128

Anthony

September 29, 2017
Tweet

Transcript

  1. 2.
  2. 3.

    • Generate the user's invoice • Charge them • Email

    them • Place account holds on delinquent users • Generate reports for internal finance teams • Perform other relevant actions Our end-of-month pipeline
  3. 5.

    class MonthClose def perform generate_invoice_items # expensive amount = generate_invoice

    # expensive success = charge_balance(amount) # external dependencies email_user(amount) # external dependencies handle_failed_charge unless success # complicated and messy end end There's a lot to do
  4. 8.

    • Persistent jobs (SQL or Redis) • Prioritized job queues

    • Immediate, recurring, or delayed scheduling • Expect failure: automatic retries • Batch jobs with success/failure callbacks Background workers
  5. 9.
  6. 11.

    # app/workers/expensive_job_worker.rb class ExpensiveJobWorker include Sidekiq::Worker sidekiq_options(queue: :high) def perform(args)

    ExpensiveJob.new(args).expensive_method end end # app/lib/expensive_job.rb class ExpensiveJob def initialize(args) @args = args end def expensive_method end end
  7. 14.

    # Run it in the background every day # whenever

    gem => https://github.com/javan/whenever every :day do runner "ExpensiveJobWorker.perform_async(args)" end
  8. 16.

    class MonthCloseWorker def perform generate_invoice_items amount = generate_invoice PaymentWorker.perform_async(amount) EmailWorker.perform_async(amount)

    end end Applying it to our use case class PaymentWorker def perform(amount) success = charge_user(amount) HandleFailedChargeWorker.perform_async unless success end end class HandleFailedChargeWorker def perform handle_failed_charge end end class EmailWorker def perform(amount) email_user(amount) end end
  9. 17.

    Before • ~30 minutes per user (on average) • 1-2

    days for entire month close process After • <10 minutes per user • <8 hours for entire month close process So much better
  10. 19.
  11. 21.

    Our mental model is an ideal world They're created from

    user stories or an ideal workflow They don't necessarily represent reality
  12. 27.

    Real world workflows Invoice is generated, but the user applied

    a credit before the payment could be made
  13. 28.

    Real world workflows Payment is attempted but the user removed

    their credit card before we realized we couldn’t charge them
  14. 30.

    Learning #1 When you pass information, you are working under

    the assumption that represents the state of the world at that time
  15. 33.

    “Well we need payments to run immediately after an invoice

    is generated, so we'll mark it highest priority”
  16. 34.

    “Well we need payments to run immediately after an invoice

    is generated, so we'll mark it highest priority” NO!
  17. 39.

    class PaymentWorker def perform(amount) current_balance = user.balance if current_balance !=

    amount # charge user? throw error? do nothing? else charge_user(amount) end end end
  18. 40.
  19. 41.
  20. 43.

    From: billing@digitalocean.com Subject: Your August 2017 Invoice Hi Anthony, Thanks

    for being a loyal customer! As of 2017-08-07 19:31:09 PST, your balance is $10.00. Thanks, DigitalOcean
  21. 44.

    From: billing@digitalocean.com Subject: Your August 2017 Invoice Hi Anthony, Thanks

    for being a loyal customer! As of 2017-08-07 19:31:09 PST, your balance is $10.00. As of 2017-08-08 03:31:09 CET, your balance is Ft2565. Thanks, DigitalOcean