Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Distributed Systems: Your Only Guarantee Is Inconsistency

Anthony
September 29, 2017

Distributed Systems: Your Only Guarantee Is Inconsistency

Anthony

September 29, 2017
Tweet

Other Decks in Programming

Transcript

  1. Distributed Systems
    Your Only Guarantee Is Inconsistency

    View full-size slide

  2. ● Generate the user's invoice
    ● Charge them
    ● Email them
    ● Place account holds on delinquent users
    ● Generate reports for internal finance teams
    ● Perform other relevant actions
    Our end-of-month pipeline

    View full-size slide

  3. Where we are...
    Architecture goals
    Where we want to be...

    View full-size slide

  4. class MonthClose
    def perform
    generate_invoice_items # expensive
    amount = generate_invoice # expensive
    success = charge_balance(amount) # external dependencies
    email_user(amount) # external dependencies
    handle_failed_charge unless success # complicated and messy
    end
    end
    There's a lot to do

    View full-size slide

  5. What if it fails halfway through?

    View full-size slide

  6. Background workers

    View full-size slide

  7. ● Persistent jobs (SQL or Redis)
    ● Prioritized job queues
    ● Immediate, recurring, or delayed scheduling
    ● Expect failure: automatic retries
    ● Batch jobs with success/failure callbacks
    Background workers

    View full-size slide

  8. ● Sidekiq
    ● Delayed Job
    ● Resque
    Background workers

    View full-size slide

  9. # app/workers/expensive_job_worker.rb
    class ExpensiveJobWorker
    include Sidekiq::Worker
    sidekiq_options(queue: :high)
    def perform(args)
    ExpensiveJob.new(args).expensive_method
    end
    end
    # app/lib/expensive_job.rb
    class ExpensiveJob
    def initialize(args)
    @args = args
    end
    def expensive_method
    end
    end

    View full-size slide

  10. # Run it in the background
    ExpensiveJobWorker.perform_async(args)

    View full-size slide

  11. # Run it in the background… in 10 minutes
    ExpensiveJobWorker.perform_in(10.minutes, args)

    View full-size slide

  12. # Run it in the background every day
    # whenever gem => https://github.com/javan/whenever
    every :day do
    runner "ExpensiveJobWorker.perform_async(args)"
    end

    View full-size slide

  13. class MonthCloseWorker
    def perform
    generate_invoice_items
    amount = generate_invoice
    charge_balance(amount)
    email_user(amount)
    handle_failed_charge
    end
    end
    We can do better

    View full-size slide

  14. class MonthCloseWorker
    def perform
    generate_invoice_items
    amount = generate_invoice
    PaymentWorker.perform_async(amount)
    EmailWorker.perform_async(amount)
    end
    end
    Applying it to our use case
    class PaymentWorker
    def perform(amount)
    success = charge_user(amount)
    HandleFailedChargeWorker.perform_async unless success
    end
    end
    class HandleFailedChargeWorker
    def perform
    handle_failed_charge
    end
    end
    class EmailWorker
    def perform(amount)
    email_user(amount)
    end
    end

    View full-size slide

  15. Before
    ● ~30 minutes per user (on average)
    ● 1-2 days for entire month close process
    After
    ● <10 minutes per user
    ● <8 hours for entire month close process
    So much better

    View full-size slide

  16. Whoops!
    We just introduced all kinds of bugs

    View full-size slide

  17. Average time between steps
    Before: 10 µs After: 5 min? 10 min?

    View full-size slide

  18. Our mental model is an ideal world
    They're created from user stories or an ideal workflow
    They don't necessarily represent reality

    View full-size slide

  19. Ideal workflows
    Invoice is generated and then payment is attempted

    View full-size slide

  20. Ideal workflows
    Payment fails and then the user is suspended

    View full-size slide

  21. Ideal workflows
    Payment succeeds and then the user is emailed a receipt

    View full-size slide

  22. Notice the and thens?

    View full-size slide

  23. Reality likes buts

    View full-size slide

  24. Real world workflows
    Invoice is generated, but the user applied a credit before the
    payment could be made

    View full-size slide

  25. Real world workflows
    Payment is attempted but the user removed their credit card
    before we realized we couldn’t charge them

    View full-size slide

  26. Real world workflows
    Payment is attempted but the user already paid manually

    View full-size slide

  27. Learning #1
    When you pass information, you are working under the
    assumption that represents the state of the world at that
    time

    View full-size slide

  28. Learning #2
    Changing methods from synchronous to asynchronous is an
    implicit change in behavior

    View full-size slide

  29. What can we do?

    View full-size slide

  30. “Well we need payments to run immediately after an invoice is generated, so
    we'll mark it highest priority”

    View full-size slide

  31. “Well we need payments to run immediately after an invoice is generated, so
    we'll mark it highest priority”
    NO!

    View full-size slide

  32. Don’t engage in a priority arms race

    View full-size slide

  33. queue_priority:
    - this_one_first_do_not_move_down
    - super_critical
    - critical
    - highest
    - higher
    - high
    - default

    View full-size slide

  34. So… what can we do?

    View full-size slide

  35. Assume the world changes

    View full-size slide

  36. class PaymentWorker
    def perform(amount)
    current_balance = user.balance
    if current_balance != amount
    # charge user? throw error? do nothing?
    else
    charge_user(amount)
    end
    end
    end

    View full-size slide

  37. Freeze your world in time

    View full-size slide

  38. From: [email protected]
    Subject: Your August 2017 Invoice
    Hi Anthony,
    Thanks for being a loyal customer!
    As of 2017-08-07 19:31:09 PST, your balance is $10.00.
    Thanks,
    DigitalOcean

    View full-size slide

  39. From: [email protected]
    Subject: Your August 2017 Invoice
    Hi Anthony,
    Thanks for being a loyal customer!
    As of 2017-08-07 19:31:09 PST, your balance is $10.00.
    As of 2017-08-08 03:31:09 CET, your balance is Ft2565.
    Thanks,
    DigitalOcean

    View full-size slide

  40. Embrace the inconsistency

    View full-size slide