Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Background jobs at scale (Montreal.rb)

Background jobs at scale (Montreal.rb)

Talks at Montreal.rb

Kerstin Puschke

February 26, 2019
Tweet

More Decks by Kerstin Puschke

Other Decks in Programming

Transcript

  1. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  2. @titanoboa42 • If order matters, queue sequentially • First job

    queues follow up jobs Out of order delivery
  3. @titanoboa42 • No breaking changes to job parameters • Changes

    need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
  4. @titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency

    guarantees vs. benefits of background jobs Eventual consistency (at best)
  5. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback Non-transactional queuing
  6. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback • Commit before queuing or 
 stage transactionally Non-transactional queuing
  7. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other Don’t lie about resource creation
  8. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other • Location: does not represent target resource Don’t lie about resource creation
  9. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside • Frequently needs workers to restart DelayedJob issues
  10. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart Resque scales
  11. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
  12. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes • Fast and less memory hungry Sidekiq scales
  13. @titanoboa42 • Aborted and requeued on shutdown • Job may

    not finish before being aborted again Long running jobs - Sidekiq
  14. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  15. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  16. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    • Keep code simple at scale Background jobs