Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Background jobs at scale (Montreal.rb)

Background jobs at scale (Montreal.rb)

Talks at Montreal.rb

Avatar for Kerstin Puschke

Kerstin Puschke

February 26, 2019
Tweet

More Decks by Kerstin Puschke

Other Decks in Programming

Transcript

  1. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  2. @titanoboa42 • If order matters, queue sequentially • First job

    queues follow up jobs Out of order delivery
  3. @titanoboa42 • No breaking changes to job parameters • Changes

    need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
  4. @titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency

    guarantees vs. benefits of background jobs Eventual consistency (at best)
  5. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback Non-transactional queuing
  6. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback • Commit before queuing or 
 stage transactionally Non-transactional queuing
  7. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other Don’t lie about resource creation
  8. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other • Location: does not represent target resource Don’t lie about resource creation
  9. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside • Frequently needs workers to restart DelayedJob issues
  10. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart Resque scales
  11. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
  12. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes • Fast and less memory hungry Sidekiq scales
  13. @titanoboa42 • Aborted and requeued on shutdown • Job may

    not finish before being aborted again Long running jobs - Sidekiq
  14. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  15. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  16. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    • Keep code simple at scale Background jobs