$30 off During Our Annual Pro Sale. View details »

Background jobs at scale (Montreal.rb)

Background jobs at scale (Montreal.rb)

Talks at Montreal.rb

Kerstin Puschke

February 26, 2019
Tweet

More Decks by Kerstin Puschke

Other Decks in Programming

Transcript

  1. Kerstin Puschke @titanoboa42 Background jobs at scale

  2. None
  3. @titanoboa42 Scaling applications using background jobs keeping code simple

  4. @titanoboa42 Outline

  5. @titanoboa42 • Introduction to background jobs Outline

  6. @titanoboa42 • Introduction to background jobs • Features Outline

  7. @titanoboa42 • Introduction to background jobs • Features • Mastering

    challenges Outline
  8. @titanoboa42 Outline

  9. @titanoboa42 • Being RESTful Outline

  10. @titanoboa42 • Being RESTful • Background jobs at scale Outline

  11. @titanoboa42 • Being RESTful • Background jobs at scale •

    Summary Outline
  12. @titanoboa42 Introduction to background jobs

  13. @titanoboa42 Background job:
 Work to be done later App Server

    Worker
  14. @titanoboa42 Asynchronous communication App Server Message Queue Worker

  15. @titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue

  16. @titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker

    Task Queue
  17. @titanoboa42 Background job backend:
 task queue & broker

  18. @titanoboa42 Encapsulating
 async communication

  19. @titanoboa42 Features

  20. @titanoboa42 Task Queue Response times App Server Worker

  21. @titanoboa42 Task Queue Spikeability App Server Worker

  22. @titanoboa42 Task Queue Parallelization App Server Worker Worker Worker

  23. @titanoboa42 Task Queue Retries App Server Worker Worker Worker

  24. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  25. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  26. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  27. @titanoboa42 Mastering challenges

  28. @titanoboa42 No exactly once delivery

  29. @titanoboa42 • “At least” vs. “at most” once delivery No

    exactly once delivery
  30. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  31. @titanoboa42 Out of order delivery

  32. @titanoboa42 • If order matters, queue sequentially Out of order

    delivery
  33. @titanoboa42 • If order matters, queue sequentially • First job

    queues follow up jobs Out of order delivery
  34. @titanoboa42 Job queued and processed by different versions

  35. @titanoboa42 • No breaking changes to job parameters Job queued

    and processed by different versions
  36. @titanoboa42 • No breaking changes to job parameters • Changes

    need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
  37. @titanoboa42 Eventual consistency (at best)

  38. @titanoboa42 • Prepare for inconsistency Eventual consistency (at best)

  39. @titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency

    guarantees vs. benefits of background jobs Eventual consistency (at best)
  40. @titanoboa42 Non-transactional queuing

  41. @titanoboa42 • Don’t queue from within a db transaction Non-transactional

    queuing
  42. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback Non-transactional queuing
  43. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback • Commit before queuing or 
 stage transactionally Non-transactional queuing
  44. @titanoboa42 Being RESTful

  45. @titanoboa42 Don’t lie about resource creation

  46. @titanoboa42 • 202 Accepted Don’t lie about resource creation

  47. @titanoboa42 • 202 Accepted • Location: temporary resource Don’t lie

    about resource creation
  48. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other Don’t lie about resource creation
  49. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other • Location: does not represent target resource Don’t lie about resource creation
  50. @titanoboa42 Callers can enforce (a)sync behaviour

  51. @titanoboa42 • Expect header Callers can enforce (a)sync behaviour

  52. @titanoboa42 • Expect header • 202-accepted Callers can enforce (a)sync

    behaviour
  53. @titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can

    enforce (a)sync behaviour
  54. @titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417

    Expectation failed Callers can enforce (a)sync behaviour
  55. @titanoboa42 Background jobs at scale

  56. @titanoboa42 DelayedJob is easy to get started

  57. @titanoboa42 • No additional infrastructure DelayedJob is easy to get

    started
  58. @titanoboa42 • No additional infrastructure • ActiveRecord DelayedJob is easy

    to get started
  59. @titanoboa42 ActiveJob makes swapping backends easy

  60. @titanoboa42 DelayedJob issues

  61. @titanoboa42 • Overhead of relational database DelayedJob issues

  62. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside DelayedJob issues
  63. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside • Frequently needs workers to restart DelayedJob issues
  64. @titanoboa42 Resque scales

  65. @titanoboa42 • Redis - no relational db Resque scales

  66. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers Resque scales
  67. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart Resque scales
  68. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
  69. @titanoboa42 Resque issues

  70. @titanoboa42 • Child processes Resque issues

  71. @titanoboa42 • Child processes • Memory hungry and slow Resque

    issues
  72. @titanoboa42 Sidekiq scales

  73. @titanoboa42 • Redis - no relational db Sidekiq scales

  74. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes Sidekiq scales
  75. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes • Fast and less memory hungry Sidekiq scales
  76. @titanoboa42 Sidekiq issues

  77. @titanoboa42 • Requires thread safe code Sidekiq issues

  78. @titanoboa42 Long running jobs - Resque

  79. @titanoboa42 • Prevent worker shutdown Long running jobs - Resque

  80. @titanoboa42 • Prevent worker shutdown • No deployments Long running

    jobs - Resque
  81. @titanoboa42 • Prevent worker shutdown • No deployments • Not

    cloud-friendly Long running jobs - Resque
  82. @titanoboa42 • Aborted and requeued on shutdown Long running jobs

    - Sidekiq
  83. @titanoboa42 • Aborted and requeued on shutdown • Job may

    not finish before being aborted again Long running jobs - Sidekiq
  84. @titanoboa42 github.com
 /Shopify/job-iteration

  85. @titanoboa42 Large collections

  86. @titanoboa42 • Split job into collection and task to be

    done Large collections
  87. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  88. @titanoboa42 Interruptible job with automatic resuming

  89. @titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming

  90. @titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job

    with automatic resuming
  91. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  92. @titanoboa42 Abstracting scaling issues
 simplifies 
 concrete background jobs

  93. @titanoboa42 github.com
 /Shopify/job-iteration

  94. @titanoboa42 Background jobs

  95. @titanoboa42 • Benefit apps of all sizes Background jobs

  96. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    Background jobs
  97. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    • Keep code simple at scale Background jobs
  98. Thanks!
 Questions?
 @titanoboa42
 
 https://www.shopify.com/careers