Background jobs at scale (Montreal.rb)

Background jobs at scale (Montreal.rb)

Talks at Montreal.rb

5e8e44a4f6632772c47925006aff31d9?s=128

Kerstin Puschke

February 26, 2019
Tweet

Transcript

  1. Kerstin Puschke @titanoboa42 Background jobs at scale

  2. None
  3. @titanoboa42 Scaling applications using background jobs keeping code simple

  4. @titanoboa42 Outline

  5. @titanoboa42 • Introduction to background jobs Outline

  6. @titanoboa42 • Introduction to background jobs • Features Outline

  7. @titanoboa42 • Introduction to background jobs • Features • Mastering

    challenges Outline
  8. @titanoboa42 Outline

  9. @titanoboa42 • Being RESTful Outline

  10. @titanoboa42 • Being RESTful • Background jobs at scale Outline

  11. @titanoboa42 • Being RESTful • Background jobs at scale •

    Summary Outline
  12. @titanoboa42 Introduction to background jobs

  13. @titanoboa42 Background job:
 Work to be done later App Server

    Worker
  14. @titanoboa42 Asynchronous communication App Server Message Queue Worker

  15. @titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue

  16. @titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker

    Task Queue
  17. @titanoboa42 Background job backend:
 task queue & broker

  18. @titanoboa42 Encapsulating
 async communication

  19. @titanoboa42 Features

  20. @titanoboa42 Task Queue Response times App Server Worker

  21. @titanoboa42 Task Queue Spikeability App Server Worker

  22. @titanoboa42 Task Queue Parallelization App Server Worker Worker Worker

  23. @titanoboa42 Task Queue Retries App Server Worker Worker Worker

  24. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  25. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  26. @titanoboa42 Prioritization App Server Worker Worker High Prio Queue Low

    Prio Queue
  27. @titanoboa42 Mastering challenges

  28. @titanoboa42 No exactly once delivery

  29. @titanoboa42 • “At least” vs. “at most” once delivery No

    exactly once delivery
  30. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  31. @titanoboa42 Out of order delivery

  32. @titanoboa42 • If order matters, queue sequentially Out of order

    delivery
  33. @titanoboa42 • If order matters, queue sequentially • First job

    queues follow up jobs Out of order delivery
  34. @titanoboa42 Job queued and processed by different versions

  35. @titanoboa42 • No breaking changes to job parameters Job queued

    and processed by different versions
  36. @titanoboa42 • No breaking changes to job parameters • Changes

    need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
  37. @titanoboa42 Eventual consistency (at best)

  38. @titanoboa42 • Prepare for inconsistency Eventual consistency (at best)

  39. @titanoboa42 • Prepare for inconsistency • Trade-off lack of consistency

    guarantees vs. benefits of background jobs Eventual consistency (at best)
  40. @titanoboa42 Non-transactional queuing

  41. @titanoboa42 • Don’t queue from within a db transaction Non-transactional

    queuing
  42. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback Non-transactional queuing
  43. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or if rollback • Commit before queuing or 
 stage transactionally Non-transactional queuing
  44. @titanoboa42 Being RESTful

  45. @titanoboa42 Don’t lie about resource creation

  46. @titanoboa42 • 202 Accepted Don’t lie about resource creation

  47. @titanoboa42 • 202 Accepted • Location: temporary resource Don’t lie

    about resource creation
  48. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other Don’t lie about resource creation
  49. @titanoboa42 • 202 Accepted • Location: temporary resource • 303

    See other • Location: does not represent target resource Don’t lie about resource creation
  50. @titanoboa42 Callers can enforce (a)sync behaviour

  51. @titanoboa42 • Expect header Callers can enforce (a)sync behaviour

  52. @titanoboa42 • Expect header • 202-accepted Callers can enforce (a)sync

    behaviour
  53. @titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content Callers can

    enforce (a)sync behaviour
  54. @titanoboa42 • Expect header • 202-accepted • 200-ok/201-created/204-no-content • 417

    Expectation failed Callers can enforce (a)sync behaviour
  55. @titanoboa42 Background jobs at scale

  56. @titanoboa42 DelayedJob is easy to get started

  57. @titanoboa42 • No additional infrastructure DelayedJob is easy to get

    started
  58. @titanoboa42 • No additional infrastructure • ActiveRecord DelayedJob is easy

    to get started
  59. @titanoboa42 ActiveJob makes swapping backends easy

  60. @titanoboa42 DelayedJob issues

  61. @titanoboa42 • Overhead of relational database DelayedJob issues

  62. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside DelayedJob issues
  63. @titanoboa42 • Overhead of relational database • Workers monitored from

    outside • Frequently needs workers to restart DelayedJob issues
  64. @titanoboa42 Resque scales

  65. @titanoboa42 • Redis - no relational db Resque scales

  66. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers Resque scales
  67. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart Resque scales
  68. @titanoboa42 • Redis - no relational db • Parent-child forking

    for workers • Rarely needs workers to restart • Workers manage their own state Resque scales
  69. @titanoboa42 Resque issues

  70. @titanoboa42 • Child processes Resque issues

  71. @titanoboa42 • Child processes • Memory hungry and slow Resque

    issues
  72. @titanoboa42 Sidekiq scales

  73. @titanoboa42 • Redis - no relational db Sidekiq scales

  74. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes Sidekiq scales
  75. @titanoboa42 • Redis - no relational db • Threads instead

    of child processes • Fast and less memory hungry Sidekiq scales
  76. @titanoboa42 Sidekiq issues

  77. @titanoboa42 • Requires thread safe code Sidekiq issues

  78. @titanoboa42 Long running jobs - Resque

  79. @titanoboa42 • Prevent worker shutdown Long running jobs - Resque

  80. @titanoboa42 • Prevent worker shutdown • No deployments Long running

    jobs - Resque
  81. @titanoboa42 • Prevent worker shutdown • No deployments • Not

    cloud-friendly Long running jobs - Resque
  82. @titanoboa42 • Aborted and requeued on shutdown Long running jobs

    - Sidekiq
  83. @titanoboa42 • Aborted and requeued on shutdown • Job may

    not finish before being aborted again Long running jobs - Sidekiq
  84. @titanoboa42 github.com
 /Shopify/job-iteration

  85. @titanoboa42 Large collections

  86. @titanoboa42 • Split job into collection and task to be

    done Large collections
  87. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  88. @titanoboa42 Interruptible job with automatic resuming

  89. @titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming

  90. @titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job

    with automatic resuming
  91. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  92. @titanoboa42 Abstracting scaling issues
 simplifies 
 concrete background jobs

  93. @titanoboa42 github.com
 /Shopify/job-iteration

  94. @titanoboa42 Background jobs

  95. @titanoboa42 • Benefit apps of all sizes Background jobs

  96. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    Background jobs
  97. @titanoboa42 • Benefit apps of all sizes • Require trade-offs

    • Keep code simple at scale Background jobs
  98. Thanks!
 Questions?
 @titanoboa42
 
 https://www.shopify.com/careers