Upgrade to Pro — share decks privately, control downloads, hide ads and more …

High availability by offloading work - background jobs, message queues, or Kafka

High availability by offloading work - background jobs, message queues, or Kafka

Talk at RubyConf Colombia 2019

5e8e44a4f6632772c47925006aff31d9?s=128

Kerstin Puschke

September 20, 2019
Tweet

Transcript

  1. Kerstin Puschke @titanoboa42 High availability by offloading work - background

    jobs, message queues, or Kafka
  2. @titanoboa42

  3. @titanoboa42 Different approaches to offload work to ensure high availability

  4. @titanoboa42 • High availability & offloading work Outline

  5. @titanoboa42 • High availability & offloading work • Background jobs

    Outline
  6. @titanoboa42 • High availability & offloading work • Background jobs

    • Message oriented middleware Outline
  7. @titanoboa42 • Event log Outline

  8. @titanoboa42 • Event log • Summary Outline

  9. @titanoboa42 High availability & offloading work

  10. @titanoboa42 High Availability

  11. @titanoboa42 
 users
 can interact
 with the system High Availability

  12. @titanoboa42 
 users
 can interact meaningfully
 with the system High

    Availability
  13. @titanoboa42 community of
 users
 can interact meaningfully
 with the system

    High Availability
  14. @titanoboa42 community of
 users
 can interact meaningfully
 with the system

    whenever needed High Availability
  15. @titanoboa42 community of
 users
 can interact meaningfully
 with the system

    whenever needed High Availability community of
 users
 can interact meaningfully
 with the system whenever needed
  16. @titanoboa42 Background jobs

  17. @titanoboa42 • Resque Background job backends

  18. @titanoboa42 • Resque • Sidekiq Background job backends

  19. @titanoboa42 • Resque • Sidekiq • … Background job backends

  20. @titanoboa42 Background job:
 Unit of work 
 to be done

    later App Server Worker
  21. @titanoboa42 Asynchronous communication App Server Message Queue Worker

  22. @titanoboa42 Asynchronous communication App Server Message Queue Worker Task Queue

  23. @titanoboa42 Asynchronous communication App Server Message Queue Worker Worker Worker

    Task Queue
  24. @titanoboa42 Background job backend:
 task queue & broker

  25. @titanoboa42 Encapsulating
 async communication

  26. @titanoboa42 Features

  27. @titanoboa42 Task Queue Response times App Server Worker

  28. @titanoboa42 Task Queue Spikeability App Server Worker

  29. @titanoboa42 Task Queue Parallelization App Server Worker Worker Worker

  30. @titanoboa42 Task Queue Retries App Server Worker Worker Worker

  31. @titanoboa42 Mastering challenges

  32. @titanoboa42 Job queued and processed by different versions

  33. @titanoboa42 • No breaking changes to job parameters Job queued

    and processed by different versions
  34. @titanoboa42 • No breaking changes to job parameters • Changes

    need to be backwards compatible until legacy jobs have been processed Job queued and processed by different versions
  35. @titanoboa42 No exactly once delivery

  36. @titanoboa42 • “At least” vs. “at most” once delivery No

    exactly once delivery
  37. @titanoboa42 • “At least” vs. “at most” once delivery •

    Idempotent jobs & at least once delivery No exactly once delivery
  38. @titanoboa42 Non-transactional queuing

  39. @titanoboa42 • Don’t queue from within a db transaction Non-transactional

    queuing
  40. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or in case of rollback Non-transactional queuing
  41. @titanoboa42 • Don’t queue from within a db transaction •

    Job runs before commit, or in case of rollback • Commit first: Job not guaranteed to be queued Non-transactional queuing
  42. @titanoboa42 Non-transactional queuing

  43. @titanoboa42 • Stage transactionally Non-transactional queuing

  44. @titanoboa42 • Stage transactionally • Scheduler queues job, updates staging

    data Non-transactional queuing
  45. @titanoboa42 Local transactions

  46. @titanoboa42 • Eventual consistency at best Local transactions

  47. @titanoboa42 • Eventual consistency at best • SAGA command/orchestration Local

    transactions
  48. @titanoboa42 Out of order delivery

  49. @titanoboa42 • SAGA events/choreography: jobs queue jobs Out of order

    delivery
  50. @titanoboa42 • SAGA events/choreography: jobs queue jobs • easy to

    build, hard to maintain or debug Out of order delivery
  51. @titanoboa42 • SAGA events/choreography: jobs queue jobs • easy to

    build, hard to maintain or debug • SAGA command/orchestrator Out of order delivery
  52. @titanoboa42 Long running jobs - Resque

  53. @titanoboa42 • Prevent worker shutdown Long running jobs - Resque

  54. @titanoboa42 • Prevent worker shutdown • No deployments Long running

    jobs - Resque
  55. @titanoboa42 • Prevent worker shutdown • No deployments • Not

    cloud-friendly Long running jobs - Resque
  56. @titanoboa42 • Aborted and requeued Long running jobs - Sidekiq

  57. @titanoboa42 • Aborted and requeued • Job may not finish

    before being aborted again Long running jobs - Sidekiq
  58. @titanoboa42 Large collections

  59. @titanoboa42 • Split job into collection and task to be

    done Large collections
  60. @titanoboa42 • Split job into collection and task to be

    done • Checkpoint after iteration & requeue Large collections
  61. @titanoboa42 Interruptible job with automatic resuming

  62. @titanoboa42 • Shutdown workers anytime Interruptible job with automatic resuming

  63. @titanoboa42 • Shutdown workers anytime • Disaster prevention Interruptible job

    with automatic resuming
  64. @titanoboa42 • Shutdown workers anytime • Disaster prevention • Data

    integrity Interruptible job with automatic resuming
  65. @titanoboa42 github.com
 /Shopify/job-iteration

  66. @titanoboa42 Abstracting scaling issues
 simplifies 
 concrete background jobs

  67. @titanoboa42 Task Queue Background jobs are ruby objects App Server

    Worker
  68. @titanoboa42 Task Queue Background jobs are ruby objects App Server

    Worker Broker Broker
  69. @titanoboa42 Offloading work to a worker running the same code

    base
  70. @titanoboa42 Background jobs
 Summary

  71. @titanoboa42 • Based on task queues Background jobs

  72. @titanoboa42 • Based on task queues • Complex overall system,

    simple concrete jobs Background jobs
  73. @titanoboa42 • Based on task queues • Complex overall system,

    simple concrete jobs • Great for monolith Background jobs
  74. @titanoboa42 Message oriented middleware

  75. @titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… Message oriented middleware

  76. @titanoboa42 • Implementations: RabbitMQ, ActiveMQ,… • Protocols: AMQP, MQTT, Stomp,…

    Message oriented middleware
  77. @titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface

    Worker Consumer
  78. @titanoboa42 Message Queue Messaging Middleware App Server Producer Data-based interface

    Worker Consumer Broker
  79. @titanoboa42 Features

  80. @titanoboa42 Commands & Events

  81. @titanoboa42 Propagating updates Business Partners Support Contracts Orders

  82. @titanoboa42 Propagating updates Business Partners Support Contracts Orders

  83. @titanoboa42 Messaging
 Middleware Resiliency Business Partners Orders

  84. @titanoboa42 Messaging
 Middleware Resiliency Business Partners

  85. @titanoboa42 Messaging
 Middleware Resiliency Business Partners Orders

  86. @titanoboa42 Topic with queues
 provides
 advanced routing App Server Business


    Partners Support
 Contracts Orders Messaging Middleware
  87. @titanoboa42 Topic with queues
 provides
 advanced routing App Server Message

    Queue Business
 Partners Support
 Contracts Orders Message Queue Messaging Middleware
  88. @titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners

    Support Contracts Orders
  89. @titanoboa42 Messaging Middleware Anonymity for producer and consumer Business Partners

    Invoices Support Contracts Orders
  90. @titanoboa42 Messaging Middleware Anonymity for producer and consumer FraudScore Orders

    Support Contracts
  91. @titanoboa42 Messaging Middleware Anonymity for producer and consumer Invoices FraudScore

    Orders Support Contracts
  92. @titanoboa42 Mastering challenges

  93. @titanoboa42 Keep breaking changes manageable

  94. @titanoboa42 • Avoid n:m routing Keep breaking changes manageable

  95. @titanoboa42 • Avoid n:m routing • Better representation of domain:


    multiple messages routed 1:n, n:1 instead Keep breaking changes manageable
  96. @titanoboa42 • No exactly once delivery Lack of guarantees

  97. @titanoboa42 • No exactly once delivery • No strong consistency

    Lack of guarantees
  98. @titanoboa42 • No exactly once delivery • No strong consistency

    • Out of order delivery Lack of guarantees
  99. @titanoboa42 No single source of truth

  100. @titanoboa42 • Messages removed after processing No single source of

    truth
  101. @titanoboa42 • Messages removed after processing • No replayability No

    single source of truth
  102. @titanoboa42 Offloading work to decoupled services with no notion of

    system wide state
  103. @titanoboa42 Message oriented middleware
 Summary

  104. @titanoboa42 • Based on queues and topics Message oriented middleware

  105. @titanoboa42 • Based on queues and topics • Complex overall

    system Message oriented middleware
  106. @titanoboa42 • Based on queues and topics • Complex overall

    system • Simple message consumers Message oriented middleware
  107. @titanoboa42 • Great for decoupled microservices Message oriented middleware

  108. @titanoboa42 • Great for decoupled microservices • No system wide

    state Message oriented middleware
  109. @titanoboa42 Event log

  110. @titanoboa42 • Kafka Event logs

  111. @titanoboa42 • Kafka • … Event logs

  112. @titanoboa42 • Events persisted into append-only log Event log

  113. @titanoboa42 • Events persisted into append-only log • Consumers read

    shared log Event log
  114. @titanoboa42 • Events persisted into append-only log • Consumers read

    shared log • Stateless broker (no queues) Event log
  115. @titanoboa42 High throughput

  116. @titanoboa42 Single source of truth

  117. @titanoboa42 • Events are not removed after processing Single source

    of truth
  118. @titanoboa42 • Events are not removed after processing • Replayability

    Single source of truth
  119. @titanoboa42 Offloading work to services keeping the notion of system

    wide state
  120. @titanoboa42 Event log
 Summary

  121. @titanoboa42 • Based on shared log, no queues Event logs

  122. @titanoboa42 • Based on shared log, no queues • Complex

    overall system Event logs
  123. @titanoboa42 Event logs

  124. @titanoboa42 • Single source of truth (e.g. for event sourcing)

    Event logs
  125. @titanoboa42 • Single source of truth (e.g. for event sourcing)

    • High throughput applications Event logs
  126. @titanoboa42 Summary

  127. @titanoboa42 • Queues Background jobs

  128. @titanoboa42 • Queues • For monolithic code base Background jobs

  129. @titanoboa42 • Topics and Queues Message oriented middleware

  130. @titanoboa42 • Topics and Queues • For decoupled microservices Message

    oriented middleware
  131. @titanoboa42 • Shared log, no queues Event logs

  132. @titanoboa42 • Shared log, no queues • For event sourcing

    & high throughput applications Event logs
  133. @titanoboa42 BFCM video

  134. @titanoboa42 BFCM video

  135. Thanks!
 Questions?
 @titanoboa42
 
 https://www.shopify.com/careers