Save 37% off PRO during our Black Friday Sale! »

Running Jobs at Scale

Running Jobs at Scale

My talk from GORUCO 2018 in New York.

C40db80709604a6868c769aca6367fb7?s=128

Kir Shatrov

June 16, 2018
Tweet

Transcript

  1. Running Jobs at Scale Kir Shatrov GORUCO 2018, @kirshatrov

  2. GORUCO 2018, @kirshatrov

  3. GORUCO 2018, @kirshatrov

  4. GORUCO 2018, @kirshatrov

  5. GORUCO 2018, @kirshatrov

  6. Jobs GORUCO 2018, @kirshatrov

  7. class ExampleJob < ActiveJob::Base def perform ... end end GORUCO

    2018, @kirshatrov
  8. class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh

    end end end GORUCO 2018, @kirshatrov
  9. class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh

    end end end minutes? hours? days? GORUCO 2018, @kirshatrov
  10. Long-running jobs GORUCO 2018, @kirshatrov

  11. Long-running jobs — Deploys and termination GORUCO 2018, @kirshatrov

  12. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost GORUCO 2018, @kirshatrov
  13. GORUCO 2018, @kirshatrov

  14. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete GORUCO 2018, @kirshatrov
  15. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete — Capacity and worker starvation GORUCO 2018, @kirshatrov
  16. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete — Capacity and worker starvation — Cloud ☁ GORUCO 2018, @kirshatrov
  17. Why is it taking long? Because it iterates over a

    long collection. GORUCO 2018, @kirshatrov
  18. What if jobs were interruptible and resumable? GORUCO 2018, @kirshatrov

  19. Split the job definition 1. Collection to process 2. Work

    to be done GORUCO 2018, @kirshatrov
  20. Split the job definition 1. Collection to process ≫ Product.all

    2. Work to be done GORUCO 2018, @kirshatrov
  21. Split the job definition 1. Collection to process ≫ Product.all

    2. Work to be done ≫ product.sync_and_refresh GORUCO 2018, @kirshatrov
  22. class ExampleJob < ActiveJob::Base include Iteration def collection Product.all end

    def each_iteration(product) product.sync_and_refresh end end GORUCO 2018, @kirshatrov
  23. — def perform — collection — each_iteration GORUCO 2018, @kirshatrov

  24. Product.all cursor: 1 GORUCO 2018, @kirshatrov

  25. Product.all cursor: 2 GORUCO 2018, @kirshatrov

  26. Product.all cursor: 3 GORUCO 2018, @kirshatrov

  27. Product.all cursor: 4 GORUCO 2018, @kirshatrov

  28. Product.all cursor: 5 GORUCO 2018, @kirshatrov

  29. Product.all cursor: 450123 GORUCO 2018, @kirshatrov

  30. class WhateverJob < ActiveJob::Base include Iteration def collection Enumerator.new do

    |enum| 3.times do |n| enum << n end end end def each_iteration(n) # do something three times! end end GORUCO 2018, @kirshatrov
  31. Endless possibilities — Interrupt and resume at any moment —

    Progress tracking — Parallel computations — Throttling by default GORUCO 2018, @kirshatrov
  32. Benefits for the infrastructure — Keep supporting long-running jobs —

    Success for Cloud runtime — Make scale invisible for developers — Opportunities to save money with short-living instances in Cloud GORUCO 2018, @kirshatrov
  33. Thank you! @kirshatrov GORUCO 2018, @kirshatrov