Running Jobs at Scale

Transcript

1. Running Jobs at Scale Kir Shatrov GORUCO 2018, @kirshatrov

2. GORUCO 2018, @kirshatrov

3. GORUCO 2018, @kirshatrov

4. GORUCO 2018, @kirshatrov

5. GORUCO 2018, @kirshatrov

6. Jobs GORUCO 2018, @kirshatrov

7. class ExampleJob < ActiveJob::Base def perform ... end end GORUCO

   2018, @kirshatrov

8. class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh

   end end end GORUCO 2018, @kirshatrov

9. class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh

   end end end minutes? hours? days? GORUCO 2018, @kirshatrov

10. Long-running jobs GORUCO 2018, @kirshatrov

11. Long-running jobs — Deploys and termination GORUCO 2018, @kirshatrov

12. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost GORUCO 2018, @kirshatrov

13. GORUCO 2018, @kirshatrov

14. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete GORUCO 2018, @kirshatrov

15. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete — Capacity and worker starvation GORUCO 2018, @kirshatrov

16. Long-running jobs — Deploys and termination — Abort and re-enqueue

    — Progress lost — Job may never complete — Capacity and worker starvation — Cloud ☁ GORUCO 2018, @kirshatrov

17. Why is it taking long? Because it iterates over a

    long collection. GORUCO 2018, @kirshatrov

18. What if jobs were interruptible and resumable? GORUCO 2018, @kirshatrov

19. Split the job definition 1. Collection to process 2. Work

    to be done GORUCO 2018, @kirshatrov

20. Split the job definition 1. Collection to process ≫ Product.all

    2. Work to be done GORUCO 2018, @kirshatrov

21. Split the job definition 1. Collection to process ≫ Product.all

    2. Work to be done ≫ product.sync_and_refresh GORUCO 2018, @kirshatrov

22. class ExampleJob < ActiveJob::Base include Iteration def collection Product.all end

    def each_iteration(product) product.sync_and_refresh end end GORUCO 2018, @kirshatrov

23. — def perform — collection — each_iteration GORUCO 2018, @kirshatrov

24. Product.all cursor: 1 GORUCO 2018, @kirshatrov

25. Product.all cursor: 2 GORUCO 2018, @kirshatrov

26. Product.all cursor: 3 GORUCO 2018, @kirshatrov

27. Product.all cursor: 4 GORUCO 2018, @kirshatrov

28. Product.all cursor: 5 GORUCO 2018, @kirshatrov

29. Product.all cursor: 450123 GORUCO 2018, @kirshatrov

30. class WhateverJob < ActiveJob::Base include Iteration def collection Enumerator.new do

    |enum| 3.times do |n| enum << n end end end def each_iteration(n) # do something three times! end end GORUCO 2018, @kirshatrov

31. Endless possibilities — Interrupt and resume at any moment —

    Progress tracking — Parallel computations — Throttling by default GORUCO 2018, @kirshatrov

32. Benefits for the infrastructure — Keep supporting long-running jobs —

    Success for Cloud runtime — Make scale invisible for developers — Opportunities to save money with short-living instances in Cloud GORUCO 2018, @kirshatrov

33. Thank you! @kirshatrov GORUCO 2018, @kirshatrov