ACIDic Jobs: A Layman's Guide to Job Bliss

Background jobs have become an essential component of any Ruby infrastructure, and, as the Sidekiq Best Practices remind us, it is essential that jobs be "idempotent and transactional." But how do we make our jobs idempotent and transactional? In this talk, we will explore various techniques to make our jobs robust and ACIDic.

Transcript

1. Stephen Margheim — RubyConf 2021 ACIDic Jobs A Layman's Guide to Job Bliss
2. Stephen Margheim — RubyConf 2021 ACIDic Jobs A Layman's Guide to Job Bliss
3. Stephen Margheim @fractaledmind
4. Jobs are essential
5. But what is a Job?
6. Job == Verb Job != ActiveJob object
7. Job == State Mutation Job != inspection or retrieval of information
8. Job => Side Effects Job !=> return value
9. • a Ruby class object, • representing a state mutation action, • that takes as input a representation of initial state • and produces side-effects representing a next state. A job is
10. ActionDoer.call *arguments # service object ActionJob.perform_now *arguments # active job ActionWorker.new.perform *arguments # sidekiq worker ActionOperation.run *arguments # operation class
11. Jobs must be idempotent & transactional
12. Operation 1 Operation 2 Operation 3 Transaction
13. • Atomicity: everything succeeds or everything fails • Consistency: the data always ends up in a valid state, as defined by your schema • Isolation: concurrent transactions won't conflict with each other • Durability: once committed always committed, even with system failures The ACIDic Guarantees
14. Jobs · Databases
15. Idempotency f(f(x)) == f(x) Job.perform & Job.perform == Job.perform
16. • Functional Idempotency: the function always returns the same result, even if called multiple times f(f(x)) == f(x) • Practical Idempotency: the side-effect(s) will happen once and only once, no matter how many times the job is performed Job.perform == Job.perform & Job.perform The Idempotent Guarantee
17. Jobs · Retries
18. class JobRun < ActiveRecord::Base # ... end
20. • Use a transaction to guarantee atomic execution • Use locks to prevent concurrent data access • Use idempotency and retries to ensure eventual completion • Ensure enqueuing other jobs is co-transactional • Split complex operations into steps ACIDic Job Principles — Nathan Griffith
21. ACIDic Jobs Level 0 — Transactional Jobs
22. def perform(from_account, to_account, amount) run = JobRun.find_or_create_by!( job_class: self.class, job_id: job_id, job_args: [from_account, to_account, amount]) run.with_lock do from_account.lock! to_account.lock! from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) end end
23. — Mike Perham “Just remember that Sidekiq will execute your job at least once, not exactly once.”
24. • Use a database record to make job runs transactional • Use a database lock to mitigate concurrency issues ACIDic Jobs Level 0 Recap
25. ACIDic Jobs Level 1 — Idempotent Jobs
26. Idempotency & Uniqueness To guarantee idempotency, we must be able to define and identify the job uniquely
27. john_account.balance # initial state # => 100_00 TransferBalanceJob.perform_later(john_account, jane_account, 10_00) TransferBalanceJob.perform_later(john_account, jane_account, 10_00) john_account.balance # resulting state # => 80_00 or 90_00 ?
28. • each job run uses a generic unique entity representing this job run • each job run uses a generic unique entity representing this job execution (based on args) Forms of Job Uniqueness
29. def perform(from_account, to_account, amount) run = JobRun.find_or_create_by!(job_class: self.class, job_id: job_id) run.with_lock do return if run.completed? from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) run.update!(completed_at: Time.current) end end Unique Job by Job Run
30. john_account.balance # initial state # => 100_00 TransferBalanceJob.perform_later(john_account, jane_account, 10_00) TransferBalanceJob.perform_later(john_account, jane_account, 10_00) john_account.balance # resulting state # => 80_00 ?
31. def perform(from_account, to_account, amount) run = JobRun.find_or_create_by!(job_class: self.class, job_args: [from_account, to_account, amount]) run.with_lock do return if run.completed? from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) run.update!(completed_at: Time.current) end end Unique Job by Execution Args
32. john_account.balance # initial state # => 100_00 TransferBalanceJob.perform_later(john_account, jane_account, 10_00) TransferBalanceJob.perform_later(john_account, jane_account, 10_00) john_account.balance # resulting state # => 90_00 ?
33. Unique Job flexibly and generically class TransferBalanceJob < ApplicationJob prepend UniqueByJobRun uniquely_identified_by_job_id # uniquely_identified_by_job_args def perform(from_account, to_account, amount) from_account.lock! to_account.lock! from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) end end
34. john_account.balance # initial state # => 100_00 TransferBalanceJob.perform_later(john_account, jane_account, 10_00) TransferBalanceJob.perform_later(john_account, jane_account, 10_00) john_account.balance # resulting state # => 80_00
35. Unique Job flexibly and generically class TransferBalanceJob < ApplicationJob prepend UniqueByJobRun # uniquely_identified_by_job_id uniquely_identified_by_job_args def perform(from_account, to_account, amount) from_account.lock! to_account.lock! from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) end end
36. john_account.balance # initial state # => 100_00 TransferBalanceJob.perform_later(john_account, jane_account, 10_00) TransferBalanceJob.perform_later(john_account, jane_account, 10_00) john_account.balance # resulting state # => 90_00
37. • Use a database record to make job runs idempotent • custom or generic • by job ID or by job args ACIDic Jobs Level 1 Recap
38. ACIDic Jobs Level 2 — Enqueuing other Jobs
39. uniquely_identified_by_job_id def perform(from_account, to_account, amount) from_account.lock! to_account.lock! from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) TransferMailer.with(account: from_account).outbound.deliver_later TransferMailer.with(account: to_account).inbound.deliver_later end
40. Failure Condition 1 job queue from.save! job process to.save! TransferMailer.deliver_later transaction commits job starts job fails job queued by web process and dequeued by background worker
41. Failure Condition 2 job queue from.save! job process to.save! TransferMailer.deliver_later job starts job fails rollback job queued by web process and dequeued by background worker
42. Solution Option 1 • A database-backed job queue • delayed_job • que • good_job But... no Sidekiq and increased db load
44. Solution Option 2 • A transactionally-staged job queue So... more Sidekiq and minimal increased db load
45. uniquely_identified_by_job_id def perform(from_account, to_account, amount) from_account.lock! to_account.lock! from_account.update!(balance: from_account.balance - amount) to_account.update!(balance: to_account.balance - amount) TransferMailer.with(account: from_account).outbound.deliver_acidic TransferMailer.with(account: to_account).inbound.deliver_acidic end
46. def deliver_acidic(options = {}) job = delivery_job_class attributes = { adapter: "activejob", job_name: job.name } job_args = if job <= ActionMailer::Parameterized::MailDeliveryJob [@mailer_class.name, @action, "deliver_now", {params: @params, args: @args}] else [@mailer_class.name, @action, "deliver_now", @params, *@args] end attributes[:job_args] = job.new(job_args).serialize StagedJob.create!(attributes) end
47. class StagedJob < ActiveRecord::Base after_create_commit :enqueue_job def enqueue_job case adapter when "activejob" ActiveJob::Base.deserialize(job_args).enqueue when "sidekiq" Sidekiq::Client.push("class" => job_name, "args" => job_args) end end end
48. • Use transactionally-staged jobs to keep job enqueuing co-transactional with standard database operations • while keeping Sidekiq, and • not requiring an independent de-staging process ACIDic Jobs Level 2 Recap
49. ACIDic Jobs Level 3 — Operational Steps
50. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end
51. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end
52. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end
53. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end
54. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end
55. def perform(order) order.lock! order.process_and_fulfill! ShopifyAPI::Fulfillment.create!({ amount: order.amount, customer: order.purchaser, }) OrderMailer.with(order: order).fulfilled.deliver_acidic end ?
57. Workflow step 1 step 2 step 3
58. Workflow — Run 1 step 1 step 2 step 3
59. Workflow — Run 2 step 1 step 2 step 3
60. Job-wise vs Step-wise Idempotency
61. Job-wise vs Step-wise vs Idempotency
62. uniquely_identified_by_job_args def perform(order) @job.with_lock do order.process_and_fulfill! @job.update!(recovery_point: :fulfill_order) end if @job.recovery_point == :start # ... end
63. uniquely_identified_by_job_args def perform(order) # ... @job.with_lock do ShopifyAPI::Fulfillment.create!({ ... }) @job.update!(recovery_point: :send_email) end if @job.recovery_point == :fulfill_order # ... end
64. uniquely_identified_by_job_args def perform(order) # ... @job.with_lock do OrderMailer.with(order: order).fulfilled.deliver_acidic @job.update!(recovery_point: :finished) end if @job.recovery_point == :send_email end
65. include WithAcidity def perform(order) with_acidity do step :process_order step :fulfill_order step :send_emails end end def process_order; # ... end def fulfill_order; # ... end def send_emails; # ... end
66. module WithAcidity def perform_step(current_step_method, next_step_method) return unless @job.recovery_point == current_step_method @job.with_lock do method(current_step_method).call @job.update!(recovery_point: next_step_method) end end end
67. • Use a recovery key to keep track of which steps in a workflow job have been successfully completed • make each step ACIDic, and • keep the entire workflow job ACIDic ACIDic Jobs Level 3 Recap
68. ACIDic Jobs Level 4 — Step Batches
69. def perform(order) with_acidity do step :process_order step :fulfill_order, awaits: [ShopifyFulfillJob] step :send_emails end end
70. — Mike Perham “Batches are Sidekiq Pro's [tool to] create a set of jobs to execute in parallel and then execute a callback when all the jobs are finished.”
71. Parallel Executing + Workflow Blocking
72. Workflow Job A Shopify Fulfill Job A Workflow Job B Workflow Job A
73. • Use Sidekiq Batches to allow parallel, separately queued jobs to be used within a multi-step workflow • keep steps serially dependent • while allowing for parallelizatino ACIDic Jobs Level 4 Recap