Slide 1
Slide 1 text
Stephen Margheim — RubyConf 2021
ACIDic Jobs
A Layman's Guide to Job Bliss
Slide 2
Slide 2 text
Stephen Margheim — RubyConf 2021
ACIDic Jobs
A Layman's Guide to Job Bliss
Slide 3
Slide 3 text
Stephen Margheim
@fractaledmind
Slide 4
Slide 4 text
Jobs are essential
Slide 5
Slide 5 text
But what is a Job?
Slide 6
Slide 6 text
Job == Verb
Job != ActiveJob object
Slide 7
Slide 7 text
Job == State Mutation
Job != inspection or retrieval of information
Slide 8
Slide 8 text
Job => Side Effects
Job !=> return value
Slide 9
Slide 9 text
• a Ruby class object,
• representing a state mutation action,
• that takes as input a representation of initial state
• and produces side-effects representing a next state.
A job is
Slide 10
Slide 10 text
ActionDoer.call *arguments # service object
ActionJob.perform_now *arguments # active job
ActionWorker.new.perform *arguments # sidekiq worker
ActionOperation.run *arguments # operation class
Slide 11
Slide 11 text
Jobs must be
idempotent &
transactional
Slide 12
Slide 12 text
Operation 1 Operation 2 Operation 3
Transaction
Slide 13
Slide 13 text
• Atomicity: everything succeeds or everything fails
• Consistency: the data always ends up in a valid
state, as defined by your schema
• Isolation: concurrent transactions won't conflict
with each other
• Durability: once committed always committed,
even with system failures
The ACIDic Guarantees
Slide 14
Slide 14 text
Jobs · Databases
Slide 15
Slide 15 text
Idempotency
f(f(x)) == f(x)
Job.perform & Job.perform ==
Job.perform
Slide 16
Slide 16 text
• Functional Idempotency: the function always
returns the same result, even if called multiple
times
f(f(x)) == f(x)
• Practical Idempotency: the side-effect(s) will
happen once and only once, no matter how many
times the job is performed
Job.perform == Job.perform & Job.perform
The Idempotent Guarantee
Slide 17
Slide 17 text
Jobs · Retries
Slide 18
Slide 18 text
class JobRun < ActiveRecord::Base
# ...
end
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
• Use a transaction to guarantee atomic execution
• Use locks to prevent concurrent data access
• Use idempotency and retries to ensure eventual
completion
• Ensure enqueuing other jobs is co-transactional
• Split complex operations into steps
ACIDic Job Principles
— Nathan Griffith
Slide 21
Slide 21 text
ACIDic Jobs
Level 0 — Transactional Jobs
Slide 22
Slide 22 text
def perform(from_account, to_account, amount)
run = JobRun.find_or_create_by!(
job_class: self.class,
job_id: job_id,
job_args: [from_account, to_account, amount])
run.with_lock do
from_account.lock!
to_account.lock!
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
end
end
Slide 23
Slide 23 text
— Mike Perham
“Just remember that Sidekiq will
execute your job at least once,
not exactly once.”
Slide 24
Slide 24 text
• Use a database record to make job runs
transactional
• Use a database lock to mitigate concurrency issues
ACIDic Jobs Level 0 Recap
Slide 25
Slide 25 text
ACIDic Jobs
Level 1 — Idempotent Jobs
Slide 26
Slide 26 text
Idempotency & Uniqueness
To guarantee idempotency, we must be able to define and
identify the job uniquely
Slide 27
Slide 27 text
john_account.balance # initial state
# => 100_00
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
john_account.balance # resulting state
# => 80_00 or 90_00 ?
Slide 28
Slide 28 text
• each job run uses a generic unique entity
representing this job run
• each job run uses a generic unique entity
representing this job execution (based on args)
Forms of Job Uniqueness
Slide 29
Slide 29 text
def perform(from_account, to_account, amount)
run = JobRun.find_or_create_by!(job_class: self.class,
job_id: job_id)
run.with_lock do
return if run.completed?
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
run.update!(completed_at: Time.current)
end
end
Unique Job by Job Run
Slide 30
Slide 30 text
john_account.balance # initial state
# => 100_00
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
john_account.balance # resulting state
# => 80_00 ?
Slide 31
Slide 31 text
def perform(from_account, to_account, amount)
run = JobRun.find_or_create_by!(job_class: self.class,
job_args: [from_account, to_account, amount])
run.with_lock do
return if run.completed?
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
run.update!(completed_at: Time.current)
end
end
Unique Job by Execution Args
Slide 32
Slide 32 text
john_account.balance # initial state
# => 100_00
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
john_account.balance # resulting state
# => 90_00 ?
Slide 33
Slide 33 text
Unique Job flexibly and generically
class TransferBalanceJob < ApplicationJob
prepend UniqueByJobRun
uniquely_identified_by_job_id
# uniquely_identified_by_job_args
def perform(from_account, to_account, amount)
from_account.lock!
to_account.lock!
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
end
end
Slide 34
Slide 34 text
john_account.balance # initial state
# => 100_00
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
john_account.balance # resulting state
# => 80_00
Slide 35
Slide 35 text
Unique Job flexibly and generically
class TransferBalanceJob < ApplicationJob
prepend UniqueByJobRun
# uniquely_identified_by_job_id
uniquely_identified_by_job_args
def perform(from_account, to_account, amount)
from_account.lock!
to_account.lock!
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
end
end
Slide 36
Slide 36 text
john_account.balance # initial state
# => 100_00
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
TransferBalanceJob.perform_later(john_account, jane_account, 10_00)
john_account.balance # resulting state
# => 90_00
Slide 37
Slide 37 text
• Use a database record to make job runs idempotent
• custom or generic
• by job ID or by job args
ACIDic Jobs Level 1 Recap
Slide 38
Slide 38 text
ACIDic Jobs
Level 2 — Enqueuing other Jobs
Slide 39
Slide 39 text
uniquely_identified_by_job_id
def perform(from_account, to_account, amount)
from_account.lock!
to_account.lock!
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
TransferMailer.with(account: from_account).outbound.deliver_later
TransferMailer.with(account: to_account).inbound.deliver_later
end
Slide 40
Slide 40 text
Failure Condition 1
job
queue
from.save!
job
process
to.save! TransferMailer.deliver_later
transaction
commits
job starts job fails
job queued by web process and
dequeued by background worker
Slide 41
Slide 41 text
Failure Condition 2
job
queue
from.save!
job
process
to.save! TransferMailer.deliver_later
job starts job fails
rollback
job queued by web process and
dequeued by background worker
Slide 42
Slide 42 text
Solution Option 1
• A database-backed job queue
• delayed_job
• que
• good_job
But... no Sidekiq and increased db load
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
Solution Option 2
• A transactionally-staged job queue
So... more Sidekiq and minimal increased db load
Slide 45
Slide 45 text
uniquely_identified_by_job_id
def perform(from_account, to_account, amount)
from_account.lock!
to_account.lock!
from_account.update!(balance: from_account.balance - amount)
to_account.update!(balance: to_account.balance - amount)
TransferMailer.with(account: from_account).outbound.deliver_acidic
TransferMailer.with(account: to_account).inbound.deliver_acidic
end
Slide 46
Slide 46 text
def deliver_acidic(options = {})
job = delivery_job_class
attributes = { adapter: "activejob", job_name: job.name }
job_args = if job <= ActionMailer::Parameterized::MailDeliveryJob
[@mailer_class.name, @action, "deliver_now", {params: @params, args:
@args}]
else
[@mailer_class.name, @action, "deliver_now", @params, *@args]
end
attributes[:job_args] = job.new(job_args).serialize
StagedJob.create!(attributes)
end
Slide 47
Slide 47 text
class StagedJob < ActiveRecord::Base
after_create_commit :enqueue_job
def enqueue_job
case adapter
when "activejob"
ActiveJob::Base.deserialize(job_args).enqueue
when "sidekiq"
Sidekiq::Client.push("class" => job_name, "args" => job_args)
end
end
end
Slide 48
Slide 48 text
• Use transactionally-staged jobs to keep job
enqueuing co-transactional with standard database
operations
• while keeping Sidekiq, and
• not requiring an independent de-staging process
ACIDic Jobs Level 2 Recap
Slide 49
Slide 49 text
ACIDic Jobs
Level 3 — Operational Steps
Slide 50
Slide 50 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
Slide 51
Slide 51 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
Slide 52
Slide 52 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
Slide 53
Slide 53 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
Slide 54
Slide 54 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
Slide 55
Slide 55 text
def perform(order)
order.lock!
order.process_and_fulfill!
ShopifyAPI::Fulfillment.create!({
amount: order.amount,
customer: order.purchaser,
})
OrderMailer.with(order: order).fulfilled.deliver_acidic
end
?
Slide 56
Slide 56 text
No content
Slide 57
Slide 57 text
Workflow
step 1
step 2
step 3
Slide 58
Slide 58 text
Workflow — Run 1
step 1
step 2
step 3
Slide 59
Slide 59 text
Workflow — Run 2
step 1
step 2
step 3
Slide 60
Slide 60 text
Job-wise vs Step-wise
Idempotency
Slide 61
Slide 61 text
Job-wise vs Step-wise vs
Idempotency
Slide 62
Slide 62 text
uniquely_identified_by_job_args
def perform(order)
@job.with_lock do
order.process_and_fulfill!
@job.update!(recovery_point: :fulfill_order)
end if @job.recovery_point == :start
# ...
end
Slide 63
Slide 63 text
uniquely_identified_by_job_args
def perform(order)
# ...
@job.with_lock do
ShopifyAPI::Fulfillment.create!({ ... })
@job.update!(recovery_point: :send_email)
end if @job.recovery_point == :fulfill_order
# ...
end
Slide 64
Slide 64 text
uniquely_identified_by_job_args
def perform(order)
# ...
@job.with_lock do
OrderMailer.with(order: order).fulfilled.deliver_acidic
@job.update!(recovery_point: :finished)
end if @job.recovery_point == :send_email
end
Slide 65
Slide 65 text
include WithAcidity
def perform(order)
with_acidity do
step :process_order
step :fulfill_order
step :send_emails
end
end
def process_order; # ... end
def fulfill_order; # ... end
def send_emails; # ... end
Slide 66
Slide 66 text
module WithAcidity
def perform_step(current_step_method, next_step_method)
return unless @job.recovery_point == current_step_method
@job.with_lock do
method(current_step_method).call
@job.update!(recovery_point: next_step_method)
end
end
end
Slide 67
Slide 67 text
• Use a recovery key to keep track of which steps in a
workflow job have been successfully completed
• make each step ACIDic, and
• keep the entire workflow job ACIDic
ACIDic Jobs Level 3 Recap
Slide 68
Slide 68 text
ACIDic Jobs
Level 4 — Step Batches
Slide 69
Slide 69 text
def perform(order)
with_acidity do
step :process_order
step :fulfill_order, awaits: [ShopifyFulfillJob]
step :send_emails
end
end
Slide 70
Slide 70 text
— Mike Perham
“Batches are Sidekiq Pro's [tool to] create a set
of jobs to execute in parallel and then execute
a callback when all the jobs are finished.”
Slide 71
Slide 71 text
Parallel Executing
+
Workflow Blocking
Slide 72
Slide 72 text
Workflow Job
A
Shopify Fulfill
Job A
Workflow Job
B
Workflow Job
A
Slide 73
Slide 73 text
• Use Sidekiq Batches to allow parallel, separately
queued jobs to be used within a multi-step
workflow
• keep steps serially dependent
• while allowing for parallelizatino
ACIDic Jobs Level 4 Recap
Slide 74
Slide 74 text
No content
Slide 75
Slide 75 text
No content