Share
Embed
Slide 1
Slide 1 text
No content
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
Alexander Baygeldin, Evil Martians
Prioritization justice:
lessons from making background jobs fair at scale
πππ
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
π§π³ π§π³ π§π³
Slide 7
Slide 7 text
π§π³ π§π³ π§π³
π
π
π
Slide 8
Slide 8 text
π§π³ π§π³ π§π³
π
π
π
Slide 9
Slide 9 text
π§π³ π§π³ π§π³
π
π
Slide 10
Slide 10 text
π§π³ π§π³ π§π³
π
π
π
Slide 11
Slide 11 text
π§π³ π§π³ π§π³
π
π
π
Slide 12
Slide 12 text
π§π³ π§π³ π§π³
π
π
Latency = the time between placing
the order and starting to prepare it
Slide 13
Slide 13 text
π§π³ π§π³ π§π³
π
π
π§π³ π§π³
Slide 14
Slide 14 text
π§π³ π§π³ π§π³
π
π
π§π³ π§π³
π§’ π§’
part-time chefs
Slide 15
Slide 15 text
π§π³ π§π³ π§π³
π
π
π§π³ π§π³
π§’ π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’ π§π³
π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’
part-time chefs
Slide 16
Slide 16 text
π§π³ π§π³ π§π³
π
π
π§π³ π§π³
π§’ π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’ π§π³
π§’
π§π³
π§’
π§π³
π§’
π§π³
π§’
part-time chefs
At some point improving Quality of Service at
more operational cost leads to diminishing returns
Slide 17
Slide 17 text
π§π³ π§π³ π§π³
π
π
π
Slide 18
Slide 18 text
π§π³ π§π³ π§π³
π
π
π
Slide 19
Slide 19 text
π
π
π
π§π³ π§π³ π§π³
Slide 20
Slide 20 text
π§π³ π§π³ π§π³
The greater the latency, the greater
the need to treat customers fairly
π
π
Slide 21
Slide 21 text
G =
n
β
i=1
n
β
j=1
xi
β xj
2n2 Β―
x
=
n
β
i=1
n
β
j=1
xi
β xj
2n
n
β
i=1
xi
π₯
(x1
, x2
, β¦, xn
) =
(βn
i=1
xi
)2
n β
βn
i=1
xi
2
=
x2
x2
=
1
1 + Μ
cv
2
Jain's index? Gini index?
Some other guy's index?
Slide 22
Slide 22 text
How do we know if we're not?
β FIFO queue
β High latency
β Greedy users
Are we OK?
Slide 23
Slide 23 text
Customer feedback!
Slide 24
Slide 24 text
π
π
π
π
π
π
π
Slide 25
Slide 25 text
π
π
π
π
π
π
π
Slide 26
Slide 26 text
π
π
π
π
π
π
π
Slide 27
Slide 27 text
π
π
π
π
π
π
π
Slide 28
Slide 28 text
π
π
π
π
π
π
π
Slide 29
Slide 29 text
π
π
π
π
π
π
π
Slide 30
Slide 30 text
π
π
π
π
π
π
π
Slide 31
Slide 31 text
π
π
π
π
π
π
π
Slide 32
Slide 32 text
π
π
π
π
π
π
π
Slide 33
Slide 33 text
π
π
π
π
π
π
π
Slide 34
Slide 34 text
π§π³ π§π³ π§π³
π
π
π
Slide 35
Slide 35 text
π§π³ π§π³ π§π³
π
π π
Slide 36
Slide 36 text
π§π³ π§π³ π§π³
π
π π
π²
Slide 37
Slide 37 text
π§π³ π§π³ π§π³
π
π π
π²
π
π
π
π
π
Slide 38
Slide 38 text
π§π³ π§π³ π§π³
π
π π
π²
π
π
π
π
π
Slide 39
Slide 39 text
π§π³ π§π³ π§π³
BRPOP queue:tenant_1 queue:tenant_2 ... queue:tenant_N 2
timeout
per-tenant queues
Slide 40
Slide 40 text
π§π³ π§π³ π§π³
LMOVE queue:tenant_1 queue:sq||tenant_1 RIGHT LEFT
LMOVE queue:tenant_2 queue:sq||tenant_2 RIGHT LEFT
...
LMOVE queue:tenant_N queue:sq||tenant_N RIGHT LEFT
worker's "in-progress" queues
per-tenant queues
Slide 41
Slide 41 text
π§π³ π§π³ π§π³
Solid Queue
SELECT job_id
FROM solid_queue_ready_executions
WHERE queue_name = 'tenant_N'
ORDER BY priority ASC, job_id ASC
LIMIT ?
FOR UPDATE SKIP LOCKED;
Slide 42
Slide 42 text
π§π³ π§π³ π§π³
GoodJob π
WITH rows AS MATERIALIZED (
SELECT id, active_job_id FROM good_jobs
WHERE queue_name = 'tenant_N' AND ()
ORDER BY priority DESC NULLS LAST, created_at ASC
LIMIT ?
)
SELECT id FROM rows
WHERE pg_try_advisory_lock()
LIMIT 1
Slide 43
Slide 43 text
π§π³ π§π³ π§π³
Slide 44
Slide 44 text
Thinking about building your own
background job processor
Actually building one
Slide 45
Slide 45 text
(with what we have)
1. Shu
ff
l
e-sharding
2. Interruptible iteration
3. Throttling
4. Per-tenant queues
Let's fix this!
Slide 46
Slide 46 text
π§π³ π§π³
π§π³
π
π
π π
π
π
Slide 47
Slide 47 text
π§π³ π§π³
π§π³
Alex π
A to I J to R S to Z
Amy π
Joe π Sam π
Sam π
Amy π
Slide 48
Slide 48 text
π§π³ π§π³
π§π³
Alex π
A to I J to R S to Z
Amy π
Joe π Sam π
Sam π
Amy π
Slide 49
Slide 49 text
π§π³ π§π³
π§π³
A to I J to R S to Z
Amy π
Amy π
π€
Sam π
Slide 50
Slide 50 text
π§π³
π· Pentagon π
π§π³
π§π³
A to I J to R S to Z
Amy π
Amy π
Sam π
π· Pentagon π
Slide 51
Slide 51 text
π§π³
π· Pentagon π
π§π³
π§π³
A to I J to R S to Z
Amy π
Amy π
Sam π
π· Pentagon π
Slide 52
Slide 52 text
π§π³
π· Pentagon π
π§π³
π§π³
A to I J to R S to Z
Amy π π· Pentagon π
π€
Joe π
Slide 53
Slide 53 text
Shuffle-sharding
(tl;dr: good when you have enough workload to
fi
ll all shards)
No hogging? No, but it's times less likely.
Perfectly fair? No, but it a
ff
ects fewer people when it's not fair.
Full resource
utilization?
Noβunderutilization is possible with too many shards.
Does it scale? Yes, but it needs at least one worker per shard.
Slide 54
Slide 54 text
π§π³ π§π³ π§π³
π
π
π
Slide 55
Slide 55 text
π§π³ π§π³ π§π³
π
π
π π π
π
Slide 56
Slide 56 text
π§π³ π§π³ π§π³
π π π
Slide 57
Slide 57 text
π§π³ π§π³ π§π³
π π π
π€
π€
π§
Slide 58
Slide 58 text
π§π³ π§π³ π§π³
π π π
π€
π€
π§
Slide 59
Slide 59 text
π§π³ π§π³ π§π³
π π π
π
π π
Slide 60
Slide 60 text
π§π³ π§π³ π§π³
π π π
π
π π
π π
Slide 61
Slide 61 text
π§π³ π§π³ π§π³
π
π π
π π
Slide 62
Slide 62 text
π§π³ π§π³ π§π³
π
π π
π π
Slide 63
Slide 63 text
π§π³ π§π³ π§π³
π π
π π
π π π π
π π π
Slide 64
Slide 64 text
Interruptible iteration
(tl;dr: good when the workload comes in large batches)
No hogging? Noβit could happen if one tenant enqueues multiple batches.
Perfectly fair? Noβit stops being fair when someone hogs the queue.
Full resource
utilization?
Yes, but must be greater than .
Does it scale? Yesβusing cursors to track progress is cheap.
Slide 65
Slide 65 text
π§π³ π§π³ π§π³
π
Slide 66
Slide 66 text
π§π³ π§π³ π§π³
π
π
π
π
Slide 67
Slide 67 text
πͺ£
πΏ
π³
Slide 68
Slide 68 text
πͺ£
πΏ
π³
π§
π§
π§
π§
π§
π§
π§
π§
π§
over time, the bucket leaks
allowing further orders
bucket has a limited capacity
each new order
fi
lls the bucket
Slide 69
Slide 69 text
πͺ£
πΏ
π³
π§
π§
π§
π§
π§
π§
π§
π§
π¦
π¦
uh-oh, over
fl
ow!
over time, the bucket leaks
allowing further orders
bucket has a limited capacity
each new order
fi
lls the bucket
Slide 70
Slide 70 text
π§π³ π§π³ π§π³
π
π
π
π
Slide 71
Slide 71 text
π§π³ π§π³ π§π³
π
π
π
π
excess orders are put here
Slide 72
Slide 72 text
π§π³ π§π³ π§π³
π
π
π
π
π²
Slide 73
Slide 73 text
π§π³ π§π³ π§π³
π
π
π
π
π²
default queue
(80% chance to be processed)
throttled queue
(20% chance to be processed)
Slide 74
Slide 74 text
π§π³ π§π³ π§π³
π
π
π
π
π²
default queue
(80% chance to be processed)
throttled queue
(20% chance to be processed)
π
Slide 75
Slide 75 text
π§π³ π§π³ π§π³
π π
π²
default queue
(80% chance to be processed)
throttled queue
(20% chance to be processed)
π
Slide 76
Slide 76 text
π§π³ π§π³ π§π³
π π
π²
π
orders stuck in
the throttled queue :(
Slide 77
Slide 77 text
π§π³ π§π³ π§π³
π π
π²
π
Slide 78
Slide 78 text
π§π³ π§π³ π§π³
π
π²
π
no longer throttled
(100% chance to be processed)
Slide 79
Slide 79 text
Throttling
(tl;dr: good when the workload is well distributed over time)
No hogging? Yesβeveryone will get at least a little work done.
Perfectly fair? Noβespecially if the workload is bursty by nature.
Full resource
utilization?
Yes, but it requires weighted queues support.
Does it scale? Yesβespecially with leaky buckets.
Slide 80
Slide 80 text
π§π³ π§π³ π§π³
π
π
Slide 81
Slide 81 text
π§π³ π§π³ π§π³
π
π
Slide 82
Slide 82 text
π§π³ π§π³ π§π³
π
π¨πΌ
π
π
Slide 83
Slide 83 text
π§π³ π§π³ π§π³
π
π¨πΌπ
π
Slide 84
Slide 84 text
π§π³ π§π³ π§π³
π
π¨πΌπ
π
π€
Slide 85
Slide 85 text
π§π³ π§π³ π§π³
π¨πΌπ
π
βΌ
Slide 86
Slide 86 text
π
π
π π§π³ π§π³ π§π³
π¨πΌ
π
π
π
Slide 87
Slide 87 text
π§π³ π§π³ π§π³
π¨πΌπ
π
π
π
π€
bottleneck!
Slide 88
Slide 88 text
π§π³ π§π³ π§π³
π
π
π
π
π¨πΌ
π§
Slide 89
Slide 89 text
π§π³ π§π³ π§π³
π
π
π
π
π§
Slide 90
Slide 90 text
π§π³ π§π³ π§π³
π
π
π
π
π§
Slide 91
Slide 91 text
π
π π
π²
π
π
π
π
π
π§π³ π§π³ π§π³
π
π
π
π
π§
Slide 92
Slide 92 text
Per-tenant queues + custom scheduler
(tl;dr: good when the workload is heavy enough to forget about the bottleneck)
No hogging? Yesβit's basically communism.
Perfectly fair? Yesβas fair as you care to make it.
Full resource
utilization?
Yesβwith zero changes to the underlying infra.
Does it scale? It depends... on how e
ff i
cient your scheduler is.
Slide 93
Slide 93 text
(where fairness matter)
β¨ AI work
fl
ows *
β¬ Data imports
πΌ Media processing
π Reports generation
Examples!
... and more!
`
* less so with async-job
Slide 94
Slide 94 text
So, which one?
Shu
ff l
e-
sharding
Interruptible
iteration
Throttling
Per-tenant
queues
No hogging? β β β
β
Perfectly fair? β β β β
Full resource
utilization?
π§ π§ β
β
Does it scale? β
β
β
π§
Slide 95
Slide 95 text
(part 1)
β sidekiq-fairplay gem β
π§β sidekiq-fairplay gem π§β
π€ sidekiq-fairplay gem π€
Resources
Slide 96
Slide 96 text
(part 2)
β’ Workload Isolation with Queue
Sharding (by Mike Perham)
β’ Workload Isolation Using Shu
ff l
e-
Sharding (by Colm MacCΓ‘rthaigh)
Resources
Slide 97
Slide 97 text
(part 3)
β’ job-iteration gem (from Shopify)
β’ Sidekiq Iterable Jobs: With Great
Power.... (by Jon Sully)
β’ Active Job Continuations
Resources
Slide 98
Slide 98 text
(part 4)
β’ βFairβ multi-tenant prioritization
of Sidekiq jobsβand our gem for
it! (by Andrey Novikov)
β’ The unreasonable e
ff
ectiveness
of leaky buckets (and how to
make one) (by Julik Tarkhanov)
Resources
Slide 99
Slide 99 text
(part 5)
β’ faqueue: researching
background jobs fairness (by
Vladimir Dementyev)
β’ fairway gem
Resources
Slide 100
Slide 100 text
No content
Slide 101
Slide 101 text
put this in your address bar
Slide 102
Slide 102 text
No content