Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Running Jobs at Scale
Search
Kir Shatrov
June 16, 2018
Programming
1
200
Running Jobs at Scale
My talk from GORUCO 2018 in New York.
Kir Shatrov
June 16, 2018
Tweet
Share
More Decks by Kir Shatrov
See All by Kir Shatrov
Operating Rails in Kubernetes
kirs
3
420
RailsClub 2016
kirs
2
300
Performance regressions in Ruby on Rails Core
kirs
0
200
Building a toolkit to detect performance regressions in Ruby on Rails core
kirs
3
5.4k
Развертывание веб-приложений и фреймворк Capistrano
kirs
1
270
Capistrano 3
kirs
4
2.7k
Other Decks in Programming
See All in Programming
How mixi2 Uses TiDB for SNS Scalability and Performance
kanmo
7
3.8k
Ruby on cygwin 2025-02
fd0
0
130
知られざるDMMデータエンジニアの生態 〜かつてツチノコと呼ばれし者〜
takaha4k
4
1.2k
ゼロからの、レトロゲームエンジンの作り方
tokujiros
3
1.2k
昭和の職場からアジャイルの世界へ
kumagoro95
1
340
iOSエンジニアから始める visionOS アプリ開発
nao_randd
3
120
Grafana Cloudとソラカメ
devoc
0
130
[Fin-JAWS 第38回 ~re:Invent 2024 金融re:Cap~]FaultInjectionServiceアップデート@pre:Invent2024
shintaro_fukatsu
0
400
ARA Ansible for the teams
kksat
0
120
『GO』アプリ バックエンドサーバのコスト削減
mot_techtalk
0
120
Conform を推す - Advocating for Conform
mizoguchicoji
3
670
Rails アプリ地図考 Flush Cut
makicamel
1
110
Featured
See All Featured
Agile that works and the tools we love
rasmusluckow
328
21k
Music & Morning Musume
bryan
46
6.3k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
31
2.1k
What's in a price? How to price your products and services
michaelherold
244
12k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
99
18k
Done Done
chrislema
182
16k
Scaling GitHub
holman
459
140k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
356
29k
Transcript
Running Jobs at Scale Kir Shatrov GORUCO 2018, @kirshatrov
GORUCO 2018, @kirshatrov
GORUCO 2018, @kirshatrov
GORUCO 2018, @kirshatrov
GORUCO 2018, @kirshatrov
Jobs GORUCO 2018, @kirshatrov
class ExampleJob < ActiveJob::Base def perform ... end end GORUCO
2018, @kirshatrov
class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh
end end end GORUCO 2018, @kirshatrov
class ExampleJob < ActiveJob::Base def perform Product.all.find_each do |product| product.sync_and_refresh
end end end minutes? hours? days? GORUCO 2018, @kirshatrov
Long-running jobs GORUCO 2018, @kirshatrov
Long-running jobs — Deploys and termination GORUCO 2018, @kirshatrov
Long-running jobs — Deploys and termination — Abort and re-enqueue
— Progress lost GORUCO 2018, @kirshatrov
GORUCO 2018, @kirshatrov
Long-running jobs — Deploys and termination — Abort and re-enqueue
— Progress lost — Job may never complete GORUCO 2018, @kirshatrov
Long-running jobs — Deploys and termination — Abort and re-enqueue
— Progress lost — Job may never complete — Capacity and worker starvation GORUCO 2018, @kirshatrov
Long-running jobs — Deploys and termination — Abort and re-enqueue
— Progress lost — Job may never complete — Capacity and worker starvation — Cloud ☁ GORUCO 2018, @kirshatrov
Why is it taking long? Because it iterates over a
long collection. GORUCO 2018, @kirshatrov
What if jobs were interruptible and resumable? GORUCO 2018, @kirshatrov
Split the job definition 1. Collection to process 2. Work
to be done GORUCO 2018, @kirshatrov
Split the job definition 1. Collection to process ≫ Product.all
2. Work to be done GORUCO 2018, @kirshatrov
Split the job definition 1. Collection to process ≫ Product.all
2. Work to be done ≫ product.sync_and_refresh GORUCO 2018, @kirshatrov
class ExampleJob < ActiveJob::Base include Iteration def collection Product.all end
def each_iteration(product) product.sync_and_refresh end end GORUCO 2018, @kirshatrov
— def perform — collection — each_iteration GORUCO 2018, @kirshatrov
Product.all cursor: 1 GORUCO 2018, @kirshatrov
Product.all cursor: 2 GORUCO 2018, @kirshatrov
Product.all cursor: 3 GORUCO 2018, @kirshatrov
Product.all cursor: 4 GORUCO 2018, @kirshatrov
Product.all cursor: 5 GORUCO 2018, @kirshatrov
Product.all cursor: 450123 GORUCO 2018, @kirshatrov
class WhateverJob < ActiveJob::Base include Iteration def collection Enumerator.new do
|enum| 3.times do |n| enum << n end end end def each_iteration(n) # do something three times! end end GORUCO 2018, @kirshatrov
Endless possibilities — Interrupt and resume at any moment —
Progress tracking — Parallel computations — Throttling by default GORUCO 2018, @kirshatrov
Benefits for the infrastructure — Keep supporting long-running jobs —
Success for Cloud runtime — Make scale invisible for developers — Opportunities to save money with short-living instances in Cloud GORUCO 2018, @kirshatrov
Thank you! @kirshatrov GORUCO 2018, @kirshatrov