Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Oleg Komarov
August 14, 2013
Programming
0
190
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
170
lpw-2012
komarov
1
330
yapc_eu_2012
komarov
2
530
Other Decks in Programming
See All in Programming
Blazing Fast UI Development with Compose Hot Reload (droidcon New York 2025)
zsmb
1
290
新メンバーも今日から大活躍!SREが支えるスケールし続ける組織のオンボーディング
honmarkhunt
5
7.2k
チームで開発し事業を加速するための"良い"設計の考え方 @ サポーターズCoLab 2025-07-08
agatan
1
420
AIともっと楽するE2Eテスト
myohei
6
2.6k
PostgreSQLのRow Level SecurityをPHPのORMで扱う Eloquent vs Doctrine #phpcon #track2
77web
2
530
ふつうの技術スタックでアート作品を作ってみる
akira888
1
840
Python型ヒント完全ガイド 初心者でも分かる、現代的で実践的な使い方
mickey_kubo
1
110
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
0
190
猫と暮らす Google Nest Cam生活🐈 / WebRTC with Google Nest Cam
yutailang0119
0
120
PipeCDのプラグイン化で目指すところ
warashi
1
270
テストから始めるAgentic Coding 〜Claude Codeと共に行うTDD〜 / Agentic Coding starts with testing
rkaga
12
4.4k
dbt民主化とLLMによる開発ブースト ~ AI Readyな分析サイクルを目指して ~
yoshyum
3
1k
Featured
See All Featured
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
The Language of Interfaces
destraynor
158
25k
Making the Leap to Tech Lead
cromwellryan
134
9.4k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.4k
Git: the NoSQL Database
bkeepers
PRO
430
65k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
6
300
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov