Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
Oleg Komarov
August 14, 2013
Programming
0
200
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
190
lpw-2012
komarov
1
350
yapc_eu_2012
komarov
2
560
Other Decks in Programming
See All in Programming
例外処理とどう使い分ける?Result型を使ったエラー設計 #burikaigi
kajitack
16
6.1k
登壇資料を作る時に意識していること #登壇資料_findy
konifar
4
1.7k
AgentCoreとHuman in the Loop
har1101
5
250
余白を設計しフロントエンド開発を 加速させる
tsukuha
7
2.1k
CSC307 Lecture 09
javiergs
PRO
1
840
Fluid Templating in TYPO3 14
s2b
0
130
KIKI_MBSD Cybersecurity Challenges 2025
ikema
0
1.3k
AIフル活用時代だからこそ学んでおきたい働き方の心得
shinoyu
0
140
CSC307 Lecture 04
javiergs
PRO
0
660
FOSDEM 2026: STUNMESH-go: Building P2P WireGuard Mesh Without Self-Hosted Infrastructure
tjjh89017
0
180
CSC307 Lecture 02
javiergs
PRO
1
780
MDN Web Docs に日本語翻訳でコントリビュート
ohmori_yusuke
0
660
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
59
6.3k
Fashionably flexible responsive web design (full day workshop)
malarkey
408
66k
Building AI with AI
inesmontani
PRO
1
710
Leveraging LLMs for student feedback in introductory data science courses - posit::conf(2025)
minecr
0
160
Music & Morning Musume
bryan
47
7.1k
What’s in a name? Adding method to the madness
productmarketing
PRO
24
3.9k
Why Our Code Smells
bkeepers
PRO
340
58k
The Pragmatic Product Professional
lauravandoore
37
7.1k
Evolving SEO for Evolving Search Engines
ryanjones
0
130
Deep Space Network (abreviated)
tonyrice
0
65
Public Speaking Without Barfing On Your Shoes - THAT 2023
reverentgeek
1
310
HDC tutorial
michielstock
1
390
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov