Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Oleg Komarov
August 14, 2013
Programming
0
150
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
130
lpw-2012
komarov
1
300
yapc_eu_2012
komarov
2
400
Other Decks in Programming
See All in Programming
両面どころかインフラもTSでできるよ ~ 全方位TypeScriptによるプロダクト開発 ~
myfinder
9
3.2k
CA.swift19 恋するAIアプリ開発の裏側
oskmr
0
330
スクラムチームと認知負荷 - ニフティのスクラムトーク Vol2. / NIFTY Tech Talk #18
niftycorp
PRO
1
120
二郎系ラーメンのコールで学ぶ AST 解析
memory1994
PRO
7
1.6k
From Spring Boot 2 to Spring Boot 3 with Java 21 and Jakarta EE
ivargrimstad
0
1.1k
データアナリストが行うDatabricksを活用したETLの自動化事例
shinoa
0
250
Micro Frontends for Java Microservices - Devnexus 2024
mraible
PRO
0
420
SpringBoot+MyBatisで例外が出たときどこを見るか
syukai
0
110
Ruby Function Composition
bkuhlmann
1
330
ドメイン・ファーストで考える問題解決に役立つモデル設計 / Domain First Model Design
suzushin54
2
2.1k
PHPの次期バージョンはこの時期どうなっているのか - Internalsの開発体制について - PHPカンファレンス小田原
youkidearitai
PRO
1
180
チーム力を高めるスクラム実践法:カンバン公開と課題攻略について - ニフティのスクラムトーク Vol. 2 - NIFTY Tech Talk #18
niftycorp
PRO
1
110
Featured
See All Featured
Being A Developer After 40
akosma
56
580k
Docker and Python
trallard
33
2.7k
Intergalactic Javascript Robots from Outer Space
tanoku
266
26k
Building a Modern Day E-commerce SEO Strategy
aleyda
16
6.3k
Side Projects
sachag
451
41k
The Power of CSS Pseudo Elements
geoffreycrofte
59
5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
34
8.8k
A better future with KSS
kneath
231
16k
Designing on Purpose - Digital PM Summit 2013
jponch
110
6.4k
For a Future-Friendly Web
brad_frost
171
8.9k
Visualization
eitanlees
135
14k
Learning to Love Humans: Emotional Interface Design
aarron
266
39k
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov