Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
yapc::eu 2013
Search
Oleg Komarov
August 14, 2013
Programming
0
190
yapc::eu 2013
reliable cron jobs in distributed environment
Oleg Komarov
August 14, 2013
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
Exploring Plack Middlewares
komarov
0
170
lpw-2012
komarov
1
330
yapc_eu_2012
komarov
2
530
Other Decks in Programming
See All in Programming
顧客の画像データをテラバイト単位で配信する 画像サーバを WebP にした際に起こった課題と その対応策 ~継続的な取り組みを添えて~
takutakahashi
4
1.3k
SQLアンチパターン第2版 データベースプログラミングで陥りがちな失敗とその対策 / Intro to SQL Antipatterns 2nd
twada
PRO
11
1.6k
DMMを支える決済基盤の技術的負債にどう立ち向かうか / Addressing Technical Debt in Payment Infrastructure
yoshiyoshifujii
3
410
GPUを計算資源として使おう!
primenumber
1
250
『自分のデータだけ見せたい!』を叶える──Laravel × Casbin で複雑権限をスッキリ解きほぐす 25 分
akitotsukahara
2
660
Git Sync を超える!OSS で実現する CDK Pull 型デプロイ / Deploying CDK with PipeCD in Pull-style
tkikuc
4
350
バイブコーディング超えてバイブデプロイ〜CloudflareMCPで実現する、未来のアプリケーションデリバリー〜
azukiazusa1
0
340
マッチングアプリにおけるフリックUIで苦労したこと
yuheiito
0
190
なぜ「共通化」を考え、失敗を繰り返すのか
rinchoku
1
680
初学者でも今すぐできる、Claude Codeの生産性を10倍上げるTips
s4yuba
16
13k
Claude Code + Container Use と Cursor で作る ローカル並列開発環境のススメ / ccc local dev
kaelaela
12
7k
TypeScriptでDXを上げろ! Hono編
yusukebe
3
770
Featured
See All Featured
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
282
13k
The Cost Of JavaScript in 2023
addyosmani
51
8.6k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
It's Worth the Effort
3n
185
28k
Writing Fast Ruby
sferik
628
62k
Navigating Team Friction
lara
187
15k
Producing Creativity
orderedlist
PRO
346
40k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
340
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
21
1.3k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
32
2.4k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.6k
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2013-08-14
What this talk is about decentralizing background job execution maintaining
concurrency (think locks and leases)
Figure: not decentralized
Figure: decentralized
Problems to solve avoid duplicate processes (locking) avoid resource conflicts
(leasing) bear in mind collecting logs monitoring
Common pitfalls of distributed locks no lock cleanup on process
death operating not actually holding the lock
Apache ZooKeeper http://zookeeper.apache.org a distributed, open-source coordination service for distributed
applications reliable, replicated, trx-ordered, single system image operations: create, get, set, get children, delete, exists features: watchers, ephemeral nodes, sequential nodes
Sequential nodes $zkh->create( "/path/basename-", $data, flags => ZOO_SEQUENTIAL ) for
(1 .. 3); creates /path/basename-0000000001 /path/basename-0000000002 /path/basename-0000000003
Synchronization http://zookeeper.apache.org/doc/r3.4.5/recipes.html CPAN Net::ZooKeeper::Lock Net::ZooKeeper::Semaphore
Non-blocking exclusive lock $zkh->create($lock_path, $node_data, acl => ZOO_OPEN_ACL_UNSAFE, flags =>
ZOO_EPHEMERAL, ); if (my $error = $zkh->get_error) { if ($error == ZNODEEXISTS) { return undef; # the lock is held by someone else } else { die "Could not acquire lock $lock_path: $error"; } } # success return $zkh->exists($lock_path, watch => $lock_watch);
Figure: lock supervising
Figure: switchman
switchman a flock-style utility for distributed locks and semaphores https://github.com/komarov/switchman
https://metacpan.org/release/switchman
Features reliable locking fair resource leasing (no starvation) lease name
macros server groups
Resources global DB API utilization local cpu memory io local
as global cpu@serverA memory@serverA
Macros names FQDN cpu FQDN mem values 1:CPU 4096:MEMMB
Figure: control flow
How to setup install switchman create a local configuration file
create a znode in zookeeper modify your crontabs
Configuration /etc/switchman.conf: { "prefix":"/switchman", "zkhosts":"zk1:2181,zk2:2181,zk3:2181", "loglevel":"info", "logfile":"/path/to/log" }
ZooKeeper znode layout /switchman /locks /queues /semaphores
Configuration /switchman znode data: { "groups":{"grp1":["host1","host2"],"grp2":"host1"}, "resources":["FQDN_mem","FQDN_cpu"] }
Usage switchman COMMAND # lockname == command’s basename switchman --lockname
LOCK -- COMMAND [ARGS] switchman --lease FQDN_cpu=1:CPU COMMAND switchman --group grp2 COMMAND # runs only on host1
Left out of scope log aggregation monitoring
Thank you! Any questions? If you really liked the talk,
don’t hesitate to like my quest :) http://questhub.io/player/komarov