Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
lpw-2012
Search
Oleg Komarov
November 24, 2012
Programming
1
320
lpw-2012
Reliable Cron Jobs in Distributed Environment
Oleg Komarov
November 24, 2012
Tweet
Share
More Decks by Oleg Komarov
See All by Oleg Komarov
yapc::eu 2013
komarov
0
170
Exploring Plack Middlewares
komarov
0
140
yapc_eu_2012
komarov
2
500
Other Decks in Programming
See All in Programming
15分で学ぶDuckDBの可愛い使い方 DuckDBの最近の更新
notrogue
3
820
Honoのおもしろいミドルウェアをみてみよう
yusukebe
1
240
5分で理解する SOLID 原則 #phpcon_nagoya
shogogg
1
410
pylint custom ruleで始めるレビュー自動化
shogoujiie
0
160
Lambdaの監視、できてますか?Datadogを用いてLambdaを見守ろう
nealle
2
730
クリーンアーキテクチャから見る依存の向きの大切さ
shimabox
5
1.1k
Go 1.24でジェネリックになった型エイリアスの紹介
syumai
2
300
TCAを用いたAmebaのリアーキテクチャ
dazy
0
220
苦しいTiDBへの移行を乗り越えて快適な運用を目指す
leveragestech
0
1.2k
楽しく向き合う例外対応
okutsu
0
730
Jakarta EE meets AI
ivargrimstad
0
650
推しメソッドsource_locationのしくみを探る - はじめてRubyのコードを読んでみた
nobu09
2
350
Featured
See All Featured
Building Your Own Lightsaber
phodgson
104
6.2k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
100
18k
How to Ace a Technical Interview
jacobian
276
23k
Code Review Best Practice
trishagee
67
18k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
13
1k
Testing 201, or: Great Expectations
jmmastey
42
7.2k
Git: the NoSQL Database
bkeepers
PRO
428
65k
Fontdeck: Realign not Redesign
paulrobertlloyd
83
5.4k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
10
540
StorybookのUI Testing Handbookを読んだ
zakiyama
28
5.5k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.5k
Writing Fast Ruby
sferik
628
61k
Transcript
Reliable Cron Jobs in Distributed Environment Oleg Komarov 2012-11-24 1/26
Presentation available at https://speakerdeck.com/komarov/lpw-2012 http://bit.ly/VLuT6g 2/26
Context 3 independent projects with shared infrastructure • over 30
boxes • over 200 scripts, 30K+ SLOC • packaged in appr. 20 deb-packages 3/26
TL;DR Reliable Cron Jobs in Distributed Environment 4/26
TL;DR Reliable Cron Jobs in Distributed Environment ... are HARD
to get right 4/26
Cron Jobs in a Vacuum • locks • logging and
output • monitoring • profiling 5/26
Logging and Output • log START and FINISH • log
enough details 6/26
Logging and Output • log START and FINISH • log
enough details • use log + STDERR for important things • use MAILTO to catch that output 6/26
Logging and Output • log START and FINISH • log
enough details • use log + STDERR for important things • use MAILTO to catch that output 6/26
Monitoring • be confident that it actually works • it
must not fail when you system fails • have a plan of action 7/26
Monitoring • be confident that it actually works • it
must not fail when you system fails • have a plan of action 7/26
What to monitor • hardware errors • free disk space
• load • crond is alive • age of generated file, queue size, etc. 8/26
Profiling • Does it need 1GB or 10GB? • What
does it take so long to complete? • How many db queries does it run? 9/26
Profiling • Does it need 1GB or 10GB? • What
does it take so long to complete? • How many db queries does it run? Measure and improve 9/26
More to consider • crash-safe • documentation • parallel execution
• resource limits (ulimit/cgroups) 10/26
Deployment Packages 11/26
Deployment Boxes 12/26
Cron Package Just populate my-project-scriptsN.cron.d file 13/26
Cron Package Just populate my-project-scriptsN.cron.d file Don’t write it by
hand, do it automatically 13/26
Cron Package Just populate my-project-scriptsN.cron.d file Don’t write it by
hand, do it automatically Put some METADATA in your scripts 13/26
Metadata =head1 METADATA <crontab> package: scriptsN params: --mod 2 --rem
0 time: */2 * * * * </crontab> <crontab> package: scriptsN params: --mod 2 --rem 1 time: */2 * * * * </crontab> =cut 14/26
Simple Setup As simple as possible: one box per package
15/26
Simple Setup As simple as possible: one box per package
apt-get purge && kill (or wait) && apt-get install 15/26
!%*#$ Back to Earth Network 16/26
With Extra Boxes Now you have some promblems to solve:
• locks • logs • load 17/26
Net::ZooKeeper::Lock Apache ZooKeeperTM is an effort to develop and maintain
an open-source server which enables highly reliable distributed coordination. Net::ZooKeeper::Lock implements distributed locks via ZooKeeper. 18/26
Introducing Switchman https://github.com/komarov/switchman 19/26
Overview 20/26
Configuration Crontabs are installed everywhere, switchman consults with config in
ZooKeeper: { "groups": { "scripts1": "box1", "scripts2": "box1", "scripts3": ["box1", "box2"] } } 21/26
Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS 22/26
Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS •
checks configuration • acquires a lock • watches configuration for changes • stops execution when it is not allowed anymore 22/26
Description switchman --config /how/to/connect/to/zk --group scriptsN -- CMD ARGS •
checks configuration • acquires a lock • watches configuration for changes • stops execution when it is not allowed anymore Easy to adopt with METADATA 22/26
One Problem Solved • locks • logs • load 23/26
Further Steps See facebook’s Scribe for collecting decentralized logs Resources
reservation and management A good monitoring system 24/26
Thanks! Questions? https://speakerdeck.com/komarov/lpw-2012 http://bit.ly/VLuT6g http://about.me/komarov om 25/26
Bonus Slide Get file age: # in days perl -E
’say -M $ARGV[0]’ /path/to/file # in seconds expr ‘date +%s‘ - ‘date +%s -r /path/to/file‘ Simple local locks: use Pid::File::Flock qw/:auto/; 26/26