The Final Crontab

About crontabber: https://github.com/mozilla/crontabber

Transcript

1. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/
2. crontabber
5. socorro1 socorro3 WAL Socorro1 .dev Socorro1. stage base_backup copy Sunday noon PT streaming rep Prod socorro2 backup4 base_backup & pg_dump backup reporting1 WAL socorro-db-zeus-rw socorro-db-zeus-ro very architecture very architecture such replicas such replicas wow wow
11. Tons more at: http://lqbs.fr/suchcomments/
13. http://github.com/mozilla/socorro
14. http://bit.ly/1fOgBSB
15. */5 * * * * socorro crontabber.sh
16. image by @CoryLoftis
17. Motivating factors
18. #ThreeWordHorrorStories
19. No unit tests
20. No unit tests
21. Bespoke shell scripts
22. Postgres stored procedures
23. Email from cron
24. 0 5000 10000 15000 20000 25000 Dec 5, 2010 May 5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014 Cron alert messages
26. Email from cron that you need to read.
28. Cron, what is it good for? • birthday reminders • status updates for a website • doxygen output for manuals every 12 hours • email nags about bugs filed wrong • ETL • Postgres -> Cloudwatch • Batch processing • Backups of RO DB • Machine heartbeat • “sweet fuck all” • “auto” updates • logging laptop IP • check for abandoned twitter accounts
29. Running jobs on a predictable schedule
30. How Socorro uses cron • Time-dependent reports or maintenance • “Simple” event detection and triggers • Status logging
31. Our use cases • Stored procedures for materialized views in Postgres • Daily map-reduces (largely deprecated) • FTP Scraping into Postgres • Bulk email responses to crash submissions pulled from Elastic Search
32. Jobs that don’t lend themselves to queue management because of time-dependencies, fragility or complexity.
33. crontabber https://github.com/mozilla/crontabber
34. On Github: Peter Bengtsson @peterbe & Lars Lohn @twobraids
35. pip install crontabber
36. Our crontabber jobs
41. configman https://github.com/mozilla/configman
42. Our config https://github.com/mozilla/socorro/blob/ master/config/crontabber.ini-dist
44. No more shell scripts
45. #!/bin/bash . /etc/socorro/socorrorc NAME=`basename $0 .sh` lock --ignore-existing $NAME ${PYTHON} ${APPDIR}/socorro/cron/crontabber.py \ --admin.conf=/etc/socorro/crontabber.ini \ >> /var/log/socorro/crontabber.log 2>&1 EXIT_CODE=$? unlock $NAME exit $EXIT_CODE
46. Retries on failure
47. Waits to run if a dependency fails
48. Nagios alerts
49. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085] socorroadm.stage.private.phx1.mozilla.com: Socorro Admin - crontab is CRITICAL: CRITICAL - correlations-addon-matview (CorrelationsAddonCronApp) (http://m.mozilla.org/Socorro+Admin+-+crontab)
50. Allow configurable number of failures before CRITICAL
51. Unit test framework for all jobs
52. Documented dependencies
55. Config can get hairy
56. One-off runs aren’t simple
57. Parallel execution coming soon! or...
58. */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber1.ini */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber2.ini */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber3.ini
59. crontabber as a module is running in our stage environment
60. Dependencies • Python 2.6 or higher • Postgres 9.2 or higher •
61. https://github.com/mozilla/crontabber Ping us in #breakpad on irc.mozilla.org Tune in: Tuesday June 10th at 7pm PDT at air.mozilla.com!
62. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/