The Final Crontab

The Final Crontab

4535e53ad45275fa955c5b05684342c5?s=128

Selena Deckelmann

May 06, 2014
Tweet

Transcript

  1. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/

  2. crontabber

  3. None
  4. None
  5. socorro1 socorro3 WAL Socorro1 .dev Socorro1. stage base_backup copy Sunday

    noon PT streaming rep Prod socorro2 backup4 base_backup & pg_dump backup reporting1 WAL socorro-db-zeus-rw socorro-db-zeus-ro very architecture very architecture such replicas such replicas wow wow
  6. None
  7. None
  8. None
  9. None
  10. None
  11. Tons more at: http://lqbs.fr/suchcomments/

  12. None
  13. http://github.com/mozilla/socorro

  14. http://bit.ly/1fOgBSB

  15. */5 * * * * socorro crontabber.sh

  16. image by @CoryLoftis

  17. Motivating factors

  18. #ThreeWordHorrorStories

  19. No unit tests

  20. No unit tests

  21. Bespoke shell scripts

  22. Postgres stored procedures

  23. Email from cron

  24. 0 5000 10000 15000 20000 25000 Dec 5, 2010 May

    5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014 Cron alert messages
  25. None
  26. Email from cron that you need to read.

  27. None
  28. Cron, what is it good for? • birthday reminders •

    status updates for a website • doxygen output for manuals every 12 hours • email nags about bugs filed wrong • ETL • Postgres -> Cloudwatch • Batch processing • Backups of RO DB • Machine heartbeat • “sweet fuck all” • “auto” updates • logging laptop IP • check for abandoned twitter accounts
  29. Running jobs on a predictable schedule

  30. How Socorro uses cron • Time-dependent reports or maintenance •

    “Simple” event detection and triggers • Status logging
  31. Our use cases • Stored procedures for materialized views in

    Postgres • Daily map-reduces (largely deprecated) • FTP Scraping into Postgres • Bulk email responses to crash submissions pulled from Elastic Search
  32. Jobs that don’t lend themselves to queue management because of

    time-dependencies, fragility or complexity.
  33. crontabber https://github.com/mozilla/crontabber

  34. On Github: Peter Bengtsson @peterbe & Lars Lohn @twobraids

  35. pip install crontabber

  36. Our crontabber jobs

  37. None
  38. None
  39. None
  40. None
  41. configman https://github.com/mozilla/configman

  42. Our config https://github.com/mozilla/socorro/blob/ master/config/crontabber.ini-dist

  43. None
  44. No more shell scripts

  45. #!/bin/bash . /etc/socorro/socorrorc NAME=`basename $0 .sh` lock --ignore-existing $NAME ${PYTHON}

    ${APPDIR}/socorro/cron/crontabber.py \ --admin.conf=/etc/socorro/crontabber.ini \ >> /var/log/socorro/crontabber.log 2>&1 EXIT_CODE=$? unlock $NAME exit $EXIT_CODE
  46. Retries on failure

  47. Waits to run if a dependency fails

  48. Nagios alerts

  49. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085] socorroadm.stage.private.phx1.mozilla.com: Socorro

    Admin - crontab is CRITICAL: CRITICAL - correlations-addon-matview (CorrelationsAddonCronApp) (http://m.mozilla.org/Socorro+Admin+-+crontab)
  50. Allow configurable number of failures before CRITICAL

  51. Unit test framework for all jobs

  52. Documented dependencies

  53. None
  54. None
  55. Config can get hairy

  56. One-off runs aren’t simple

  57. Parallel execution coming soon! or...

  58. */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber1.ini */5

    * * * * socorro crontabber \ --admin.conf=/etc/crontabber2.ini */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber3.ini
  59. crontabber as a module is running in our stage environment

  60. Dependencies • Python 2.6 or higher • Postgres 9.2 or

    higher •
  61. https://github.com/mozilla/crontabber Ping us in #breakpad on irc.mozilla.org Tune in: Tuesday

    June 10th at 7pm PDT at air.mozilla.com!
  62. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/