$30 off During Our Annual Pro Sale. View Details »

The Final Crontab

The Final Crontab

Selena Deckelmann

May 06, 2014
Tweet

More Decks by Selena Deckelmann

Other Decks in Technology

Transcript

  1. The Final Crontab
    Selena Deckelmann
    Data Architect, Mozilla
    @selenamarie
    http://chesnok.com/

    View Slide

  2. crontabber

    View Slide

  3. View Slide

  4. View Slide

  5. socorro1 socorro3
    WAL
    Socorro1
    .dev
    Socorro1.
    stage
    base_backup copy
    Sunday noon PT
    streaming rep
    Prod
    socorro2
    backup4
    base_backup &
    pg_dump backup
    reporting1
    WAL
    socorro-db-zeus-rw
    socorro-db-zeus-ro
    very architecture
    very architecture
    such replicas
    such replicas
    wow
    wow

    View Slide

  6. View Slide

  7. View Slide

  8. View Slide

  9. View Slide

  10. View Slide

  11. Tons more at:
    http://lqbs.fr/suchcomments/

    View Slide

  12. View Slide

  13. http://github.com/mozilla/socorro

    View Slide

  14. http://bit.ly/1fOgBSB

    View Slide

  15. */5 * * * * socorro crontabber.sh

    View Slide

  16. image by @CoryLoftis

    View Slide

  17. Motivating factors

    View Slide

  18. #ThreeWordHorrorStories

    View Slide

  19. No unit tests

    View Slide

  20. No unit tests

    View Slide

  21. Bespoke shell scripts

    View Slide

  22. Postgres stored procedures

    View Slide

  23. Email from cron

    View Slide

  24. 0
    5000
    10000
    15000
    20000
    25000
    Dec 5, 2010 May 5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014
    Cron alert messages

    View Slide

  25. View Slide

  26. Email from cron
    that you need to read.

    View Slide

  27. View Slide

  28. Cron, what is it good for?
    • birthday reminders
    • status updates for a
    website
    • doxygen output for
    manuals every 12 hours
    • email nags about bugs
    filed wrong
    • ETL
    • Postgres -> Cloudwatch
    • Batch processing
    • Backups of RO DB
    • Machine heartbeat
    • “sweet fuck all”
    • “auto” updates
    • logging laptop IP
    • check for abandoned
    twitter accounts

    View Slide

  29. Running jobs on a
    predictable schedule

    View Slide

  30. How Socorro uses cron
    • Time-dependent reports or maintenance
    • “Simple” event detection and triggers
    • Status logging

    View Slide

  31. Our use cases
    • Stored procedures for materialized views in
    Postgres
    • Daily map-reduces (largely deprecated)
    • FTP Scraping into Postgres
    • Bulk email responses to crash submissions
    pulled from Elastic Search

    View Slide

  32. Jobs that don’t lend themselves to
    queue management because of
    time-dependencies, fragility or
    complexity.

    View Slide

  33. crontabber
    https://github.com/mozilla/crontabber

    View Slide

  34. On Github:
    Peter Bengtsson
    @peterbe
    &
    Lars Lohn
    @twobraids

    View Slide

  35. pip install crontabber

    View Slide

  36. Our crontabber jobs

    View Slide

  37. View Slide

  38. View Slide

  39. View Slide

  40. View Slide

  41. configman
    https://github.com/mozilla/configman

    View Slide

  42. Our config
    https://github.com/mozilla/socorro/blob/
    master/config/crontabber.ini-dist

    View Slide

  43. View Slide

  44. No more shell scripts

    View Slide

  45. #!/bin/bash
    . /etc/socorro/socorrorc
    NAME=`basename $0 .sh`
    lock --ignore-existing $NAME
    ${PYTHON} ${APPDIR}/socorro/cron/crontabber.py \
    --admin.conf=/etc/socorro/crontabber.ini \
    >> /var/log/socorro/crontabber.log 2>&1
    EXIT_CODE=$?
    unlock $NAME
    exit $EXIT_CODE

    View Slide

  46. Retries on failure

    View Slide

  47. Waits to run if a
    dependency fails

    View Slide

  48. Nagios alerts

    View Slide

  49. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085]
    socorroadm.stage.private.phx1.mozilla.com:
    Socorro Admin - crontab is CRITICAL:
    CRITICAL - correlations-addon-matview
    (CorrelationsAddonCronApp)
    (http://m.mozilla.org/Socorro+Admin+-+crontab)

    View Slide

  50. Allow configurable
    number of failures
    before CRITICAL

    View Slide

  51. Unit test framework
    for all jobs

    View Slide

  52. Documented
    dependencies

    View Slide

  53. View Slide

  54. View Slide

  55. Config can get hairy

    View Slide

  56. One-off runs aren’t
    simple

    View Slide

  57. Parallel execution
    coming soon!
    or...

    View Slide

  58. */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber1.ini
    */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber2.ini
    */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber3.ini

    View Slide

  59. crontabber as a module
    is running in our stage
    environment

    View Slide

  60. Dependencies
    • Python 2.6 or higher
    • Postgres 9.2 or higher

    View Slide

  61. https://github.com/mozilla/crontabber
    Ping us in #breakpad on
    irc.mozilla.org
    Tune in: Tuesday June 10th at
    7pm PDT at air.mozilla.com!

    View Slide

  62. The Final Crontab
    Selena Deckelmann
    Data Architect, Mozilla
    @selenamarie
    http://chesnok.com/

    View Slide