The Final Crontab

About crontabber: https://github.com/mozilla/crontabber

Transcript

1. The Final Crontab
   Selena Deckelmann
   Data Architect, Mozilla
   @selenamarie
   http://chesnok.com/
   

   View Slide

2. crontabber
   

   View Slide

3. View Slide

4. View Slide

5. socorro1 socorro3
   WAL
   Socorro1
   .dev
   Socorro1.
   stage
   base_backup copy
   Sunday noon PT
   streaming rep
   Prod
   socorro2
   backup4
   base_backup &
   pg_dump backup
   reporting1
   WAL
   socorro-db-zeus-rw
   socorro-db-zeus-ro
   very architecture
   very architecture
   such replicas
   such replicas
   wow
   wow
   

   View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. Tons more at:
    http://lqbs.fr/suchcomments/
    

    View Slide

12. View Slide

13. http://github.com/mozilla/socorro
    

    View Slide

14. http://bit.ly/1fOgBSB
    

    View Slide

15. */5 * * * * socorro crontabber.sh
    

    View Slide

16. image by @CoryLoftis
    

    View Slide

17. Motivating factors
    

    View Slide

18. #ThreeWordHorrorStories
    

    View Slide

19. No unit tests
    

    View Slide

20. No unit tests
    

    View Slide

21. Bespoke shell scripts
    

    View Slide

22. Postgres stored procedures
    

    View Slide

23. Email from cron
    

    View Slide

24. 0
    5000
    10000
    15000
    20000
    25000
    Dec 5, 2010 May 5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014
    Cron alert messages
    

    View Slide

25. View Slide

26. Email from cron
    that you need to read.
    

    View Slide

27. View Slide

28. Cron, what is it good for?
    • birthday reminders
    • status updates for a
    website
    • doxygen output for
    manuals every 12 hours
    • email nags about bugs
    filed wrong
    • ETL
    • Postgres -> Cloudwatch
    • Batch processing
    • Backups of RO DB
    • Machine heartbeat
    • “sweet fuck all”
    • “auto” updates
    • logging laptop IP
    • check for abandoned
    twitter accounts
    

    View Slide

29. Running jobs on a
    predictable schedule
    

    View Slide

30. How Socorro uses cron
    • Time-dependent reports or maintenance
    • “Simple” event detection and triggers
    • Status logging
    

    View Slide

31. Our use cases
    • Stored procedures for materialized views in
    Postgres
    • Daily map-reduces (largely deprecated)
    • FTP Scraping into Postgres
    • Bulk email responses to crash submissions
    pulled from Elastic Search
    

    View Slide

32. Jobs that don’t lend themselves to
    queue management because of
    time-dependencies, fragility or
    complexity.
    

    View Slide

33. crontabber
    https://github.com/mozilla/crontabber
    

    View Slide

34. On Github:
    Peter Bengtsson
    @peterbe
    &
    Lars Lohn
    @twobraids
    

    View Slide

35. pip install crontabber
    

    View Slide

36. Our crontabber jobs
    

    View Slide

37. View Slide

38. View Slide

39. View Slide

40. View Slide

41. configman
    https://github.com/mozilla/configman
    

    View Slide

42. Our config
    https://github.com/mozilla/socorro/blob/
    master/config/crontabber.ini-dist
    

    View Slide

43. View Slide

44. No more shell scripts
    

    View Slide

45. #!/bin/bash
    . /etc/socorro/socorrorc
    NAME=`basename $0 .sh`
    lock --ignore-existing $NAME
    ${PYTHON} ${APPDIR}/socorro/cron/crontabber.py \
    --admin.conf=/etc/socorro/crontabber.ini \
    >> /var/log/socorro/crontabber.log 2>&1
    EXIT_CODE=$?
    unlock $NAME
    exit $EXIT_CODE
    

    View Slide

46. Retries on failure
    

    View Slide

47. Waits to run if a
    dependency fails
    

    View Slide

48. Nagios alerts
    

    View Slide

49. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085]
    socorroadm.stage.private.phx1.mozilla.com:
    Socorro Admin - crontab is CRITICAL:
    CRITICAL - correlations-addon-matview
    (CorrelationsAddonCronApp)
    (http://m.mozilla.org/Socorro+Admin+-+crontab)
    

    View Slide

50. Allow configurable
    number of failures
    before CRITICAL
    

    View Slide

51. Unit test framework
    for all jobs
    

    View Slide

52. Documented
    dependencies
    

    View Slide

53. View Slide

54. View Slide

55. Config can get hairy
    

    View Slide

56. One-off runs aren’t
    simple
    

    View Slide

57. Parallel execution
    coming soon!
    or...
    

    View Slide

58. */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber1.ini
    */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber2.ini
    */5 * * * * socorro crontabber \
    --admin.conf=/etc/crontabber3.ini
    

    View Slide

59. crontabber as a module
    is running in our stage
    environment
    

    View Slide

60. Dependencies
    • Python 2.6 or higher
    • Postgres 9.2 or higher
    •
    

    View Slide

61. https://github.com/mozilla/crontabber
    Ping us in #breakpad on
    irc.mozilla.org
    Tune in: Tuesday June 10th at
    7pm PDT at air.mozilla.com!
    

    View Slide

62. The Final Crontab
    Selena Deckelmann
    Data Architect, Mozilla
    @selenamarie
    http://chesnok.com/
    

    View Slide