The Final Crontab

Transcript

1. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/

2. crontabber

3. None
4. None

5. socorro1 socorro3 WAL Socorro1 .dev Socorro1. stage base_backup copy Sunday

   noon PT streaming rep Prod socorro2 backup4 base_backup & pg_dump backup reporting1 WAL socorro-db-zeus-rw socorro-db-zeus-ro very architecture very architecture such replicas such replicas wow wow
6. None
7. None
8. None
9. None
10. None

11. Tons more at: http://lqbs.fr/suchcomments/

12. None

13. http://github.com/mozilla/socorro

14. http://bit.ly/1fOgBSB

15. */5 * * * * socorro crontabber.sh

16. image by @CoryLoftis

17. Motivating factors

18. #ThreeWordHorrorStories

19. No unit tests

20. No unit tests

21. Bespoke shell scripts

22. Postgres stored procedures

23. Email from cron

24. 0 5000 10000 15000 20000 25000 Dec 5, 2010 May

    5, 2011 Oct 5, 2011 Mar 5, 2012 Aug 5, 2012 Jan 5, 2013 Jun 5, 2013 Nov 5, 2013 Apr 5, 2014 Cron alert messages
25. None

26. Email from cron that you need to read.

27. None

28. Cron, what is it good for? • birthday reminders •

    status updates for a website • doxygen output for manuals every 12 hours • email nags about bugs filed wrong • ETL • Postgres -> Cloudwatch • Batch processing • Backups of RO DB • Machine heartbeat • “sweet fuck all” • “auto” updates • logging laptop IP • check for abandoned twitter accounts

29. Running jobs on a predictable schedule

30. How Socorro uses cron • Time-dependent reports or maintenance •

    “Simple” event detection and triggers • Status logging

31. Our use cases • Stored procedures for materialized views in

    Postgres • Daily map-reduces (largely deprecated) • FTP Scraping into Postgres • Bulk email responses to crash submissions pulled from Elastic Search

32. Jobs that don’t lend themselves to queue management because of

    time-dependencies, fragility or complexity.

33. crontabber https://github.com/mozilla/crontabber

34. On Github: Peter Bengtsson @peterbe & Lars Lohn @twobraids

35. pip install crontabber

36. Our crontabber jobs

37. None
38. None
39. None
40. None

41. configman https://github.com/mozilla/configman

42. Our config https://github.com/mozilla/socorro/blob/ master/config/crontabber.ini-dist

43. None

44. No more shell scripts

45. #!/bin/bash . /etc/socorro/socorrorc NAME=`basename $0 .sh` lock --ignore-existing $NAME ${PYTHON}

    ${APPDIR}/socorro/cron/crontabber.py \ --admin.conf=/etc/socorro/crontabber.ini \ >> /var/log/socorro/crontabber.log 2>&1 EXIT_CODE=$? unlock $NAME exit $EXIT_CODE

46. Retries on failure

47. Waits to run if a dependency fails

48. Nagios alerts

49. 15:58 < nagios-phx1> | Sun 15:58:44 PDT [1085] socorroadm.stage.private.phx1.mozilla.com: Socorro

    Admin - crontab is CRITICAL: CRITICAL - correlations-addon-matview (CorrelationsAddonCronApp) (http://m.mozilla.org/Socorro+Admin+-+crontab)

50. Allow configurable number of failures before CRITICAL

51. Unit test framework for all jobs

52. Documented dependencies

53. None
54. None

55. Config can get hairy

56. One-off runs aren’t simple

57. Parallel execution coming soon! or...

58. */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber1.ini */5

    * * * * socorro crontabber \ --admin.conf=/etc/crontabber2.ini */5 * * * * socorro crontabber \ --admin.conf=/etc/crontabber3.ini

59. crontabber as a module is running in our stage environment

60. Dependencies • Python 2.6 or higher • Postgres 9.2 or

    higher •

61. https://github.com/mozilla/crontabber Ping us in #breakpad on irc.mozilla.org Tune in: Tuesday

    June 10th at 7pm PDT at air.mozilla.com!

62. The Final Crontab Selena Deckelmann Data Architect, Mozilla @selenamarie http://chesnok.com/