Building painless scheduling systems in Node

Scheduling stuff sounds like a pretty straight forward thing to do, doesn’t it? Simply set up a cron with a schedule (after looking at the cron syntax online, of course), tell it what to do and forget about it.

Well, if you have ever worked with scheduling systems, you know that isn’t the end of the story. Monitoring, failure isloation, retries, choking preventions, etc. make sure you think about that 5 AM job in your sleep.

Idempotency can help - slightly. This talk tries to encapsulate some of my learnings building a distributed scheduler in Node, some of the pain points I faced and how embracing idempotency helped relieve some of those pains.

Transcript

1. P R E S E N TAT I O N T E M P L AT E Building painless scheduling systems in Node Presented By Deepak Pathania @debugPathania
2. P R E S E N TAT I O N T E M P L AT E Case study !== best practices
3. Scheduling systems? - Executes specified tasks at specified intervals. - Used to automate recurring tasks. - Fancy name for systems that run a bunch of crons.
4. Simplified problem statement - Build a system that allows users to setup custom schedules for tasks. -Tasks are database intensive, eventual consistency is acceptable, need common post processing. - Experimental project, limited resources. (1 backend dev)
5. Build vs buy? -Managed enterprise level solutions available like Google Cloud Scheduler, EasyCron etc. -High setup time and cost for an experimental project whose requirements were constantly evolving. -Didn’t want to connect third party to database, sending records over the wire was bandwidth intensive.
6. Enter Cron - duhh
8. Simplified Flow
9. 1. User creates a task with a schedule and a job.
10. 2. Cron invokes a process function after set interval.
11. 3. The process function: • filters out eligible records based on nextRunningTime and status. • runs the associated job for each filtered task. • updates the nextRunningTime of the task based on the schedule. • updates the status flag to mark eligibility for post processing.
13. What if jobs failed for certain tasks?
14. Since the post processing status and nextRunningTime is updated only on successful job completions, the task would be auto picked in next process cycle. Embrace Failure
15. By increasing the frequency of process and bailing out early in case no eligible records are found after filtering, we can build an implicit retry mechanism. Embrace Failure
18. How do I monitor my crons?
19. Slack web hooks - monitoring 101
20. How do I test my crons?
21. 1.Test functionality, trust the timing. - REST first approach. - Modularise scheduled tasks. - Stub everything.
23. Questions? @debugPathania