3回つくってわかった、 定期実行ジョブ基盤の本質情報 I Built a Scheduled Job Platform Three Times, and Now I Know the Essential Truths 

Over ten years in engineering leadership, including roles as CTO and VP of Engineering. Now a Sta Engineer at IVRy, working on Staff Engineer, IVRy engineering organization. ff Naoto Takai product development and improving our 

No content

No content

No content

Remember yesterday's keynote? 

DHH Matz Who? 

A Look Back at the Early Ruby Days • Babie was an active Ruby community member in the 2000s. • In December 2004, He introduced O/R mapping tools in Rubyist Magazine. • My rorm was featured alongside Active Record, just six months after its release. https://magazine.rubyist.net/articles/0004/0004- 

3回つくってわかった、 定期実行ジョブ基盤の本質情報 I Built a Scheduled Job Platform Three Times, and Now I Know the Essential Truths 

ff Do you ever do business-y stu at work? 

Business Applications Need Recurring Jobs • Business runs on cycles: • Transactions happen continuously. • Business periods repeat daily, weekly, monthly, and yearly. • Data has to be aggregated, reconciled, and nalized. • Systems exchange and synchronize data. fi Recurring jobs come out of business processes. 

Rails Makes This Easy, Right? • Rails 8 + Solid Queue makes recurring jobs simple. # app/jobs/daily_report_job.rb class DailyReportJob < ApplicationJob queue_as :default def perform(*args) DailyReport.generate! end end # config/recurring.yml production: daily_report: class: DailyReportJob queue: default schedule: every day at 2am fi De ne a job, add a schedule, and you're done! 

Real-World Jobs Are Often Work ows • A business process usually has several steps: 1. Call an external API 2. Fetch and validate data 3. Transform and store it 4. Call another service 5. Generate a report or send a noti cation fl fi fl A business process is a work ow, not a single job. 

Scheduling Is Easy. Operations Are Hard. • What if the previous run is still running? • What if the job fails halfway through? • Can we safely rerun it? • Can we see what is running and what failed? • Who gets noti ed when something goes wrong? fi Recurring jobs need more than a scheduler. 

How We Used Sidekiq at IVRy • We used Sidekiq Pro for event-driven async processing triggered by phone calls. • User-triggered jobs and scheduled jobs ran on Sidekiq. Rails App Queue Queue Worker Worker Worker Sidekiq Scheduler Different workloads shared the same workers. 

Problems We Actually Hit • Case 1: fi Recurring jobs delayed user-triggered jobs around month-end and month-start. • Case 2: A recurring job sometimes started before its previous run had nished. • Case 3: When a job failed, we could not tell whether it was safe to rerun. 

A Job Class Knows What, Not When • A job class knows what to do. • It doesn't know: • when it should run • whether another run is still going • what happened last time • whether it is safe to run again Scheduled jobs need execution management. 

Not an Async Job with a Timer Async Job Scheduled Job Event-driven Time-driven Single task Sequential steps Many at once Only one (no overlap) Seconds to minutes Minutes to hours Retry whole job Retry failed steps History Optional History Required 

I've Built This Three Times • Kuroko (Cookpad, 2012) • Distributed architecture with RabbitMQ, a work ow fl engine, and EC2 • Kuroko2 (Cookpad, 2014) • Monolithic architecture centered on the database • IVRy Job Scheduler (IVRy, 2026) • Serverless architecture built on AWS managed services 

Kuroko • In 2012, I was young, and so was AWS. 

Kuroko2 • Kuroko2 is built around a relational database, using a monolithic architecture to keep state management simple. 

IVRy Job Scheduler • Built entirely on AWS managed services, it is serverless and requires little operational overhead. Console API Sidekiq Job Schedule Workflow Scripts Status 

What Each Platform Was Built From • The responsibilities stayed the same, even when the fl technology changed. Kuroko Kuroko2 IVRy Job Scheduler rufus-scheduler Custom scheduler EventBridge Scheduler Ruote Custom work ow Step Functions EC2 workers Custom executor ECS / Lambda Fluentd CloudWatch Logs CloudWatch Logs / Datadog Custom console Custom console Custom console 

The Same Five Components Keep Appearing • All three have a scheduler, a work ow engine, an executor, logs, and a console. Scheduler Workflow Executor Console fl Logs 

Simple Intent, Complex Reality • What developers want: Run this job every day at 2 AM. • Then the questions start: • Where does it run? What if it runs twice? Who owns the schedule? What if it fails? Retry? How many times? When do we give up? What state is it in? Where are the logs? What happened last night? How do we recover? Rerun everything? Just one step? Who gets noti ed? fi The gap creates cognitive load for application developers. 

Cognitive Load Hurts DevEx • The DevEx Framework identi es three core dimensions: Flow State DevEx Cognitive Load Feedback Loops fi Adapted from Noda et al., “DevEx: What Actually Drives Productivity,” ACM Queue, 2023. 

We need to build scheduled job infrastructure as a platform. 

Platform Engineering Bridges Intent and Reality Run this job every day at 2 AM. Platform Developer Intent Workload Concurrency Logs Scheduling Retries Recovery State Timeouts Notifications Developers express intent. The platform handles reality. 

How we built it in the IVRy Job Scheduler 

No content

No content

How Application Developers De ne Jobs • Creating a job takes three steps: fi fi 1. Write a normal Sidekiq job 2. De ne the job with the DSL 3. Open and merge a pull request 

Write A Normal Sidekiq Job • Application developers reuse existing Sidekiq jobs and keep working in Rails. class DailyReportJob < ApplicationJob include Sidekiq::Job def perform(*args) DailyReport.generate! end end 

De ne The Job With The DSL • Developers write the intent in Jsonnet. fi function(dsl, jobId) dsl.Job(jobId, { steps: [ dsl.SidekiqTask('DailyReportJob'), ], schedules: [ { expression: 'cron(0 2 * * ? *)' }, ], execution_policy: 'SKIP_IF_RUNNING', }) 

Open and Merge a Pull Request • Merging the de nition updates the platform. Terraform JSON Apply AWS Resources Generate Job Definition Jsonnet Generate fi Job Metadata JSON Import Web Console 

Three Problems, Three Responsibilities • What the platform must handle • Case1: Recurring jobs delayed event-driven jobs fi → Isolate workloads • Case2: A job started before the previous run nished → Prevent overlapping runs • Case3: We couldn't tell if a failed job was safe to rerun → Make failures visible and recovery easy 

How a Scheduled Job Runs • The platform executes existing Sidekiq jobs through a thin runner. GitHub EventBridge Scheduler Trigger Step Functions CI / CD ECR Run Task Sidekiq Runner on ECS Fargate Invoke Sidekiq Job 

A Thin Runner Connects the Platform to Rails • The runner loads Rails and executes an existing Sidekiq job directly. # Simplified version require "bundler/setup" require_relative "../config/environment" job_class = Object.const_get(ARGV[0]) job_class.perform_inline(*ARGV.drop(1)) 

Three Problems, Three Responsibilities • What the platform must handle • Case1: Recurring jobs delayed user-facing jobs fi → Isolate workloads • Case2: A job started before the previous run nished → Prevent overlapping runs • Case3: We couldn't tell if a failed job was safe to rerun → Make failures visible and recovery easy 

A Few Lines of Intent Become a Full Work ow • A short DSL de nition expands into hundreds of lines of work ow JSON with state checks, retries, and execution tracking. { "ResultPath": "$.prepared", "Type": "Pass" }, { }, "SetRunningState": { "Catch": [ { "ErrorEquals": [ "States.ALL" ], "Next": "EnsureDatadogContext", "ResultPath": "$.error" } ], "Next": "EnsureDatadogContext", "Parameters": { "Item": { "executionArn.$": "$$.Execution.Id", "jobName": { "S.$": "$$.StateMachine.Name" }, "lastUpdated.$": "$$.State.EnteredTime", "startTime.$": "$$.Execution.StartTime", "status": { "S": "RUNNING" }, "stopTime": { "S": "" } }, "TableName": "job-scheduler-state" }, "Resource": "arn:aws:states:::dynamodb:putItem", "ResultPath": "$.putResult", "Retry": [ { "BackoffRate": 2, "ErrorEquals": [ "DynamoDB.SdkClientException", "DynamoDB.AmazonServiceException", "DynamoDB.InternalServerError", "DynamoDB.ServiceUnavailable", "IsPresent": true, "Variable": "$.dbResult.Item.status.S" }, { "Or": [ { "StringEquals": "FAILED", "Variable": "$.dbResult.Item.status.S" }, { "StringEquals": "TIMED_OUT", "Variable": "$.dbResult.Item.status.S" }, { "StringEquals": "ABORTED", "Variable": "$.dbResult.Item.status.S" } ] } ], "Next": "SkipExecution" } ], "Default": "SetRunningState", "Type": "Choice" }, "EnsureDatadogContext": { "Next": "PrepareTraceContext", "OutputPath": "$.result", "Parameters": { "result.$": "States.JsonMerge(States.StringToJson('{\"_datadog\":{}}'), $, false)" }, "Type": "Pass" }, "GetJobState": { "Catch": [ { "ErrorEquals": [ "States.ALL" ], "Next": "SetRunningState", "ResultPath": "$.error" } ], "Next": "CheckExecutionPolicy", "Parameters": { "Key": { "jobName": { "S.$": "$$.StateMachine.Name" } }, "TableName": "job-scheduler-state" }, "Resource": "arn:aws:states:::dynamodb:getItem", "ResultPath": "$.dbResult", "Type": "Task" }, "PrepareTraceContext": { "Next": "SidekiqTask-Contracts::DailyReportJob", "Parameters": { "ddTraceContext.$": "States.JsonToString($ ['_datadog'])" }, "DynamoDB.ProvisionedThroughputExceededException", "DynamoDB.RequestLimitExceeded", "DynamoDB.ThrottlingException" ], "IntervalSeconds": 1, "JitterStrategy": "FULL", "MaxAttempts": 3 } ], "Type": "Task" }, "SidekiqTask-Contracts::DailyReportJob": { "End": true, "Parameters": { "Cluster": "arn:aws:ecs:apnortheast-1:146154011163:cluster/job-scheduler", "LaunchType": "FARGATE", "NetworkConfiguration": { "AwsvpcConfiguration": { "AssignPublicIp": "ENABLED", "SecurityGroups": [ "sg-00ddd5fe5b773d7a4" ], "Subnets": [ "subnet-064f8c06baca91b95", "subnet-02544c3bc2d96f992", fl fi fl function(dsl, jobId) dsl.Job(jobId, { steps: [ dsl.SidekiqTask('DailyReportJob'), ], schedules: [ { expression: 'cron(0 9 * * ? *)' }, ], execution_policy: 'SKIP_IF_RUNNING', }) "Variable": "$.executionPolicy" "Comment": "Daily Report Job execution workflow", "StartAt": "GetJobState", "States": { "CheckExecutionPolicy": { "Choices": [ { "Next": "SetRunningState", "StringEquals": "ALWAYS_RUN", "Variable": "$.executionPolicy" }, { "Next": "SetRunningState", "Not": { "IsPresent": true, "Variable": "$.dbResult.Item.status" } }, { "And": [ { "StringEquals": "SKIP_IF_RUNNING_OR_FAILED", "Variable": "$.executionPolicy" }, { "IsPresent": true, "Variable": "$.dbResult.Item.status.S" }, { "Or": [ { "StringEquals": "RUNNING", "Variable": "$.dbResult.Item.status.S" }, { "StringEquals": "FAILED", "Variable": "$.dbResult.Item.status.S" }, { "StringEquals": "TIMED_OUT", "Variable": "$.dbResult.Item.status.S" }, { "StringEquals": "ABORTED", "Variable": "$.dbResult.Item.status.S" } ] } ], "Next": "SkipExecution" }, { "And": [ { "StringEquals": "SKIP_IF_RUNNING", "Variable": "$.executionPolicy" }, { "StringEquals": "RUNNING", "Variable": "$.dbResult.Item.status.S" } ], "Next": "SkipExecution" }, { "And": [ { "StringEquals": "SKIP_IF_FAILED", "subnet-09bc46d9ba952162f" ] } }, "Overrides": { "ContainerOverrides": [ { "Command": [ "./scripts/sidekiq-runner", "Contracts::DailyReportJob" ], "Environment": [ { "Name": "JOB_EXECUTION_ARN", "Value.$": "$$.Execution.Id" }, { "Name": "JOB_START_TIME", "Value.$": "$$.Execution.StartTime" }, { "Name": "DD_TRACE_CONTEXT", "Value.$": "$.prepared.ddTraceContext" } ], "Name": "first" } ] }, "TaskDefinition": "arn:aws:ecs:apnortheast-1:146154011163:task-definition/pro-ivry-job-runneroneshot" }, "Resource": "arn:aws:states:::ecs:runTask.sync", "ResultPath": "$.ecsResult", "Retry": [ { "BackoffRate": 2, "ErrorEquals": [ "ECS.ServerException", "ECS.SdkClientException" ], "IntervalSeconds": 10, "JitterStrategy": "FULL", "MaxAttempts": 3 } ], "Type": "Task" }, "SkipExecution": { "End": true, "Parameters": { "executionPolicy.$": "$.executionPolicy", "jobName.$": "$.jobName", "skipResult": { "prevStatus.$": "$.dbResult.Item.status.S", "reason.$": "States.Format('Skipped due to previous status: {}', $.dbResult.Item.status.S)", "skipped": true }, "triggerType.$": "$.triggerType", "triggeredBy.$": "$.triggeredBy" }, "Type": "Pass" } } } 

A Few Lines of Intent Become a Full Work ow • A short DSL de nition expands into hundreds of lines of work ow JSON with state checks, retries, and execution tracking. Start Choice state CheckEventStatus $.detail.status == "SUCCEEDED" and $.detail.outputDetails.included Default== true Pass state Pass state ExtractWithOutput ExtractWithoutOutput Pass state ParseTriggerContext Choice state CheckIfSkipped $.status == "FAILED" function(dsl, jobId) dsl.Job(jobId, { steps: [ dsl.SidekiqTask('DailyReportJob'), ], schedules: [ { expression: 'cron(0 9 * * ? *)' }, ], execution_policy: 'SKIP_IF_RUNNING', }) Start AWS Step Functions: DescribeExecution DescribeFailedExecution Amazon DynamoDB: GetItem GetJobState Choice state CheckForInfraFailureRedrive Choice state CheckExecutionPolicy $.executionPolicy == "ALWAYS_RUN" not($.dbResult.Item.status is present) Default Default $.executionDetails.Cause is present and $.executionDetails.Cause ~= "*TaskFailedToStart*" and $.executionDetails.RedriveCount Catch #1 <2 Catch #1 AWS Step Functions: RedriveExecution $.executionPolicy == "SKIP_IF_RUNNING_OR_FAILED" $.executionPolicy and $.dbResult.Item.status.S == "SKIP_IF_RUNNING" is present and $.executionPolicy $.dbResult.Item.status.S and ($.dbResult.Item.status.S == "SKIP_IF_FAILED" == "RUNNING" == "RUNNING" and $.dbResult.Item.status.S or $.dbResult.Item.status.S is present == and"FAILED" ($.dbResult.Item.status.S or $.dbResult.Item.status.S == "FAILED" ==or "TIMED_OUT" $.dbResult.Item.status.S or $.dbResult.Item.status.S == "TIMED_OUT" ==or "ABORTED") $.dbResult.Item.status.S == "ABORTED") Default $.parsedOutput.skipResult.skipped is present and $.parsedOutput.skipResult.skipped == true RedriveOriginalExecution Amazon DynamoDB: PutItem Pass state SetRunningState SkipExecution Catch #1 Amazon DynamoDB: PutItem Catch #1 UpdateJobState Pass state EnsureDatadogContext Catch #1 Amazon DynamoDB: PutItem WriteExecutionHistory Pass state PrepareTraceContext Catch #1 Amazon ECS: RunTask SidekiqTask-HealthCheckJob $.status == "FAILED" End Choice state Amazon DynamoDB: PutItem CheckIfFailed WriteSkippedHistory $.status == "TIMED_OUT" $.status == "ABORTED" fl fi Catch #1 Amazon SNS: Publish Succeed state SendFailureNotification Success End fl Default 

Three Problems, Three Responsibilities • What the platform must handle • Case1: Recurring jobs delayed user-facing jobs fi → Isolate workloads • Case2: A job started before the previous run nished → Prevent overlapping runs • Case3: We couldn't tell if a failed job was safe to rerun → Make failures visible and recovery easy 

Failures Go Straight to Slack • Failed jobs are reported to Slack. 

Recover from the Portal • From the alert, operators open the failed execution in the portal. 

Operate Jobs from the Portal • The portal has what anyone needs to operate a job. 

What I Learned After Building It Three Times 1. Scheduled jobs come with their own operational challenges. 2. There’s a gap between what developers want and what reliable execution requires. 3. A platform can close that gap. If operating scheduled jobs is getting painful, consider building a platform. 

Takeaways • Every domain has its own complexity. • That complexity shapes the architecture. • A platform turns developer intent into working systems.