• Web-based (not “enterprisy”)
• Embedded workflow engine (ruote).
• Distributed architecture
• with on demand EC2 provisioning.
• communicates with message queue.
Kuroko1
Slide 3
Slide 3 text
kuroko-console
RabbitMQ
kuoko-worker
workflow engine worker process
EC2 API
launch
exec command
web interface web
workflow
Kuroko1 Overview
Slide 4
Slide 4 text
kuroko-console
RabbitMQ
kuoko-worker
workflow engine worker process
EC2 API
launch
exec command
web interface web
workflow
Kuroko1 Pitfalls
×
×
×
× ×
× ×
×
× ×
Slide 5
Slide 5 text
• It’s really hard to isolate the problem in
a distributed environment.
• “Programmable” means programming is
always needed.
• Nothing is reliable except MySQL.
Lessons from Kuroko1
Slide 6
Slide 6 text
There is data inconsistency between
the actual state and the stored state.
Lessons from Kuroko1 (cont.)
Slide 7
Slide 7 text
State management is a key factor.
Slide 8
Slide 8 text
• Monolithic is the best.
• Data oriented, not process oriented.
• Monitor as possible.
Kuroko2
Slide 9
Slide 9 text
Monolithic Architecture
kuroko-console kuoko-worker
workflow engine worker process
exec command
web interface
DB
Slide 10
Slide 10 text
Kuroko2 Job State
job_definitions job_instances tokens
noop
noop
noop
noop /0-noop
A job definition A launched job A job state
Slide 11
Slide 11 text
it do
engine.process(token)
expect(token.path).to eq '/0-noop'
engine.process(token)
expect(token.path).to eq '/1-noop'
engine.process(token)
expect(token).to be_finished
expect(Kuroko2::Token.all.count).to eq 0
end
Slide 12
Slide 12 text
Kuroko2 Execution State
kuroko-worker
kuroko-console
executions
workflow-processor command-executor
exec command
Slide 13
Slide 13 text
loop do
execution = Execution.unstarted.take()
execution.touch(:started_at)
execute_shell(execution.shell)
execution.update(exit_status: exit_status,
finished_at: Time.current)
execution.finish()
end
Kuroko2 Worker Processes
command-executor
kill process
monitor process
shell process
exec command
Slide 16
Slide 16 text
def check_assignment_delay(exec_no_pid)
if exec_no_pid.started_at < 1.minutes.ago
Kuroko2::Notifications
.not_assigned(exec_no_pid,
@hostname).deliver_now
exec_no_pid.touch(:mailed_at)
end
end
Slide 17
Slide 17 text
def check_process_absence(execution)
begin
Process.kill(0, execution.pid)
rescue Errno::EPERM
true
rescue Errno::ESRCH
if Execution.exists?(execution.id)
notify_process_absence(execution)
end
end
end
Slide 18
Slide 18 text
• /v1/stats/waiting_execution
• /v1/stats/instance
Kuroko2 Monitoring API
Slide 19
Slide 19 text
• State management is a key factor:
• monolithic is the best.
• data oriented, not process oriented.
• monitor as possible.
Kuroko2
Slide 20
Slide 20 text
is open source product.
We are waiting your contributions!