The Design Philosophy of Kuroko2

Transcript

1. The Design Philosophy of

2. • Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •

   Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1

3. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

   exec command web interface web workflow Kuroko1 Overview

4. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

   exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×

5. • It’s really hard to isolate the problem in a

   distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1

6. There is data inconsistency between the actual state and the

   stored state. Lessons from Kuroko1 (cont.)

7. State management is a key factor.

8. • Monolithic is the best. • Data oriented, not process

   oriented. • Monitor as possible. Kuroko2

9. Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command

   web interface DB

10. Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop

    /0-noop A job definition A launched job A job state

11. it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'

    engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end

12. Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command

13. loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at:

    Time.current) execution.finish() end

14. kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls ×

    × ×

15. Kuroko2 Worker Processes command-executor kill process monitor process shell process

    exec command

16. def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at)

    end end

17. def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH

    if Execution.exists?(execution.id) notify_process_absence(execution) end end end

18. • /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API

19. • State management is a key factor: • monolithic is

    the best. • data oriented, not process oriented. • monitor as possible. Kuroko2

20. is open source product. We are waiting your contributions!