The Design Philosophy of Kuroko2

Cf7b553387b247d737c60cfceabb2cea?s=47 Naoto Takai
January 18, 2017

The Design Philosophy of Kuroko2

Internal lightning talks in Cookpad

Cf7b553387b247d737c60cfceabb2cea?s=128

Naoto Takai

January 18, 2017
Tweet

Transcript

  1. The Design Philosophy of

  2. • Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •

    Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
  3. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Overview
  4. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
  5. • It’s really hard to isolate the problem in a

    distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
  6. There is data inconsistency between the actual state and the

    stored state. Lessons from Kuroko1 (cont.)
  7. State management is a key factor.

  8. • Monolithic is the best. • Data oriented, not process

    oriented. • Monitor as possible. Kuroko2
  9. Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command

    web interface DB
  10. Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop

    /0-noop A job definition A launched job A job state
  11. it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'

    engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
  12. Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command

  13. loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at:

    Time.current) execution.finish() end
  14. kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls ×

    × ×
  15. Kuroko2 Worker Processes command-executor kill process monitor process shell process

    exec command
  16. def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at)

    end end
  17. def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH

    if Execution.exists?(execution.id) notify_process_absence(execution) end end end
  18. • /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API

  19. • State management is a key factor: • monolithic is

    the best. • data oriented, not process oriented. • monitor as possible. Kuroko2
  20. is open source product. We are waiting your contributions!