$30 off During Our Annual Pro Sale. View Details »

The Design Philosophy of Kuroko2

Naoto Takai
January 18, 2017

The Design Philosophy of Kuroko2

Internal lightning talks in Cookpad

Naoto Takai

January 18, 2017
Tweet

More Decks by Naoto Takai

Other Decks in Programming

Transcript

  1. The Design Philosophy of

  2. • Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •

    Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
  3. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Overview
  4. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
  5. • It’s really hard to isolate the problem in a

    distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
  6. There is data inconsistency between the actual state and the

    stored state. Lessons from Kuroko1 (cont.)
  7. State management is a key factor.

  8. • Monolithic is the best. • Data oriented, not process

    oriented. • Monitor as possible. Kuroko2
  9. Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command

    web interface DB
  10. Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop

    /0-noop A job definition A launched job A job state
  11. it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'

    engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
  12. Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command

  13. loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at:

    Time.current) execution.finish() end
  14. kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls ×

    × ×
  15. Kuroko2 Worker Processes command-executor kill process monitor process shell process

    exec command
  16. def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at)

    end end
  17. def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH

    if Execution.exists?(execution.id) notify_process_absence(execution) end end end
  18. • /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API

  19. • State management is a key factor: • monolithic is

    the best. • data oriented, not process oriented. • monitor as possible. Kuroko2
  20. is open source product. We are waiting your contributions!