Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Design Philosophy of Kuroko2

The Design Philosophy of Kuroko2

Internal lightning talks in Cookpad

Naoto Takai

January 18, 2017
Tweet

More Decks by Naoto Takai

Other Decks in Programming

Transcript

  1. • Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •

    Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
  2. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Overview
  3. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch

    exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
  4. • It’s really hard to isolate the problem in a

    distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
  5. There is data inconsistency between the actual state and the

    stored state. Lessons from Kuroko1 (cont.)
  6. • Monolithic is the best. • Data oriented, not process

    oriented. • Monitor as possible. Kuroko2
  7. Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop

    /0-noop A job definition A launched job A job state
  8. it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'

    engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
  9. def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH

    if Execution.exists?(execution.id) notify_process_absence(execution) end end end
  10. • State management is a key factor: • monolithic is

    the best. • data oriented, not process oriented. • monitor as possible. Kuroko2