The Design Philosophy of Kuroko2

Internal lightning talks in Cookpad

Transcript

1. The Design Philosophy of
   

   View Slide

2. • Web-based (not “enterprisy”)
   • Embedded workflow engine (ruote).
   • Distributed architecture
   • with on demand EC2 provisioning.
   • communicates with message queue.
   Kuroko1
   

   View Slide

3. kuroko-console
   RabbitMQ
   kuoko-worker
   workflow engine worker process
   EC2 API
   launch
   exec command
   web interface web
   workflow
   Kuroko1 Overview
   

   View Slide

4. kuroko-console
   RabbitMQ
   kuoko-worker
   workflow engine worker process
   EC2 API
   launch
   exec command
   web interface web
   workflow
   Kuroko1 Pitfalls
   ×
   ×
   ×
   × ×
   × ×
   ×
   × ×
   

   View Slide

5. • It’s really hard to isolate the problem in
   a distributed environment.
   • “Programmable” means programming is
   always needed.
   • Nothing is reliable except MySQL.
   Lessons from Kuroko1
   

   View Slide

6. There is data inconsistency between
   the actual state and the stored state.
   Lessons from Kuroko1 (cont.)
   

   View Slide

7. State management is a key factor.
   

   View Slide

8. • Monolithic is the best.
   • Data oriented, not process oriented.
   • Monitor as possible.
   Kuroko2
   

   View Slide

9. Monolithic Architecture
   kuroko-console kuoko-worker
   workflow engine worker process
   exec command
   web interface
   DB
   

   View Slide

10. Kuroko2 Job State
    job_definitions job_instances tokens
    noop
    noop
    noop
    noop /0-noop
    A job definition A launched job A job state
    

    View Slide

11. it do
    engine.process(token)
    expect(token.path).to eq '/0-noop'
    engine.process(token)
    expect(token.path).to eq '/1-noop'
    engine.process(token)
    expect(token).to be_finished
    expect(Kuroko2::Token.all.count).to eq 0
    end
    

    View Slide

12. Kuroko2 Execution State
    kuroko-worker
    kuroko-console
    executions
    workflow-processor command-executor
    exec command
    

    View Slide

13. loop do
    execution = Execution.unstarted.take()
    execution.touch(:started_at)
    execute_shell(execution.shell)
    execution.update(exit_status: exit_status,
    finished_at: Time.current)
    execution.finish()
    end
    

    View Slide

14. kuroko-worker
    kuroko-console
    executions
    workflow-processor command-executor
    exec command
    Kuroko2 Pitfalls
    × ×
    ×
    

    View Slide

15. Kuroko2 Worker Processes
    command-executor
    kill process
    monitor process
    shell process
    exec command
    

    View Slide

16. def check_assignment_delay(exec_no_pid)
    if exec_no_pid.started_at < 1.minutes.ago
    Kuroko2::Notifications
    .not_assigned(exec_no_pid,
    @hostname).deliver_now
    exec_no_pid.touch(:mailed_at)
    end
    end
    

    View Slide

17. def check_process_absence(execution)
    begin
    Process.kill(0, execution.pid)
    rescue Errno::EPERM
    true
    rescue Errno::ESRCH
    if Execution.exists?(execution.id)
    notify_process_absence(execution)
    end
    end
    end
    

    View Slide

18. • /v1/stats/waiting_execution
    • /v1/stats/instance
    Kuroko2 Monitoring API
    

    View Slide

19. • State management is a key factor:
    • monolithic is the best.
    • data oriented, not process oriented.
    • monitor as possible.
    Kuroko2
    

    View Slide

20. is open source product.
    We are waiting your contributions!
    

    View Slide