Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Design Philosophy of Kuroko2
Search
Naoto Takai
January 18, 2017
Programming
3
2.1k
The Design Philosophy of Kuroko2
Internal lightning talks in Cookpad
Naoto Takai
January 18, 2017
Tweet
Share
More Decks by Naoto Takai
See All by Naoto Takai
Building the Perfect Custom Keyboard
takai
1
550
キースイッチ潤滑入門 / A Brief Introduction to Lube MX Switches
takai
2
780
みなさん、分かっていますか / What is a difference?
takai
1
880
Agile and DevOps with Ruby
takai
2
1.1k
Software Quality and Testing #1
takai
5
650
Microservices in Action
takai
37
5.3k
Ruby App on Netflix Microservices Stack
takai
6
1k
How We Use Jenkins?
takai
21
8.3k
COOKPAD and Test Automation
takai
19
2.7k
Other Decks in Programming
See All in Programming
生成AIでGitHubソースコード取得して仕様書を作成
shukob
0
600
CQRS+ES の力を使って効果を感じる / Feel the effects of using the power of CQRS+ES
seike460
PRO
0
230
LLM Supervised Fine-tuningの理論と実践
datanalyticslabo
8
1.8k
Findy Team+ Awardを受賞したかった!ベストプラクティス応募内容をふりかえり、開発生産性向上もふりかえる / Findy Team Plus Award BestPractice and DPE Retrospective 2024
honyanya
0
130
Kaigi on Railsに初参加したら、その日にLT登壇が決定した件について
tama50505
0
130
range over funcの使い道と非同期N+1リゾルバーの夢 / about a range over func
mackee
0
200
Cloudflare MCP ServerでClaude Desktop からWeb APIを構築
kutakutat
1
630
20年もののレガシープロダクトに 0からPHPStanを入れるまで / phpcon2024
hirobe1999
0
960
iOS開発におけるCopilot For XcodeとCode Completion / copilot for xcode
fuyan777
1
1.1k
Итераторы в Go 1.23: зачем они нужны, как использовать, и насколько они быстрые?
lamodatech
0
1.2k
PHPで学ぶプログラミングの教訓 / Lessons in Programming Learned through PHP
nrslib
4
1k
Spatial Rendering for Apple Vision Pro
warrenm
0
340
Featured
See All Featured
Designing for humans not robots
tammielis
250
25k
jQuery: Nuts, Bolts and Bling
dougneiner
62
7.6k
Into the Great Unknown - MozCon
thekraken
34
1.6k
No one is an island. Learnings from fostering a developers community.
thoeni
19
3.1k
Measuring & Analyzing Core Web Vitals
bluesmoon
5
190
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
850
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
How STYLIGHT went responsive
nonsquared
96
5.3k
Code Reviewing Like a Champion
maltzj
521
39k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
28
2.2k
The World Runs on Bad Software
bkeepers
PRO
66
11k
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
Transcript
The Design Philosophy of
• Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •
Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch
exec command web interface web workflow Kuroko1 Overview
kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch
exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
• It’s really hard to isolate the problem in a
distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
There is data inconsistency between the actual state and the
stored state. Lessons from Kuroko1 (cont.)
State management is a key factor.
• Monolithic is the best. • Data oriented, not process
oriented. • Monitor as possible. Kuroko2
Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command
web interface DB
Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop
/0-noop A job definition A launched job A job state
it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'
engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command
loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at:
Time.current) execution.finish() end
kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls ×
× ×
Kuroko2 Worker Processes command-executor kill process monitor process shell process
exec command
def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at)
end end
def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH
if Execution.exists?(execution.id) notify_process_absence(execution) end end end
• /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API
• State management is a key factor: • monolithic is
the best. • data oriented, not process oriented. • monitor as possible. Kuroko2
is open source product. We are waiting your contributions!