Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow による機械学習ワークフロー管理
Search
Livesense Inc.
PRO
June 27, 2019
Technology
2
3.5k
Argo Workflow による機械学習ワークフロー管理
2019/06/27
Data Pipeline Casual Talk #3
Livesense Inc.
PRO
June 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
1.2k
データ基盤の負債解消のためのリプレイス
livesense
PRO
0
330
26新卒_総合職採用_会社説明資料
livesense
PRO
0
7k
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
20k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
10k
中途セールス職_会社説明資料
livesense
PRO
0
230
EM候補者向け転職会議説明資料
livesense
PRO
0
110
コロナで失われたノベルティ作成ノウハウを復活させた話
livesense
PRO
0
230
転職会議でGPT-3を活用した企業口コミ要約機能をリリースした話
livesense
PRO
0
1.3k
Other Decks in Technology
See All in Technology
MCPが変えるAIとの協働
knishioka
1
150
AI駆動で進化する開発プロセス ~クラスメソッドでの実践と成功事例~ / aidd-in-classmethod
tomoki10
1
1k
kernelvm-brain-net
raspython3
0
510
エンジニアリングで組織のアウトカムを最速で最大化する!
ham0215
1
300
グループ ポリシー再確認 (2)
murachiakira
0
230
AI 코딩 에이전트 더 똑똑하게 쓰기
nacyot
0
540
ペアーズにおける評価ドリブンな AI Agent 開発のご紹介
fukubaka0825
9
2.5k
Previewでもここまで追える! Azure AI Foundryで始めるLLMトレース
tomodo_ysys
2
630
製造業向けIoTソリューション提案資料.pdf
haruki_uiru
0
240
[新卒向け研修資料] テスト文字列に「うんこ」と入れるな(2025年版)
infiniteloop_inc
4
14k
20 Years of Domain-Driven Design: What I’ve Learned About DDD
ewolff
1
310
AIとSREで「今」できること
honmarkhunt
3
720
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
420
[RailsConf 2023] Rails as a piece of cake
palkan
54
5.5k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
5
600
YesSQL, Process and Tooling at Scale
rocio
172
14k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.4k
Site-Speed That Sticks
csswizardry
6
540
Making the Leap to Tech Lead
cromwellryan
133
9.3k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
41
2.3k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Transcript
Argo Workflow ʹΑΔ ػցֶशϫʔΫϑϩʔཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε
Data Pipeline Casual Talk #3
͢͜ͱ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ϦϒηϯεͷαʔϏεͱMLγεςϜ • MLγεςϜͷ։ൃɾӡ༻ࣄ •
MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • Argo Workflow ͷجຊػೳ • ϦϒηϯεͰͷ Argo Workflow ӡ༻ ※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢
αʔϏεͱMLγεςϜ
ϦϒηϯεͷαʔϏε
ϦϒηϯεͰͷMLར༻ αʔϏεͱML • ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ • MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ • ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙ ओͳMLγεςϜ •
ٻਓϨίϝϯυΤϯδϯ • Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • A/BςετɾόϯσΟοτπʔϧ
ٻਓϨίϝϯυΤϯδϯ • ϚοϋόΠτɾస৬φϏͳͲͷϢʔβʹٻਓΛਪન • ϝʔϧɾWebαΠτɾωΠςΟϒΞϓϦͳͲ༷ʑͳॴͰಋೖ
ٻਓϨίϝϯυΤϯδϯͷ෦ • ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़ • user-item item-item
ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • ϚοϋόΠτɾస৬φϏͳͲͷٻਓͷCVRԠื୯ՁΛࢉग़ • ݕࡧ݁ՌͷॱҐ੍ޚࠂग़ߘͷ࠷దԽʹ׆༻
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧͷ෦ • ϩάΛ༻͍ͯϕΠζਪఆɾϩδεςΟοΫճؼͰ༧ଌɾਪఆ • σΟϨΫλʔ͚ʹ؆қతͳϏϡʔϫΛWebΞϓϦͱͯ͠։ൃ
A/BςετɾόϯσΟοτπʔϧ • A/Bςετͷύλʔϯ৴ൺΛόϯσΟοτΞϧΰϦζϜͰࣗಈௐ • WebαΠτɾωΠςΟϒΞϓϦͷ࠷దԽΛޮԽ
A/BςετɾόϯσΟοτπʔϧͷ෦ • ཧը໘͔ΒύλʔϯΛొ͠ɺWeb APIͰϥϯμϜʹ৴ • CVϩάΛੳج൫Ͱूܭ͠ɺύλʔϯ৴ൺΛࣗಈߋ৽
MLγεςϜͷ։ൃɾӡ༻ࣄ
ϦϒηϯεͷMLγεςϜͷಛ ֶशɾ༧ଌͱେ෦͕όονॲཧ • ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ • ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉ όονॲཧͷߏ͕ෳࡶ •
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ϨίϝϯυΤϯδϯͰෳͷΞϧΰϦζϜΛΈ߹Θͤͯ͏ • ϑΟϧλϦϯάϦετͷϚʔδΛߦͬͯϨίϝϯυϦετΛੜ
ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • ಉαʔϏεͰA/BςετͷͨΊΞϧΰϦζϜ͚ͩมߋ • ผαʔϏεͷԣల։ͷࡍʹΞϧΰϦζϜΛ࠶ར༻
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ ٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞ → ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ → ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ • ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮
• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ • Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ • ਪఆɾ༧ଌϞσϧͰ Stan Λར༻ • ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ
ෳࡶԽ͢Δߏͷରॲ Ҏલ֤γεςϜ͕୯ҰϨϙδτϦͰཧ͞ΕΔϞϊϦγοΫͳߏ → ߏͷෳࡶԽͰ։ൃɾӡ༻͕·ΘΒͳ͘ͳ͖ͬͯͨ • MLͷίΞ෦ͱDBIO͕ີ݁߹͠ݸผ࣮ߦͰ͖ͳ͍ • γεςϜؒͰڞ௨͢ΔΞϧΰϦζϜ͕ίϐϖ͞ΕΔ • ಉҰͷόονॲཧͰεςοϓ͝ͱʹݴޠΛม͑ʹ͍͘
→ γεςϜΛػೳ͝ͱʹׂɾ࠶ߏங͢Δ͜ͱʹ
ίϯϙʔωϯτׂͱίϯςφԽ
ίϯϙʔωϯτͷׂ ·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠ • ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ • ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δ name role
input file output file sqlkit DBIO SQL CSV nlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧ recommender Ϩίϝϯυ ධՁ ਪનείΞ
ίϯϙʔωϯτͷίϯςφԽ ͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠ • ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ
• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ # load dataset docker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csv docker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv # preprocess docker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv # run recommender docker run -v $(pwd):/workdir recommender predict config.yaml /workdir
ίϯϙʔωϯτͷׂͱίϯςφԽ
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁ ίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ • ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ • ڞ௨෦ͷ࠶ར༻ՄೳԽ • ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ ͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ •
୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ • ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ • ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ
ͦΜͳ͋Δ (2017)
Argo Workflow Λൃݟ https://argoproj.github.io/
Argo Workflow "Container native workflow engine for Kubernetes" Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δ
ͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ • ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ • ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ • ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ • ϞχλϦϯά༻ Web UI
Argo Workflow ͷಛ CRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ • argo submit Ͱ࡞͞Εͨ
Workflow ϦιʔεΛ controller ͕࣮ߦ • ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞ ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍ • ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ • Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ
Argo Workflow ͷجຊػೳ
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec: entrypoint:
entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆ templates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ - name: entrypoint container: image: alpine:latest command: ["echo", "hello world"] ※Ҏ߱ͷྫ spec ԼͷΈهࡌ
None
ϫʔΫϑϩʔʹύϥϝʔλΛ͢ entrypoint: entrypoint arguments: # ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello
ͷΑ͏ʹͤΔ parameters: - name: message templates: - name: entrypoint container: image: alpine:latest command: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
εςοϓʹύϥϝʔλΛ͢ entrypoint: entrypoint templates: - name: entrypoint inputs: # ޙड़ͷ
steps, dag ͳͲ͔Β͢ parameters: - name: message value: hello container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
steps: εςοϓͷྻɾฒྻ࣮ߦ templates: - name: entrypoint steps: - - name:
hello1 template: echo # ίϯςφςϯϓϨʔτΛࢦఆ arguments: {parameters: [{name: "message", value: "hello1"}]} - - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2a"}]} - name: hello2b # hello2a, hello2b ฒྻ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2b"}]} - name: echo inputs: {parameters: [{name: "message"}]} container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"]
None
dag: ͰλεΫͷDAG࣮ߦ templates: - name: entrypoint dag: tasks: - name:
A template: echo arguments: {parameters: [{name: message, value: A}]} - name: B dependencies: [A] # ґଘλεΫΛࢦఆ template: echo arguments: {parameters: [{name: message, value: B}]} - name: C dependencies: [A] template: echo arguments: {parameters: [{name: message, value: C}]} - name: D dependencies: [B, C] # ґଘλεΫΛෳࢦఆ template: echo arguments: {parameters: [{name: message, value: D}]}
None
artifact: εςοϓؒͰϑΝΠϧΛड͚͠ templates: - name: entrypoint steps: - - {name:
generate-artifact, template: generate-artifact} - - {name: consume-artifact, template: consume-artifact} - name: generate-artifact container: image: alpine:latest command: ["sh", "-c", "echo hello > /tmp/output.txt"] outputs: artifacts: - {name: "result", path: "/tmp/output.txt"} - name: consume-artifact container: image: alpine:latest command: ["sh", "-c", "cat /tmp/input.txt"] inputs: artifacts: - {name: "result", path: "/tmp/input.txt"}
when: ϫʔΫϑϩʔͷذ templates: - name: entrypoint steps: - - name:
flip-coin template: flip-coin # when Ͱશεςοϓͷ݁ՌΛͱʹذ - - when: "{{steps.flip-coin.outputs.result}} == heads" name: heads - when: "{{steps.flip-coin.outputs.result}} == tails" name: tails - name: flip-coin script: image: python:latest command: [python] source: "import random; print(random.choice(['heads', 'tails']))"
None
withItems, withParams: εςοϓͷ܁Γฦ͠ templates: - name: entrypoint steps: # withItems
Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ - - withItems: ["hello world", "goodbye world", "ok world"] name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]} # withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ - - withParams: "{{workflow.parameters.params}}" name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]}
None
exitHandler : ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯά onExit: exit-handler templates: - name: entrypoint container:
image: alpine:latest command: ["exit", "1"] - name: exit-handler steps: # workflow.status Λͱʹذ - - when: "{{workflow.status}} == Succeeded" template: echo arguments: {parameters: [{name: "message", value: "SUCCESS"}]} - when: "{{workflow.status}} != Succeeded" template: echo arguments: {parameters: [{name: "message", value: "ERROR!"}]}
None
ϦτϥΠɾλΠϜΞτͳͲ templates: - name: entrypoint # ϦτϥΠճͳͲΛઃఆ retryStrategy: limit: 2
# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡) activeDeadlineSeconds: 28800 # ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡) nodeSelector: cloud.google.com/gke-nodepool: highmem-pool # Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡) container: resources: limits: memory: "32Gi"
None
ͦͷଞ • ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ • ϘϦϡʔϜʹΑΔσʔλͷड͚͠ • ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)
• ֎෦ετϨʔδͷར༻ • etc. ৄ͘͠ެࣜͷ example Λࢀর https://github.com/argoproj/argo/tree/master/examples
ϦϒηϯεͰͷ Argo Workflow ӡ༻
MLγεςϜͷ࣮ߦج൫ GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங • ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू • όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻ Argo Workflow ͷར༻
• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ • ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ • ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit
GCP, GKE, Argo Workflow ͷߏਤ
ӡ༻ࢦ όονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ • खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮ •
·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔ ӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ • DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠ • ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱ ҎԼɺࣄྫͱӡ༻ϊϋΛհ
CASE: ίϯϙʔωϯτͷΈ߹Θͤ • ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ • SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։ templates: - name: entrypoint
steps: - - name: load-config - - name: sqlkit withItems: - sqlfile: /workspace/sql/ratings.sql - sqlfile: /workspace/sql/contents.sql - - name: nlpkit - - name: recommender
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ • ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ • MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠Δ onExit: exit-handler templates: -
name: entrypoint steps: - - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLج൫ΤϯδχΞ͕࣮ - name: exit-handler # MLج൫ΤϯδχΞ͕࣮ steps: - - when: "{{workflow.status}} != Succeeded" name: notify-error
CASE: MLϞσϧͷ؆қతͳCD • ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻ • ਪఆॲཧྃ࣌ʹ kubectl set
env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ templates: - name: entrypoint steps: - - name: train-predict - - name: import-to-db - - name: update-viewer - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏ • ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻ • ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ • αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ
- name: train-predict activeDeadlineSeconds: 28800 # 8h retryStrategy: limit: 2 nodeSelector: cloud.google.com/gke-nodepool: highmem-pool container: resources: limits: memory: "32Gi"
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ • όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ • ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦ templates: - name: entrypoint
steps: # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ # લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ - - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦ name: calc-weights arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ӡ༻ TIPS Argo Workflow ͷ Web UI ͷΞΫηε • σϑΥϧτͰ
kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ • ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ • GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔ ݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ • ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ • ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ • argo delete --older Φϓγϣϯ͕ศར
Pros, Cons, ·ͱΊ
Argo Workflow - Pros ଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘ • ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ • ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢ όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ
• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ • ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ
Argo Workflow - Cons ଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍ • Airflow, Luigi ͷΑ͏ʹ Python
DSL ͕ॻ͚ͨΓ͠ͳ͍ • ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍ ࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍ • ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ • Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍ • ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘ • WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ
·ͱΊ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ෳͷαʔϏεͰMLγεςϜΛར༻ • ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ •
։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங • MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻