Slide 1
Slide 1 text
Argo Workflow ʹΑΔ
ػցֶशϫʔΫϑϩʔཧ
Shotaro Tanaka / @yubessy / Ϧϒηϯε
Data Pipeline Casual Talk #3
Slide 2
Slide 2 text
͢͜ͱ
ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ
• ϦϒηϯεͷαʔϏεͱMLγεςϜ
• MLγεςϜͷ։ൃɾӡ༻ࣄ
• MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽ
Argo Workflow ΛͲ͏͍ͬͯΔ͔
• Argo Workflow ͷجຊػೳ
• ϦϒηϯεͰͷ Argo Workflow ӡ༻
※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢
Slide 3
Slide 3 text
αʔϏεͱMLγεςϜ
Slide 4
Slide 4 text
ϦϒηϯεͷαʔϏε
Slide 5
Slide 5 text
ϦϒηϯεͰͷMLར༻
αʔϏεͱML
• ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ
• MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ
• ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙ
ओͳMLγεςϜ
• ٻਓϨίϝϯυΤϯδϯ
• Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ
• A/BςετɾόϯσΟοτπʔϧ
Slide 6
Slide 6 text
ٻਓϨίϝϯυΤϯδϯ
• ϚοϋόΠτɾస৬φϏͳͲͷϢʔβʹٻਓΛਪન
• ϝʔϧɾWebαΠτɾωΠςΟϒΞϓϦͳͲ༷ʑͳॴͰಋೖ
Slide 7
Slide 7 text
ٻਓϨίϝϯυΤϯδϯͷ෦
• ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़
• user-item item-item ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ
Slide 8
Slide 8 text
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ
• ϚοϋόΠτɾస৬φϏͳͲͷٻਓͷCVRԠื୯ՁΛࢉग़
• ݕࡧ݁ՌͷॱҐ੍ޚࠂग़ߘͷ࠷దԽʹ׆༻
Slide 9
Slide 9 text
Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧͷ෦
• ϩάΛ༻͍ͯϕΠζਪఆɾϩδεςΟοΫճؼͰ༧ଌɾਪఆ
• σΟϨΫλʔ͚ʹ؆қతͳϏϡʔϫΛWebΞϓϦͱͯ͠։ൃ
Slide 10
Slide 10 text
A/BςετɾόϯσΟοτπʔϧ
• A/Bςετͷύλʔϯ৴ൺΛόϯσΟοτΞϧΰϦζϜͰࣗಈௐ
• WebαΠτɾωΠςΟϒΞϓϦͷ࠷దԽΛޮԽ
Slide 11
Slide 11 text
A/BςετɾόϯσΟοτπʔϧͷ෦
• ཧը໘͔ΒύλʔϯΛొ͠ɺWeb APIͰϥϯμϜʹ৴
• CVϩάΛੳج൫Ͱूܭ͠ɺύλʔϯ৴ൺΛࣗಈߋ৽
Slide 12
Slide 12 text
MLγεςϜͷ։ൃɾӡ༻ࣄ
Slide 13
Slide 13 text
ϦϒηϯεͷMLγεςϜͷಛ
ֶशɾ༧ଌͱେ෦͕όονॲཧ
• ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ
• ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉ
όονॲཧͷߏ͕ෳࡶ
• ୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ
• ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍
• తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
Slide 14
Slide 14 text
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ
• ϨίϝϯυΤϯδϯͰෳͷΞϧΰϦζϜΛΈ߹Θͤͯ͏
• ϑΟϧλϦϯάϦετͷϚʔδΛߦͬͯϨίϝϯυϦετΛੜ
Slide 15
Slide 15 text
ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍
• ಉαʔϏεͰA/BςετͷͨΊΞϧΰϦζϜ͚ͩมߋ
• ผαʔϏεͷԣల։ͷࡍʹΞϧΰϦζϜΛ࠶ར༻
Slide 16
Slide 16 text
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ
ٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞
→ ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ
→ ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ
• ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮
• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ
• Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ
• ਪఆɾ༧ଌϞσϧͰ Stan Λར༻
• ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ
Slide 17
Slide 17 text
ෳࡶԽ͢Δߏͷରॲ
Ҏલ֤γεςϜ͕୯ҰϨϙδτϦͰཧ͞ΕΔϞϊϦγοΫͳߏ
→ ߏͷෳࡶԽͰ։ൃɾӡ༻͕·ΘΒͳ͘ͳ͖ͬͯͨ
• MLͷίΞ෦ͱDBIO͕ີ݁߹͠ݸผ࣮ߦͰ͖ͳ͍
• γεςϜؒͰڞ௨͢ΔΞϧΰϦζϜ͕ίϐϖ͞ΕΔ
• ಉҰͷόονॲཧͰεςοϓ͝ͱʹݴޠΛม͑ʹ͍͘
→ γεςϜΛػೳ͝ͱʹׂɾ࠶ߏங͢Δ͜ͱʹ
Slide 18
Slide 18 text
ίϯϙʔωϯτׂͱίϯςφԽ
Slide 19
Slide 19 text
ίϯϙʔωϯτͷׂ
·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠
• ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ
• ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δ
name role input file output file
sqlkit DBIO SQL CSV
nlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧ
recommender Ϩίϝϯυ ධՁ ਪનείΞ
Slide 20
Slide 20 text
ίϯϙʔωϯτͷίϯςφԽ
͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠
• ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ
• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ
# load dataset
docker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csv
docker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv
# preprocess
docker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv
# run recommender
docker run -v $(pwd):/workdir recommender predict config.yaml /workdir
Slide 21
Slide 21 text
ίϯϙʔωϯτͷׂͱίϯςφԽ
Slide 22
Slide 22 text
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁ
ίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ
• ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ
• ڞ௨෦ͷ࠶ར༻ՄೳԽ
• ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ
͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ
• ୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ
• ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ
• ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ
Slide 23
Slide 23 text
ͦΜͳ͋Δ (2017)
Slide 24
Slide 24 text
Argo Workflow Λൃݟ
https://argoproj.github.io/
Slide 25
Slide 25 text
Argo Workflow
"Container native workflow engine for Kubernetes"
Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δ
ͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ
• ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ
• ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ
• ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ
• ϞχλϦϯά༻ Web UI
Slide 26
Slide 26 text
Argo Workflow ͷಛ
CRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ
• argo submit Ͱ࡞͞Εͨ Workflow ϦιʔεΛ controller ͕࣮ߦ
• ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞
ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍
• ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ
• Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ
Slide 27
Slide 27 text
Argo Workflow ͷجຊػೳ
Slide 28
Slide 28 text
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔ
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-world-
spec:
entrypoint: entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆ
templates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ
- name: entrypoint
container:
image: alpine:latest
command: ["echo", "hello world"]
※Ҏ߱ͷྫ spec ԼͷΈهࡌ
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
ϫʔΫϑϩʔʹύϥϝʔλΛ͢
entrypoint: entrypoint
arguments:
# ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello ͷΑ͏ʹͤΔ
parameters:
- name: message
templates:
- name: entrypoint
container:
image: alpine:latest
command: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
Slide 31
Slide 31 text
εςοϓʹύϥϝʔλΛ͢
entrypoint: entrypoint
templates:
- name: entrypoint
inputs:
# ޙड़ͷ steps, dag ͳͲ͔Β͢
parameters:
- name: message
value: hello
container:
image: alpine:latest
command: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ
Slide 32
Slide 32 text
steps: εςοϓͷྻɾฒྻ࣮ߦ
templates:
- name: entrypoint
steps:
- - name: hello1
template: echo # ίϯςφςϯϓϨʔτΛࢦఆ
arguments: {parameters: [{name: "message", value: "hello1"}]}
- - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦ
template: echo
arguments: {parameters: [{name: "message", value: "hello2a"}]}
- name: hello2b # hello2a, hello2b ฒྻ࣮ߦ
template: echo
arguments: {parameters: [{name: "message", value: "hello2b"}]}
- name: echo
inputs: {parameters: [{name: "message"}]}
container:
image: alpine:latest
command: ["echo", "{{inputs.parameters.message}}"]
Slide 33
Slide 33 text
No content
Slide 34
Slide 34 text
dag: ͰλεΫͷDAG࣮ߦ
templates:
- name: entrypoint
dag:
tasks:
- name: A
template: echo
arguments: {parameters: [{name: message, value: A}]}
- name: B
dependencies: [A] # ґଘλεΫΛࢦఆ
template: echo
arguments: {parameters: [{name: message, value: B}]}
- name: C
dependencies: [A]
template: echo
arguments: {parameters: [{name: message, value: C}]}
- name: D
dependencies: [B, C] # ґଘλεΫΛෳࢦఆ
template: echo
arguments: {parameters: [{name: message, value: D}]}
Slide 35
Slide 35 text
No content
Slide 36
Slide 36 text
artifact: εςοϓؒͰϑΝΠϧΛड͚͠
templates:
- name: entrypoint
steps:
- - {name: generate-artifact, template: generate-artifact}
- - {name: consume-artifact, template: consume-artifact}
- name: generate-artifact
container:
image: alpine:latest
command: ["sh", "-c", "echo hello > /tmp/output.txt"]
outputs:
artifacts:
- {name: "result", path: "/tmp/output.txt"}
- name: consume-artifact
container:
image: alpine:latest
command: ["sh", "-c", "cat /tmp/input.txt"]
inputs:
artifacts:
- {name: "result", path: "/tmp/input.txt"}
Slide 37
Slide 37 text
when: ϫʔΫϑϩʔͷذ
templates:
- name: entrypoint
steps:
- - name: flip-coin
template: flip-coin
# when Ͱશεςοϓͷ݁ՌΛͱʹذ
- - when: "{{steps.flip-coin.outputs.result}} == heads"
name: heads
- when: "{{steps.flip-coin.outputs.result}} == tails"
name: tails
- name: flip-coin
script:
image: python:latest
command: [python]
source: "import random; print(random.choice(['heads', 'tails']))"
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
withItems, withParams: εςοϓͷ܁Γฦ͠
templates:
- name: entrypoint
steps:
# withItems Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ
- - withItems: ["hello world", "goodbye world", "ok world"]
name: each
template: echo
arguments: {parameters: [{name: "message", value: "{{item}}"}]}
# withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ
- - withParams: "{{workflow.parameters.params}}"
name: each
template: echo
arguments: {parameters: [{name: "message", value: "{{item}}"}]}
Slide 40
Slide 40 text
No content
Slide 41
Slide 41 text
exitHandler
: ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯά
onExit: exit-handler
templates:
- name: entrypoint
container:
image: alpine:latest
command: ["exit", "1"]
- name: exit-handler
steps:
# workflow.status Λͱʹذ
- - when: "{{workflow.status}} == Succeeded"
template: echo
arguments: {parameters: [{name: "message", value: "SUCCESS"}]}
- when: "{{workflow.status}} != Succeeded"
template: echo
arguments: {parameters: [{name: "message", value: "ERROR!"}]}
Slide 42
Slide 42 text
No content
Slide 43
Slide 43 text
ϦτϥΠɾλΠϜΞτͳͲ
templates:
- name: entrypoint
# ϦτϥΠճͳͲΛઃఆ
retryStrategy:
limit: 2
# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡)
activeDeadlineSeconds: 28800
# ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡)
nodeSelector:
cloud.google.com/gke-nodepool: highmem-pool
# Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡)
container:
resources:
limits:
memory: "32Gi"
Slide 44
Slide 44 text
No content
Slide 45
Slide 45 text
ͦͷଞ
• ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ
• ϘϦϡʔϜʹΑΔσʔλͷड͚͠
• ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)
• ֎෦ετϨʔδͷར༻
• etc.
ৄ͘͠ެࣜͷ example Λࢀর
https://github.com/argoproj/argo/tree/master/examples
Slide 46
Slide 46 text
ϦϒηϯεͰͷ Argo Workflow ӡ༻
Slide 47
Slide 47 text
MLγεςϜͷ࣮ߦج൫
GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங
• ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू
• όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻
Argo Workflow ͷར༻
• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ
• ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ
• ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit
Slide 48
Slide 48 text
GCP, GKE, Argo Workflow ͷߏਤ
Slide 49
Slide 49 text
ӡ༻ࢦ
όονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ
• खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮
• ·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔ
ӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ
• DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠
• ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱ
ҎԼɺࣄྫͱӡ༻ϊϋΛհ
Slide 50
Slide 50 text
CASE: ίϯϙʔωϯτͷΈ߹Θͤ
• ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ
• SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։
templates:
- name: entrypoint
steps:
- - name: load-config
- - name: sqlkit
withItems:
- sqlfile: /workspace/sql/ratings.sql
- sqlfile: /workspace/sql/contents.sql
- - name: nlpkit
- - name: recommender
Slide 51
Slide 51 text
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ
• ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ
• MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠Δ
onExit: exit-handler
templates:
- name: entrypoint
steps:
- - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV)
- - name: import-to-db # MLج൫ΤϯδχΞ͕࣮
- name: exit-handler # MLج൫ΤϯδχΞ͕࣮
steps:
- - when: "{{workflow.status}} != Succeeded"
name: notify-error
Slide 52
Slide 52 text
CASE: MLϞσϧͷ؆қతͳCD
• ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻
• ਪఆॲཧྃ࣌ʹ kubectl set env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ
• Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ
templates:
- name: entrypoint
steps:
- - name: train-predict
- - name: import-to-db
- - name: update-viewer
- name: update-viewer
container:
image: kubectl
command: ["sh", "-c"]
args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
Slide 53
Slide 53 text
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏
• ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻
• ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ
• αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ
- name: train-predict
activeDeadlineSeconds: 28800 # 8h
retryStrategy:
limit: 2
nodeSelector:
cloud.google.com/gke-nodepool: highmem-pool
container:
resources:
limits:
memory: "32Gi"
Slide 54
Slide 54 text
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ
• όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ
• ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦ
templates:
- name: entrypoint
steps:
# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ
- - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ
# લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ
- - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}"
# Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦ
name: calc-weights
arguments:
parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
Slide 55
Slide 55 text
ӡ༻ TIPS
Argo Workflow ͷ Web UI ͷΞΫηε
• σϑΥϧτͰ kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ
• ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ
• GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔ
ݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ
• ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ
• ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ
• argo delete --older Φϓγϣϯ͕ศར
Slide 56
Slide 56 text
Pros, Cons, ·ͱΊ
Slide 57
Slide 57 text
Argo Workflow - Pros
ଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘
• ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ
• ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢
όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ
• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ
• ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ
Slide 58
Slide 58 text
Argo Workflow - Cons
ଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍
• Airflow, Luigi ͷΑ͏ʹ Python DSL ͕ॻ͚ͨΓ͠ͳ͍
• ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍
࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍
• ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ
• Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍
• ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘
• WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ
Slide 59
Slide 59 text
·ͱΊ
ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ
• ෳͷαʔϏεͰMLγεςϜΛར༻
• ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ
• ։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽ
Argo Workflow ΛͲ͏͍ͬͯΔ͔
• ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங
• MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻