Share
Embed
Argo Workflow による機械学習ワークフロー管理
2019/06/27
Data Pipeline Casual Talk #3
Livesense Inc. PRO
June 27, 2019
Transcript
59 slides
-
Argo Workflow ʹΑΔ ػցֶशϫʔΫϑϩʔཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε
Data Pipeline Casual Talk #3 -
͢͜ͱ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ϦϒηϯεͷαʔϏεͱMLγεςϜ • MLγεςϜͷ։ൃɾӡ༻ࣄ •
MLγεςϜͷίϯϙʔωϯτׂͱίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • Argo Workflow ͷجຊػೳ • ϦϒηϯεͰͷ Argo Workflow ӡ༻ ※ Kubernetes ͷجૅࣝΛલఏͱ͍ͯ͠·͢ -
ϦϒηϯεͰͷMLར༻ αʔϏεͱML • ٻਓɾෆಈ࢈ྖҬͰෳͷWebαʔϏεΛӡӦ • MLγεςϜͷ։ൃɾӡ༻νʔϜԣஅ৫ͱͯ͠αʔϏε͔Βಠཱ • ֤αʔϏεʹϨίϝϯυޮՌ༧ଌϞσϧͳͲෳͷMLγεςϜΛఏڙ ओͳMLγεςϜ •
ٻਓϨίϝϯυΤϯδϯ • Ԡืɾ࠾༻ͷޮՌਪఆɾ༧ଌϞσϧ • A/BςετɾόϯσΟοτπʔϧ -
ٻਓϨίϝϯυΤϯδϯͷ෦ • ධՁɾίϯςϯπΛͱʹ MF, FM ͳͲͷΞϧΰϦζϜΛద༻͠είΞΛࢉग़ • user-item item-item
ͷϦετΛόονॲཧͰੜ֤͠αʔϏεʹఏڙ -
ϦϒηϯεͷMLγεςϜͷಛ ֶशɾ༧ଌͱେ෦͕όονॲཧ • ֶश: CVR༧ଌϞσϧͷֶशϨίϝϯυͷҼࢠղ • ༧ଌ: ݕࡧɾϨίϝϯυ༻ͷείΞΛࣄલʹҰׅܭࢉ όονॲཧͷߏ͕ෳࡶ •
୯Ұͷόονॲཧ͕ଟͷεςοϓͰߏ • ෳͷόονॲཧؒͰڞ௨෦͕ଟ͍ • తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ -
తʹԠͯ͡ݴޠɾϥΠϒϥϦΛ͍͚Δ ٻਓαʔϏεࠂαʔϏεͳͲͱൺ୯Ձ͕େ͖͘CVR͕খ͍͞ → ࠷ਪఆϕʔεͷҰൠతͳMLϥΠϒϥϦ͕ద͠ͳ͍͜ͱ → ϞσϧɾΞϧΰϦζϜͷࣗલ࣮ͷͨΊݴޠɾϥΠϒϥϦΛ͍͚Δ • ϨίϝϯυΞϧΰϦζϜΛ Julia Ͱ࣮
• Alternating Least SquaresʹΑΔFactorization Machinesͷύϥϝʔλਪఆ • Factorization MachinesΛϨίϝϯσʔγϣϯͰ͏ͱ͖ͷධՁਪఆܭࢉ • ਪఆɾ༧ଌϞσϧͰ Stan Λར༻ • ֊ϕΠζʹΑΔখඪຊσʔλͷൺͷਪఆ -
ίϯϙʔωϯτͷׂ ·ͣγεςϜΛ࣍ͷΑ͏ͳ୯ػೳίϯϙʔωϯτʹׂͨ͠ • ֤ίϯϙʔωϯτ CLI Ͱ୯ಠ࣮ߦͰ͖Δ • ίϯϙʔωϯτؒͷೖग़ྗͯ͢ϑΝΠϧΛհ͢Δ name role
input file output file sqlkit DBIO SQL CSV nlpkit ࣗવݴޠॲཧ ςΩετ BoWϕΫτϧ recommender Ϩίϝϯυ ධՁ ਪનείΞ -
ίϯϙʔωϯτͷίϯςφԽ ͞Βʹ֤ίϯϙʔωϯτΛ୯ҰͷίϯςφΠϝʔδʹͨ͠ • ֤ίϯςφίϯϙʔωϯτ docker run kubectl run Ͱ࣮ߦͰ͖Δ
• γεςϜ͝ͱͷࠩ΄΅ઃఆϑΝΠϧSQL͚ͩͰදݱ # load dataset docker run -v $(pwd):/workdir sqlkit select ratings.sql /workdir/ratings.csv docker run -v $(pwd):/workdir sqlkit select content.sql /workdir/content.csv # preprocess docker run -v $(pwd):/workdir nlpkit vectorize /workdir/content.csv /workdir/features.csv # run recommender docker run -v $(pwd):/workdir recommender predict config.yaml /workdir -
ϫʔΫϑϩʔΛͲ͏࣮ݱ͢Δ͔ʁ ίϯϙʔωϯτͷׂͱίϯςφԽʹΑΓෳͷ՝ΛղܾͰ͖ͨ • ີ݁߹ͷղফɾεςοϓ࣮ߦͷՄೳԽ • ڞ௨෦ͷ࠶ར༻ՄೳԽ • ݴޠɾϥΠϒϥϦͷ͍͚ͷ༰қԽ ͔͠͠ɺෳࡶͳϫʔΫϑϩʔΛͲ͏ߏஙɾཧ͢Δ͔ͷ՝Δ •
୯७ͳόονॲཧͳΒ docker run kubectl run Λஞ࣮࣍ߦ͢Δ͚ͩ • ࣮ࡍʹ͜ͷํࣜͰຊ൪Քಇ͍ͯ͠ΔγεςϜଘࡏ • ฒྻԽɾϦτϥΠͳͲͷߴͳϫʔΫϑϩʔΛ࣮ݱ͍ͨ͠߹ʁ -
Argo Workflow "Container native workflow engine for Kubernetes" Kubernetes ্Ͱෳͷίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ࣮ߦͰ͖Δ
ͻͱ͜ͱͰݴ͏ͱʮߴػೳͳ k8s Jobʯ • ෳίϯςφͷྻɾฒྻɾDAG࣮ߦ • ذɾϧʔϓɾϑοΫͷ੍ޚϑϩʔ • ϦτϥΠɾλΠϜΞτɾϫʔΧʔϊʔυબ • ϞχλϦϯά༻ Web UI -
Argo Workflow ͷಛ CRD controller ͱ࣮ͯ͠͞Ε͍ͯΔ • argo submit Ͱ࡞͞Εͨ
Workflow ϦιʔεΛ controller ͕࣮ߦ • ϫʔΫϑϩʔͷ֤εςοϓ Pod ͱͯ͠ಈ࡞ ϫʔΫϑϩʔͷ࣮ߦʹઐ೦͠ɺτϦΨʔఆظ࣮ߦͷػೳͨͳ͍ • ͍উख Airflow, Digdag ΑΓ Luigi ʹ͍ۙ • Argo Events ͱ͍͏ผπʔϧͰ༷ʑͳτϦΨʔΛఏڙ -
୯ҰίϯςφΛ࣮ߦ͢Δ࠷؆୯ͳϫʔΫϑϩʔ apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec: entrypoint:
entrypoint # ࠷ॳʹ࣮ߦ͢ΔίϯςφςϯϓϨʔτΛࢦఆ templates: # ̍ͭҎ্ͷίϯςφςϯϓϨʔτΛఆٛ - name: entrypoint container: image: alpine:latest command: ["echo", "hello world"] ※Ҏ߱ͷྫ spec ԼͷΈهࡌ -
ϫʔΫϑϩʔʹύϥϝʔλΛ͢ entrypoint: entrypoint arguments: # ϫʔΫϑϩʔ࣮ߦ࣌ʹ argo submit -p message=hello
ͷΑ͏ʹͤΔ parameters: - name: message templates: - name: entrypoint container: image: alpine:latest command: ["echo", "{{workflow.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ -
εςοϓʹύϥϝʔλΛ͢ entrypoint: entrypoint templates: - name: entrypoint inputs: # ޙड़ͷ
steps, dag ͳͲ͔Β͢ parameters: - name: message value: hello container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"] # ύϥϝʔλͷຒΊࠐΈ -
steps: εςοϓͷྻɾฒྻ࣮ߦ templates: - name: entrypoint steps: - - name:
hello1 template: echo # ίϯςφςϯϓϨʔτΛࢦఆ arguments: {parameters: [{name: "message", value: "hello1"}]} - - name: hello2a # hello1 ͷ࣍ʹ hello2a, hello2b Λ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2a"}]} - name: hello2b # hello2a, hello2b ฒྻ࣮ߦ template: echo arguments: {parameters: [{name: "message", value: "hello2b"}]} - name: echo inputs: {parameters: [{name: "message"}]} container: image: alpine:latest command: ["echo", "{{inputs.parameters.message}}"] -
dag: ͰλεΫͷDAG࣮ߦ templates: - name: entrypoint dag: tasks: - name:
A template: echo arguments: {parameters: [{name: message, value: A}]} - name: B dependencies: [A] # ґଘλεΫΛࢦఆ template: echo arguments: {parameters: [{name: message, value: B}]} - name: C dependencies: [A] template: echo arguments: {parameters: [{name: message, value: C}]} - name: D dependencies: [B, C] # ґଘλεΫΛෳࢦఆ template: echo arguments: {parameters: [{name: message, value: D}]} -
artifact: εςοϓؒͰϑΝΠϧΛड͚͠ templates: - name: entrypoint steps: - - {name:
generate-artifact, template: generate-artifact} - - {name: consume-artifact, template: consume-artifact} - name: generate-artifact container: image: alpine:latest command: ["sh", "-c", "echo hello > /tmp/output.txt"] outputs: artifacts: - {name: "result", path: "/tmp/output.txt"} - name: consume-artifact container: image: alpine:latest command: ["sh", "-c", "cat /tmp/input.txt"] inputs: artifacts: - {name: "result", path: "/tmp/input.txt"} -
when: ϫʔΫϑϩʔͷذ templates: - name: entrypoint steps: - - name:
flip-coin template: flip-coin # when Ͱશεςοϓͷ݁ՌΛͱʹذ - - when: "{{steps.flip-coin.outputs.result}} == heads" name: heads - when: "{{steps.flip-coin.outputs.result}} == tails" name: tails - name: flip-coin script: image: python:latest command: [python] source: "import random; print(random.choice(['heads', 'tails']))" -
withItems, withParams: εςοϓͷ܁Γฦ͠ templates: - name: entrypoint steps: # withItems
Ͱͨ͠ item ͷ͚ͩεςοϓΛฒྻ࣮ߦ - - withItems: ["hello world", "goodbye world", "ok world"] name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]} # withParams ʹ ["hello world", "goodbye world"] ͷΑ͏ͳ JSON Λ͢͜ͱՄೳ - - withParams: "{{workflow.parameters.params}}" name: each template: echo arguments: {parameters: [{name: "message", value: "{{item}}"}]} -
exitHandler : ϫʔΫϑϩʔͷޭɾࣦഊ࣌ͷϋϯυϦϯά onExit: exit-handler templates: - name: entrypoint container:
image: alpine:latest command: ["exit", "1"] - name: exit-handler steps: # workflow.status Λͱʹذ - - when: "{{workflow.status}} == Succeeded" template: echo arguments: {parameters: [{name: "message", value: "SUCCESS"}]} - when: "{{workflow.status}} != Succeeded" template: echo arguments: {parameters: [{name: "message", value: "ERROR!"}]} -
ϦτϥΠɾλΠϜΞτͳͲ templates: - name: entrypoint # ϦτϥΠճͳͲΛઃఆ retryStrategy: limit: 2
# λΠϜΞτΛઃఆ (Pod ͷه๏ͱಉ͡) activeDeadlineSeconds: 28800 # ϊʔυͷࢦఆ (Pod ͷه๏ͱಉ͡) nodeSelector: cloud.google.com/gke-nodepool: highmem-pool # Ϧιʔε੍ݶ (Pod ͷه๏ͱಉ͡) container: resources: limits: memory: "32Gi" -
ͦͷଞ • ฒྻ࣮ߦ࣌ͷฒྻ্ݶΛઃఆ • ϘϦϡʔϜʹΑΔσʔλͷड͚͠ • ิॿίϯςφͷར༻ (Sidecar, Daemon, ...)
• ֎෦ετϨʔδͷར༻ • etc. ৄ͘͠ެࣜͷ example Λࢀর https://github.com/argoproj/argo/tree/master/examples -
MLγεςϜͷ࣮ߦج൫ GCP্ͰGKE Λத৺ͱ͢Δػցֶशج൫Λߏங • ෳͷMLγεςϜΛ୯ҰͷGKEΫϥελʹू • όονॲཧ͚ͩͰͳ͘WebΞϓϦಉ͡ΫϥελͰӡ༻ Argo Workflow ͷར༻
• ίϯςφίϯϙʔωϯτGCBͰϏϧυ͠GCR ʹొ • ϫʔΫϑϩʔఆٛଞͷ manifest ͱಉ͡ϨϙδτϦͰཧ • ఆظ࣮ߦ͢ΔϫʔΫϑϩʔ CronJob Ͱ argo submit -
ӡ༻ࢦ όονॲཧͱΓ͋͑ͣ Workflow ͱͯ͠ఆٛ • खݩͰ docker run ͚ͩͰࢼݧ࣮ߦͰ͖ΔΑ͏γεςϜΛ࣮ •
·ͣ୯Ұεςοϓͷ Workflow ͱͯ͠ӡ༻ʹࡌͤΔ ӡ༻͠ͳ͕ΒίϯϙʔωϯτԽΛਐΊͯຊମΛεϦϜԽ • DBIO௨ͳͲͷڞ௨ॲཧΛஈ֊తʹΓग़͍ͯ͘͠ • ฒྻԽɾϦτϥΠͳͲͳΔ͘ Workflow ଆͷػೳͰ࣮ݱ ҎԼɺࣄྫͱӡ༻ϊϋΛհ -
CASE: ίϯϙʔωϯτͷΈ߹Θͤ • ϨίϝϯυΤϯδϯಛʹίϯϙʔωϯτԽ͕ਐΜͰ͍Δ • SQLઃఆϑΝΠϧͻͱͭͷίϯςφʹ·ͱΊͯ࠷ॳʹల։ templates: - name: entrypoint
steps: - - name: load-config - - name: sqlkit withItems: - sqlfile: /workspace/sql/ratings.sql - sqlfile: /workspace/sql/contents.sql - - name: nlpkit - - name: recommender -
CASE: ϝΠϯͷόονॲཧͷεϦϜԽ • ਪఆɾ༧ଌϞσϧDBIO௨ͳͲΛΓग़ͯ͠ϝΠϯͷόονॲཧΛεϦϜԽ • MLΤϯδχΞɾMLج൫ΤϯδχΞͰͷ୲Λ͍ͯ͘͢͠͠Δ onExit: exit-handler templates: -
name: entrypoint steps: - - name: train-predict # MLΤϯδχΞ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLج൫ΤϯδχΞ͕࣮ - name: exit-handler # MLج൫ΤϯδχΞ͕࣮ steps: - - when: "{{workflow.status}} != Succeeded" name: notify-error -
CASE: MLϞσϧͷ؆қతͳCD • ਪఆɾ༧ଌϞσϧͷ݁ՌϏϡʔϫ Deployment ͱͯ͠ӡ༻ • ਪఆॲཧྃ࣌ʹ kubectl set
env ͰϏϡʔϫʹ৽͍͠ϞσϧΛಡΈࠐ·ͤΔ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ templates: - name: entrypoint steps: - - name: train-predict - - name: import-to-db - - name: update-viewer - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"] -
CASE: ॏ͍ɾෆ҆ఆͳMLॲཧΛѻ͏ • ਪఆɾ༧ଌϞσϧͳͲͰ Stan Λଟ༻ • ϝϞϦɾCPUΛେྔʹফඅ͢Δ߹ઐ༻ͷϊʔυͰ࣮ߦ • αϯϓϦϯά͕֬తʹࣦഊ͢ΔͷͰϦτϥΠɾλΠϜΞτ͕ඞཁ
- name: train-predict activeDeadlineSeconds: 28800 # 8h retryStrategy: limit: 2 nodeSelector: cloud.google.com/gke-nodepool: highmem-pool container: resources: limits: memory: "32Gi" -
CASE: Ϟσϧਪఆͷಈతͳฒྻ࣮ߦ • όϯσΟοτπʔϧͰ࣮ࢪதͷςετ͝ͱʹਪఆॲཧ͕ඞཁ • ֤ςετͷਪఆॲཧΛಈతʹฒྻ࣮ߦ templates: - name: entrypoint
steps: # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ # લͷεςοϓͷग़ྗ͔ΒύϥϝʔλͷϦετΛಡΈࠐΈ - - withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷཁૉ͝ͱʹޙଓͷεςοϓΛ࣮ߦ name: calc-weights arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}] -
ӡ༻ TIPS Argo Workflow ͷ Web UI ͷΞΫηε • σϑΥϧτͰ
kubectl port-forward ͰΞΫηε͢Δඞཁ͕͋Δ • ΠϯλʔωοτΞΫηεΛՄೳʹ͢Δʹ Ingress ͰϩʔυόϥϯαΛཱͯΔ • GCP ͷ Identity-Aware Proxy Λ͏ͱϩʔυόϥϯαଆͰೝূΛ͔͚ΒΕΔ ݹ͍ϫʔΫϑϩʔͷΫϦʔϯΞοϓ • ࣮ߦࡁΈͷ Workflow ͱͦͷཧ͢Δ Pod Successful ͷ··Γଓ͚Δ • ఆظతʹݹ͍ Workflow Λআ͢Δ CronJob Λཱ͍ͯͯΔ • argo delete --older Φϓγϣϯ͕ศར -
Argo Workflow - Pros ଞͷϫʔΫϑϩʔΤϯδϯͱൺϩοΫΠϯ͞Εʹ͍͘ • ίϯςφԽ͞Ε͍ͯΕԿͰಈ͔ͤΔ • ࠓޙଞͷϫʔΫϑϩʔΤϯδϯ͕ग़͖ͯͯΓ͍͑͢ όονॲཧͱWebΞϓϦΛಉ͡ΫϥελͰཧͰ͖Δ
• σϓϩΠɾϩΪϯάɾϞχλϦϯάɾΤϥʔϨϙʔτͳͲΛҰݩԽ • ΦʔτεέʔϦϯάͳͲͱΈ߹ΘͤͯϦιʔεར༻ΛޮԽ -
Argo Workflow - Cons ଞͷϫʔΫϑϩʔΤϯδϯ΄ͲϓϩάϥϚϒϧͰͳ͍ • Airflow, Luigi ͷΑ͏ʹ Python
DSL ͕ॻ͚ͨΓ͠ͳ͍ • ֤ΫϥυαʔϏεઐ༻ͷΦϖϨʔλ༻ҙ͞Ε͍ͯͳ͍ ࡞͞ΕͨϫʔΫϑϩʔΛଈ࣮࣌ߦ͢ΔҎ֎ͷػೳͨͳ͍ • ఆظ࣮ߦʹ CronJob ͳͲΛ͏ඞཁ͕͋Δ • Web UI ϞχλϦϯάͷΈͰϦτϥΠͳͲͷૢ࡞Ͱ͖ͳ͍ • ϫʔΫϑϩʔࣗମͷςϯϓϨʔτԽɾ࠶ར༻͕͠ʹ͍͘ • WorkflowTemplate ͕ఏҊ͞Ε͍ͯΔͷͰظ -
·ͱΊ ͳͥ Argo Workflow ͕ඞཁ͔ͩͬͨ • ෳͷαʔϏεͰMLγεςϜΛར༻ • ଟ͘ͷεςοϓ͔ΒͳΔόονॲཧ͕ෳଘࡏ •
։ൃɾӡ༻ΛޮԽ͢ΔͨΊίϯϙʔωϯτΛׂͯ͠ίϯςφԽ Argo Workflow ΛͲ͏͍ͬͯΔ͔ • ίϯςφίϯϙʔωϯτΛΈ߹ΘͤͯϫʔΫϑϩʔΛߏங • MLγεςϜͷ։ൃɾӡ༻্ͷ߹ʹ߹Θ֤ͤͯछػೳΛ׆༻