Argo Workflow によるMLジョブ管理

MACHINE LEARNING Meetup KANSAI #4
2019/3/27

Transcript

1. Argo Workflow ʹΑΔMLδϣϒ؅ཧ
   Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژ౎ΦϑΟε)
   MACHINE LEARNING Meetup KANSAI #4 LT
   

   View Slide

2. ͜Εͷ঺հ͠·͢
   

   View Slide

3. https://argoproj.github.io/
   

   View Slide

4. Կ͕Ͱ͖Δͷ͔
   "Container native workflow engine for Kubernetes"
   • ෳ਺ͷίϯςφΛ௚ྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ
   • σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λ૝ఆ
   • ৽όʔδϣϯͰ͸ DAG ΋αϙʔτ
   • Argo ϕʔεͷ༷ʑͳϓϩμΫτ
   • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ
   • Argo Event: ϫʔΫϑϩʔͷτϦΨ
   

   View Slide

5. apiVersion: argoproj.io/v1alpha1
   kind: Workflow
   metadata:
   generateName: ml-workflow-
   spec:
   entrypoint: main
   templates:
   - name: main
   steps:
   - - name: load-dataset
   template: load-dataset
   - - name: train-model-1
   template: train-model
   arguments:
   parameters: [{name: model, value: model1}]
   - name: train-model-2
   template: train-model
   arguments:
   parameters: [{name: model, value: model2}]
   ...
   

   View Slide

6. ...
   - name: load-dataset
   container:
   image: postgres:latest
   command: [sh, -c]
   args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"]
   - name: train-model
   inputs:
   parameters: [{name: model}]
   container:
   image: train-model
   command: [sh -c]
   args: ["python train_model.py --model={{inputs.parameters.model}}"]
   

   View Slide

7. View Slide

8. ͳͥ࢖͏ͷ͔
   ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ৐͍ͤͨʯ
   • MLϞσϧͷ։ൃऀ
   • SQL Ͱσʔλऔಘ ʙ Ϟσϧ΍༧ଌ஋ΛϑΝΠϧʹग़ྗ
   • Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠
   • MLγεςϜͷ։ൃऀ
   • DBIO ΍Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮૷
   • Argo Ͱ͢΂ͯΛ૊Έ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ
   → ίϯςφ୯ҐͰ໾ׂ෼୲
   

   View Slide

9. ϦϒηϯεͰͷར༻ྫ
   • ग़ྗͷDBॻ͖ࠐΈॲཧͷ෼཭
   • Ϟσϧͷ Continuous Delivery
   • ฒߦॲཧ
   

   View Slide

10. ग़ྗͷDBॻ͖ࠐΈॲཧͷ෼཭
    • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ
    • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ
    • Ϟσϧͷ։ൃऀ͸CSVग़ྗ·Ͱ࣮૷ͯ͠ Docker Խ͓ͯ͘͠
    • ॻ͖ࠐΈॲཧ΍ΫϨσϯγϟϧ؅ཧ͸γεςϜͷ։ൃऀ͕࣮૷
    steps:
    - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮૷
    - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮૷ (ग़ྗ͸CSV)
    - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮૷
    # ※ग़ྗϑΝΠϧ͸ڞ༗ϘϦϡʔϜͰड͚౉͠
    

    View Slide

11. Ϟσϧͷ Continuous Delivery
    • Ӧۀઓུɾ޿ࠂग़ߘΛ૝ఆͨ͠ٻਓޮՌਪఆϞσϧ
    • ϚʔέςΟϯά୲౰ऀ޲͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻
    • ਪఆॲཧ͕׬ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽
    steps:
    - - name: estimate # ਪఆॲཧ
    - - name: upload-model # ࡞੒͞ΕͨϞσϧΛετϨʔδʹอଘ
    - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͠௚͢
    

    View Slide

12. Ϟσϧͷ Continuous Delivery (ଓ͖)
    • Ϗϡʔϫ΋ಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
    • kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ
    • Rolling Update ʹΑΓμ΢ϯλΠϜແ͠ͷϞσϧߋ৽΋Մೳ
    - name: update-viewer
    container:
    image: kubectl
    command: ["sh", "-c"]
    args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
    

    View Slide

13. ฒߦॲཧ
    • Webςετͷଟ࿹όϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ
    • ෳ਺ͷςετ͕૸͓ͬͯΓɺ֤ςετͷਪఆॲཧ͸ฒߦ࣮ߦ͍ͨ͠
    steps:
    - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ
    - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ਺͚ͩฒߦ࣮ߦ͢Δ
    # ग़ྗύϥϝʔλͷϦετΛ౉͢ͱͦͷ਺͚ͩίϯςφ্ཱ͕͕ͪΔ
    # Ϧετ͸ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON
    withParams: "{{steps.list-experiments.outputs.parameters.experiments}}"
    # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͠౉͢
    arguments:
    parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
    

    View Slide

14. ฒߦॲཧ (ଓ͖)
    templates:
    - name: list-experiments
    container: ...
    outputs:
    parameters:
    - name: experiments
    # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ
    valueFrom: {path: /output/experiments.json}
    - name: calc-weights
    container: ...
    inputs:
    parameters:
    # ύϥϝʔλΛೖྗ஋ͱͯ͠ड͚औΔ
    - name: experimentId
    

    View Slide

15. View Slide

16. ·ͱΊ
    • ෳ਺ίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹ૊ΊΔ
    • ͭͬͨ͘MLϞσϧΛ͢͹΍͘ӡ༻͍ͨ͠ͱ͖ʹศར
    هࣄ΋͋Γ·͢:
    Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔ؅ཧ
    

    View Slide