Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Argo Workflow によるMLジョブ管理

Argo Workflow によるMLジョブ管理

MACHINE LEARNING Meetup KANSAI #4
2019/3/27

Livesense Inc.
PRO

March 27, 2019
Tweet

More Decks by Livesense Inc.

Other Decks in Technology

Transcript

  1. Argo Workflow ʹΑΔMLδϣϒ؅ཧ
    Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژ౎ΦϑΟε)
    MACHINE LEARNING Meetup KANSAI #4 LT

    View Slide

  2. ͜Εͷ঺հ͠·͢

    View Slide

  3. https://argoproj.github.io/

    View Slide

  4. Կ͕Ͱ͖Δͷ͔
    "Container native workflow engine for Kubernetes"
    • ෳ਺ͷίϯςφΛ௚ྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ
    • σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λ૝ఆ
    • ৽όʔδϣϯͰ͸ DAG ΋αϙʔτ
    • Argo ϕʔεͷ༷ʑͳϓϩμΫτ
    • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ
    • Argo Event: ϫʔΫϑϩʔͷτϦΨ

    View Slide

  5. apiVersion: argoproj.io/v1alpha1
    kind: Workflow
    metadata:
    generateName: ml-workflow-
    spec:
    entrypoint: main
    templates:
    - name: main
    steps:
    - - name: load-dataset
    template: load-dataset
    - - name: train-model-1
    template: train-model
    arguments:
    parameters: [{name: model, value: model1}]
    - name: train-model-2
    template: train-model
    arguments:
    parameters: [{name: model, value: model2}]
    ...

    View Slide

  6. ...
    - name: load-dataset
    container:
    image: postgres:latest
    command: [sh, -c]
    args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"]
    - name: train-model
    inputs:
    parameters: [{name: model}]
    container:
    image: train-model
    command: [sh -c]
    args: ["python train_model.py --model={{inputs.parameters.model}}"]

    View Slide

  7. View Slide

  8. ͳͥ࢖͏ͷ͔
    ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ৐͍ͤͨʯ
    • MLϞσϧͷ։ൃऀ
    • SQL Ͱσʔλऔಘ ʙ Ϟσϧ΍༧ଌ஋ΛϑΝΠϧʹग़ྗ
    • Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠
    • MLγεςϜͷ։ൃऀ
    • DBIO ΍Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮૷
    • Argo Ͱ͢΂ͯΛ૊Έ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ
    → ίϯςφ୯ҐͰ໾ׂ෼୲

    View Slide

  9. ϦϒηϯεͰͷར༻ྫ
    • ग़ྗͷDBॻ͖ࠐΈॲཧͷ෼཭
    • Ϟσϧͷ Continuous Delivery
    • ฒߦॲཧ

    View Slide

  10. ग़ྗͷDBॻ͖ࠐΈॲཧͷ෼཭
    • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ
    • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ
    • Ϟσϧͷ։ൃऀ͸CSVग़ྗ·Ͱ࣮૷ͯ͠ Docker Խ͓ͯ͘͠
    • ॻ͖ࠐΈॲཧ΍ΫϨσϯγϟϧ؅ཧ͸γεςϜͷ։ൃऀ͕࣮૷
    steps:
    - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮૷
    - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮૷ (ग़ྗ͸CSV)
    - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮૷
    # ※ग़ྗϑΝΠϧ͸ڞ༗ϘϦϡʔϜͰड͚౉͠

    View Slide

  11. Ϟσϧͷ Continuous Delivery
    • Ӧۀઓུɾ޿ࠂग़ߘΛ૝ఆͨ͠ٻਓޮՌਪఆϞσϧ
    • ϚʔέςΟϯά୲౰ऀ޲͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻
    • ਪఆॲཧ͕׬ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽
    steps:
    - - name: estimate # ਪఆॲཧ
    - - name: upload-model # ࡞੒͞ΕͨϞσϧΛετϨʔδʹอଘ
    - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͠௚͢

    View Slide

  12. Ϟσϧͷ Continuous Delivery (ଓ͖)
    • Ϗϡʔϫ΋ಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
    • kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ
    • Rolling Update ʹΑΓμ΢ϯλΠϜແ͠ͷϞσϧߋ৽΋Մೳ
    - name: update-viewer
    container:
    image: kubectl
    command: ["sh", "-c"]
    args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]

    View Slide

  13. ฒߦॲཧ
    • Webςετͷଟ࿹όϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ
    • ෳ਺ͷςετ͕૸͓ͬͯΓɺ֤ςετͷਪఆॲཧ͸ฒߦ࣮ߦ͍ͨ͠
    steps:
    - - name: list-experiments # ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ
    - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ਺͚ͩฒߦ࣮ߦ͢Δ
    # ग़ྗύϥϝʔλͷϦετΛ౉͢ͱͦͷ਺͚ͩίϯςφ্ཱ͕͕ͪΔ
    # Ϧετ͸ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON
    withParams: "{{steps.list-experiments.outputs.parameters.experiments}}"
    # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͠౉͢
    arguments:
    parameters: [{name: experimentId, value: "{{item.experimentId}}"}]

    View Slide

  14. ฒߦॲཧ (ଓ͖)
    templates:
    - name: list-experiments
    container: ...
    outputs:
    parameters:
    - name: experiments
    # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ
    valueFrom: {path: /output/experiments.json}
    - name: calc-weights
    container: ...
    inputs:
    parameters:
    # ύϥϝʔλΛೖྗ஋ͱͯ͠ड͚औΔ
    - name: experimentId

    View Slide

  15. View Slide

  16. ·ͱΊ
    • ෳ਺ίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹ૊ΊΔ
    • ͭͬͨ͘MLϞσϧΛ͢͹΍͘ӡ༻͍ͨ͠ͱ͖ʹศར
    هࣄ΋͋Γ·͢:
    Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔ؅ཧ

    View Slide