$30 off During Our Annual Pro Sale. View Details »

ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1

ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1

Kubernetes Meetup Tokyo #54 https://k8sjp.connpass.com/event/264501/ で発表したLT資料

Keita Watanabe

November 24, 2022
Tweet

More Decks by Keita Watanabe

Other Decks in Technology

Transcript

  1. Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT
    KubeflowͰ͸͡ΊΔ

    ML on Kubernetes
    #3 Kube
    fl
    ow Pipelines: Part 1

    View Slide

  2. Keita Watanabe
    Machine Learning Solutions Architect
    ▶ AWS JapanͰSelf-managedͳMachine Learning

    ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect

    ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠

    ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ

    ▶ Twitter: keitaw09

    ▶ Linkedin: keitawatanabe

    αϯσΟΤΰͰग़ձͬͨτϦ
    झຯ

    View Slide

  3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢
    KubeflowͰ͸͡ΊΔML on Kubernetes
    • #1 Kube
    fl
    owͷ֓ཁͱηοτΞοϓ

    • #2 Kube
    fl
    ow Notebooks

    • #3 Kube
    fl
    ow Pipelines (ࠓճʂ)

    View Slide

  4. ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ

    View Slide

  5. Kubeflow Pipelines ʢKFPʣͱ͸
    KFPͷߏ੒ཁૉ
    • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI

    • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine

    • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
    Python SDK

    • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ
    Notebook αϙʔτ
    Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·
    ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ
    UI: pipeline graph view Ұ෦ൈਮ

    View Slide

  6. KFP SDKͷόʔδϣϯʹ͍ͭͯ
    • 2022/11/24ݱࡏɺKFP SDKʹ͸ҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ

    v1ʢStable Statusʣ/ v2ʢBeta Statusʣ

    • KFP SDK v2͸ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ

    KFP v1্Ͱಈ࡞Մೳͱͨ͠΋ͷ

    • ͜͜Ͱ͸KFP SDK v2Λ༻͍Δ

    • KFP SDK v2ͷར༻ʹ͸KFP 1.6Ҏ߱

    ͕ඞཁʢࠓճ͸1.8.4Λ࢖༻ʣ
    https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/

    View Slide

  7. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ
    (1) ίϯϙʔωϯτͷఆٛ

    ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ
    ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ

    (2) ύΠϓϥΠϯͷఆٛ

    ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ
    ͦͷؔ܎Λఆٛ͢Δ

    (3) ύΠϓϥΠϯͷ࣮ߦ

    ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ
    (1)
    (2)
    (3)

    View Slide

  8. ͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ…
    • “/var/run/secrets/kube
    fl
    ow/pipelines/token”
    ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ

    • ࣮͸ϚϧνϢʔβʔ؀ڥͰSDKΛ࣮ߦ͢Δࡍ
    ʹ͸ɺServiceAccount token͕

    ”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢
    Δඞཁ͕͋Δ

    ʢ͜ͷ؀ڥม਺ͷσϑΥϧτͷύε͕

    “/var/run/secrets/kube
    fl
    ow/pipelines/token”ʣ

    • σϑΥϧτͷNotebook؀ڥʹ͸͜ͷτʔΫϯ
    ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ
    https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube
    fl
    ow-subfrom-inside-clustersub

    View Slide

  9. PodDefault
    • Kubernetesʹ͸Podʹରͯ͠ɺ
    ؀ڥม਺΍Volumeͷ৘ใͳͲ
    Λ͋ͱ͔Β௥Ճ͢ΔPodPreset
    ͱ͍͏Ϧιʔεʢݱࡏ

    Alpha stageʣ͕ଘࡏ͢Δ

    • PodDefault͸PodPreset૬౰ͷ
    ػೳΛKube
    fl
    owଆͰ

    ࣮૷ͨ͠΋ͷ
    https://github.com/kube
    fl
    ow/kube
    fl
    ow/blob/master/components/admission-webhook/README.md

    View Slide

  10. PodDefaultͷ࢖͍ํ
    ৽نNotebook࡞੒࣌ͷ

    ”Con
    fi
    gurations”ΑΓ࡞੒ͨ͠
    PodDefaultΛબ୒͢Δ͜ͱ͕Ͱ͖Δ

    View Slide

  11. τʔΫϯ͕Ϛ΢ϯτ͞Ε͍ͯΔ͜ͱ
    ͕֬ೝͰ͖Δ
    ύΠϓϥΠϯͷ࣮ߦʹ੒ޭ͠ɺ
    ExperimentͱRunͷৄࡉ΁ͷϦϯΫ
    ͕දࣔ͞Εͨ

    View Slide

  12. Run details

    View Slide

  13. ࣍ճ͸…
    ࣮ફతͳPipelinesߏஙͷ࿩Λ͠·͢

    View Slide

  14. ࢀߟࢿྉ
    • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube
    fl
    ow-Pipelines

    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk/connect-api/
    #full-kube
    fl
    ow-subfrom-inside-clustersub

    • https://www.kube
    fl
    ow.org/docs/components/notebooks/overview

    • https://github.com/aws-samples/aws-do-kube
    fl
    ow

    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk-v2/v2-
    compatibility/

    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk/connect-api/
    #full-kube
    fl
    ow-subfrom-inside-clustersub

    View Slide