ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1

Kubernetes Meetup Tokyo #54 https://k8sjp.connpass.com/event/264501/ で発表したLT資料

Transcript

1. Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT KubeflowͰ͸͡ΊΔ
 ML on Kubernetes #3 Kube fl ow Pipelines: Part 1
2. Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
 ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠
 ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ • #2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines (ࠓճʂ)
4. ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
5. Kubeflow Pipelines ʢKFPʣͱ͸ KFPͷߏ੒ཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
6. KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹ͸ҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ
 v1ʢStable Statusʣ/ v2ʢBeta Statusʣ • KFP SDK v2͸ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ
 KFP v1্Ͱಈ࡞Մೳͱͨ͠΋ͷ • ͜͜Ͱ͸KFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹ͸KFP 1.6Ҏ߱
 ͕ඞཁʢࠓճ͸1.8.4Λ࢖༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
7. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ (1) ίϯϙʔωϯτͷఆٛ
 ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ
 ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔ܎Λఆٛ͢Δ (3) ύΠϓϥΠϯͷ࣮ߦ
 ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
8. ͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮͸ϚϧνϢʔβʔ؀ڥͰSDKΛ࣮ߦ͢Δࡍ ʹ͸ɺServiceAccount token͕
 ”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ
 ʢ͜ͷ؀ڥม਺ͷσϑΥϧτͷύε͕
 “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebook؀ڥʹ͸͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
9. PodDefault • Kubernetesʹ͸Podʹରͯ͠ɺ ؀ڥม਺΍Volumeͷ৘ใͳͲ Λ͋ͱ͔Β௥Ճ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ
 Alpha stageʣ͕ଘࡏ͢Δ • PodDefault͸PodPreset૬౰ͷ ػೳΛKube fl owଆͰ
 ࣮૷ͨ͠΋ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
10. PodDefaultͷ࢖͍ํ ৽نNotebook࡞੒࣌ͷ
 ”Con fi gurations”ΑΓ࡞੒ͨ͠ PodDefaultΛબ୒͢Δ͜ͱ͕Ͱ͖Δ
11. τʔΫϯ͕Ϛ΢ϯτ͞Ε͍ͯΔ͜ͱ ͕֬ೝͰ͖Δ ύΠϓϥΠϯͷ࣮ߦʹ੒ޭ͠ɺ ExperimentͱRunͷৄࡉ΁ͷϦϯΫ ͕දࣔ͞Εͨ
12. Run details
13. ࣍ճ͸… ࣮ફతͳPipelinesߏஙͷ࿩Λ͠·͢
14. ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub