Kubernetes Meetup Tokyo #54 https://k8sjp.connpass.com/event/264501/ で発表したLT資料
Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LTKubeflowͰ͡ΊΔ ML on Kubernetes#3 Kubeflow Pipelines: Part 1
View Slide
Keita WatanabeMachine Learning Solutions Architect▶ AWS JapanͰSelf-managedͳMachine Learning ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ▶ Twitter: keitaw09▶ Linkedin: keitawatanabeαϯσΟΤΰͰग़ձͬͨτϦझຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢KubeflowͰ͡ΊΔML on Kubernetes• #1 Kubeflowͷ֓ཁͱηοτΞοϓ• #2 Kubeflow Notebooks• #3 Kubeflow Pipelines (ࠓճʂ)
ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
Kubeflow Pipelines ʢKFPʣͱKFPͷߏཁૉ• Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI• Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine• ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕ՄೳͳPython SDK• SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢ΔNotebook αϙʔτComponentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢ΔπʔϧUI: pipeline graph view Ұ෦ൈਮ
KFP SDKͷόʔδϣϯʹ͍ͭͯ• 2022/11/24ݱࡏɺKFP SDKʹҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ v1ʢStable Statusʣ/ v2ʢBeta Statusʣ• KFP SDK v2ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ KFP v1্Ͱಈ࡞Մೳͱͨ͠ͷ• ͜͜ͰKFP SDK v2Λ༻͍Δ• KFP SDK v2ͷར༻ʹKFP 1.6Ҏ߱ ͕ඞཁʢࠓճ1.8.4Λ༻ʣhttps://www.kubeflow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ(1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ(2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱͦͷؔΛఆٛ͢Δ(3) ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ(1)(2)(3)
͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ…• “/var/run/secrets/kubeflow/pipelines/token”͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ• ࣮ϚϧνϢʔβʔڥͰSDKΛ࣮ߦ͢ΔࡍʹɺServiceAccount token͕ ”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢Δඞཁ͕͋Δ ʢ͜ͷڥมͷσϑΥϧτͷύε͕ “/var/run/secrets/kubeflow/pipelines/token”ʣ• σϑΥϧτͷNotebookڥʹ͜ͷτʔΫϯ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δhttps://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kubeflow-subfrom-inside-clustersub
PodDefault• KubernetesʹPodʹରͯ͠ɺڥมVolumeͷใͳͲΛ͋ͱ͔ΒՃ͢ΔPodPresetͱ͍͏Ϧιʔεʢݱࡏ Alpha stageʣ͕ଘࡏ͢Δ• PodDefaultPodPreset૬ͷػೳΛKubeflowଆͰ ࣮ͨ͠ͷhttps://github.com/kubeflow/kubeflow/blob/master/components/admission-webhook/README.md
PodDefaultͷ͍ํ৽نNotebook࡞࣌ͷ ”Configurations”ΑΓ࡞ͨ͠PodDefaultΛબ͢Δ͜ͱ͕Ͱ͖Δ
τʔΫϯ͕Ϛϯτ͞Ε͍ͯΔ͜ͱ͕֬ೝͰ͖ΔύΠϓϥΠϯͷ࣮ߦʹޭ͠ɺExperimentͱRunͷৄࡉͷϦϯΫ͕දࣔ͞Εͨ
Run details
࣍ճ…࣮ફతͳPipelinesߏஙͷΛ͠·͢
ࢀߟࢿྉ• https://techblog.zozo.com/entry/aip-pipelines-impl#Kubeflow-Pipelines• https://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kubeflow-subfrom-inside-clustersub• https://www.kubeflow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kubeflow• https://www.kubeflow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/• https://www.kubeflow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kubeflow-subfrom-inside-clustersub