ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1

Kubernetes Meetup Tokyo #54 https://k8sjp.connpass.com/event/264501/ で発表したLT資料

Transcript

1. Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT
   KubeflowͰ͸͡ΊΔ

   ML on Kubernetes
   #3 Kube
   fl
   ow Pipelines: Part 1
   

   View Slide

2. Keita Watanabe
   Machine Learning Solutions Architect
   ▶ AWS JapanͰSelf-managedͳMachine Learning

   ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect
   
   ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠

   ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ
   
   ▶ Twitter: keitaw09
   
   ▶ Linkedin: keitawatanabe
   
   αϯσΟΤΰͰग़ձͬͨτϦ
   झຯ
   

   View Slide

3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢
   KubeflowͰ͸͡ΊΔML on Kubernetes
   • #1 Kube
   fl
   owͷ֓ཁͱηοτΞοϓ
   
   • #2 Kube
   fl
   ow Notebooks
   
   • #3 Kube
   fl
   ow Pipelines (ࠓճʂ)
   

   View Slide

4. ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
   

   View Slide

5. Kubeflow Pipelines ʢKFPʣͱ͸
   KFPͷߏ੒ཁૉ
   • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI
   
   • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine
   
   • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
   Python SDK
   
   • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ
   Notebook αϙʔτ
   Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·
   ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ
   UI: pipeline graph view Ұ෦ൈਮ
   

   View Slide

6. KFP SDKͷόʔδϣϯʹ͍ͭͯ
   • 2022/11/24ݱࡏɺKFP SDKʹ͸ҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ

   v1ʢStable Statusʣ/ v2ʢBeta Statusʣ
   
   • KFP SDK v2͸ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ

   KFP v1্Ͱಈ࡞Մೳͱͨ͠΋ͷ
   
   • ͜͜Ͱ͸KFP SDK v2Λ༻͍Δ
   
   • KFP SDK v2ͷར༻ʹ͸KFP 1.6Ҏ߱

   ͕ඞཁʢࠓճ͸1.8.4Λ࢖༻ʣ
   https://www.kube
   fl
   ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
   

   View Slide

7. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ
   (1) ίϯϙʔωϯτͷఆٛ

   ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ
   ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ
   
   (2) ύΠϓϥΠϯͷఆٛ

   ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ
   ͦͷؔ܎Λఆٛ͢Δ
   
   (3) ύΠϓϥΠϯͷ࣮ߦ

   ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ
   (1)
   (2)
   (3)
   

   View Slide

8. ͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ…
   • “/var/run/secrets/kube
   fl
   ow/pipelines/token”
   ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ
   
   • ࣮͸ϚϧνϢʔβʔ؀ڥͰSDKΛ࣮ߦ͢Δࡍ
   ʹ͸ɺServiceAccount token͕

   ”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢
   Δඞཁ͕͋Δ

   ʢ͜ͷ؀ڥม਺ͷσϑΥϧτͷύε͕

   “/var/run/secrets/kube
   fl
   ow/pipelines/token”ʣ
   
   • σϑΥϧτͷNotebook؀ڥʹ͸͜ͷτʔΫϯ
   ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ
   https://www.kube
   fl
   ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube
   fl
   ow-subfrom-inside-clustersub
   

   View Slide

9. PodDefault
   • Kubernetesʹ͸Podʹରͯ͠ɺ
   ؀ڥม਺΍Volumeͷ৘ใͳͲ
   Λ͋ͱ͔Β௥Ճ͢ΔPodPreset
   ͱ͍͏Ϧιʔεʢݱࡏ

   Alpha stageʣ͕ଘࡏ͢Δ
   
   • PodDefault͸PodPreset૬౰ͷ
   ػೳΛKube
   fl
   owଆͰ

   ࣮૷ͨ͠΋ͷ
   https://github.com/kube
   fl
   ow/kube
   fl
   ow/blob/master/components/admission-webhook/README.md
   

   View Slide

10. PodDefaultͷ࢖͍ํ
    ৽نNotebook࡞੒࣌ͷ

    ”Con
    fi
    gurations”ΑΓ࡞੒ͨ͠
    PodDefaultΛબ୒͢Δ͜ͱ͕Ͱ͖Δ
    

    View Slide

11. τʔΫϯ͕Ϛ΢ϯτ͞Ε͍ͯΔ͜ͱ
    ͕֬ೝͰ͖Δ
    ύΠϓϥΠϯͷ࣮ߦʹ੒ޭ͠ɺ
    ExperimentͱRunͷৄࡉ΁ͷϦϯΫ
    ͕දࣔ͞Εͨ
    

    View Slide

12. Run details
    

    View Slide

13. ࣍ճ͸…
    ࣮ફతͳPipelinesߏஙͷ࿩Λ͠·͢
    

    View Slide

14. ࢀߟࢿྉ
    • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube
    fl
    ow-Pipelines
    
    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk/connect-api/
    #full-kube
    fl
    ow-subfrom-inside-clustersub
    
    • https://www.kube
    fl
    ow.org/docs/components/notebooks/overview
    
    • https://github.com/aws-samples/aws-do-kube
    fl
    ow
    
    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk-v2/v2-
    compatibility/
    
    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk/connect-api/
    #full-kube
    fl
    ow-subfrom-inside-clustersub
    

    View Slide