ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Kubernetes Meetup Tokyo #55 （https://k8sjp.connpass.com/event/267620/）にて発表した資料です。 引き続き、Kubeflow Pipelinesについて取り上げています。

Transcript

1. Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT
   KubeflowͰ͸͡ΊΔ

   ML on Kubernetes
   #4 Kube
   fl
   ow Pipelines: Part 2
   

   View Slide

2. Keita Watanabe
   Machine Learning Solutions Architect
   ▶ AWS JapanͰSelf-managedͳMachine Learning

   ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect
   
   ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠

   ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ
   
   ▶ Twitter: keitaw09
   
   ▶ Linkedin: keitawatanabe
   
   αϯσΟΤΰͰग़ձͬͨτϦ
   झຯ
   

   View Slide

3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢
   KubeflowͰ͸͡ΊΔML on Kubernetes
   • #1 Kube
   fl
   owͷ֓ཁͱηοτΞοϓ
   
   • #2 Kube
   fl
   ow Notebooks
   
   • #3 Kube
   fl
   ow Pipelines: Part1 (Kube
   fl
   ow Pipelinesͷಋೖฤ)
   
   • #4 Kube
   fl
   ow Pipelines: Part2 (Kube
   fl
   ow Pipelinesͷ࣮ફฤ)
   ࠓճʂ
   

   View Slide

4. લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ
   Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·
   ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ
   લճͷ಺༰:
   
   KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ
   

   View Slide

5. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ
   (1) ίϯϙʔωϯτͷఆٛ

   ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ
   ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ
   
   (2) ύΠϓϥΠϯͷఆٛ

   ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ
   ͦͷؔ܎Λఆٛ͢Δ
   
   (3) ύΠϓϥΠϯͷ࣮ߦ

   ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ
   (1)
   (2)
   (3)
   

   View Slide

6. ೖग़ྗͷܕʹ͍ͭͯ
   • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ
   
   • Parameters
   
   • Int, double,
   fl
   oat, str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍
   Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ
   
   • Artifact
   
   • σʔλͷ΍ΓऔΓʹ༻͍Δɻ
   
   • ϑΝΠϧೖग़ྗͷྫ
   
   • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
   

   View Slide

7. ෳ਺ͷValueΛฦ͍ͨ͠৔߹
   

   View Slide

8. ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ
   • Input/Output
   
   • δΣωϦοΫͳܕώϯτ
   
   • InputBinaryFile/OutputBinaryFile
   
   • ೖग़ྗ͕io.ByteIOͷͱ͖
   
   • InputPath/OutputPath
   
   • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ
   ͖
   
   • InputTextFile/OutputTextFile
   
   • ೖग़ྗ͕io.TextIOWrapperͷͱ͖
   ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
   

   View Slide

9. Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘
   ύΠϓϥΠϯͷఆٛ
   ίϯϙʔωϯτͷఆٛ
   

   View Slide

10. Component ಡΈࠐΈํ๏
    ύΠϓϥΠϯͷఆٛ • load_component_from_
    fi
    le

    ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ
    
    • load_component_from_url

    URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ
    
    • load_component_from_text

    YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
    

    View Slide

11. Component ಡΈࠐΈํ๏
    • load_component_from_
    fi
    le

    ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ
    
    • load_component_from_url

    URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ
    
    • load_component_from_text

    YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
    

    View Slide

12. ύΠϓϥΠϯͷ෼ذɾϧʔϓ
    

    View Slide

13. ࣍ճ͸…
    • Kube
    fl
    ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢
    

    View Slide

14. ࢀߟࢿྉ
    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk-v2/component-
    development/
    
    • https://medium.datadriveninvestor.com/conditional-kube
    fl
    ow-pipeline-for-
    dummies-972fd81f9de2
    
    • https://github.com/kube
    fl
    ow/pipelines/blob/
    d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/
    train_until_good/train_until_good.py
    

    View Slide