Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Kubernetes Meetup Tokyo #55 (https://k8sjp.connpass.com/event/267620/)にて発表した資料です。 引き続き、Kubeflow Pipelinesについて取り上げています。

Keita Watanabe

December 22, 2022
Tweet

More Decks by Keita Watanabe

Other Decks in Technology

Transcript

  1. Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT
    KubeflowͰ͸͡ΊΔ

    ML on Kubernetes
    #4 Kube
    fl
    ow Pipelines: Part 2

    View Slide

  2. Keita Watanabe
    Machine Learning Solutions Architect
    ▶ AWS JapanͰSelf-managedͳMachine Learning

    ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect

    ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠

    ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ

    ▶ Twitter: keitaw09

    ▶ Linkedin: keitawatanabe

    αϯσΟΤΰͰग़ձͬͨτϦ
    झຯ

    View Slide

  3. ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢
    KubeflowͰ͸͡ΊΔML on Kubernetes
    • #1 Kube
    fl
    owͷ֓ཁͱηοτΞοϓ

    • #2 Kube
    fl
    ow Notebooks

    • #3 Kube
    fl
    ow Pipelines: Part1 (Kube
    fl
    ow Pipelinesͷಋೖฤ)

    • #4 Kube
    fl
    ow Pipelines: Part2 (Kube
    fl
    ow Pipelinesͷ࣮ફฤ)
    ࠓճʂ

    View Slide

  4. લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ
    Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·
    ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ
    લճͷ಺༰:

    KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ

    View Slide

  5. SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ
    (1) ίϯϙʔωϯτͷఆٛ

    ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ
    ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ

    (2) ύΠϓϥΠϯͷఆٛ

    ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ
    ͦͷؔ܎Λఆٛ͢Δ

    (3) ύΠϓϥΠϯͷ࣮ߦ

    ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ
    (1)
    (2)
    (3)

    View Slide

  6. ೖग़ྗͷܕʹ͍ͭͯ
    • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ

    • Parameters

    • Int, double,
    fl
    oat, str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍
    Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ

    • Artifact

    • σʔλͷ΍ΓऔΓʹ༻͍Δɻ

    • ϑΝΠϧೖग़ྗͷྫ

    • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

    View Slide

  7. ෳ਺ͷValueΛฦ͍ͨ͠৔߹

    View Slide

  8. ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ
    • Input/Output

    • δΣωϦοΫͳܕώϯτ

    • InputBinaryFile/OutputBinaryFile

    • ೖग़ྗ͕io.ByteIOͷͱ͖

    • InputPath/OutputPath

    • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ
    ͖

    • InputTextFile/OutputTextFile

    • ೖग़ྗ͕io.TextIOWrapperͷͱ͖
    ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

    View Slide

  9. Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘
    ύΠϓϥΠϯͷఆٛ
    ίϯϙʔωϯτͷఆٛ

    View Slide

  10. Component ಡΈࠐΈํ๏
    ύΠϓϥΠϯͷఆٛ • load_component_from_
    fi
    le

    ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ

    • load_component_from_url

    URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ

    • load_component_from_text

    YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

    View Slide

  11. Component ಡΈࠐΈํ๏
    • load_component_from_
    fi
    le

    ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ

    • load_component_from_url

    URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ

    • load_component_from_text

    YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

    View Slide

  12. ύΠϓϥΠϯͷ෼ذɾϧʔϓ

    View Slide

  13. ࣍ճ͸…
    • Kube
    fl
    ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢

    View Slide

  14. ࢀߟࢿྉ
    • https://www.kube
    fl
    ow.org/docs/components/pipelines/v1/sdk-v2/component-
    development/

    • https://medium.datadriveninvestor.com/conditional-kube
    fl
    ow-pipeline-for-
    dummies-972fd81f9de2

    • https://github.com/kube
    fl
    ow/pipelines/blob/
    d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/
    train_until_good/train_until_good.py

    View Slide