ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT KubeflowͰ͸͡ΊΔ 
ML on Kubernetes #4 Kube fl ow Pipelines: Part 2

Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning 
ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠  ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ

ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •
#2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines: Part1 (Kube fl ow Pipelinesͷಋೖฤ) • #4 Kube fl ow Pipelines: Part2 (Kube fl ow Pipelinesͷ࣮ફฤ) ࠓճʂ

લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ લճͷ಺༰: KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ

SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ (1) ίϯϙʔωϯτͷఆٛ  ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ  ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔ܎Λఆٛ͢Δ (3)
ύΠϓϥΠϯͷ࣮ߦ  ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)

ೖग़ྗͷܕʹ͍ͭͯ • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ • Parameters • Int, double, fl oat,
str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍ Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ • Artifact • σʔλͷ΍ΓऔΓʹ༻͍Δɻ • ϑΝΠϧೖग़ྗͷྫ • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

ෳ਺ͷValueΛฦ͍ͨ͠৔߹

ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ • Input/Output • δΣωϦοΫͳܕώϯτ • InputBinaryFile/OutputBinaryFile • ೖग़ྗ͕io.ByteIOͷͱ͖ •
InputPath/OutputPath • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ ͖ • InputTextFile/OutputTextFile • ೖग़ྗ͕io.TextIOWrapperͷͱ͖ ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘ ύΠϓϥΠϯͷఆٛ ίϯϙʔωϯτͷఆٛ

Component ಡΈࠐΈํ๏ ύΠϓϥΠϯͷఆٛ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url 
URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text  YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

Component ಡΈࠐΈํ๏ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url  URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ
• load_component_from_text  YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

ύΠϓϥΠϯͷ෼ذɾϧʔϓ

࣍ճ͸… • Kube fl ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢

ࢀߟࢿྉ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/component- development/ • https://medium.datadriveninvestor.com/conditional-kube fl ow-pipeline-for-
dummies-972fd81f9de2 • https://github.com/kube fl ow/pipelines/blob/ d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/ train_until_good/train_until_good.py

ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Keita Watanabe

More Decks by Keita Watanabe

Other Decks in Technology

Featured

Transcript

Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT KubeflowͰ͸͡ΊΔ

Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning

ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •

લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ લճͷ಺༰: KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ

SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ (1) ίϯϙʔωϯτͷఆٛ  ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ  ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔ܎Λఆٛ͢Δ (3)

ೖग़ྗͷܕʹ͍ͭͯ • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ • Parameters • Int, double, fl oat,

ෳ਺ͷValueΛฦ͍ͨ͠৔߹

ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ • Input/Output • δΣωϦοΫͳܕώϯτ • InputBinaryFile/OutputBinaryFile • ೖग़ྗ͕io.ByteIOͷͱ͖ •

Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘ ύΠϓϥΠϯͷఆٛ ίϯϙʔωϯτͷఆٛ

Component ಡΈࠐΈํ๏ ύΠϓϥΠϯͷఆٛ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url

Component ಡΈࠐΈํ๏ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url  URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ

ύΠϓϥΠϯͷ෼ذɾϧʔϓ

࣍ճ͸… • Kube fl ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢

ࢀߟࢿྉ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/component- development/ • https://medium.datadriveninvestor.com/conditional-kube fl ow-pipeline-for-