ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2

Slide 1

Slide 1 text

Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LT KubeflowͰ͸͡ΊΔ  ML on Kubernetes #4 Kube fl ow Pipelines: Part 2

Slide 2

Slide 2 text

Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning  ͷҊ݅Λ୲౰͍ͯ͠ΔSolutions Architect ▶ લ৬Ͱ͸Datascientist/ML Researcherͱͯ͠  ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ

Slide 3

Slide 3 text

ͪ͜ΒͷLT͸γϦʔζ΋ͷͰ͢ KubeflowͰ͸͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ • #2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines: Part1 (Kube fl ow Pipelinesͷಋೖฤ) • #4 Kube fl ow Pipelines: Part2 (Kube fl ow Pipelinesͷ࣮ફฤ) ࠓճʂ

Slide 4

Slide 4 text

લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖ Componentͱͯ͠ɺ֤εςοϓΛ࣮૷͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ લճͷ಺༰: KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ

Slide 5

Slide 5 text

SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮૷ྫ (1) ίϯϙʔωϯτͷఆٛ  ͜͜Ͱ͸଍͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ  ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔ܎Λఆٛ͢Δ (3) ύΠϓϥΠϯͷ࣮ߦ  ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)

Slide 6

Slide 6 text

ೖग़ྗͷܕʹ͍ͭͯ • ೖग़ྗ͸ɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ • Parameters • Int, double, fl oat, str౳ɻ௨ৗύΠϓϥΠϯͷৼΔ෣͍Λ੍ޚ͢Δͷʹ༻͍ Δɻ-> ෳ਺ͷParametersΛฦ͍ͨ͠৔߹͸ʁ • Artifact • σʔλͷ΍ΓऔΓʹ༻͍Δɻ • ϑΝΠϧೖग़ྗͷྫ • ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

Slide 7

Slide 7 text

ෳ਺ͷValueΛฦ͍ͨ͠৔߹

Slide 8

Slide 8 text

ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ • Input/Output • δΣωϦοΫͳܕώϯτ • InputBinaryFile/OutputBinaryFile • ೖग़ྗ͕io.ByteIOͷͱ͖ • InputPath/OutputPath • ೖग़ྗ͕ϑΝΠϧ΁ͷύεͷͱ ͖ • InputTextFile/OutputTextFile • ೖग़ྗ͕io.TextIOWrapperͷͱ͖ ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ

Slide 9

Slide 9 text

Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘ ύΠϓϥΠϯͷఆٛ ίϯϙʔωϯτͷఆٛ

Slide 10

Slide 10 text

Component ಡΈࠐΈํ๏ ύΠϓϥΠϯͷఆٛ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url  URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text  YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

Slide 11

Slide 11 text

Component ಡΈࠐΈํ๏ • load_component_from_ fi le  ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_url  URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ • load_component_from_text  YamlϑΝΠϧΛString͔ΒಡΈࠐΉ

Slide 12

Slide 12 text

ύΠϓϥΠϯͷ෼ذɾϧʔϓ

Slide 13

Slide 13 text

࣍ճ͸… • Kube fl ow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢

Slide 14

Slide 14 text

ࢀߟࢿྉ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/component- development/ • https://medium.datadriveninvestor.com/conditional-kube fl ow-pipeline-for- dummies-972fd81f9de2 • https://github.com/kube fl ow/pipelines/blob/ d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/ train_until_good/train_until_good.py