Kubernetes Meetup Tokyo #55 (https://k8sjp.connpass.com/event/267620/)にて発表した資料です。 引き続き、Kubeflow Pipelinesについて取り上げています。
Keita Watanabe 2022/12/22 Kubernetes Meetup Tokyo # 55 LTKubeflowͰ͡ΊΔ ML on Kubernetes#4 Kubeflow Pipelines: Part 2
View Slide
Keita WatanabeMachine Learning Solutions Architect▶ AWS JapanͰSelf-managedͳMachine Learning ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ▶ Twitter: keitaw09▶ Linkedin: keitawatanabeαϯσΟΤΰͰग़ձͬͨτϦझຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢KubeflowͰ͡ΊΔML on Kubernetes• #1 Kubeflowͷ֓ཁͱηοτΞοϓ• #2 Kubeflow Notebooks• #3 Kubeflow Pipelines: Part1 (Kubeflow Pipelinesͷಋೖฤ)• #4 Kubeflow Pipelines: Part2 (Kubeflow Pipelinesͷ࣮ફฤ)ࠓճʂ
લճ·Ͱͷ෮श: Kubeflow Pipelines (KFP)ͷಋೖComponentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ·ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧલճͷ༰:KFPͷಋೖͱαϯϓϧύΠϓϥΠϯͷ࣮ߦ
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ(1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ(2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱͦͷؔΛఆٛ͢Δ(3) ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ(1)(2)(3)
ೖग़ྗͷܕʹ͍ͭͯ• ೖग़ྗɺσʔλͷछྨʹΑͬͯparametersͱartifactsʹେผ͞ΕΔ• Parameters• Int, double,float, strɻ௨ৗύΠϓϥΠϯͷৼΔ͍Λ੍ޚ͢Δͷʹ༻͍Δɻ-> ෳͷParametersΛฦ͍ͨ͠߹ʁ• Artifact• σʔλͷΓऔΓʹ༻͍Δɻ• ϑΝΠϧೖग़ྗͷྫ• ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
ෳͷValueΛฦ͍ͨ͠߹
ArtifactʢDatasetʣͷྫɿϑΝΠϧͷೖग़ྗ• Input/Output• δΣωϦοΫͳܕώϯτ• InputBinaryFile/OutputBinaryFile• ೖग़ྗ͕io.ByteIOͷͱ͖• InputPath/OutputPath• ೖग़ྗ͕ϑΝΠϧͷύεͷͱ͖• InputTextFile/OutputTextFile• ೖग़ྗ͕io.TextIOWrapperͷͱ͖ϑΝΠϧೖग़ྗʹ༻͍Δ͜ͱͷͰ͖Δܕώϯτ
Componentͷߏஙํ๏ ͦͷ2: YamlΛॻ͘ύΠϓϥΠϯͷఆٛίϯϙʔωϯτͷఆٛ
Component ಡΈࠐΈํ๏ύΠϓϥΠϯͷఆٛ • load_component_from_file ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ• load_component_from_url URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ• load_component_from_text YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
Component ಡΈࠐΈํ๏• load_component_from_file ϩʔΧϧͷYamlϑΝΠϧΛಡΈࠐΉ• load_component_from_url URL͔ΒYamlϑΝΠϧΛಡΈࠐΉ• load_component_from_text YamlϑΝΠϧΛString͔ΒಡΈࠐΉ
ύΠϓϥΠϯͷذɾϧʔϓ
࣍ճ…• Kubeflow্Ͱͷ࣮ࡍͷML Model TrainingΛऔΓ্͍͛ͨͱࢥ͍·͢
ࢀߟࢿྉ• https://www.kubeflow.org/docs/components/pipelines/v1/sdk-v2/component-development/• https://medium.datadriveninvestor.com/conditional-kubeflow-pipeline-for-dummies-972fd81f9de2• https://github.com/kubeflow/pipelines/blob/d5b428416871fdc17325b1f04674b5ac62f17894/samples/core/train_until_good/train_until_good.py