Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ML on Kubernetes with Kubeflow #3 Kubeflow Pipe...
Search
Keita Watanabe
November 24, 2022
Technology
0
1.1k
ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1
Kubernetes Meetup Tokyo #54
https://k8sjp.connpass.com/event/264501/
で発表したLT資料
Keita Watanabe
November 24, 2022
Tweet
Share
More Decks by Keita Watanabe
See All by Keita Watanabe
Building foundation model on AWS
keitaw
0
3
[re:Invent2024 Chalktalk] Cost-effectively deploy PyTorch LLMs on AWS Inferentia using Amazon EKS
keitaw
0
36
AWS Summit New York 2024: CMP 301 Demystifying the ML software stack on Amazon EC2 accelerated instances
keitaw
0
230
re:Invent 2023 CMP319 Deploy LLMs with AWS Inferentia & Ray to optimize performance and cost
keitaw
0
18
re:Invent 2023: CMP332 De-mystifying ML software stack on Amazon EC2 accelerated instances
keitaw
0
93
AWS における LLM・GenAI 大規模学習への取り組み / Large scale GenAI・LLM training on AWS
keitaw
1
500
Amazon EC2 シリコン革命 / Amazon EC2 Silicon Innovation
keitaw
0
160
Introduction to Kserve
keitaw
0
490
ML on Kubernetes with Kubeflow #4 Kubeflow Pipelines: Part2
keitaw
0
1.1k
Other Decks in Technology
See All in Technology
株式会社EventHub・エンジニア採用資料
eventhub
0
4.3k
Developers Summit 2025 浅野卓也(13-B-7 LegalOn Technologies)
legalontechnologies
PRO
1
1.4k
ユーザーストーリーマッピングから始めるアジャイルチームと並走するQA / Starting QA with User Story Mapping
katawara
0
270
2.5Dモデルのすべて
yu4u
2
950
短縮URLをお手軽に導入しよう
nakasho
0
110
OpenID BizDay#17 みんなの銀行による身元確認結果の活用 / 20250219-BizDay17-KYC-minna-no-ginko
oidfj
0
170
クラウドサービス事業者におけるOSS
tagomoris
3
960
Culture Deck
optfit
0
500
OpenID BizDay#17 KYC WG活動報告(法人) / 20250219-BizDay17-KYC-legalidentity
oidfj
0
410
2025-02-21 ゆるSRE勉強会 Enhancing SRE Using AI
yoshiiryo1
1
430
Oracle Cloud Infrastructure:2025年2月度サービス・アップデート
oracle4engineer
PRO
1
380
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
15
4.1k
Featured
See All Featured
A Modern Web Designer's Workflow
chriscoyier
693
190k
Building Applications with DynamoDB
mza
93
6.2k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.4k
We Have a Design System, Now What?
morganepeng
51
7.4k
A Philosophy of Restraint
colly
203
16k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Optimising Largest Contentful Paint
csswizardry
34
3.1k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.1k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
114
50k
Become a Pro
speakerdeck
PRO
26
5.1k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Transcript
Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT KubeflowͰ͡ΊΔ
ML on Kubernetes #3 Kube fl ow Pipelines: Part 1
Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect ▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢ KubeflowͰ͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •
#2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines (ࠓճʂ)
ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
Kubeflow Pipelines ʢKFPʣͱ KFPͷߏཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ v1ʢStable Statusʣ/ v2ʢBeta Statusʣ •
KFP SDK v2ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ KFP v1্Ͱಈ࡞Մೳͱͨ͠ͷ • ͜͜ͰKFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹKFP 1.6Ҏ߱ ͕ඞཁʢࠓճ1.8.4Λ༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ (1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔΛఆٛ͢Δ (3)
ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮ϚϧνϢʔβʔڥͰSDKΛ࣮ߦ͢Δࡍ ʹɺServiceAccount token͕
”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ ʢ͜ͷڥมͷσϑΥϧτͷύε͕ “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebookڥʹ͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
PodDefault • KubernetesʹPodʹରͯ͠ɺ ڥมVolumeͷใͳͲ Λ͋ͱ͔ΒՃ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ Alpha stageʣ͕ଘࡏ͢Δ • PodDefaultPodPreset૬ͷ
ػೳΛKube fl owଆͰ ࣮ͨ͠ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
PodDefaultͷ͍ํ ৽نNotebook࡞࣌ͷ ”Con fi gurations”ΑΓ࡞ͨ͠ PodDefaultΛબ͢Δ͜ͱ͕Ͱ͖Δ
τʔΫϯ͕Ϛϯτ͞Ε͍ͯΔ͜ͱ ͕֬ೝͰ͖Δ ύΠϓϥΠϯͷ࣮ߦʹޭ͠ɺ ExperimentͱRunͷৄࡉͷϦϯΫ ͕දࣔ͞Εͨ
Run details
࣍ճ… ࣮ફతͳPipelinesߏஙͷΛ͠·͢
ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube
fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub