Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ML on Kubernetes with Kubeflow #3 Kubeflow Pipe...
Search
Keita Watanabe
November 24, 2022
Technology
0
1.1k
ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1
Kubernetes Meetup Tokyo #54
https://k8sjp.connpass.com/event/264501/
で発表したLT資料
Keita Watanabe
November 24, 2022
Tweet
Share
More Decks by Keita Watanabe
See All by Keita Watanabe
Scalable Infrastructure for Large-Scale AI Training with AWS Sagemaker Hyperpod @ Singapore AI Hour
keitaw
0
13
[AWS Summit Japan 2025] Optimizing Foundation Model Development with Amazon SageMaker HyperPod: Insights from Training the Amazon Nova Model
keitaw
0
16
Building foundation models on AWS
keitaw
0
250
[re:Invent2024 Chalktalk] Cost-effectively deploy PyTorch LLMs on AWS Inferentia using Amazon EKS
keitaw
0
77
AWS Summit New York 2024: CMP 301 Demystifying the ML software stack on Amazon EC2 accelerated instances
keitaw
0
350
re:Invent 2023 CMP319 Deploy LLMs with AWS Inferentia & Ray to optimize performance and cost
keitaw
0
36
re:Invent 2023: CMP332 De-mystifying ML software stack on Amazon EC2 accelerated instances
keitaw
1
150
AWS における LLM・GenAI 大規模学習への取り組み / Large scale GenAI・LLM training on AWS
keitaw
1
540
Amazon EC2 シリコン革命 / Amazon EC2 Silicon Innovation
keitaw
0
180
Other Decks in Technology
See All in Technology
2時間で300+テーブルをデータ基盤に連携するためのAI活用 / FukuokaDataEngineer
sansan_randd
0
130
Claude Codeから我々が学ぶべきこと
s4yuba
6
1.8k
AWS re:Inforce 2025 re:Cap Update Pickup & AWS Control Tower の運用における考慮ポイント
htan
1
200
【CEDEC2025】『Shadowverse: Worlds Beyond』二度目のDCG開発でゲームをリデザインする~遊びやすさと競技性の両立~
cygames
PRO
1
290
僕たちが「開発しやすさ」を求め 模索し続けたアーキテクチャ #アーキテクチャ勉強会_findy
bengo4com
0
1.9k
解消したはずが…技術と人間のエラーが交錯する恐怖体験
lamaglama39
0
190
【新卒研修資料】数理最適化 / Mathematical Optimization
brainpadpr
25
11k
S3 Glacier のデータを Athena からクエリしようとしたらどうなるのか/try-to-query-s3-glacier-from-athena
emiki
0
180
生成AI時代におけるAI・機械学習技術を用いたプロダクト開発の深化と進化 #BetAIDay
layerx
PRO
1
1k
KubeCon + CloudNativeCon Japan 2025 Recap
donkomura
0
160
Strands Agents & Bedrock AgentCoreを1分でおさらい
minorun365
PRO
6
230
dipにおけるSRE変革の軌跡
dip_tech
PRO
1
230
Featured
See All Featured
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
18
1k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
251
21k
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.5k
Six Lessons from altMBA
skipperchong
28
3.9k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Connecting the Dots Between Site Speed, User Experience & Your Business [WebExpo 2025]
tammyeverts
8
420
Side Projects
sachag
455
43k
How STYLIGHT went responsive
nonsquared
100
5.7k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.8k
Transcript
Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT KubeflowͰ͡ΊΔ
ML on Kubernetes #3 Kube fl ow Pipelines: Part 1
Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect ▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢ KubeflowͰ͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •
#2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines (ࠓճʂ)
ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
Kubeflow Pipelines ʢKFPʣͱ KFPͷߏཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ v1ʢStable Statusʣ/ v2ʢBeta Statusʣ •
KFP SDK v2ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ KFP v1্Ͱಈ࡞Մೳͱͨ͠ͷ • ͜͜ͰKFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹKFP 1.6Ҏ߱ ͕ඞཁʢࠓճ1.8.4Λ༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ (1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔΛఆٛ͢Δ (3)
ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮ϚϧνϢʔβʔڥͰSDKΛ࣮ߦ͢Δࡍ ʹɺServiceAccount token͕
”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ ʢ͜ͷڥมͷσϑΥϧτͷύε͕ “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebookڥʹ͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
PodDefault • KubernetesʹPodʹରͯ͠ɺ ڥมVolumeͷใͳͲ Λ͋ͱ͔ΒՃ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ Alpha stageʣ͕ଘࡏ͢Δ • PodDefaultPodPreset૬ͷ
ػೳΛKube fl owଆͰ ࣮ͨ͠ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
PodDefaultͷ͍ํ ৽نNotebook࡞࣌ͷ ”Con fi gurations”ΑΓ࡞ͨ͠ PodDefaultΛબ͢Δ͜ͱ͕Ͱ͖Δ
τʔΫϯ͕Ϛϯτ͞Ε͍ͯΔ͜ͱ ͕֬ೝͰ͖Δ ύΠϓϥΠϯͷ࣮ߦʹޭ͠ɺ ExperimentͱRunͷৄࡉͷϦϯΫ ͕දࣔ͞Εͨ
Run details
࣍ճ… ࣮ફతͳPipelinesߏஙͷΛ͠·͢
ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube
fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub