Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ML on Kubernetes with Kubeflow #3 Kubeflow Pipe...
Search
Keita Watanabe
November 24, 2022
Technology
0
1.1k
ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1
Kubernetes Meetup Tokyo #54
https://k8sjp.connpass.com/event/264501/
で発表したLT資料
Keita Watanabe
November 24, 2022
Tweet
Share
More Decks by Keita Watanabe
See All by Keita Watanabe
Scalable Infrastructure for Large-Scale AI Training with AWS Sagemaker Hyperpod @ Singapore AI Hour
keitaw
0
17
[AWS Summit Japan 2025] Optimizing Foundation Model Development with Amazon SageMaker HyperPod: Insights from Training the Amazon Nova Model
keitaw
0
22
Building foundation models on AWS
keitaw
0
310
[re:Invent2024 Chalktalk] Cost-effectively deploy PyTorch LLMs on AWS Inferentia using Amazon EKS
keitaw
0
89
AWS Summit New York 2024: CMP 301 Demystifying the ML software stack on Amazon EC2 accelerated instances
keitaw
0
370
re:Invent 2023 CMP319 Deploy LLMs with AWS Inferentia & Ray to optimize performance and cost
keitaw
0
37
re:Invent 2023: CMP332 De-mystifying ML software stack on Amazon EC2 accelerated instances
keitaw
1
170
AWS における LLM・GenAI 大規模学習への取り組み / Large scale GenAI・LLM training on AWS
keitaw
1
540
Amazon EC2 シリコン革命 / Amazon EC2 Silicon Innovation
keitaw
0
180
Other Decks in Technology
See All in Technology
「魔法少女まどか☆マギカ Magia Exedra」での負荷試験の実践と学び
gree_tech
PRO
0
620
サンドボックス技術でAI利活用を促進する
koh_naga
0
190
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
19
9.4k
エラーとアクセシビリティ
schktjm
0
1.1k
シークレット管理だけじゃない!HashiCorp Vault でデータ暗号化をしよう / Beyond Secret Management! Let's Encrypt Data with HashiCorp Vault
nnstt1
3
230
20250903_1つのAWSアカウントに複数システムがある環境におけるアクセス制御をABACで実現.pdf
yhana
3
460
生成AIでセキュリティ運用を効率化する話
sakaitakeshi
0
290
LLM翻訳ツールの開発と海外のお客様対応等への社内導入事例
gree_tech
PRO
0
580
落ちる 落ちるよ サーバーは落ちる
suehiromasatoshi
0
140
AWSで推進するデータマネジメント
kawanago
1
1.2k
複数サービスを支えるマルチテナント型Batch MLプラットフォーム
lycorptech_jp
PRO
0
220
DevIO2025_継続的なサービス開発のための技術的意思決定のポイント / how-to-tech-decision-makaing-devio2025
nologyance
1
300
Featured
See All Featured
For a Future-Friendly Web
brad_frost
180
9.9k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Speed Design
sergeychernyshev
32
1.1k
Documentation Writing (for coders)
carmenintech
74
5k
Building an army of robots
kneath
306
46k
The Language of Interfaces
destraynor
161
25k
RailsConf 2023
tenderlove
30
1.2k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
The Art of Programming - Codeland 2020
erikaheidi
55
13k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
111
20k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
252
21k
How to Ace a Technical Interview
jacobian
279
23k
Transcript
Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT KubeflowͰ͡ΊΔ
ML on Kubernetes #3 Kube fl ow Pipelines: Part 1
Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect ▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢ KubeflowͰ͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •
#2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines (ࠓճʂ)
ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
Kubeflow Pipelines ʢKFPʣͱ KFPͷߏཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ v1ʢStable Statusʣ/ v2ʢBeta Statusʣ •
KFP SDK v2ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ KFP v1্Ͱಈ࡞Մೳͱͨ͠ͷ • ͜͜ͰKFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹKFP 1.6Ҏ߱ ͕ඞཁʢࠓճ1.8.4Λ༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ (1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔΛఆٛ͢Δ (3)
ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮ϚϧνϢʔβʔڥͰSDKΛ࣮ߦ͢Δࡍ ʹɺServiceAccount token͕
”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ ʢ͜ͷڥมͷσϑΥϧτͷύε͕ “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebookڥʹ͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
PodDefault • KubernetesʹPodʹରͯ͠ɺ ڥมVolumeͷใͳͲ Λ͋ͱ͔ΒՃ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ Alpha stageʣ͕ଘࡏ͢Δ • PodDefaultPodPreset૬ͷ
ػೳΛKube fl owଆͰ ࣮ͨ͠ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
PodDefaultͷ͍ํ ৽نNotebook࡞࣌ͷ ”Con fi gurations”ΑΓ࡞ͨ͠ PodDefaultΛબ͢Δ͜ͱ͕Ͱ͖Δ
τʔΫϯ͕Ϛϯτ͞Ε͍ͯΔ͜ͱ ͕֬ೝͰ͖Δ ύΠϓϥΠϯͷ࣮ߦʹޭ͠ɺ ExperimentͱRunͷৄࡉͷϦϯΫ ͕දࣔ͞Εͨ
Run details
࣍ճ… ࣮ફతͳPipelinesߏஙͷΛ͠·͢
ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube
fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub