Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ML on Kubernetes with Kubeflow #3 Kubeflow Pipe...
Search
Keita Watanabe
November 24, 2022
Technology
0
1.1k
ML on Kubernetes with Kubeflow #3 Kubeflow Pipelines: Part1
Kubernetes Meetup Tokyo #54
https://k8sjp.connpass.com/event/264501/
で発表したLT資料
Keita Watanabe
November 24, 2022
Tweet
Share
More Decks by Keita Watanabe
See All by Keita Watanabe
Scalable Infrastructure for Large-Scale AI Training with AWS Sagemaker Hyperpod @ Singapore AI Hour
keitaw
0
17
[AWS Summit Japan 2025] Optimizing Foundation Model Development with Amazon SageMaker HyperPod: Insights from Training the Amazon Nova Model
keitaw
0
22
Building foundation models on AWS
keitaw
0
320
[re:Invent2024 Chalktalk] Cost-effectively deploy PyTorch LLMs on AWS Inferentia using Amazon EKS
keitaw
0
90
AWS Summit New York 2024: CMP 301 Demystifying the ML software stack on Amazon EC2 accelerated instances
keitaw
0
380
re:Invent 2023 CMP319 Deploy LLMs with AWS Inferentia & Ray to optimize performance and cost
keitaw
0
37
re:Invent 2023: CMP332 De-mystifying ML software stack on Amazon EC2 accelerated instances
keitaw
1
170
AWS における LLM・GenAI 大規模学習への取り組み / Large scale GenAI・LLM training on AWS
keitaw
1
540
Amazon EC2 シリコン革命 / Amazon EC2 Silicon Innovation
keitaw
0
180
Other Decks in Technology
See All in Technology
「どこから読む?」コードとカルチャーに最速で馴染むための実践ガイド
zozotech
PRO
0
550
Evolución del razonamiento matemático de GPT-4.1 a GPT-5 - Data Aventura Summit 2025 & VSCode DevDays
lauchacarro
0
210
💡Ruby 川辺で灯すPicoRubyからの光
bash0c7
0
120
Claude Code でアプリ開発をオートパイロットにするためのTips集 Zennの場合 / Claude Code Tips in Zenn
wadayusuke
4
490
CDK CLIで使ってたあの機能、CDK Toolkit Libraryではどうやるの?
smt7174
4
190
Oracle Cloud Infrastructure IaaS 新機能アップデート 2025/06 - 2025/08
oracle4engineer
PRO
0
110
大「個人開発サービス」時代に僕たちはどう生きるか
sotarok
20
10k
AI時代を生き抜くエンジニアキャリアの築き方 (AI-Native 時代、エンジニアという道は 「最大の挑戦の場」となる) / Building an Engineering Career to Thrive in the Age of AI (In the AI-Native Era, the Path of Engineering Becomes the Ultimate Arena of Challenge)
jeongjaesoon
0
230
「Linux」という言葉が指すもの
sat
PRO
4
140
「全員プロダクトマネージャー」を実現する、Cursorによる仕様検討の自動運転
applism118
22
12k
なぜスクラムはこうなったのか?歴史が教えてくれたこと/Shall we explore the roots of Scrum
sanogemaru
5
1.7k
新アイテムをどう使っていくか?みんなであーだこーだ言ってみよう / 20250911-rpi-jam-tokyo
akkiesoft
0
330
Featured
See All Featured
Typedesign – Prime Four
hannesfritz
42
2.8k
Faster Mobile Websites
deanohume
309
31k
The Pragmatic Product Professional
lauravandoore
36
6.9k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
31
2.2k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.7k
Designing for Performance
lara
610
69k
How GitHub (no longer) Works
holman
315
140k
Optimizing for Happiness
mojombo
379
70k
Optimising Largest Contentful Paint
csswizardry
37
3.4k
The Art of Programming - Codeland 2020
erikaheidi
56
13k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
29
2.9k
Transcript
Keita Watanabe 2022/11/24 Kubernetes Meetup Tokyo # 54 LT KubeflowͰ͡ΊΔ
ML on Kubernetes #3 Kube fl ow Pipelines: Part 1
Keita Watanabe Machine Learning Solutions Architect ▶ AWS JapanͰSelf-managedͳMachine Learning
ͷҊ݅Λ୲͍ͯ͠ΔSolutions Architect ▶ લ৬ͰDatascientist/ML Researcherͱͯ͠ ECαΠτ্ͷը૾ݕࡧػೳͷ։ൃʹैࣄ ▶ Twitter: keitaw09 ▶ Linkedin: keitawatanabe αϯσΟΤΰͰग़ձͬͨτϦ झຯ
ͪ͜ΒͷLTγϦʔζͷͰ͢ KubeflowͰ͡ΊΔML on Kubernetes • #1 Kube fl owͷ֓ཁͱηοτΞοϓ •
#2 Kube fl ow Notebooks • #3 Kube fl ow Pipelines (ࠓճʂ)
ࠓճͷςʔϚɿ Kubeflow Pipelines ʢͷಋೖʣ
Kubeflow Pipelines ʢKFPʣͱ KFPͷߏཁૉ • Pipelineͷ֬ೝɾ࣮ߦʹ༻͍ΔUI • Pipeline࣮ߦΛεέδϡʔϦϯά͢ΔEngine • ύΠϓϥΠϯͷఆٛɺϏϧυɺσϓϩΠ͕Մೳͳ
Python SDK • SDKͰͷύΠϓϥΠϯ։ൃɺ͓Αͼ࣮ߦʹؔ͢Δ Notebook αϙʔτ Componentͱͯ͠ɺ֤εςοϓΛ࣮͠ɺͦΕΒΛPipelineͱͯ͠Ұ࿈ͷॲཧʹ· ͱΊΔ͜ͱͰMLύΠϓϥΠϯΛߏங͢Δπʔϧ UI: pipeline graph view Ұ෦ൈਮ
KFP SDKͷόʔδϣϯʹ͍ͭͯ • 2022/11/24ݱࡏɺKFP SDKʹҎԼͷ̎ͭͷVersion͕ଘࡏ͢Δ v1ʢStable Statusʣ/ v2ʢBeta Statusʣ •
KFP SDK v2ɺݱࡏPre-release stageͷKFP v2ͷίΞػೳΛ KFP v1্Ͱಈ࡞Մೳͱͨ͠ͷ • ͜͜ͰKFP SDK v2Λ༻͍Δ • KFP SDK v2ͷར༻ʹKFP 1.6Ҏ߱ ͕ඞཁʢࠓճ1.8.4Λ༻ʣ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2-compatibility/
SDKΛ༻͍ͨγϯϓϧͳPipelineͷ࣮ྫ (1) ίϯϙʔωϯτͷఆٛ ͜͜Ͱ͠ࢉΛ”python:3.9”Πϝʔδ্Ͱ ࣮ߦ͢ΔίϯϙʔωϯτΛఆٛ͢Δ (2) ύΠϓϥΠϯͷఆٛ ύΠϓϥΠϯͰ༻͍Δίϯϙʔωϯτ܈ͱ ͦͷؔΛఆٛ͢Δ (3)
ύΠϓϥΠϯͷ࣮ߦ ύΠϓϥΠϯʹೖྗΛ༩͑ɺ࣮ߦ͢Δ (1) (2) (3)
͜ͷίʔυΛ࣮ߦͯ͠ΈΔͱ… • “/var/run/secrets/kube fl ow/pipelines/token” ͕ແ͍ͱౖΒΕɺ࣮ߦʹࣦഊ͢Δ • ࣮ϚϧνϢʔβʔڥͰSDKΛ࣮ߦ͢Δࡍ ʹɺServiceAccount token͕
”KF_PIPELINES_SA_TOKEN_PATH”ʹଘࡏ͢ Δඞཁ͕͋Δ ʢ͜ͷڥมͷσϑΥϧτͷύε͕ “/var/run/secrets/kube fl ow/pipelines/token”ʣ • σϑΥϧτͷNotebookڥʹ͜ͷτʔΫϯ ͕ઃఆ͞Ε͍ͯͳ͍ˠ PodDefaultΛ༻͍Δ https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/#full-kube fl ow-subfrom-inside-clustersub
PodDefault • KubernetesʹPodʹରͯ͠ɺ ڥมVolumeͷใͳͲ Λ͋ͱ͔ΒՃ͢ΔPodPreset ͱ͍͏Ϧιʔεʢݱࡏ Alpha stageʣ͕ଘࡏ͢Δ • PodDefaultPodPreset૬ͷ
ػೳΛKube fl owଆͰ ࣮ͨ͠ͷ https://github.com/kube fl ow/kube fl ow/blob/master/components/admission-webhook/README.md
PodDefaultͷ͍ํ ৽نNotebook࡞࣌ͷ ”Con fi gurations”ΑΓ࡞ͨ͠ PodDefaultΛબ͢Δ͜ͱ͕Ͱ͖Δ
τʔΫϯ͕Ϛϯτ͞Ε͍ͯΔ͜ͱ ͕֬ೝͰ͖Δ ύΠϓϥΠϯͷ࣮ߦʹޭ͠ɺ ExperimentͱRunͷৄࡉͷϦϯΫ ͕දࣔ͞Εͨ
Run details
࣍ճ… ࣮ફతͳPipelinesߏஙͷΛ͠·͢
ࢀߟࢿྉ • https://techblog.zozo.com/entry/aip-pipelines-impl#Kube fl ow-Pipelines • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube
fl ow-subfrom-inside-clustersub • https://www.kube fl ow.org/docs/components/notebooks/overview • https://github.com/aws-samples/aws-do-kube fl ow • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk-v2/v2- compatibility/ • https://www.kube fl ow.org/docs/components/pipelines/v1/sdk/connect-api/ #full-kube fl ow-subfrom-inside-clustersub