Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow によるMLジョブ管理
Search
Livesense Inc.
PRO
March 27, 2019
Technology
2
840
Argo Workflow によるMLジョブ管理
MACHINE LEARNING Meetup KANSAI #4
2019/3/27
Livesense Inc.
PRO
March 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
27新卒_総合職採用_会社説明資料
livesense
PRO
0
4
27新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
0
3.5k
株式会社リブセンス・転職会議 採用候補者様向け資料
livesense
PRO
0
71
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
1.6k
データ基盤の負債解消のためのリプレイス
livesense
PRO
0
470
26新卒_総合職採用_会社説明資料
livesense
PRO
0
12k
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
41k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
13k
中途セールス職_会社説明資料
livesense
PRO
0
270
Other Decks in Technology
See All in Technology
生成AI時代のセキュアコーディングとDevSecOps
yuriemori
0
130
AIツールでどこまでデザインを忠実に実装できるのか
oikon48
6
3.5k
HR Force における DWH の併用事例 ~ サービス基盤としての BigQuery / 分析基盤としての Snowflake ~@Cross Data Platforms Meetup #2「BigQueryと愉快な仲間たち」
ryo_suzuki
0
240
Node.js 2025: What's new and what's next
ruyadorno
0
530
プロダクトのコードから見るGoによるデザインパターンの実践 #go_night_talk
bengo4com
1
2.7k
今この時代に技術とどう向き合うべきか
gree_tech
PRO
2
2.1k
AWS Control Tower に学ぶ! IAM Identity Center 権限設計の第一歩 / IAM Identity Center with Control Tower
y___u
1
200
名刺メーカーDevグループ 紹介資料
sansan33
PRO
0
930
Databricks AI/BI Genie の「値ディクショナリー」をAmazonの奥地(S3)まで見に行く
kameitomohiro
1
270
防災デジタル分野での官民共創の取り組み (2)DIT/CCとD-CERTについて
ditccsugii
0
310
エンタメとAIのための3Dパラレルワールド構築(GPU UNITE 2025 特別講演)
pfn
PRO
0
540
AWSでAgentic AIを開発するための前提知識の整理
nasuvitz
2
210
Featured
See All Featured
The MySQL Ecosystem @ GitHub 2015
samlambert
251
13k
Building Adaptive Systems
keathley
44
2.8k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
132
19k
Typedesign – Prime Four
hannesfritz
42
2.8k
Side Projects
sachag
455
43k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
10
870
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
980
Optimising Largest Contentful Paint
csswizardry
37
3.5k
The Pragmatic Product Professional
lauravandoore
36
6.9k
How to Ace a Technical Interview
jacobian
280
24k
Building a Modern Day E-commerce SEO Strategy
aleyda
44
7.8k
Transcript
Argo Workflow ʹΑΔMLδϣϒཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژΦϑΟε)
MACHINE LEARNING Meetup KANSAI #4 LT
͜Εͷհ͠·͢
https://argoproj.github.io/
Կ͕Ͱ͖Δͷ͔ "Container native workflow engine for Kubernetes" • ෳͷίϯςφΛྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ •
σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λఆ • ৽όʔδϣϯͰ DAG αϙʔτ • Argo ϕʔεͷ༷ʑͳϓϩμΫτ • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ • Argo Event: ϫʔΫϑϩʔͷτϦΨ
apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ml-workflow- spec: entrypoint: main
templates: - name: main steps: - - name: load-dataset template: load-dataset - - name: train-model-1 template: train-model arguments: parameters: [{name: model, value: model1}] - name: train-model-2 template: train-model arguments: parameters: [{name: model, value: model2}] ...
... - name: load-dataset container: image: postgres:latest command: [sh, -c]
args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"] - name: train-model inputs: parameters: [{name: model}] container: image: train-model command: [sh -c] args: ["python train_model.py --model={{inputs.parameters.model}}"]
None
ͳͥ͏ͷ͔ ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ͍ͤͨʯ • MLϞσϧͷ։ൃऀ • SQL Ͱσʔλऔಘ ʙ Ϟσϧ༧ଌΛϑΝΠϧʹग़ྗ •
Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠ • MLγεςϜͷ։ൃऀ • DBIO Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮ • Argo Ͱͯ͢ΛΈ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ → ίϯςφ୯ҐͰׂ୲
ϦϒηϯεͰͷར༻ྫ • ग़ྗͷDBॻ͖ࠐΈॲཧͷ • Ϟσϧͷ Continuous Delivery • ฒߦॲཧ
ग़ྗͷDBॻ͖ࠐΈॲཧͷ • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ • Ϟσϧͷ։ൃऀCSVग़ྗ·Ͱ࣮ͯ͠ Docker Խ͓ͯ͘͠ •
ॻ͖ࠐΈॲཧΫϨσϯγϟϧཧγεςϜͷ։ൃऀ͕࣮ steps: - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮ - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮ # ※ग़ྗϑΝΠϧڞ༗ϘϦϡʔϜͰड͚͠
Ϟσϧͷ Continuous Delivery • Ӧۀઓུɾࠂग़ߘΛఆͨ͠ٻਓޮՌਪఆϞσϧ • ϚʔέςΟϯά୲ऀ͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻ •
ਪఆॲཧ͕ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽ steps: - - name: estimate # ਪఆॲཧ - - name: upload-model # ࡞͞ΕͨϞσϧΛετϨʔδʹอଘ - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͢͠
Ϟσϧͷ Continuous Delivery (ଓ͖) • Ϗϡʔϫಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
• kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
ฒߦॲཧ • WebςετͷଟόϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ • ෳͷςετ͕͓ͬͯΓɺ֤ςετͷਪఆॲཧฒߦ࣮ߦ͍ͨ͠ steps: - - name: list-experiments
# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ͚ͩฒߦ࣮ߦ͢Δ # ग़ྗύϥϝʔλͷϦετΛ͢ͱͦͷ͚ͩίϯςφ্ཱ͕͕ͪΔ # Ϧετ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͢͠ arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ฒߦॲཧ (ଓ͖) templates: - name: list-experiments container: ... outputs: parameters:
- name: experiments # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ valueFrom: {path: /output/experiments.json} - name: calc-weights container: ... inputs: parameters: # ύϥϝʔλΛೖྗͱͯ͠ड͚औΔ - name: experimentId
None
·ͱΊ • ෳίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹΊΔ • ͭͬͨ͘MLϞσϧΛ͘͢ӡ༻͍ͨ͠ͱ͖ʹศར هࣄ͋Γ·͢: Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔཧ