Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow によるMLジョブ管理
Search
Livesense Inc.
PRO
March 27, 2019
Technology
2
800
Argo Workflow によるMLジョブ管理
MACHINE LEARNING Meetup KANSAI #4
2019/3/27
Livesense Inc.
PRO
March 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
990
データ基盤の負債解消のためのリプレイス
livesense
PRO
0
250
26新卒_総合職採用_会社説明資料
livesense
PRO
0
4.3k
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
13k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
7.9k
中途セールス職_会社説明資料
livesense
PRO
0
190
EM候補者向け転職会議説明資料
livesense
PRO
0
84
コロナで失われたノベルティ作成ノウハウを復活させた話
livesense
PRO
0
200
転職会議でGPT-3を活用した企業口コミ要約機能をリリースした話
livesense
PRO
0
1.3k
Other Decks in Technology
See All in Technology
Perlの生きのこり - エンジニアがこの先生きのこるためのカンファレンス2025
kfly8
1
150
2025-02-21 ゆるSRE勉強会 Enhancing SRE Using AI
yoshiiryo1
1
410
Swiftの “private” を テストする / Testing Swift "private"
yutailang0119
0
130
TAMとre:Capセキュリティ編 〜拡張脅威検出デモを添えて〜
fujiihda
2
340
なぜ私は自分が使わないサービスを作るのか? / Why would I create a service that I would not use?
aiandrox
0
820
抽象化をするということ - 具体と抽象の往復を身につける / Abstraction and concretization
soudai
26
13k
君も受託系GISエンジニアにならないか
sudataka
2
450
人はなぜISUCONに夢中になるのか
kakehashi
PRO
6
1.7k
依存パッケージの更新はコツコツが勝つコツ! / phpcon_nagoya2025
blue_goheimochi
3
170
管理者しか知らないOutlookの裏側のAIを覗く#AzureTravelers
hirotomotaguchi
2
490
The Future of SEO: The Impact of AI on Search
badams
0
230
システム・ML活用を広げるdbtのデータモデリング / Expanding System & ML Use with dbt Modeling
i125
0
170
Featured
See All Featured
Building an army of robots
kneath
303
45k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
30
2.2k
Visualization
eitanlees
146
15k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
27
1.9k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
4 Signs Your Business is Dying
shpigford
182
22k
KATA
mclloyd
29
14k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
The Cult of Friendly URLs
andyhume
78
6.2k
The Language of Interfaces
destraynor
156
24k
A Modern Web Designer's Workflow
chriscoyier
693
190k
Transcript
Argo Workflow ʹΑΔMLδϣϒཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژΦϑΟε)
MACHINE LEARNING Meetup KANSAI #4 LT
͜Εͷհ͠·͢
https://argoproj.github.io/
Կ͕Ͱ͖Δͷ͔ "Container native workflow engine for Kubernetes" • ෳͷίϯςφΛྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ •
σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λఆ • ৽όʔδϣϯͰ DAG αϙʔτ • Argo ϕʔεͷ༷ʑͳϓϩμΫτ • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ • Argo Event: ϫʔΫϑϩʔͷτϦΨ
apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ml-workflow- spec: entrypoint: main
templates: - name: main steps: - - name: load-dataset template: load-dataset - - name: train-model-1 template: train-model arguments: parameters: [{name: model, value: model1}] - name: train-model-2 template: train-model arguments: parameters: [{name: model, value: model2}] ...
... - name: load-dataset container: image: postgres:latest command: [sh, -c]
args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"] - name: train-model inputs: parameters: [{name: model}] container: image: train-model command: [sh -c] args: ["python train_model.py --model={{inputs.parameters.model}}"]
None
ͳͥ͏ͷ͔ ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ͍ͤͨʯ • MLϞσϧͷ։ൃऀ • SQL Ͱσʔλऔಘ ʙ Ϟσϧ༧ଌΛϑΝΠϧʹग़ྗ •
Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠ • MLγεςϜͷ։ൃऀ • DBIO Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮ • Argo Ͱͯ͢ΛΈ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ → ίϯςφ୯ҐͰׂ୲
ϦϒηϯεͰͷར༻ྫ • ग़ྗͷDBॻ͖ࠐΈॲཧͷ • Ϟσϧͷ Continuous Delivery • ฒߦॲཧ
ग़ྗͷDBॻ͖ࠐΈॲཧͷ • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ • Ϟσϧͷ։ൃऀCSVग़ྗ·Ͱ࣮ͯ͠ Docker Խ͓ͯ͘͠ •
ॻ͖ࠐΈॲཧΫϨσϯγϟϧཧγεςϜͷ։ൃऀ͕࣮ steps: - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮ - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮ # ※ग़ྗϑΝΠϧڞ༗ϘϦϡʔϜͰड͚͠
Ϟσϧͷ Continuous Delivery • Ӧۀઓུɾࠂग़ߘΛఆͨ͠ٻਓޮՌਪఆϞσϧ • ϚʔέςΟϯά୲ऀ͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻ •
ਪఆॲཧ͕ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽ steps: - - name: estimate # ਪఆॲཧ - - name: upload-model # ࡞͞ΕͨϞσϧΛετϨʔδʹอଘ - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͢͠
Ϟσϧͷ Continuous Delivery (ଓ͖) • Ϗϡʔϫಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
• kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
ฒߦॲཧ • WebςετͷଟόϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ • ෳͷςετ͕͓ͬͯΓɺ֤ςετͷਪఆॲཧฒߦ࣮ߦ͍ͨ͠ steps: - - name: list-experiments
# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ͚ͩฒߦ࣮ߦ͢Δ # ग़ྗύϥϝʔλͷϦετΛ͢ͱͦͷ͚ͩίϯςφ্ཱ͕͕ͪΔ # Ϧετ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͢͠ arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ฒߦॲཧ (ଓ͖) templates: - name: list-experiments container: ... outputs: parameters:
- name: experiments # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ valueFrom: {path: /output/experiments.json} - name: calc-weights container: ... inputs: parameters: # ύϥϝʔλΛೖྗͱͯ͠ड͚औΔ - name: experimentId
None
·ͱΊ • ෳίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹΊΔ • ͭͬͨ͘MLϞσϧΛ͘͢ӡ༻͍ͨ͠ͱ͖ʹศར هࣄ͋Γ·͢: Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔཧ