Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Argo Workflow によるMLジョブ管理
Search
Livesense Inc.
PRO
March 27, 2019
Technology
2
830
Argo Workflow によるMLジョブ管理
MACHINE LEARNING Meetup KANSAI #4
2019/3/27
Livesense Inc.
PRO
March 27, 2019
Tweet
Share
More Decks by Livesense Inc.
See All by Livesense Inc.
27新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
0
36
株式会社リブセンス・転職会議 採用候補者様向け資料
livesense
PRO
0
13
株式会社リブセンス 会社説明資料(報道関係者様向け)
livesense
PRO
0
1.4k
データ基盤の負債解消のためのリプレイス
livesense
PRO
0
390
26新卒_総合職採用_会社説明資料
livesense
PRO
0
8.7k
株式会社リブセンス会社紹介資料 / Invent the next common.
livesense
PRO
1
27k
26新卒_Webエンジニア職採用_会社説明資料
livesense
PRO
1
12k
中途セールス職_会社説明資料
livesense
PRO
0
250
EM候補者向け転職会議説明資料
livesense
PRO
0
120
Other Decks in Technology
See All in Technology
標準技術と独自システムで作る「つらくない」SaaS アカウント管理 / Effortless SaaS Account Management with Standard Technologies & Custom Systems
yuyatakeyama
3
1.3k
フィンテック養成勉強会#54
finengine
0
180
Node-REDのFunctionノードでMCPサーバーの実装を試してみた / Node-RED × MCP 勉強会 vol.1
you
PRO
0
120
Agentic Workflowという選択肢を考える
tkikuchi1002
1
540
変化する開発、進化する体系時代に適応するソフトウェアエンジニアの知識と考え方(JaSST'25 Kansai)
mizunori
1
230
GitHub Copilot の概要
tomokusaba
1
140
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
lycorptech_jp
PRO
0
110
TechLION vol.41~MySQLユーザ会のほうから来ました / techlion41_mysql
sakaik
0
190
生成AI時代 文字コードを学ぶ意義を見出せるか?
hrsued
1
570
Delegating the chores of authenticating users to Keycloak
ahus1
0
130
ひとり情シスなCTOがLLMと始めるオペレーション最適化 / CTO's LLM-Powered Ops
yamitzky
0
440
Lambda Web Adapterについて自分なりに理解してみた
smt7174
4
120
Featured
See All Featured
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
The Art of Programming - Codeland 2020
erikaheidi
54
13k
Building an army of robots
kneath
306
45k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
53
2.8k
How to Ace a Technical Interview
jacobian
277
23k
The Straight Up "How To Draw Better" Workshop
denniskardys
234
140k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.8k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
The Invisible Side of Design
smashingmag
300
51k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
30
2.1k
Transcript
Argo Workflow ʹΑΔMLδϣϒཧ Shotaro Tanaka / @yubessy / Ϧϒηϯε (ژΦϑΟε)
MACHINE LEARNING Meetup KANSAI #4 LT
͜Εͷհ͠·͢
https://argoproj.github.io/
Կ͕Ͱ͖Δͷ͔ "Container native workflow engine for Kubernetes" • ෳͷίϯςφΛྻ/ฒྻ࣮ߦ͢ΔϫʔΫϑϩʔΛఆٛͰ͖Δ •
σʔλύΠϓϥΠϯ, CI/CD ͳͲͷར༻Λఆ • ৽όʔδϣϯͰ DAG αϙʔτ • Argo ϕʔεͷ༷ʑͳϓϩμΫτ • Argo CD: GitOps ʹΑΔ CD Λ࣮ݱ • Argo Event: ϫʔΫϑϩʔͷτϦΨ
apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: ml-workflow- spec: entrypoint: main
templates: - name: main steps: - - name: load-dataset template: load-dataset - - name: train-model-1 template: train-model arguments: parameters: [{name: model, value: model1}] - name: train-model-2 template: train-model arguments: parameters: [{name: model, value: model2}] ...
... - name: load-dataset container: image: postgres:latest command: [sh, -c]
args: ["psql db -c 'SELECT * FROM dataset' -A -F, > dataset.csv"] - name: train-model inputs: parameters: [{name: model}] container: image: train-model command: [sh -c] args: ["python train_model.py --model={{inputs.parameters.model}}"]
None
ͳͥ͏ͷ͔ ʮϞσϧ͕Ͱ͖ͨͷͰɺαΫοͱӡ༻ʹ͍ͤͨʯ • MLϞσϧͷ։ൃऀ • SQL Ͱσʔλऔಘ ʙ Ϟσϧ༧ଌΛϑΝΠϧʹग़ྗ •
Docker Ͱಈ͘Α͏ʹ͓ͯ͘͠ • MLγεςϜͷ։ൃऀ • DBIO Ϟσϧɾ༧ଌ݁ՌͷσϦόϦॲཧΛ࣮ • Argo Ͱͯ͢ΛΈ߹ΘͤͨϫʔΫϑϩʔΛ࡞Δ → ίϯςφ୯ҐͰׂ୲
ϦϒηϯεͰͷར༻ྫ • ग़ྗͷDBॻ͖ࠐΈॲཧͷ • Ϟσϧͷ Continuous Delivery • ฒߦॲཧ
ग़ྗͷDBॻ͖ࠐΈॲཧͷ • ٻਓαΠτͷݕࡧॱҐ੍ޚ༻༧ଌϞσϧ • όονͰֶशɾ༧ଌ͠ग़ྗΛDBʹॻ͖ࠐΈ • Ϟσϧͷ։ൃऀCSVग़ྗ·Ͱ࣮ͯ͠ Docker Խ͓ͯ͘͠ •
ॻ͖ࠐΈॲཧΫϨσϯγϟϧཧγεςϜͷ։ൃऀ͕࣮ steps: - - name: train-model # MLϞσϧͷ։ൃऀ͕࣮ - - name: predict-rates # MLϞσϧͷ։ൃऀ͕࣮ (ग़ྗCSV) - - name: import-to-db # MLγεςϜͷ։ൃऀ͕࣮ # ※ग़ྗϑΝΠϧڞ༗ϘϦϡʔϜͰड͚͠
Ϟσϧͷ Continuous Delivery • Ӧۀઓུɾࠂग़ߘΛఆͨ͠ٻਓޮՌਪఆϞσϧ • ϚʔέςΟϯά୲ऀ͚ͷϏϡʔϫΛ R-Shiny Ͱ։ൃɾӡ༻ •
ਪఆॲཧ͕ྃ͢ΔͨͼʹϏϡʔϫΛσϓϩΠͯ͠ϞσϧΛߋ৽ steps: - - name: estimate # ਪఆॲཧ - - name: upload-model # ࡞͞ΕͨϞσϧΛετϨʔδʹอଘ - - name: update-viewer # ϏϡʔϫΛσϓϩΠ͢͠
Ϟσϧͷ Continuous Delivery (ଓ͖) • Ϗϡʔϫಉ͡ Kubernetes ΫϥελͰ Deployment ͱ͍ͯಈ͍͍ͯΔ
• kubectl set env Ͱ Deployment Λߋ৽͢Δ͜ͱͰ৽͍͠ϞσϧΛಡΈࠐΉ • Rolling Update ʹΑΓμϯλΠϜແ͠ͷϞσϧߋ৽Մೳ - name: update-viewer container: image: kubectl command: ["sh", "-c"] args: ["kubectl set env deployment/viewer-app MODEL={{workflow.parameters.model}}"]
ฒߦॲཧ • WebςετͷଟόϯσΟοτ࠷దԽͷॏΈߋ৽δϣϒ • ෳͷςετ͕͓ͬͯΓɺ֤ςετͷਪఆॲཧฒߦ࣮ߦ͍ͨ͠ steps: - - name: list-experiments
# ਪఆॲཧ͕ඞཁͳςετΛϦετΞοϓ - - name: calc-weights # ͜ΕΛϦετΞοϓ͞Εͨςετͷ͚ͩฒߦ࣮ߦ͢Δ # ग़ྗύϥϝʔλͷϦετΛ͢ͱͦͷ͚ͩίϯςφ্ཱ͕͕ͪΔ # Ϧετ [{"experimentId": 1}, {"experimentId": 2}] ͷΑ͏ͳ JSON withParams: "{{steps.list-experiments.outputs.parameters.experiments}}" # Ϧετͷ֤ΞΠςϜ͔ΒύϥϝʔλΛऔΓग़ͯ͢͠ arguments: parameters: [{name: experimentId, value: "{{item.experimentId}}"}]
ฒߦॲཧ (ଓ͖) templates: - name: list-experiments container: ... outputs: parameters:
- name: experiments # ग़ྗύϥϝʔλͷϦετΛϑΝΠϧࢦఆ valueFrom: {path: /output/experiments.json} - name: calc-weights container: ... inputs: parameters: # ύϥϝʔλΛೖྗͱͯ͠ड͚औΔ - name: experimentId
None
·ͱΊ • ෳίϯςφ͔ΒͳΔϫʔΫϑϩʔΛ؆୯ʹΊΔ • ͭͬͨ͘MLϞσϧΛ͘͢ӡ༻͍ͨ͠ͱ͖ʹศར هࣄ͋Γ·͢: Argo ʹΑΔίϯςφωΠςΟϒͳσʔλύΠϓϥΠϯͷϫʔΫϑϩʔཧ