Slide 1
Slide 1 text
Getting Started
with Argo
Slide 2
Slide 2 text
profile:
name: 松井渉（WataruMatsui）
trn: さび開発（3bi.tec）
twitter: @watawuwu
role: [ Server, Infra, SRE ]
lang: [ Rust, Scala, Golang, PHP ]
Slide 3
Slide 3 text
Agenda
⊸ What’s Argo
⊸ What’s Workflow Engine
⊸ Why use Argo
⊸ Argo projects
▫ Argo Workflows
▫ Argo CD/Argo Events
⊸ Conclusions
Slide 4
Slide 4 text
What’s Argo？
（Argo => Argo workflows）
Slide 5
Slide 5 text
Container-native
Workflow Engine
for K8S
Slide 6
Slide 6 text
What’s
Workflow Engine?
Slide 7
Slide 7 text
Workflow engines mainly
have three functions
● Verification of the current status
● Determine the authority of users
● Executing condition script
By Wikipedia
Slide 8
Slide 8 text
Just managing
Data-Pipelines
Build
Test
Package
Notif
Deploy
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
Workflow Engine scope?
CI/CD
ETL
Job Scheduler
ML
Batch
Slide 11
Slide 11 text
Similar tool
● Apache Airflow
● Digdag
● Luigi
● Azkaban
Slide 12
Slide 12 text
Why use Argo?
And Key Features
Slide 13
Slide 13 text
Kubernetesの
Custom Resource
Definition(CRD)
で構築されている
Why use Argo?
POINT!!
Slide 14
Slide 14 text
K8SのResource
を利用できる
Why use Argo?
Slide 15
Slide 15 text
Why use Argo?
K8S Resources
● Persistent volume
● ConfigMap
● Secret
Slide 16
Slide 16 text
Workflowのマニフェストを
K8Sのマニフェストと同じ
同じツールで管理
できる可能性が高い
Why use Argo?
kustomize
Slide 17
Slide 17 text
K8Sの機能を利用できる
Why use Argo?
Slide 18
Slide 18 text
Why use Argo?
K8S 機能
● Pod metadata
● Node selector
● Sidecar
Slide 19
Slide 19 text
Kubernetes Userは
導入が簡単そう！
Why use Argo?
Slide 20
Slide 20 text
Why use Argo?
ちなみに
GoogleとNVIDIA
開発にJOINしてる
Slide 21
Slide 21 text
Argo Projects
Slide 22
Slide 22 text
Argo projects
● Argo Workflows(Stable)
● Argo CI(Not active?)
● Argo CD(Early development)
● Argo Events(Planning)
Slide 23
Slide 23 text
Main topic
Argo Workflows
Slide 24
Slide 24 text
ex) Hello world App!!
Slide 25
Slide 25 text
$ argo install
Install Argo
Slide 26
Slide 26 text
Installing Argo v2.1.0 into namespace 'kube-system'
Proceeding with Kubernetes version 1.9.6
CustomResourceDefinition 'workflows.argoproj.io' created
ServiceAccount 'argo' created
ClusterRole 'argo-cluster-role' created
ClusterRoleBinding 'argo-binding' created
Deployment 'workflow-controller' created
ServiceAccount 'argo-ui' created
ClusterRole 'argo-ui-cluster-role' created
ClusterRoleBinding 'argo-ui-binding' created
Deployment 'argo-ui' created
Service 'argo-ui' created
Install Argo
Slide 27
Slide 27 text
Configuration workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: hello-world-
spec:
entrypoint: hello
templates:
- name: hello
container:
image: docker/whalesay:latest
command: [cowsay]
args: ["hello world"]
Slide 28
Slide 28 text
$ argo submit hello.yaml
or
$ kubectl create -f hello.yaml
Submit
Slide 29
Slide 29 text
No content
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
とっても簡単！！
Slide 32
Slide 32 text
Architecture
Slide 33
Slide 33 text
No content
Slide 34
Slide 34 text
Argo Workflowsは
何ができるか？
Slide 35
Slide 35 text
templates:
- name: hello
inputs:
parameters:
- name: message
container:
image: docker/whalesay:latest
command: [cowsay]
args: ["{{inputs.parameters.message}}"]
Args parameter
Slide 36
Slide 36 text
$ argo submit \
--parameter message="(°▽°)" \
param.yaml
Submit with params
Slide 37
Slide 37 text
entrypoint: step-run
templates
- name: container-run
container: { image: alpine }
- name: step-run
steps:
- - name: step-A
template: container-run
- - name: step-B
template: container-run
Steps
Slide 38
Slide 38 text
entrypoint: step-run
templates:
- name: container-run
container: { image: alpine }
- name: step-run
steps:
- - name: step-A
template: container-run
- - name: step-B
template: container-run
- name: step-C
template: container-run
Parallel
Slide 39
Slide 39 text
entrypoint: dag-run
templates:
- name: container-run
container: { image: alpine }
- name: dag-run
dag:
tasks:
- name: dag-A
template: container-run
- name: dag-B
template: container-run
dependencies: [dag-A]
- name: dag-C
template: container-run
dependencies: [dag-A]
DAG
1
2a
2b
Slide 40
Slide 40 text
entrypoint: pwd-ls
templates:
- name: pwd-ls
script:
image: debian:9.4
command: [bash]
source: |
pwd
ls
entrypoint: container-ls
templates:
- name: container-ls
container:
image: debian:9.4
command: [bash]
args:
- -c
- |
pwd
ls
Script bash
Slide 41
Slide 41 text
templates:
- name: random
script:
image: python:alpine3.6
command: [python]
source: |
import random
i = random.randint(1, 100)
print(i)
templates:
- name: random
script:
image: node:9.1-alpine
command: [node]
source: |
var rand = Math.floor(Math.random() * 100);
console.log(rand);
Script
Slide 42
Slide 42 text
apiVersion: v1
kind: Secret
metadata:
name: my-secret
type: Opaque
data:
# 1f2d1e2e67df
my-password: MWYyZDFlMmU2N2Rm
Secret
K8S側のSecretマニフェスト
Slide 43
Slide 43 text
entrypoint: whalesay
volumes:
- name: my-secret-vol
secret:
secretName: my-secret
templates:
- name: whalesay
container:
image: alpine:3.7
command: [sh, -c]
args: ['echo "secret from file: `cat
/secret/mountpath/my-password`"']
volumeMounts:
- name: my-secret-vol
mountPath: "/secret/mountpath"
Secret from volume
K8S側のSecretの名前
Slide 44
Slide 44 text
templates:
- name: whalesay
container:
image: alpine:3.7
command: [sh, -c]
args: ['echo "secret from env: $PASS";']
env:
- name: PASS
valueFrom:
secretKeyRef:
name: my-secret
key: my-password
Secret from env
K8S側のSecretの名前
Slide 45
Slide 45 text
Other
● Data sharing between job
● Loops
● Conditions
● Recursion
● Exit handlers
● Timeouts
● Daemon container
● Sidecar
● Artifacts(S3, HTTP, Git)
● Kubernetes resource
● Docker in Docker
Slide 46
Slide 46 text
Argo Workflowsは
何ができない？
Slide 47
Slide 47 text
Argo Workflowsには
トリガー機能がない
GithubはArgo CIでWebhook起動できる
Argo Events側でトリガーが実装される？
Slide 48
Slide 48 text
Workflowの設定は
永続化されない
（Roadmapにあったような）
Slide 49
Slide 49 text
失敗したジョブを
GUIから再実行できない
Slide 50
Slide 50 text
Argo CI
Slide 51
Slide 51 text
開発が止まっている？
Argo Eventsの
機能と被ってる？
Slide 52
Slide 52 text
Argo CD
Slide 53
Slide 53 text
No content
Slide 54
Slide 54 text
argocd-demo
Slide 55
Slide 55 text
期待しているものと違う
● workflowsが使えない
● Ksonnetに限定してる
● Deployメカニズムのニーズ
○ Istioを使うケースを考えるとニーズが低そう
Slide 56
Slide 56 text
Roadmap
● PreSync, PostSync, OutOfSync
hooks
● Customized application actions as
Argo workflows
● Blue/Green & canary upgrades
Slide 57
Slide 57 text
Argo Events
Slide 58
Slide 58 text
Overview
Slide 59
Slide 59 text
Trigger
Slide 60
Slide 60 text
CI/CDに
期待できそう！！
Argo
Events
Argo
Workflows
Slide 61
Slide 61 text
Conclusions
Slide 62
Slide 62 text
Argo projects state
● Argo Workflows
● Argo CI
● Argo CD
● Argo Events
HOLD
Trial
ASSESS
HOLD
Slide 63
Slide 63 text
Argo Workflowsは
シンプルで使いやすい！！
Slide 64
Slide 64 text
CI/CDで使うには
どのProject使っても
2018-06-01時点では不十分
Argo Eventsに期待する
Slide 65
Slide 65 text
Argo Workflows
CI/CD
ETL
ML
Batch
どのユースケースでも使えるが
どのユースケースでも機能が物足りない
Slide 66
Slide 66 text
組み合わせで使う？
Argo
Events
Argo
Workflows
？？？
OR
Slide 67
Slide 67 text
K8SのJOBを使ってる人
パイプラインとGUIが手に入るの
Argo Workflowsを！！
JOB Pipeline
ユースケース
Slide 68
Slide 68 text
Airflowなどを使ってる人は
まずは一部の処理をArgoで！！
Trigger
Workflows
ユースケース
Slide 69
Slide 69 text
MLの用途で考えている人は
Kubeflowを！！
ユースケース
Slide 70
Slide 70 text
Thanks!
Rust、GCP、DevOpsな
エンジニアは必要ありませんか？
松井 渉 @watawuwu