Argo入門/Getting Started with Argo

Transcript

1. Getting Started with Argo
2. profile: name: 松井渉（WataruMatsui） trn: さび開発（3bi.tec） twitter: @watawuwu role: [ Server, Infra, SRE ] lang: [ Rust, Scala, Golang, PHP ]
3. Agenda ⊸ What’s Argo ⊸ What’s Workflow Engine ⊸ Why use Argo ⊸ Argo projects ▫ Argo Workflows ▫ Argo CD/Argo Events ⊸ Conclusions
4. What’s Argo？ （Argo => Argo workflows）
5. Container-native Workflow Engine for K8S
6. What’s Workflow Engine?
7. Workflow engines mainly have three functions • Verification of the current status • Determine the authority of users • Executing condition script By Wikipedia
8. Just managing Data-Pipelines Build Test Package Notif Deploy
10. Workflow Engine scope? CI/CD ETL Job Scheduler ML Batch
11. Similar tool • Apache Airflow • Digdag • Luigi • Azkaban
12. Why use Argo? And Key Features
13. Kubernetesの Custom Resource Definition(CRD) で構築されている Why use Argo? POINT!!
14. K8SのResource を利用できる Why use Argo?
15. Why use Argo? K8S Resources • Persistent volume • ConfigMap • Secret
16. Workflowのマニフェストを K8Sのマニフェストと同じ 同じツールで管理 できる可能性が高い Why use Argo? kustomize
17. K8Sの機能を利用できる Why use Argo?
18. Why use Argo? K8S 機能 • Pod metadata • Node selector • Sidecar
19. Kubernetes Userは 導入が簡単そう！ Why use Argo?
20. Why use Argo? ちなみに GoogleとNVIDIA 開発にJOINしてる
21. Argo Projects
22. Argo projects • Argo Workflows(Stable) • Argo CI(Not active?) • Argo CD(Early development) • Argo Events(Planning)
23. Main topic Argo Workflows
24. ex) Hello world App!!
25. $ argo install Install Argo
26. Installing Argo v2.1.0 into namespace 'kube-system' Proceeding with Kubernetes version 1.9.6 CustomResourceDefinition 'workflows.argoproj.io' created ServiceAccount 'argo' created ClusterRole 'argo-cluster-role' created ClusterRoleBinding 'argo-binding' created Deployment 'workflow-controller' created ServiceAccount 'argo-ui' created ClusterRole 'argo-ui-cluster-role' created ClusterRoleBinding 'argo-ui-binding' created Deployment 'argo-ui' created Service 'argo-ui' created Install Argo
27. Configuration workflow apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec: entrypoint: hello templates: - name: hello container: image: docker/whalesay:latest command: [cowsay] args: ["hello world"]
28. $ argo submit hello.yaml or $ kubectl create -f hello.yaml Submit
31. とっても簡単！！
32. Architecture
34. Argo Workflowsは 何ができるか？
35. templates: - name: hello inputs: parameters: - name: message container: image: docker/whalesay:latest command: [cowsay] args: ["{{inputs.parameters.message}}"] Args parameter
36. $ argo submit \ --parameter message="(°▽°)" \ param.yaml Submit with params
37. entrypoint: step-run templates - name: container-run container: { image: alpine } - name: step-run steps: - - name: step-A template: container-run - - name: step-B template: container-run Steps
38. entrypoint: step-run templates: - name: container-run container: { image: alpine } - name: step-run steps: - - name: step-A template: container-run - - name: step-B template: container-run - name: step-C template: container-run Parallel
39. entrypoint: dag-run templates: - name: container-run container: { image: alpine } - name: dag-run dag: tasks: - name: dag-A template: container-run - name: dag-B template: container-run dependencies: [dag-A] - name: dag-C template: container-run dependencies: [dag-A] DAG 1 2a 2b
40. entrypoint: pwd-ls templates: - name: pwd-ls script: image: debian:9.4 command: [bash] source: | pwd ls entrypoint: container-ls templates: - name: container-ls container: image: debian:9.4 command: [bash] args: - -c - | pwd ls Script bash
41. templates: - name: random script: image: python:alpine3.6 command: [python] source: | import random i = random.randint(1, 100) print(i) templates: - name: random script: image: node:9.1-alpine command: [node] source: | var rand = Math.floor(Math.random() * 100); console.log(rand); Script
42. apiVersion: v1 kind: Secret metadata: name: my-secret type: Opaque data: # 1f2d1e2e67df my-password: MWYyZDFlMmU2N2Rm Secret K8S側のSecretマニフェスト
43. entrypoint: whalesay volumes: - name: my-secret-vol secret: secretName: my-secret templates: - name: whalesay container: image: alpine:3.7 command: [sh, -c] args: ['echo "secret from file: `cat /secret/mountpath/my-password`"'] volumeMounts: - name: my-secret-vol mountPath: "/secret/mountpath" Secret from volume K8S側のSecretの名前
44. templates: - name: whalesay container: image: alpine:3.7 command: [sh, -c] args: ['echo "secret from env: $PASS";'] env: - name: PASS valueFrom: secretKeyRef: name: my-secret key: my-password Secret from env K8S側のSecretの名前
45. Other • Data sharing between job • Loops • Conditions • Recursion • Exit handlers • Timeouts • Daemon container • Sidecar • Artifacts(S3, HTTP, Git) • Kubernetes resource • Docker in Docker
46. Argo Workflowsは 何ができない？
47. Argo Workflowsには トリガー機能がない GithubはArgo CIでWebhook起動できる Argo Events側でトリガーが実装される？
48. Workflowの設定は 永続化されない （Roadmapにあったような）
49. 失敗したジョブを GUIから再実行できない
50. Argo CI
51. 開発が止まっている？ Argo Eventsの 機能と被ってる？
52. Argo CD
54. argocd-demo
55. 期待しているものと違う • workflowsが使えない • Ksonnetに限定してる • Deployメカニズムのニーズ ◦ Istioを使うケースを考えるとニーズが低そう
56. Roadmap • PreSync, PostSync, OutOfSync hooks • Customized application actions as Argo workflows • Blue/Green & canary upgrades
57. Argo Events
58. Overview
59. Trigger
60. CI/CDに 期待できそう！！ Argo Events Argo Workflows
61. Conclusions
62. Argo projects state • Argo Workflows • Argo CI • Argo CD • Argo Events HOLD Trial ASSESS HOLD
63. Argo Workflowsは シンプルで使いやすい！！
64. CI/CDで使うには どのProject使っても 2018-06-01時点では不十分 Argo Eventsに期待する
65. Argo Workflows CI/CD ETL ML Batch どのユースケースでも使えるが どのユースケースでも機能が物足りない
66. 組み合わせで使う？ Argo Events Argo Workflows ？？？ OR
67. K8SのJOBを使ってる人 パイプラインとGUIが手に入るの Argo Workflowsを！！ JOB Pipeline ユースケース
68. Airflowなどを使ってる人は まずは一部の処理をArgoで！！ Trigger Workflows ユースケース
69. MLの用途で考えている人は Kubeflowを！！ ユースケース
70. Thanks! Rust、GCP、DevOpsな エンジニアは必要ありませんか？ 松井 渉　@watawuwu