Argo入門/Getting Started with Argo

Argo入門/Getting Started with Argo

E7a86f7090ecce18bc3848324741b04c?s=128

watawuwu

June 01, 2018
Tweet

Transcript

  1. Getting Started with Argo

  2. profile: name: 松井渉(WataruMatsui) trn: さび開発(3bi.tec) twitter: @watawuwu role: [ Server,

    Infra, SRE ] lang: [ Rust, Scala, Golang, PHP ]
  3. Agenda ⊸ What’s Argo ⊸ What’s Workflow Engine ⊸ Why

    use Argo ⊸ Argo projects ▫ Argo Workflows ▫ Argo CD/Argo Events ⊸ Conclusions
  4. What’s Argo? (Argo => Argo workflows)

  5. Container-native Workflow Engine for K8S

  6. What’s Workflow Engine?

  7. Workflow engines mainly have three functions • Verification of the

    current status • Determine the authority of users • Executing condition script By Wikipedia
  8. Just managing Data-Pipelines Build Test Package Notif Deploy

  9. None
  10. Workflow Engine scope? CI/CD ETL Job Scheduler ML Batch

  11. Similar tool • Apache Airflow • Digdag • Luigi •

    Azkaban
  12. Why use Argo? And Key Features

  13. Kubernetesの Custom Resource Definition(CRD) で構築されている Why use Argo? POINT!!

  14. K8SのResource を利用できる Why use Argo?

  15. Why use Argo? K8S Resources • Persistent volume • ConfigMap

    • Secret
  16. Workflowのマニフェストを K8Sのマニフェストと同じ 同じツールで管理 できる可能性が高い Why use Argo? kustomize

  17. K8Sの機能を利用できる Why use Argo?

  18. Why use Argo? K8S 機能 • Pod metadata • Node

    selector • Sidecar
  19. Kubernetes Userは 導入が簡単そう! Why use Argo?

  20. Why use Argo? ちなみに GoogleとNVIDIA 開発にJOINしてる

  21. Argo Projects

  22. Argo projects • Argo Workflows(Stable) • Argo CI(Not active?) •

    Argo CD(Early development) • Argo Events(Planning)
  23. Main topic Argo Workflows

  24. ex) Hello world App!!

  25. $ argo install Install Argo

  26. Installing Argo v2.1.0 into namespace 'kube-system' Proceeding with Kubernetes version

    1.9.6 CustomResourceDefinition 'workflows.argoproj.io' created ServiceAccount 'argo' created ClusterRole 'argo-cluster-role' created ClusterRoleBinding 'argo-binding' created Deployment 'workflow-controller' created ServiceAccount 'argo-ui' created ClusterRole 'argo-ui-cluster-role' created ClusterRoleBinding 'argo-ui-binding' created Deployment 'argo-ui' created Service 'argo-ui' created Install Argo
  27. Configuration workflow apiVersion: argoproj.io/v1alpha1 kind: Workflow metadata: generateName: hello-world- spec:

    entrypoint: hello templates: - name: hello container: image: docker/whalesay:latest command: [cowsay] args: ["hello world"]
  28. $ argo submit hello.yaml or $ kubectl create -f hello.yaml

    Submit
  29. None
  30. None
  31. とっても簡単!!

  32. Architecture

  33. None
  34. Argo Workflowsは 何ができるか?

  35. templates: - name: hello inputs: parameters: - name: message container:

    image: docker/whalesay:latest command: [cowsay] args: ["{{inputs.parameters.message}}"] Args parameter
  36. $ argo submit \ --parameter message="(°▽°)" \ param.yaml Submit with

    params
  37. entrypoint: step-run templates - name: container-run container: { image: alpine

    } - name: step-run steps: - - name: step-A template: container-run - - name: step-B template: container-run Steps
  38. entrypoint: step-run templates: - name: container-run container: { image: alpine

    } - name: step-run steps: - - name: step-A template: container-run - - name: step-B template: container-run - name: step-C template: container-run Parallel
  39. entrypoint: dag-run templates: - name: container-run container: { image: alpine

    } - name: dag-run dag: tasks: - name: dag-A template: container-run - name: dag-B template: container-run dependencies: [dag-A] - name: dag-C template: container-run dependencies: [dag-A] DAG 1 2a 2b
  40. entrypoint: pwd-ls templates: - name: pwd-ls script: image: debian:9.4 command:

    [bash] source: | pwd ls entrypoint: container-ls templates: - name: container-ls container: image: debian:9.4 command: [bash] args: - -c - | pwd ls Script bash
  41. templates: - name: random script: image: python:alpine3.6 command: [python] source:

    | import random i = random.randint(1, 100) print(i) templates: - name: random script: image: node:9.1-alpine command: [node] source: | var rand = Math.floor(Math.random() * 100); console.log(rand); Script
  42. apiVersion: v1 kind: Secret metadata: name: my-secret type: Opaque data:

    # 1f2d1e2e67df my-password: MWYyZDFlMmU2N2Rm Secret K8S側のSecretマニフェスト
  43. entrypoint: whalesay volumes: - name: my-secret-vol secret: secretName: my-secret templates:

    - name: whalesay container: image: alpine:3.7 command: [sh, -c] args: ['echo "secret from file: `cat /secret/mountpath/my-password`"'] volumeMounts: - name: my-secret-vol mountPath: "/secret/mountpath" Secret from volume K8S側のSecretの名前
  44. templates: - name: whalesay container: image: alpine:3.7 command: [sh, -c]

    args: ['echo "secret from env: $PASS";'] env: - name: PASS valueFrom: secretKeyRef: name: my-secret key: my-password Secret from env K8S側のSecretの名前
  45. Other • Data sharing between job • Loops • Conditions

    • Recursion • Exit handlers • Timeouts • Daemon container • Sidecar • Artifacts(S3, HTTP, Git) • Kubernetes resource • Docker in Docker
  46. Argo Workflowsは 何ができない?

  47. Argo Workflowsには トリガー機能がない GithubはArgo CIでWebhook起動できる Argo Events側でトリガーが実装される?

  48. Workflowの設定は 永続化されない (Roadmapにあったような)

  49. 失敗したジョブを GUIから再実行できない

  50. Argo CI

  51. 開発が止まっている? Argo Eventsの 機能と被ってる?

  52. Argo CD

  53. None
  54. argocd-demo

  55. 期待しているものと違う • workflowsが使えない • Ksonnetに限定してる • Deployメカニズムのニーズ ◦ Istioを使うケースを考えるとニーズが低そう

  56. Roadmap • PreSync, PostSync, OutOfSync hooks • Customized application actions

    as Argo workflows • Blue/Green & canary upgrades
  57. Argo Events

  58. Overview

  59. Trigger

  60. CI/CDに 期待できそう!! Argo Events Argo Workflows

  61. Conclusions

  62. Argo projects state • Argo Workflows • Argo CI •

    Argo CD • Argo Events HOLD Trial ASSESS HOLD
  63. Argo Workflowsは シンプルで使いやすい!!

  64. CI/CDで使うには どのProject使っても 2018-06-01時点では不十分 Argo Eventsに期待する

  65. Argo Workflows CI/CD ETL ML Batch どのユースケースでも使えるが どのユースケースでも機能が物足りない

  66. 組み合わせで使う? Argo Events Argo Workflows ??? OR

  67. K8SのJOBを使ってる人 パイプラインとGUIが手に入るの Argo Workflowsを!! JOB Pipeline ユースケース

  68. Airflowなどを使ってる人は まずは一部の処理をArgoで!! Trigger Workflows ユースケース

  69. MLの用途で考えている人は Kubeflowを!! ユースケース

  70. Thanks! Rust、GCP、DevOpsな エンジニアは必要ありませんか? 松井 渉 @watawuwu