Introduction to Weave Flux

Weave Flux 是 CNCF 底下的 Sandbox Project，其主要目標是希望透過 GitOps 來完成 Kubernetes 資源的自動化版本更新及佈署。

在此議程會簡單介紹什麼是 GitOps、Weave Flux 以及介紹 Weave Flux 的運作流程，帶大家初步認識 Flux 這個專案。

# Cloud Native Taiwan User Group Meetup #25

Transcript

1. Introduction to Weave Flux 施柏賢 Pohsien Shih 2020/02/22 CNTUG Meetup #25
2. @ Pohsien Who am I 施柏賢 Pohsien Shih • System Engineer • A member of Cloud Native Taiwan User Group (CNTUG) Blog: https://pohsienshih.github.io/ Github: https://github.com/pohsienshih Email: pohsien324@gmail.com 2
3. @ Pohsien Outline • GitOps • Weave Flux • Installation • Mechanism Discussion • Demo • Deficiencies and Advanced Usage • Conclusion • References 3
4. @ Pohsien What is GitOps? 4
5. @ Pohsien GitOps GitOps, a term coined by Weaveworks. The concept is using Git as the single source of truth for declarative infrastructure and applications. It means the lifecycle of the program (from development to deployment) can be version controlled in Git, and the automated workflow can also be triggered by Git event (Push, PR, etc). Benefits: 1. Fast deployment 2. Easier rollback/review the changing 3. Easier compliance and auditing 4. Friendly for developers who are already familiar with git 5 Ref: https://www.weave.works/blog/gitops-operations-by-pull-request https://www.weave.works/technologies/gitops/
6. @ Pohsien Weave Flux 6
7. @ Pohsien Weaveworks Flux “Flux is the Kubernetes operator that makes GitOps happen in your cluster. It ensures that the cluster config matches the one in git and automates your deployments (Continue Delivery).” - Weaveworks Official Website. 7 Ref: https://www.weave.works/oss/flux/ Features: • Automated git->cluster synchronisation. • Automated deployment of new container images.
8. @ Pohsien Weaveworks Flux 8 Ref: https://www.weave.works/oss/flux/ 2017/01 0.1.0 release 1.0.0 release 2017/08 1.5.0 release 2018/08 1.10.0 release 2019/02 1.14.0 release 2019/08 2019/12 2020/02 1.17.0 release 1.18.0 release 2019/08/28 Join CNCF
9. @ Pohsien Weave Flux History 9 Ref: https://www.weave.works/blog/flux-joins-the-cncf-sandbox Flux joins the CNCF sandbox project on Wednesday, August 28.
10. @ Pohsien Weave Flux Workflow 10 Repository A (for application) Repository B (for manifest) Container Registry CI Tools Developer 1. Push Application Code 2. Build Image 3. Push Image 4. Push Manifest 5. Clone Manifest 6. Apply Manifest 7. Polling New Image 8. Update Manifest (Add commit) 9. Apply New Manifest
11. @ Pohsien GitOps with Weave Flux 11 Ref: https://github.com/fluxcd/flux/blob/master/docs/_files/flux-cd-diagram.png
12. @ Pohsien Installation 12
13. @ Pohsien Components of Flux • deployment.apps/memcached • service/memcached • serviceaccount/flux • clusterrole.rbac.authorization.k8s.io/flux • clusterrolebinding.rbac.authorization.k8s.io/flux • deployment.apps/flux • secret/flux-git-deploy 13
14. @ Pohsien Flux Installation 14 $ fluxctl install \ --git-user=”your Git username” \ --git-email=”your Git email” \ --git-url=git@ <git server>:<your manifest repository> \ --namespace=flux | kubectl apply -f - Install fluxctl. $ kubectl creat ns flux $ snap install fluxctl Create a namespace (e.g: flux) for Flux workloads. Deploy Flux into your Kubernetes cluster. Ref: https://docs.fluxcd.io/en/1.18.0/tutorials/get-started.html
15. @ Pohsien 15 $ kubectl edit deploy flux -n flux Flux Installation (cont.) Append the specifics flags that you want to configure into Flux deployment. e.g., --sync-garbage-collection, --git-ci-skip, etc .) kind: Deployment metadata: name: flux ... spec: ... template: ... spec: containers: - args: - <some flags...> - <some flags...> - ... Append the flags into spec.template.spec.containers.args
16. @ Pohsien Flux Installation (cont.) 16 $ kubectl exec -it <flux pod name> git clone git@ <git server>:<your manifest repository> -n flux Check the state of Flux workloads. $ fluxctl identity --k8s-fwd-ns flux ssh-rsa AAAAB3N... $ kubectl get all -n flux Get the SSH public key of Flux. Add this key to your Git repository. Make sure Flux daemon has privilege to clone the repository.
17. @ Pohsien 17 # List the state of workloads in cluster $ fluxctl list-workloads -n <the namespace of workloads> \ --k8s-fwd-ns <flux daemon namespace> # List the image metadata for workload $ fluxctl list-images --workload <workload name> \ --k8s-fwd-ns <flux daemon namespace> Flux Installation (cont.) List the resources.
18. @ Pohsien Mechanism Discussion 18 • Sync with Git Manifest Repo • New Image Scanning
19. @ Pohsien Sync with Git Manifest Repo 1. Implementation 2. Synchronization Interval 3. Garbage Collection 4. Temporarily ignore the manifest 5. Lock/Unlock the workloads 19
20. @ Pohsien Sync with Git Manifest Repo 20 git fetch & git pull & kubectl apply & git tag Equivalent to the following commands: 1. Get the latest commit from repo. 2. Apply the new version manifest. 3. Push a tag (flux/flux-sync) to the commit which means it has been processed by flux. Implementation
21. @ Pohsien Sync with Git Manifest Repo 21 Synchronization Interval Git Polling Interval • How often does flux looks for new commits. • Five minutes by default. • Control the interval by using flag: --git-poll-interval. e.g. --git-poll-interval=1m30s Flux Sync Interval • How often does flux applies what’s in git repo if there are no new commits. • This can recover the resource affected by unexpected factors. • Five minutes by default. • Control the interval by using flag: --sync-interval. e.g. --sync-interval=1m30s Ref: https://docs.fluxcd.io/en/1.18.0/faq.html#how-often-does-flux-check-for-new-images
22. @ Pohsien Sync with Git Manifest Repo 22 Synchronization Interval Trigger the synchronization by manual: $ fluxctl sync --k8s-fwd-ns <flux daemon deployment namespace> Revision of master to apply is 0cff093 Waiting for 0cff093 to be applied ... Done.
23. @ Pohsien Sync with Git Manifest Repo 23 By default, Flux will not get rid of the resources from the cluster if they have been removed in the repository You can enable the garbage collection by using flag: --sync-garbage-collection. Then Flux will use the label: fluxcd.io/sync-gc-mark to recognize the resource created by fluxd and destroy it. Garbage Collection Ref: https://docs.fluxcd.io/en/1.18.0/references/garbagecollection.html apiVersion: apps/v1 kind: Deployment metadata: labels: app: web-service-deployment fluxcd.io/sync-gc-mark: <hashcode> spec: ...
24. @ Pohsien Sync with Git Manifest Repo 24 Flux support the feature that can temporarily ignore the manifest. This means when flux detect a new image, it will only update the Git manifest repo, and will not update the resource in the cluster. Temporarily Ignore the Manifest Ref: https://docs.fluxcd.io/en/stable/faq.html#can-i-temporarily-make-flux-ignore-a-manifest apiVersion: apps/v1 kind: Deployment metadata: name: my-web ... annotations: fluxcd.io/ignore: true spec: ... Use the following annotation to make flux ignores the manifest.
25. @ Pohsien Sync with Git Manifest Repo 25 The other feature similar to ignore is lock. Locking a workload will stop manual or automated releases to that workload. Changes made in the file will still be synced. Lock the resource Ref: https://docs.fluxcd.io/en/stable/references/fluxctl.html#locking-a-workload apiVersion: apps/v1 kind: Deployment metadata: name: my-web ... annotations: fluxcd.io/locked: 'true' fluxcd.io/locked_user: pohsien <pohsien@example.com> Or you can simply use the command: fluxctl lock --workload= <workload name> --k8s-fwd-ns \ <flux daemon namespace> Use the following annotation to make flux lock the workload:
26. @ Pohsien Sync with Git Manifest Repo 26 Unlock the resource Ref: https://docs.fluxcd.io/en/stable/references/fluxctl.html#locking-a-workload apiVersion: apps/v1 kind: Deployment metadata: name: my-web ... annotations: fluxcd.io/locked: 'true' fluxcd.io/locked_user: pohsien <pohsien@example.com> Or you can simply use the command: fluxctl unlock --workload= <workload name> --k8s-fwd-ns \ <flux daemon namespace> Remove the following annotation to make flux unlock the workload:
27. @ Pohsien Sync with Git Manifest Repo 27 Ignore vs Lock Ingore Lock Scan a new Image V V Auto update the manifest and add a commit V X Sync between manifest and workload X V Workload: The workloads executed on Cluster. Manifest: The YAML files in Git repo.
28. @ Pohsien Image Monitoring 1. Image Automatically Update 2. Implementation 3. Image Scanning Interval 4. Disable Registry Scanning 5. Image Filter 28
29. @ Pohsien Image Monitoring 29 Image Automatically Update Flux can be used to automate container image updates in the cluster. apiVersion: apps/v1 kind: Deployment metadata: name: my-web ... annotations: fluxcd.io/automated: "true" Or you can simply use the command: fluxctl automate --workload= <workload name> --k8s-fwd-ns \ <flux daemon namespace> Use the following annotation to enable auto update mechanism: Ref: https://docs.fluxcd.io/en/stable/references/automated-image-update.html#
30. @ Pohsien Image Monitoring 30 Disable Automatically Update apiVersion: apps/v1 kind: Deployment metadata: name: my-web ... annotations: fluxcd.io/automated: "true" Or you can simply use the command: fluxctl deautomate --workload= <workload name> --k8s-fwd-ns \ <flux daemon namespace> Remove the following annotation to disable auto update mechanism:
31. @ Pohsien Image Monitoring 31 1. Use Docker API to get the metadata of Image. 2. Update the manifest and push a commit to Git repo. 3. Apply the new version manifest to Kubernetes cluster. 4. Push a tag (flux/flux-sync) to the commit which means it has been processed by flux. Implementation Enable automatic deployment. *Flux doesn’t support the latest tag. Every image tag must be unique.
32. @ Pohsien Image Monitoring 32 Image Scanning Interval Polling Rate • The rate limiting that flux scans registry for Image metadata. • As quickly as it can. • Control the interval by using flags: --registry-rps and --registry-burst To avoid to get blacklist by registry, it’s not recommended to increase the rate limiting. Workloads (enable auto-deploy) Update Interval • How often does flux check update for automated workloads. • This can recover the workloads affected by unexpected factors. • Five minutes by default. • Control the interval by using flag: --automation-interval. e.g. --automation-interval=1m30s Ref: https://docs.fluxcd.io/en/1.18.0/faq.html#how-often-does-flux-check-for-new-images
33. @ Pohsien Image Monitoring 33 Ref: https://docs.fluxcd.io/en/1.18.0/faq.html#how-often-does-flux-check-for-new-images If you want to update image by manual, you can use the following commands: $ fluxctl release --workload= <workload name> --update-image= <new image name> --k8s-fwd-ns \ <flux daemon namespace> Submitting release ... WORKLOAD STATUS UPDATES pohsien:deployment/web-service-deployment success web-service: pohsien/my-web:23 -> 20 Commit pushed: 04b59a1 Commit applied: 04b59a1 Image Scanning Interval
34. @ Pohsien Image Monitoring 34 Ref: https://docs.fluxcd.io/en/1.18.0/faq.html#how-often-does-flux-check-for-new-images Disable Registry Scanning • Exclude images from specific registry --registry-disable-scanning=docker.io/*,quay.io/* • Exclude specific image --registry-exclude-image=<image name>:<tag> e.g. --registry-exclude-image=*test* You can disable Image scanning by configure the following flag:
35. @ Pohsien Image Monitoring 35 Image Filter Auto-update image against specific subset of tags. apiVersion: apps/v1 kind: Deployment metadata: annotations: fluxcd.io/automated: "true" fluxcd.io/tag. <container name> : glob:<tag name> fluxcd.io/tag.nginx: glob:test-* ... containers: - name: nginx image: docker.io/pohsien/my-nginx:test-1 Or you can simply use the command: fluxctl policy --workload= <workload name> --tag-all='tag name' --k8s-fwd-ns <flux daemon namespace> Use the following annotation to enable auto update mechanism:
36. @ Pohsien Demo 36
37. @ Pohsien Deficiencies and Advanced Usage 37
38. @ Pohsien Deficiencies 1. HA mode is not yet supported. 2. Only one Git repo can be processed at a time. 3. Only support Docker container registry. 4. fluxctl doesn't support all flags of Flux daemon yet. 38
39. @ Pohsien Helm Operator Flux can also combine to the Helm Operator to release your chart via GitOps. Steps: 1. Use CRD to declare your Helm. 2. Put the manifest into the Git repo. 3. Assign the repo to Flux. 39 Ref: https://docs.fluxcd.io/projects/helm-operator/en/1.0.0-rc9/index.html# https://docs.fluxcd.io/en/1.18.0/references/helm-operator-integration.html# kind: HelmRelease metadata: name: rabbit namespace: default spec: releaseName: rabbitmq targetNamespace: mq timeout: 300 ... chart: repository: https://kubernetes-charts.storage.goo gleapis.com/ name: rabbitmq version: 3.3.6 values: replicas: 1 Custom Resources: HelmRelease
40. @ Pohsien Conclusion Flux 是一個 Kubernetes GitOps Operator ， Kubernetes 能夠透過 Flux 自動的去同 步及更新 Cluster Workload ，大幅降低維運成本，而開發人員也只需要將心力放在開 發應用程式以及 Git 版控上。雖然 Flux 目前還有許多地方需要改進，但相信在加入 CNCF 後，這個專案能越來越茁壯，並且能夠支援更多的功能，這是未來可以期待的。 40
41. @ Pohsien Reference • Guide To GitOps • GitOps - Operations by Pull Request • Weave Online User Group • Flux Official Website • Flux Document • Kubeconf 2019 - Introducing Flux • Amazon EKS Workshop - GITOPS WITH WEAVE FLUX • GitHub - Flux • Helm Operator 41