How Honestbee does CI/CD on Kubernetes - Voxxed Days Singapore 2018

recording of the talk here: https://youtu.be/YtqOoiF--Mw

Honestbee has been running Kubernetes in production since mid 2016. In this presentation we take a look at how Helm enabled us to manage our deployment configurations as code, and demonstrate how it can be used to power your continuous delivery (CI/CD) pipeline. We tackle the following examples and use cases:

- Continuous Delivery using Helm with Drone.io
- Secret management with VaultController
- Building and hosting our own chart repositories (and the iterations we did)
- Plugins to help manage values
- Chart best practices

Learn from our practical Kubernetes usage, what benefits Helm and Kubernetes provide us. What challenges you may face adopting these technologies in your company and best practices from our experience

Transcript

1. How Honestbee Does CI/CD On Kubernetes @vincentdesmet
2. $ whoami DevOps at Honestbee DevOps at Swatmobile @vincentdesmet / github.com/so0k Singapore-Kubernetes-User-Group Cloud-Native-Singapore
3. What is Honestbee?
4. What is Swat? https://swatmobile.io
5. Overview - Honestbee (~2 years): - Containers - Drone - Kubernetes - Helm - Vault - Swat (~3 weeks): - docker cloud + stacks → Kubernetes + Helm - Pluggable CD using Keel
6. What are Containers? • Packages up software binaries & dependencies • Immutable & testable • Isolate software from each other • Portable across environments a self-contained process ref: kubernetes-comic
7. Why Containers? image: ruby:2.1 services: - postgres stages: - Build - Test - Staging - Production ... source: GitLab CI Lightweight Reproducible builds
8. Why Containers? $ make get-tags-prod getting apse1a tag: master-ebf3205b getting apse1b tag: master-ebf3205b $ make get-tags-staging getting apse1a tag: staging-98a2aa4f getting apse1b tag: staging-98a2aa4f Note: "latest" is not a version Versioned packages
10. More than just packaging and Isolation - Scheduling - Resource Optimisation - Monitoring - Lifecycle and health - Auth{n,z} - Scaling - Discovery - … Source
11. ref: kubernetes-comic
13. Version 1: sed kube-install: update-manifests kubectl create -f manifests/${SHORT_NAME}-deployment.tmp.yaml kube-update: update-manifests kubectl patch deployment ${SHORT_NAME} -p '{"spec":{"template":{"spec":{"containers":[{"name":"'"${SHORT_NAME}"'","image":"' "${IMAGE}"'"}]}}}}' update-manifests : @sed 's#\(image:\) .*#\1 $(IMAGE)#' manifests/${SHORT_NAME}-deployment.yaml > manifests/${SHORT_NAME}-deployment.tmp.yaml
14. Version 2: jinja
15. $ kubectl get deploy -l release=backend NAME DESIRED CURRENT UP-TO-DATE AVAILABLE backend-admin 16 16 16 16 backend-core 32 32 32 32 backend-core-bee 10 10 10 10 backend-karafka 1 1 1 1 backend-scheduler 1 1 1 1 backend-worker-all 8 8 8 8 backend-worker-critical 2 2 2 2 backend-worker-food-import 1 1 1 1 backend-worker-high 2 2 2 2 backend-worker-low 12 12 12 12 $ kubectl get hpa -l release=backend NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS backend-admin Deployment/backend-admin 9% / 70% 16 24 16 backend-core Deployment/backend-core 28% / 70% 32 48 32 backend-core-bee Deployment/backend-core-bee 20% / 80% 10 16 10 $ kubectl get scheduledscaler -l release=backend NAME AGE backend-admin 9d backend-core 9d backend-core-bee 9d
16. Different requirements
17. apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: service-core name: backend-core spec: replicas: 32 ... template: ... spec: containers: - env: - name: NEW_RELIC_APP value: backend-core-apse1a;backend-core ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items: - key: config apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: scheduler name: backend-scheduler spec: replicas: 1 ... template: ... spec: containers: - command: - bin/rake - resque:scheduler env: - name: NEW_RELIC_APP ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items:
18. Kubernetes Deployment challenges - Each application has multiple components - Every component has its own k8s resources How to: - Deploy, Manage, Edit and Update multiple k8s configurations - Deploy multiple k8s configurations as a single application - Parameterise and support multiple environments - Manage application releases: rollout, rollback, history, ... - ...
19. Helm Package Manager for Kubernetes - Aims to provide Apt/Yum/Homebrew User Experience - Ensure collaboration - Shareable Packages Deployment Manager - Repeatable deployments - Manage multiple configurations - Update, Rollback and test application deployments
20. Chart, Repositories, Releases - Chart: “Package”, “Bundle” - Repository: Package Repository - Release: Installed Chart (same chart can be installed multiple times)
21. Helm Charts Render(Templates + Values) = Release Templates: - The Go Template language: {{ .foo | quote }} - Variables, simple control structures (looping, conditionals, ... ) - 50+ functions from Go/Sprig template libraries ...
22. Easy to manage deployments $ helm install backend --values production.yaml $ helm history backend REVISION UPDATED STATUS CHART DESCRIPTION 110 Tue Apr 24 17:53 SUPERSEDED backend-2.0.7 Upgrade complete 111 Wed Apr 25 10:53 SUPERSEDED backend-2.0.7 Upgrade complete 112 Wed Apr 25 14:58 SUPERSEDED backend-2.0.7 Upgrade complete 113 Wed Apr 25 15:02 SUPERSEDED backend-2.0.7 Upgrade complete ... $ helm rollback backend 112
23. Tip: Labels Crucial for: - Service Discovery - Graphing / Alerting - Troubleshooting Define standard set of labels include them for consistency ## _helpers.tpl {{- /* labels.standard prints the standard Helm labels. The standard labels are frequently used in metadata. */ -}} {{- define "labels.standard" }} app: {{ template "fullname" . }} chart: {{ template "chartref" . }} heritage: {{ .Release.Service | quote }} release: {{ .Release.Name | quote }} {{- end }} ## deploy.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: {{ template "backend.fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} ...
24. Public Charts hub.kubeapps.com
25. Tip: Powerful templating For example: iterate yaml maps using the range function {{- range $key, $value := .Values.map }} Key: $key Value: $value {{- end }} map: foo: "bar" baz: "qux" Key: foo Value: bar Key: baz Value: qux
26. {{- range $api_name, $api := .Values.api.types }} {{- if gt $api.replicas 0.0 -}} apiVersion: apps/v1beta1 kind: Deployment metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: service-{{ $api_name }} spec: replicas: {{ $api.replicas }} ... --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: {{ $api_name }}-service spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: {{ template "fullname" $ }}-{{ $api_name }} minReplicas: {{ $api.autoscaling.min }} maxReplicas: {{ $api.autoscaling.max }} targetCPUUtilizationPercentage: {{ $api.autoscaling.cpuTarget }} --- {{- with $api.autoscaling.steps }} apiVersion: "scaling.k8s.restdev.com/v1alpha1" kind: ScheduledScaler ... steps: {{ toYaml . | indent 2 }} {{- end }} --- {{ end -}} {{- end -}} api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
27. ## Tip: Use Helm Values to automate monitoring / alerts ## i.e. Terraform + https://coveo.github.io/gotemplate/ {{- $Values := yaml "production.yaml" }} {{- define "container_graphs" }} {{ $container := . }} graph { title = "Memory - Honestbee-{{ $container }}" viz = "timeseries" request { q = "avg:kubernetes.memory.requests { $helm_release, kube_container_name:{{ $container }} }" type = "line" ... } } {{- end }} resource "datadog_timeboard" "backend_production" { title = "Backend Timeboard (Terraform)" {{- range $api, $definition := $Values.api.types }} {{- template "container_graphs" $api }} {{- end }} template_variable { name = "helm_release" prefix = "helm_release" default = "backend-production" } } api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
28. $ gotemplate -P templates/timeboard.tf.template > timeboard.tf $ terraform apply datadog_timeboard.backend_production: Refreshing state... (ID: 102245) An execution plan has been generated and is shown Below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: ~ datadog_timeboard.backend_production graph.0.request.0.q: "max:kubernetes.cpu..." => "avg:kubernetes.cpu..." Plan: 0 to add, 1 to change, 0 to destroy. Tip: Automate monitoring / alerts
29. Tip: Helm Values Defines your application configuration per environment! - Should be source controlled or tracked - roboll/helmfile - skuid/helm-value-store - Should not contain secrets SECRETS
30. Private Chart Repository Honestbee: Version 1 - Self-host: - Using s3 Bucket + VPC endpoint - CI/CD: - Package - Generate Index - Push to s3 articles/devops/2017-07/drone-helm- repository resource "aws_s3_bucket" "b" { bucket = "${var.bucket_name_prefix}.${var.domain_name}" policy = "${data.aws_iam_policy_document.s3-read.json}" website { index_document = "index.html" } cors_rule { allowed_headers = ["*"] allowed_methods = ["PUT", "POST"] allowed_origins = [ "https://${var.bucket_name_prefix}.${var.domain_name}", ] } } data "aws_iam_policy_document" "s3-read" { statement { sid = "Access-from-specific-VPC-only" effect = "Allow" actions = [ "s3:GetObject" ] resources = [ "arn:aws:s3:::${var.bucket_name_prefix}.${var.domain_name}/*", ] condition { test = "StringEquals" variable = "aws:sourceVpce" values = [ "${values(data.terraform_remote_state.kops-state.vpc_endpoint)}", ] } } }
31. Integration with CI/CD
32. Private Chart Repository Honestbee Version 2: - Use ChartMuseum - CI/CD: - Package and push honestbee/drone-chartmuseum ChartMuseum is an open-source Helm Chart Repository written in Go (Golang), with support for cloud storage backends, including Google Cloud Storage and Amazon S3. Works as a valid Helm Chart Repository, and also provides an API for uploading new chart packages to storage etc.
33. Secrets? Version 1 (k8s 1.7): Vault + VaultController - Hashicorp Vault - Supports k8s auth - roboll/kube-vault-controller - Add k8s custom resource: secretClaim kind: SecretClaim apiVersion: vaultproject.io/v1 metadata: name: {{ template "fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} spec: type: Opaque path: "secret/{{ .Values.env }}/{{ .Release.Name }}" renew: 3900
35. Swat May 7 → May 25 docker cloud + stack → kubernetes + helm routing-worker: autoredeploy: true deployment_strategy: every_node environment: - MASTER_URL=master - MASTER_SECRET=foo expose: - "3352" image: swat/routing-worker:staging links: - routing-master:master tags: - staging routing-master: autoredeploy: true environment: - VIRTUAL_HOST=http://bar.com,https://bar.com - ... services: routing-master: replicaCount: 1 image: repository: swat/routing-master tag: master pullPolicy: Always port: 3352 healthChecks: livenessPath: / readinessPath: / resources: limits: cpu: 500m memory: 500Mi ingress: enabled: true hosts: - "foo.com" environment: MASTER_URL: "master" services: routing-master: image: tag: staging resources: limits: cpu: 250m memory: 300Mi ingress: hosts: - "bar.com" default staging staging 1:1
36. Swat May 7 → May 25 docker cloud + stack → kubernetes + helm - Automated haproxy config through VIRTUAL_HOST envvars - Automated Let's Encrypt Certificates - Manual DNS configuration - Challenges in Secret Management - zalando/kube-ingress-aws-controller - Automated AWS ALB configuration - Automated AWS Certificate Management - kubernetes/external-dns - Automated DNS configuration using AWS Route53 - mozilla/sops - Automated secret encrypt / decrypt using AWS KMS 1:1
37. Swat: Secrets vNext (k8s 1.8+): Bank-vaults + vault-sidekick - banzaicloud/bank-vaults: - Automated unsealing - Automated Vault initialisation - UKHomeOffice/vault-sidekick - Vault audit by Service
38. Swat - Pluggable CD May 25 → May 31 docker cloud + stack → helm + keel.sh - Automated deployment on image push - Git branch → Image tag (no sha) - Poll registry for image changes - Trigger Kubernetes rolling update using newer image digest through helm - Provides slack notifications (optional approval workflow): Re-use existing CI
39. Join us