How Honestbee does CI/CD on Kubernetes

Transcript

1. How Honestbee Does CI/CD On Kubernetes @vincentdesmet
2. $ whoami DevOps at Honestbee DevOps at Swatmobile @vincentdesmet / github.com/so0k Singapore-Kubernetes-User-Group Cloud-Native-Singapore
3. What is Honestbee?
4. What is Swat? https://swatmobile.io
5. Overview - Honestbee (~2 years): - Containers - Drone - Kubernetes - Helm - Vault - Swat (~3 weeks): - docker cloud + stacks → Kubernetes + Helm - Pluggable CD using Keel
6. What are Containers? • Packages up software binaries & dependencies • Immutable & testable • Isolate software from each other • Portable across environments a self-contained process ref: kubernetes-comic
7. Why Containers? image: ruby:2.1 services: - postgres stages: - Build - Test - Staging - Production ... source: GitLab CI Lightweight Reproducible builds
8. Why Containers? $ make get-tags-prod getting apse1a tag: master-ebf3205b getting apse1b tag: master-ebf3205b $ make get-tags-staging getting apse1a tag: staging-98a2aa4f getting apse1b tag: staging-98a2aa4f Note: "latest" is not a version Versioned packages
10. More than just packaging and Isolation - Scheduling - Resource Optimisation - Monitoring - Lifecycle and health - Auth{n,z} - Scaling - Discovery - … Source
11. ref: kubernetes-comic
13. Version 1: sed kube-install: update-manifests kubectl create -f manifests/${SHORT_NAME}-deployment.tmp.yaml kube-update: update-manifests kubectl patch deployment ${SHORT_NAME} -p '{"spec":{"template":{"spec":{"containers":[{"name":"'"${SHORT_NAME}"'","image":"' "${IMAGE}"'"}]}}}}' update-manifests : @sed 's#\(image:\) .*#\1 $(IMAGE)#' manifests/${SHORT_NAME}-deployment.yaml > manifests/${SHORT_NAME}-deployment.tmp.yaml
14. Version 2: jinja
15. $ kubectl get deploy -l release=backend NAME DESIRED CURRENT UP-TO-DATE AVAILABLE backend-admin 16 16 16 16 backend-core 32 32 32 32 backend-core-bee 10 10 10 10 backend-karafka 1 1 1 1 backend-scheduler 1 1 1 1 backend-worker-all 8 8 8 8 backend-worker-critical 2 2 2 2 backend-worker-food-import 1 1 1 1 backend-worker-high 2 2 2 2 backend-worker-low 12 12 12 12 $ kubectl get hpa -l release=backend NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS backend-admin Deployment/backend-admin 9% / 70% 16 24 16 backend-core Deployment/backend-core 28% / 70% 32 48 32 backend-core-bee Deployment/backend-core-bee 20% / 80% 10 16 10 $ kubectl get scheduledscaler -l release=backend NAME AGE backend-admin 9d backend-core 9d backend-core-bee 9d
16. Different requirements
17. apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: service-core name: backend-core spec: replicas: 32 ... template: ... spec: containers: - env: - name: NEW_RELIC_APP value: backend-core-apse1a;backend-core ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items: - key: config apiVersion: apps/v1beta1 kind: Deployment metadata: labels: app: backend component: scheduler name: backend-scheduler spec: replicas: 1 ... template: ... spec: containers: - command: - bin/rake - resque:scheduler env: - name: NEW_RELIC_APP ... image: myregistry/ backend:master-ebf3205b volumeMounts: - mountPath: /app/config/application.yml name: backend-config subPath: application.yml volumes: - name: backend-config secret: items:
18. Kubernetes Deployment challenges - Each application has multiple components - Every component has its own k8s resources How to: - Deploy, Manage, Edit and Update multiple k8s configurations - Deploy multiple k8s configurations as a single application - Parameterise and support multiple environments - Manage application releases: rollout, rollback, history, ... - ...
19. Helm Package Manager for Kubernetes - Aims to provide Apt/Yum/Homebrew User Experience - Ensure collaboration - Shareable Packages Deployment Manager - Repeatable deployments - Manage multiple configurations - Update, Rollback and test application deployments
20. Chart, Repositories, Releases - Chart: “Package”, “Bundle” - Repository: Package Repository - Release: Installed Chart (same chart can be installed multiple times)
21. Helm Charts Render(Templates + Values) = Release Templates: - The Go Template language: {{ .foo | quote }} - Variables, simple control structures (looping, conditionals, ... ) - 50+ functions from Go/Sprig template libraries ...
22. Easy to manage deployments $ helm install backend --values production.yaml $ helm history backend REVISION UPDATED STATUS CHART DESCRIPTION 110 Tue Apr 24 17:53 SUPERSEDED backend-2.0.7 Upgrade complete 111 Wed Apr 25 10:53 SUPERSEDED backend-2.0.7 Upgrade complete 112 Wed Apr 25 14:58 SUPERSEDED backend-2.0.7 Upgrade complete 113 Wed Apr 25 15:02 SUPERSEDED backend-2.0.7 Upgrade complete ... $ helm rollback backend 112
23. Tip: Labels Crucial for: - Service Discovery - Graphing / Alerting - Troubleshooting Define standard set of labels include them for consistency ## _helpers.tpl {{- /* labels.standard prints the standard Helm labels. The standard labels are frequently used in metadata. */ -}} {{- define "labels.standard" }} app: {{ template "fullname" . }} chart: {{ template "chartref" . }} heritage: {{ .Release.Service | quote }} release: {{ .Release.Name | quote }} {{- end }} ## deploy.yaml apiVersion: extensions/v1beta1 kind: Deployment metadata: name: {{ template "backend.fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} ...
24. Public Charts hub.kubeapps.com
25. Tip: Powerful templating For example: iterate yaml maps using the range function {{- range $key, $value := .Values.map }} Key: $key Value: $value {{- end }} map: foo: "bar" baz: "qux" Key: foo Value: bar Key: baz Value: qux
26. {{- range $api_name, $api := .Values.api.types }} {{- if gt $api.replicas 0.0 -}} apiVersion: apps/v1beta1 kind: Deployment metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: service-{{ $api_name }} spec: replicas: {{ $api.replicas }} ... --- apiVersion: autoscaling/v1 kind: HorizontalPodAutoscaler metadata: name: {{ template "fullname" $ }}-{{ $api_name }} labels: {{ include "labels.standard" $ | indent 4 }} component: {{ $api_name }}-service spec: scaleTargetRef: apiVersion: apps/v1beta1 kind: Deployment name: {{ template "fullname" $ }}-{{ $api_name }} minReplicas: {{ $api.autoscaling.min }} maxReplicas: {{ $api.autoscaling.max }} targetCPUUtilizationPercentage: {{ $api.autoscaling.cpuTarget }} --- {{- with $api.autoscaling.steps }} apiVersion: "scaling.k8s.restdev.com/v1alpha1" kind: ScheduledScaler ... steps: {{ toYaml . | indent 2 }} {{- end }} --- {{ end -}} {{- end -}} api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
27. ## Tip: Use Helm Values to automate monitoring / alerts ## i.e. Terraform + https://coveo.github.io/gotemplate/ {{- $Values := yaml "production.yaml" }} {{- define "container_graphs" }} {{ $container := . }} graph { title = "Memory - Honestbee-{{ $container }}" viz = "timeseries" request { q = "avg:kubernetes.memory.requests { $helm_release, kube_container_name:{{ $container }} }" type = "line" ... } } {{- end }} resource "datadog_timeboard" "backend_production" { title = "Backend Timeboard (Terraform)" {{- range $api, $definition := $Values.api.types }} {{- template "container_graphs" $api }} {{- end }} template_variable { name = "helm_release" prefix = "helm_release" default = "backend-production" } } api: types: admin: autoscaling: min: 4 max: 12 cpuTarget: 70 steps: - runat: '0 30 11 * * *' mode: range minReplicas: 8 # 11:30am: pre-lunch schedule maxReplicas: 12 ... resources: limits: cpu: 2 memory: 6Gi requests: cpu: 2 memory: 6Gi config: admin-panel-enabled: true url: foo.honestbee.com core-bee: autoscaling: min: 2 max: 16 cpuTarget: 80 steps: - runat: '0 00 08 * * *' mode: range minReplicas: 4 # 08:00am: day schedule maxReplicas: 16
28. $ gotemplate -P templates/timeboard.tf.template > timeboard.tf $ terraform apply datadog_timeboard.backend_production: Refreshing state... (ID: 102245) An execution plan has been generated and is shown Below. Resource actions are indicated with the following symbols: ~ update in-place Terraform will perform the following actions: ~ datadog_timeboard.backend_production graph.0.request.0.q: "max:kubernetes.cpu..." => "avg:kubernetes.cpu..." Plan: 0 to add, 1 to change, 0 to destroy. Tip: Automate monitoring / alerts
29. Tip: Helm Values Defines your application configuration per environment! - Should be source controlled or tracked - roboll/helmfile - skuid/helm-value-store - Should not contain secrets SECRETS
30. Private Chart Repository Honestbee: Version 1 - Self-host: - Using s3 Bucket + VPC endpoint - CI/CD: - Package - Generate Index - Push to s3 articles/devops/2017-07/drone-helm- repository resource "aws_s3_bucket" "b" { bucket = "${var.bucket_name_prefix}.${var.domain_name}" policy = "${data.aws_iam_policy_document.s3-read.json}" website { index_document = "index.html" } cors_rule { allowed_headers = ["*"] allowed_methods = ["PUT", "POST"] allowed_origins = [ "https://${var.bucket_name_prefix}.${var.domain_name}", ] } } data "aws_iam_policy_document" "s3-read" { statement { sid = "Access-from-specific-VPC-only" effect = "Allow" actions = [ "s3:GetObject" ] resources = [ "arn:aws:s3:::${var.bucket_name_prefix}.${var.domain_name}/*", ] condition { test = "StringEquals" variable = "aws:sourceVpce" values = [ "${values(data.terraform_remote_state.kops-state.vpc_endpoint)}", ] } } }
31. Integration with CI/CD
32. Private Chart Repository Honestbee Version 2: - Use ChartMuseum - CI/CD: - Package and push honestbee/drone-chartmuseum ChartMuseum is an open-source Helm Chart Repository written in Go (Golang), with support for cloud storage backends, including Google Cloud Storage and Amazon S3. Works as a valid Helm Chart Repository, and also provides an API for uploading new chart packages to storage etc.
33. Secrets? Version 1 (k8s 1.7): Vault + VaultController - Hashicorp Vault - Supports k8s auth - roboll/kube-vault-controller - Add k8s custom resource: secretClaim kind: SecretClaim apiVersion: vaultproject.io/v1 metadata: name: {{ template "fullname" . }} labels: {{- include "labels.standard" . | indent 4 }} spec: type: Opaque path: "secret/{{ .Values.env }}/{{ .Release.Name }}" renew: 3900
35. Swat May 7 → May 25 docker cloud + stack → kubernetes + helm routing-worker: autoredeploy: true deployment_strategy: every_node environment: - MASTER_URL=master - MASTER_SECRET=foo expose: - "3352" image: swat/routing-worker:staging links: - routing-master:master tags: - staging routing-master: autoredeploy: true environment: - VIRTUAL_HOST=http://bar.com,https://bar.com - ... services: routing-master: replicaCount: 1 image: repository: swat/routing-master tag: master pullPolicy: Always port: 3352 healthChecks: livenessPath: / readinessPath: / resources: limits: cpu: 500m memory: 500Mi ingress: enabled: true hosts: - "foo.com" environment: MASTER_URL: "master" services: routing-master: image: tag: staging resources: limits: cpu: 250m memory: 300Mi ingress: hosts: - "bar.com" default staging staging 1:1
36. Swat May 7 → May 25 docker cloud + stack → kubernetes + helm - Automated haproxy config through VIRTUAL_HOST envvars - Automated Let's Encrypt Certificates - Manual DNS configuration - Challenges in Secret Management - zalando/kube-ingress-aws-controller - Automated AWS ALB configuration - Automated AWS Certificate Management - kubernetes/external-dns - Automated DNS configuration using AWS Route53 - mozilla/sops - Automated secret encrypt / decrypt using AWS KMS 1:1
37. Swat: Secrets vNext (k8s 1.8+): Bank-vaults + vault-sidekick - banzaicloud/bank-vaults: - Automated unsealing - Automated Vault initialisation - UKHomeOffice/vault-sidekick - Vault audit by Service
38. Swat - Pluggable CD May 25 → May 31 docker cloud + stack → helm + keel.sh - Automated deployment on image push - Git branch → Image tag (no sha) - Poll registry for image changes - Trigger Kubernetes rolling update using newer image digest through helm - Provides slack notifications (optional approval workflow): Re-use existing CI
39. Join us