Kustomizing your Kubernetes Deployments Cloud Native Computing Switzerland Meetup, 22 November 2018 David Schweikert @dschweikert AdNovum Informatik AG

“Configuration management” in Kubernetes?

We don’t need it! $ kubectl apply -f /

But… how to parametrize?

No content

Parametrization is important! § Minimize differences between environments § Test what you deploy in prod! § Avoid code duplication

Me in 2017: !

Me in 2017: ! § OpenShift Templates § Helm § Self-made? Jinja2? § Even thinking of using Ansible…

Me in 2018: ! § Kustomize

What are the options?

Ansible Forge Helm K8comp KPM KY Kapitan Kdeploy Kedge Kenv Kexpand Kit-Deploymentizer Kompose Konfd Kontemplate Ksonnet Ktmpl Kubecfg Kubegen Kubernetes-deploy Kubetpl Kustomize Mortar OpenShift templates Psykube Spread Terraform …

Why Kustomize?

Reason #1: Embrace Kubernetes API Object Descriptions (YAML files)

kind: Service apiVersion: v1 metadata: name: my-service spec: selector: app: MyApp ports: - protocol: TCP port: 80 targetPort: 9376 my-service.yaml:

local params = std.extVar("__ksonnet/params").components.demo; local k = import "k.libsonnet"; local service = k.core.v1.service; local servicePort = k.core.v1.service.mixin.spec.portsType; local targetPort = params.containerPort; local labels = {app: params.name}; local appService = service .new( params.name, labels, servicePort.new(params.servicePort, targetPort)) .withType(params.type); k.core.v1.list.new([appService]) Ksonnet:

keep your YAML files the way they are

Reason #2: Keep using kubectl

$ kustomize build . | kubectl apply -f - Typical workflow:

Helm: § forget about kubectl, now you need to always use “helm install”, “helm ls”, “helm status”

Reason #3: Declarative and Templates-free

YAML can be painful sometimes

No content

Templated YAML is much worse…

apiVersion: v1 kind: Service metadata: name: {{ template "grafana.fullname" . }} labels: app: {{ template "grafana.name" . }} chart: {{ template "grafana.chart" . }} release: {{ .Release.Name }} heritage: {{ .Release.Service }} {{- if .Values.service.labels }} {{ toYaml .Values.service.labels | indent 4 }} {{- end }} {{- with .Values.service.annotations }} annotations: {{ toYaml . | indent 4 }} {{- end }} spec: {{- if (or (eq .Values.service.type "ClusterIP") (empty .Values.service.type)) }} type: ClusterIP {{- if .Values.service.clusterIP }} clusterIP: {{ .Values.service.clusterIP }} {{end}} {{- else if eq .Values.service.type "LoadBalancer" }} …

Kustomize’s cure: § No templating at all!

Kustomize’s cure: § Overlays § Transformations § Generators § Patches

Overlays

Overlays myApp | ├── base │ ├── deployment.yaml │ ├── kustomization.yaml │ └── service.yaml | ├── development │ ├── ingress.yaml │ └── kustomization.yaml | └── production ├── ingress.yaml └── kustomization.yaml resources: - deployment.yaml - service.yaml

Overlays myApp | ├── base │ ├── deployment.yaml │ ├── kustomization.yaml │ └── service.yaml | ├── development │ ├── ingress.yaml │ └── kustomization.yaml | └── production ├── ingress.yaml └── kustomization.yaml bases: - ../base resources: - ingress.yaml bases: - ../base resources: - ingress.yaml

$ kustomize build development apiVersion: v1 kind: Service metadata: … --- apiVersion: apps/v1 kind: Deployment metadata: … --- apiVersion: apps/v1 kind: Ingress metadata: …

Transformations

Transformations myApp| ├── base │ ├── deployment.yaml │ └── kustomization.yaml | ├── development │ └── kustomization.yaml bases: - ../base namePrefix: dev- § All resource names are now prefixed with “dev-”

Transformations It’s what makes kustomize so powerful: § Because it knows Kubernetes semantics § A single line, use-case specific (e.g. namePrefix) causes big changes § All references are preserved

Generators kustomization.yaml: configMapGenerator: - name: myconfig files: - configs/configfile - configs/another_configfile § generates: myconfig-b62k6t7g8f (and fixes all references to it) § b62k6t7g8f is a hash of the contents

Generators kustomization.yaml: configMapGenerator: - name: myconfig files: - configs/configfile - configs/another_configfile § generates: myconfig-b62k6t7g8f (and fixes all references to it) § b62k6t7g8f is a hash of the contents !!!

Patches

Patches myApp| ├── base │ ├── deployment.yaml │ └── kustomization.yaml | ├── development | ├── deployment.patch.yaml │ └── kustomization.yaml bases: - ../base patches: - deployment.patch.yaml

Patches myApp| ├── base │ ├── deployment.yaml │ └── kustomization.yaml | ├── development | ├── deployment.patch.yaml │ └── kustomization.yaml apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: replicas: 1

Patches Same syntax as “kubectl patch” § Strategic merge patches § JSON patches (RFC 6902)

Challenges

Challenges § Documentation is (currently) not super great

Challenges § Things that Kustomize doesn’t know about § OpenShift objects § CRDs

Challenges § Things that Kustomize doesn’t know about § OpenShift objects § CRDs It is now possible to extend Kustomize knowledge about Kubernetes objects see also: https://github.com/adnovum/kustomize-openshift

Summary When to use kustomize § It’s the perfect tool to parametrize your own application When not to use kustomize § Packaging an application for the general public (use Helm for that)

Questions? More about this topic: § Declarative application management in Kubernetes August 2017, by Brian Grant § Introducing kustomize; Template-free Configuration Customization for Kubernetes May 2018, by Jeff Regan and Phil Wittrock Contacting me: § david@schweikert.ch, @dschweikert