Kustomizing your Kubernetes Deployments

How to parametrize your Kubernetes object definitions using kustomize, a template-free customization tool.

Talk presented at the Cloud Native Computing Meetup November 2018.

Transcript

1. Kustomizing your Kubernetes Deployments
   Cloud Native Computing Switzerland Meetup, 22 November 2018
   David Schweikert @dschweikert
   AdNovum Informatik AG
   

   View Slide

2. “Configuration management” in Kubernetes?
   

   View Slide

3. We don’t need it!
   $ kubectl apply -f /
   

   View Slide

4. But… how to parametrize?
   

   View Slide

5. View Slide

6. Parametrization is important!
   § Minimize differences between environments
   § Test what you deploy in prod!
   § Avoid code duplication
   

   View Slide

7. Me in 2017:
   !
   

   View Slide

8. Me in 2017:
   !
   § OpenShift Templates
   § Helm
   § Self-made? Jinja2?
   § Even thinking of using Ansible…
   

   View Slide

9. Me in 2018:
   !
   § Kustomize
   

   View Slide

10. What are the options?
    

    View Slide

11. Ansible
    Forge
    Helm
    K8comp
    KPM
    KY
    Kapitan
    Kdeploy
    Kedge
    Kenv
    Kexpand
    Kit-Deploymentizer
    Kompose
    Konfd
    Kontemplate
    Ksonnet
    Ktmpl
    Kubecfg
    Kubegen
    Kubernetes-deploy
    Kubetpl
    Kustomize
    Mortar
    OpenShift
    templates
    Psykube
    Spread
    Terraform
    …
    

    View Slide

12. Why Kustomize?
    

    View Slide

13. Reason #1: Embrace Kubernetes API Object Descriptions (YAML files)
    

    View Slide

14. kind: Service
    apiVersion: v1
    metadata:
    name: my-service
    spec:
    selector:
    app: MyApp
    ports:
    - protocol: TCP
    port: 80
    targetPort: 9376
    my-service.yaml:
    

    View Slide

15. local params = std.extVar("__ksonnet/params").components.demo;
    local k = import "k.libsonnet";
    local service = k.core.v1.service;
    local servicePort = k.core.v1.service.mixin.spec.portsType;
    local targetPort = params.containerPort;
    local labels = {app: params.name};
    local appService = service
    .new(
    params.name,
    labels,
    servicePort.new(params.servicePort, targetPort))
    .withType(params.type);
    k.core.v1.list.new([appService])
    Ksonnet:
    

    View Slide

16. keep your YAML files the way they are
    

    View Slide

17. Reason #2: Keep using kubectl
    

    View Slide

18. $ kustomize build . | kubectl apply -f -
    Typical workflow:
    

    View Slide

19. Helm:
    § forget about kubectl, now you need to always use “helm install”, “helm
    ls”, “helm status”
    

    View Slide

20. Reason #3: Declarative and Templates-free
    

    View Slide

21. YAML can be painful sometimes
    

    View Slide

22. View Slide

23. Templated YAML is much worse…
    

    View Slide

24. apiVersion: v1
    kind: Service
    metadata:
    name: {{ template "grafana.fullname" . }}
    labels:
    app: {{ template "grafana.name" . }}
    chart: {{ template "grafana.chart" . }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
    {{- if .Values.service.labels }}
    {{ toYaml .Values.service.labels | indent 4 }}
    {{- end }}
    {{- with .Values.service.annotations }}
    annotations:
    {{ toYaml . | indent 4 }}
    {{- end }}
    spec:
    {{- if (or (eq .Values.service.type "ClusterIP") (empty .Values.service.type)) }}
    type: ClusterIP
    {{- if .Values.service.clusterIP }}
    clusterIP: {{ .Values.service.clusterIP }}
    {{end}}
    {{- else if eq .Values.service.type "LoadBalancer" }}
    …
    

    View Slide

25. Kustomize’s cure:
    § No templating at all!
    

    View Slide

26. Kustomize’s cure:
    § Overlays
    § Transformations
    § Generators
    § Patches
    

    View Slide

27. Overlays
    

    View Slide

28. Overlays
    myApp
    |
    ├── base
    │ ├── deployment.yaml
    │ ├── kustomization.yaml
    │ └── service.yaml
    |
    ├── development
    │ ├── ingress.yaml
    │ └── kustomization.yaml
    |
    └── production
    ├── ingress.yaml
    └── kustomization.yaml
    resources:
    - deployment.yaml
    - service.yaml
    

    View Slide

29. Overlays
    myApp
    |
    ├── base
    │ ├── deployment.yaml
    │ ├── kustomization.yaml
    │ └── service.yaml
    |
    ├── development
    │ ├── ingress.yaml
    │ └── kustomization.yaml
    |
    └── production
    ├── ingress.yaml
    └── kustomization.yaml
    bases:
    - ../base
    resources:
    - ingress.yaml
    bases:
    - ../base
    resources:
    - ingress.yaml
    

    View Slide

30. $ kustomize build development
    apiVersion: v1
    kind: Service
    metadata:
    …
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    …
    ---
    apiVersion: apps/v1
    kind: Ingress
    metadata:
    …
    

    View Slide

31. Transformations
    

    View Slide

32. Transformations
    myApp|
    ├── base
    │ ├── deployment.yaml
    │ └── kustomization.yaml
    |
    ├── development
    │ └── kustomization.yaml
    bases:
    - ../base
    namePrefix: dev-
    § All resource names are now prefixed with “dev-”
    

    View Slide

33. Transformations
    It’s what makes kustomize so powerful:
    § Because it knows Kubernetes semantics
    § A single line, use-case specific (e.g. namePrefix) causes big changes
    § All references are preserved
    

    View Slide

34. Generators
    kustomization.yaml:
    configMapGenerator:
    - name: myconfig
    files:
    - configs/configfile
    - configs/another_configfile
    § generates: myconfig-b62k6t7g8f (and fixes all references to it)
    § b62k6t7g8f is a hash of the contents
    

    View Slide

35. Generators
    kustomization.yaml:
    configMapGenerator:
    - name: myconfig
    files:
    - configs/configfile
    - configs/another_configfile
    § generates: myconfig-b62k6t7g8f (and fixes all references to it)
    § b62k6t7g8f is a hash of the contents !!!
    

    View Slide

36. Patches
    

    View Slide

37. Patches
    myApp|
    ├── base
    │ ├── deployment.yaml
    │ └── kustomization.yaml
    |
    ├── development
    | ├── deployment.patch.yaml
    │ └── kustomization.yaml bases:
    - ../base
    patches:
    - deployment.patch.yaml
    

    View Slide

38. Patches
    myApp|
    ├── base
    │ ├── deployment.yaml
    │ └── kustomization.yaml
    |
    ├── development
    | ├── deployment.patch.yaml
    │ └── kustomization.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
    name: myapp
    spec:
    replicas: 1
    

    View Slide

39. Patches
    Same syntax as “kubectl patch”
    § Strategic merge patches
    § JSON patches (RFC 6902)
    

    View Slide

40. Challenges
    

    View Slide

41. Challenges
    § Documentation is (currently) not super great
    

    View Slide

42. Challenges
    § Things that Kustomize doesn’t know about
    § OpenShift objects
    § CRDs
    

    View Slide

43. Challenges
    § Things that Kustomize doesn’t know about
    § OpenShift objects
    § CRDs
    It is now possible to extend Kustomize knowledge about Kubernetes
    objects
    see also: https://github.com/adnovum/kustomize-openshift
    

    View Slide

44. Summary
    When to use kustomize
    § It’s the perfect tool to parametrize your own application
    When not to use kustomize
    § Packaging an application for the general public (use Helm for that)
    

    View Slide

45. Questions?
    More about this topic:
    § Declarative application management in Kubernetes
    August 2017, by Brian Grant
    § Introducing kustomize; Template-free Configuration Customization for
    Kubernetes
    May 2018, by Jeff Regan and Phil Wittrock
    Contacting me:
    § david@schweikert.ch, @dschweikert
    

    View Slide