Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Developer Tooling for Kubernetes Configuration

Developer Tooling for Kubernetes Configuration

Talk from Configuration Management Camp, a rerun and rebranding of the talk I gave at KubeCon. All about testing and linting tools for Kubernetes configs, taking inspiration from the various configuration management tools.

Gareth Rushgrove

February 05, 2018
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Infrastructure as code is the process of managing and provisioning

    computers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools https://en.wikipedia.org/wiki/Infrastructure_as_Code
  2. Increasingly configuration: - Contains logic - Takes arguments - Interfaces

    with other configuration - Spans multiple versions of software
  3. As the community adopts higher-level tools like Helm, ksonnet, Compose,

    Kedge, Kapitan, etc. this becomes more relevant to Kubernetes users
  4. Take bitnami/kube-manifests as an example. It contains 7000+ lines of

    JSON in 159 files generated from 2000 lines of Jsonnet.
  5. I posit that the lessons learnt applying testing practices to

    infrastructure as code apply to Kubernetes configs
  6. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis

    role: master tier: backend spec: ports: - port: 6379 targetPort: "6379" selector: app: redis role: master1 tier: backend Is this a valid Kubernetes configuration file?
  7. Is this a valid Kubernetes configuration file? apiVersion: v1 kind:

    ReplicationController spec: replicas: none selector: app: nginx loadbalancer: lb-1 templates: name: nginx labels: backend app: nginx spec: containers: - name: nginx image: nginx
  8. Is this Helm template valid for Kubernetes? apiVersion: v1 kind:

    Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
  9. Is this Puppet code valid for Kubernetes? kubernetes_pod { 'sample-pod':

    ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
  10. That’s a lot of JSON PS> Get-Content -Path swagger.json |

    Measure-Object -line).Lines 85340 PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count 26181 PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines 7296392
  11. Validate directly using the schemas $ jsonschema -F "{error.message}" -i

    hello-nginx.json 1.6.6-standalone/deployment.json u'template' is a required property
  12. Validation is useful for - Catching typos - Fast feedback

    - Missing properties - Checking against multiple K8 versions
  13. A nice CLI for validating K8S configurations $ kubeval --help

    Validate a Kubernetes YAML file against the relevant schema Usage: kubeval <file> [file...] [flags] Flags: -h, --help help for kubeval -v, --kubernetes-version string Version of Kubernetes to validate against --openshift Use OpenShift schemas instead of upstream Kubernetes --schema-location string Base URL used to download schemas. Can also be specified with the environment variable KUBEVAL_SCHEMA_LOCATION --version Display the kubeval version information and exit
  14. Validate YAML files on the command line $ kubeval my-invalid-rc.yaml

    The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1 $ cat my-invalid-rc.yaml | kubeval The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1
  15. Validate multiple resources in the same file $ kubeval multi.yaml

    The document fixtures/multi.yaml contains a valid Service The document fixtures/multi.yaml contains an invalid Deployment --> spec.template.spec.containers.0.env.0.value: Invalid type. Expected: string, given: integer The document fixtures/multi.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string The document fixtures/multi.json contains a valid Deployment The document fixtures/multi.yaml contains a valid ReplicationController
  16. Validate against multiple versions of Kubernetes $ kubeval --v 1.7.9

    my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ kubeval --v 1.8.1 my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string
  17. Use as a library in other Go tools import (

    "github.com/garethr/kubeval/kubeval" ) results, err := kubeval.Validate(fileContents, fileName)
  18. Lots of teams have a script to check certain properties

    of their K8S configurations against internal policies
  19. Our internal tooling includes a linter that renders helm charts

    and validates the resources produced pass certain internal rules.
  20. - Podspec needs resource requests - Ensure certain labels are

    set - Prevent usage of latest images - Prohibit privileged containers - Enforce naming conventions
  21. Run tests against your configurations $ kubetest rc.yaml --verbose INFO

    rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
  22. Skylark is a dialect of Python. It is an untyped

    dynamic language with high-level data types, first-class functions with lexical scope, and garbage collection
  23. A Skylark interpreter is typically embedded within an application which

    may define additional domain-specific functions and data types beyond those provided by the core language
  24. Example tests for kubetest #// vim: set ft=python: def test_for_latest_image():

    if spec["kind"] == "ReplicationController": for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", "should not use latest images") def test_minimum_replicas(): if spec["kind"] == "ReplicationController": test = spec["spec"]["replicas"] >= 4 assert_true(test, "ReplicationController should have at least 4 replicas") test_for_latest_image() test_minimum_replicas()
  25. Tests enforcing a team label #// vim: set ft=python: def

    test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
  26. Using kubeval with Helm $ git clone https://github.com/kubernetes/helm.git $ cd

    helm/docs/examples $ ls alpine nginx README.md $ helm template nginx | kubeval The document stdin contains a valid Secret The document stdin contains a valid ConfigMap The document stdin contains a valid Service The document stdin contains a valid Pod The document stdin contains a valid Deployment The document stdin contains a valid Job
  27. Tests for our Helm Chart #// vim: set ft=python: def

    test_for_latest_image(): if spec["kind"] in ["Job", "Deployment"]: for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", spec["kind"] + " should not use latest images") test_for_latest_image()
  28. Using kubetest with Helm $ helm template nginx | kubetest

    --verbose INFO stdin Deployment should not use latest images INFO stdin Job should not use latest images
  29. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
  30. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"}) $ jsonnet deployment.jsonnet | kubeval The document stdin contains a valid Deployment
  31. Using kubetest with ksonnet $ jsonnet deployment.jsonnet | kubetest --verbose

    INFO stdin should not use latest images WARN stdin ReplicationController should have at least 7 replicas
  32. Using kubeval with Puppet $ cat guestbook.pp kubernetes_service { 'frontend':

    ensure => 'present', metadata => { 'labels' => {'app' => 'guestbook', 'tier' => 'frontend'}, 'namespace' => 'default', }, spec => { 'type' => 'LoadBalancer', 'ports' => [ {'port' => 80, 'protocol' => 'TCP'} ], 'selector' => { 'app' => 'guestbook', 'tier' => 'frontend' }, },
  33. Using kubeval with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubeval The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service
  34. Using kubetest with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubetest --verbose INFO stdin should not use latest images INFO stdin should not use latest images INFO stdin should not use latest images