Developer Tooling for Kubernetes Configuration

Developer Tooling for Kubernetes Configuration

Talk from Configuration Management Camp, a rerun and rebranding of the talk I gave at KubeCon. All about testing and linting tools for Kubernetes configs, taking inspiration from the various configuration management tools.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

February 05, 2018
Tweet

Transcript

  1. Developer tooling for Kubernetes configuration Gareth Rushgrove

  2. @garethr

  3. None
  4. - Infrastructure as code - Validation, linting and unit testing

    - Demos and examples
  5. Testing infrastructure as code A bit of useful history

  6. Infrastructure as code is the process of managing and provisioning

    computers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools https://en.wikipedia.org/wiki/Infrastructure_as_Code
  7. rspec-puppet - originally written in 2011

  8. Examples span communities ChefSpec, Puppet Lint, Puppet Syntax, Food Critic,

    Cookstyle
  9. Why test a declarative configuration?

  10. Increasingly configuration: - Contains logic - Takes arguments - Interfaces

    with other configuration - Spans multiple versions of software
  11. Those points from a Puppet talk from 4 years ago

  12. As the community adopts higher-level tools like Helm, ksonnet, Compose,

    Kedge, Kapitan, etc. this becomes more relevant to Kubernetes users
  13. Take bitnami/kube-manifests as an example. It contains 7000+ lines of

    JSON in 159 files generated from 2000 lines of Jsonnet.
  14. Sidenote Brian Grant from Google found 43 tools for managing

    K8S configurations
  15. We’re now tracking more than 60

  16. Relevant discussion in the App Def Working Group

  17. I posit that the lessons learnt applying testing practices to

    infrastructure as code apply to Kubernetes configs
  18. Widely adopted tools break down into - Validation - Linting

    - Unit testing - Acceptance testing
  19. What would tools to address these problems look like for

    Kubernetes?
  20. Validation Using the type schemas when writing configs

  21. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis

    role: master tier: backend spec: ports: - port: 6379 targetPort: "6379" selector: app: redis role: master1 tier: backend Is this a valid Kubernetes configuration file?
  22. Is this a valid Kubernetes configuration file? apiVersion: v1 kind:

    ReplicationController spec: replicas: none selector: app: nginx loadbalancer: lb-1 templates: name: nginx labels: backend app: nginx spec: containers: - name: nginx image: nginx
  23. Is this Helm template valid for Kubernetes? apiVersion: v1 kind:

    Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
  24. Is this Puppet code valid for Kubernetes? kubernetes_pod { 'sample-pod':

    ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
  25. Kubernetes has a well-defined set of API primitives; Pods, Deployments,

    Services, ReplicationControllers, etc.
  26. Kubernetes uses OpenAPI to describe the API

  27. OpenAPI uses JSON Schema internally

  28. Kubernetes JSON Schema

  29. That’s a lot of JSON PS> Get-Content -Path swagger.json |

    Measure-Object -line).Lines 85340 PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count 26181 PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines 7296392
  30. OpenShift JSON Schema

  31. Validate directly using the schemas $ jsonschema -F "{error.message}" -i

    hello-nginx.json 1.6.6-standalone/deployment.json u'template' is a required property
  32. Validation is useful for - Catching typos - Fast feedback

    - Missing properties - Checking against multiple K8 versions
  33. Introducing Kubeval

  34. A nice CLI for validating K8S configurations $ kubeval --help

    Validate a Kubernetes YAML file against the relevant schema Usage: kubeval <file> [file...] [flags] Flags: -h, --help help for kubeval -v, --kubernetes-version string Version of Kubernetes to validate against --openshift Use OpenShift schemas instead of upstream Kubernetes --schema-location string Base URL used to download schemas. Can also be specified with the environment variable KUBEVAL_SCHEMA_LOCATION --version Display the kubeval version information and exit
  35. Validate YAML files on the command line $ kubeval my-invalid-rc.yaml

    The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1 $ cat my-invalid-rc.yaml | kubeval The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1
  36. Validate multiple resources in the same file $ kubeval multi.yaml

    The document fixtures/multi.yaml contains a valid Service The document fixtures/multi.yaml contains an invalid Deployment --> spec.template.spec.containers.0.env.0.value: Invalid type. Expected: string, given: integer The document fixtures/multi.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string The document fixtures/multi.json contains a valid Deployment The document fixtures/multi.yaml contains a valid ReplicationController
  37. Validate against multiple versions of Kubernetes $ kubeval --v 1.7.9

    my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ kubeval --v 1.8.1 my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string
  38. Use as a library in other Go tools import (

    "github.com/garethr/kubeval/kubeval" ) results, err := kubeval.Validate(fileContents, fileName)
  39. Unit testing Custom business rules for configuration

  40. Validation says something is valid, not that it’s what you

    intended
  41. Lots of teams have a script to check certain properties

    of their K8S configurations against internal policies
  42. Our internal tooling includes a linter that renders helm charts

    and validates the resources produced pass certain internal rules.
  43. - Podspec needs resource requests - Ensure certain labels are

    set - Prevent usage of latest images - Prohibit privileged containers - Enforce naming conventions
  44. Introducing kubetest

  45. Run tests against your configurations $ kubetest rc.yaml --verbose INFO

    rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
  46. Tests are written in Skylark

  47. Skylark is a dialect of Python. It is an untyped

    dynamic language with high-level data types, first-class functions with lexical scope, and garbage collection
  48. A Skylark interpreter is typically embedded within an application which

    may define additional domain-specific functions and data types beyond those provided by the core language
  49. Example tests for kubetest #// vim: set ft=python: def test_for_latest_image():

    if spec["kind"] == "ReplicationController": for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", "should not use latest images") def test_minimum_replicas(): if spec["kind"] == "ReplicationController": test = spec["spec"]["replicas"] >= 4 assert_true(test, "ReplicationController should have at least 4 replicas") test_for_latest_image() test_minimum_replicas()
  50. Tests enforcing a team label #// vim: set ft=python: def

    test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
  51. Linting Building common community assertions

  52. Organisation-specific assertions are useful, but require you to write the

    tests yourself
  53. Many assertions are common, they are the result of emerging

    community best-practice
  54. I’d like to build an out-of-the-box experience for kubetest

  55. Integration with K8 tools Demos and examples

  56. Using kubeval with Helm $ git clone https://github.com/kubernetes/helm.git $ cd

    helm/docs/examples $ ls alpine nginx README.md $ helm template nginx | kubeval The document stdin contains a valid Secret The document stdin contains a valid ConfigMap The document stdin contains a valid Service The document stdin contains a valid Pod The document stdin contains a valid Deployment The document stdin contains a valid Job
  57. Tests for our Helm Chart #// vim: set ft=python: def

    test_for_latest_image(): if spec["kind"] in ["Job", "Deployment"]: for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", spec["kind"] + " should not use latest images") test_for_latest_image()
  58. Using kubetest with Helm $ helm template nginx | kubetest

    --verbose INFO stdin Deployment should not use latest images INFO stdin Job should not use latest images
  59. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
  60. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"}) $ jsonnet deployment.jsonnet | kubeval The document stdin contains a valid Deployment
  61. Using kubetest with ksonnet $ jsonnet deployment.jsonnet | kubetest --verbose

    INFO stdin should not use latest images WARN stdin ReplicationController should have at least 7 replicas
  62. Using kubeval with Puppet $ cat guestbook.pp kubernetes_service { 'frontend':

    ensure => 'present', metadata => { 'labels' => {'app' => 'guestbook', 'tier' => 'frontend'}, 'namespace' => 'default', }, spec => { 'type' => 'LoadBalancer', 'ports' => [ {'port' => 80, 'protocol' => 'TCP'} ], 'selector' => { 'app' => 'guestbook', 'tier' => 'frontend' }, },
  63. Using kubeval with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubeval The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service
  64. Using kubetest with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubetest --verbose INFO stdin should not use latest images INFO stdin should not use latest images INFO stdin should not use latest images
  65. kubeval is a great tool to validate your Kubernetes configuration

    files as part of your CI pipeline
  66. Using kubeval and kubetest in CI

  67. Conclusions If all you remember is...

  68. The community is still exploring ways to describe Kubernetes configuration

    in code
  69. The Configuration Complexity Clock is real

  70. Some of the current tools lend themselves to native testing,

    others are just data and templates
  71. Having testing tools that work with different configuration approaches can

    make moving between tools easier
  72. Lots more interesting opportunities around testing Kubernetes configs

  73. And lots of inspiration we can take from other communities

    and tools
  74. Any questions? And thanks for listening