Developer tooling for Kubernetes configurations

Developer tooling for Kubernetes configurations

Talk from KubeCon in Austin, about learning from other programming environments and using tools to validate and test your configurations, whether you're writing them by hand or using higher level tools like ksonnet and Helm.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

December 07, 2017
Tweet

Transcript

  1. Gareth Rushgrove Developer tooling for Kubernetes configuration Tools for developer-friendly

    workflows
  2. None
  3. @garethr

  4. - Infrastructure as code - Validation, linting and unit testing

    - Demos and examples
  5. Testing infrastructure as code A bit of useful history

  6. Infrastructure as code is the process of managing and provisioning

    computers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools https://en.wikipedia.org/wiki/Infrastructure_as_Code
  7. rspec-puppet - originally written in 2011

  8. Examples span communities ChefSpec, Puppet Lint, Puppet Syntax, Food Critic,

    Cookstyle
  9. Why test a declarative configuration?

  10. Increasingly configuration: - Contains logic - Takes arguments - Interfaces

    with other configuration - Spans multiple versions of software
  11. Those points from a Puppet talk from 4 years ago

  12. As the community adopts higher-level tools like Helm, ksonnet, Compose,

    Kedge, Kapitan, etc. this becomes more relevant to Kubernetes users
  13. Take bitnami/kube-manifests as an example. It contains 7000+ lines of

    JSON in 159 files generated from 2000 lines of Jsonnet.
  14. Sidenote Brian Grant found 43 higher-levels tools for managing K8S

    configurations
  15. Relevant discussion in the App Def Working Group

  16. I posit that the lessons learnt applying testing practices to

    infrastructure as code apply to Kubernetes configs
  17. Widely adopted tools break down into - Validation - Linting

    - Unit testing - Acceptance testing
  18. What would tools to address these problems look like for

    Kubernetes?
  19. Validation Using the type schemas when writing configs

  20. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis

    role: master tier: backend spec: ports: - port: 6379 targetPort: "6379" selector: app: redis role: master1 tier: backend Is this a valid Kubernetes configuration file?
  21. Is this a valid Kubernetes configuration file? apiVersion: v1 kind:

    ReplicationController spec: replicas: none selector: app: nginx loadbalancer: lb-1 templates: name: nginx labels: backend app: nginx spec: containers: - name: nginx image: nginx
  22. Is this Helm template valid for Kubernetes? apiVersion: v1 kind:

    Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
  23. Is this Puppet code valid for Kubernetes? kubernetes_pod { 'sample-pod':

    ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
  24. Kubernetes has a well-defined set of API primitives; Pods, Deployments,

    Services, ReplicationControllers, etc.
  25. Kubernetes uses OpenAPI to describe the API

  26. OpenAPI uses JSON Schema internally

  27. Kubernetes JSON Schema

  28. That’s a lot of JSON PS> Get-Content -Path swagger.json |

    Measure-Object -line).Lines 85340 PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count 26181 PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines 7296392
  29. OpenShift JSON Schema

  30. Validate directly using the schemas $ jsonschema -F "{error.message}" -i

    hello-nginx.json 1.6.6-standalone/deployment.json u'template' is a required property
  31. Validation is useful for - Catching typos - Fast feedback

    - Missing properties - Checking against multiple K8 versions
  32. Introducing Kubeval

  33. A nice CLI for validating K8S configurations $ kubeval --help

    Validate a Kubernetes YAML file against the relevant schema Usage: kubeval <file> [file...] [flags] Flags: -h, --help help for kubeval -v, --kubernetes-version string Version of Kubernetes to validate against --openshift Use OpenShift schemas instead of upstream Kubernetes --schema-location string Base URL used to download schemas. Can also be specified with the environment variable KUBEVAL_SCHEMA_LOCATION --version Display the kubeval version information and exit
  34. Validate YAML files on the command line $ kubeval my-invalid-rc.yaml

    The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1 $ cat my-invalid-rc.yaml | kubeval The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1
  35. Validate multiple resources in the same file $ kubeval multi.yaml

    The document fixtures/multi.yaml contains a valid Service The document fixtures/multi.yaml contains an invalid Deployment --> spec.template.spec.containers.0.env.0.value: Invalid type. Expected: string, given: integer The document fixtures/multi.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string The document fixtures/multi.json contains a valid Deployment The document fixtures/multi.yaml contains a valid ReplicationController
  36. Validate against multiple versions of Kubernetes $ kubeval --v 1.7.9

    my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ kubeval --v 1.8.1 my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string
  37. Use as a library in other Go tools import (

    "github.com/garethr/kubeval/kubeval" ) results, err := kubeval.Validate(fileContents, fileName)
  38. Unit testing Custom business rules for configuration

  39. Validation says something is valid, not that it’s what you

    intended
  40. Lots of teams have a script to check certain properties

    of their K8S configurations against internal policies
  41. Our internal tooling includes a linter that renders helm charts

    and validates the resources produced pass certain internal rules.
  42. - Podspec needs resource requests - Ensure certain labels are

    set - Prevent usage of latest images - Prohibit privileged containers - Enforce naming conventions
  43. Introducing kubetest

  44. Run tests against your configurations $ kubetest rc.yaml --verbose INFO

    rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
  45. Tests are written in Skylark

  46. Skylark is a dialect of Python. It is an untyped

    dynamic language with high-level data types, first-class functions with lexical scope, and garbage collection
  47. A Skylark interpreter is typically embedded within an application which

    may define additional domain-specific functions and data types beyond those provided by the core language
  48. Example tests for kubetest #// vim: set ft=python: def test_for_latest_image():

    if spec["kind"] == "ReplicationController": for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", "should not use latest images") def test_minimum_replicas(): if spec["kind"] == "ReplicationController": test = spec["spec"]["replicas"] >= 4 assert_true(test, "ReplicationController should have at least 4 replicas") test_for_latest_image() test_minimum_replicas()
  49. Tests enforcing a team label #// vim: set ft=python: def

    test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
  50. Linting Building common community assertions

  51. Organisation-specific assertions are useful, but require you to write the

    tests yourself
  52. Many assertions are common, they are the result of emerging

    community best-practice
  53. I’d like to build an out-of-the-box experience for kubetest

  54. Integration with K8 tools Demos and examples

  55. Using kubeval with Helm $ git clone https://github.com/kubernetes/helm.git $ cd

    helm/docs/examples $ ls alpine nginx README.md $ helm template nginx | kubeval The document stdin contains a valid Secret The document stdin contains a valid ConfigMap The document stdin contains a valid Service The document stdin contains a valid Pod The document stdin contains a valid Deployment The document stdin contains a valid Job
  56. Tests for our Helm Chart #// vim: set ft=python: def

    test_for_latest_image(): if spec["kind"] in ["Job", "Deployment"]: for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", spec["kind"] + " should not use latest images") test_for_latest_image()
  57. Using kubetest with Helm $ helm template nginx | kubetest

    --verbose INFO stdin Deployment should not use latest images INFO stdin Job should not use latest images
  58. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
  59. Using kubeval with ksonnet $ cat deployment.jsonnet local k =

    import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"}) $ jsonnet deployment.jsonnet | kubeval The document stdin contains a valid Deployment
  60. Using kubetest with ksonnet $ jsonnet deployment.jsonnet | kubetest --verbose

    INFO stdin should not use latest images WARN stdin ReplicationController should have at least 7 replicas
  61. Using kubeval with Puppet $ cat guestbook.pp kubernetes_service { 'frontend':

    ensure => 'present', metadata => { 'labels' => {'app' => 'guestbook', 'tier' => 'frontend'}, 'namespace' => 'default', }, spec => { 'type' => 'LoadBalancer', 'ports' => [ {'port' => 80, 'protocol' => 'TCP'} ], 'selector' => { 'app' => 'guestbook', 'tier' => 'frontend' }, },
  62. Using kubeval with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubeval The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service
  63. Using kubetest with Puppet $ puppet kubernetes convert --manifest guestbook.pp

    | kubetest --verbose INFO stdin should not use latest images INFO stdin should not use latest images INFO stdin should not use latest images
  64. kubeval is a great tool to validate your Kubernetes configuration

    files as part of your CI pipeline
  65. Using kubeval and kubetest in CI

  66. Conclusions If all you remember is...

  67. The community is still exploring ways to describe Kubernetes configuration

    in code
  68. The Configuration Complexity Clock is real

  69. Some of the current tools lend themselves to native testing,

    others are just data and templates
  70. Having testing tools that work with different configuration approaches can

    make moving between tools easier
  71. Lots more interesting opportunities around testing Kubernetes configs

  72. And lots of inspiration we can take from other communities

    and tools
  73. Any questions? And thanks for listening