Developer Tooling for Kubernetes Configuration

Talk from Configuration Management Camp, a rerun and rebranding of the talk I gave at KubeCon. All about testing and linting tools for Kubernetes configs, taking inspiration from the various configuration management tools.

1. Developer tooling for Kubernetes configuration Gareth Rushgrove
2. @garethr
4. - Infrastructure as code - Validation, linting and unit testing - Demos and examples
5. Testing infrastructure as code A bit of useful history
6. Infrastructure as code is the process of managing and provisioning computers through machine-readable definition files, rather than physical hardware configuration or interactive configuration tools https://en.wikipedia.org/wiki/Infrastructure_as_Code
7. rspec-puppet - originally written in 2011
8. Examples span communities ChefSpec, Puppet Lint, Puppet Syntax, Food Critic, Cookstyle
9. Why test a declarative configuration?
10. Increasingly configuration: - Contains logic - Takes arguments - Interfaces with other configuration - Spans multiple versions of software
11. Those points from a Puppet talk from 4 years ago
12. As the community adopts higher-level tools like Helm, ksonnet, Compose, Kedge, Kapitan, etc. this becomes more relevant to Kubernetes users
13. Take bitnami/kube-manifests as an example. It contains 7000+ lines of JSON in 159 files generated from 2000 lines of Jsonnet.
14. Sidenote Brian Grant from Google found 43 tools for managing K8S configurations
15. We’re now tracking more than 60
16. Relevant discussion in the App Def Working Group
17. I posit that the lessons learnt applying testing practices to infrastructure as code apply to Kubernetes configs
18. Widely adopted tools break down into - Validation - Linting - Unit testing - Acceptance testing
19. What would tools to address these problems look like for Kubernetes?
20. Validation Using the type schemas when writing configs
21. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis role: master tier: backend spec: ports: - port: 6379 targetPort: "6379" selector: app: redis role: master1 tier: backend Is this a valid Kubernetes configuration file?
22. Is this a valid Kubernetes configuration file? apiVersion: v1 kind: ReplicationController spec: replicas: none selector: app: nginx loadbalancer: lb-1 templates: name: nginx labels: backend app: nginx spec: containers: - name: nginx image: nginx
23. Is this Helm template valid for Kubernetes? apiVersion: v1 kind: Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
24. Is this Puppet code valid for Kubernetes? kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, }
25. Kubernetes has a well-defined set of API primitives; Pods, Deployments, Services, ReplicationControllers, etc.
26. Kubernetes uses OpenAPI to describe the API
27. OpenAPI uses JSON Schema internally
28. Kubernetes JSON Schema
29. That’s a lot of JSON PS> Get-Content -Path swagger.json | Measure-Object -line).Lines 85340 PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count 26181 PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines 7296392
30. OpenShift JSON Schema
31. Validate directly using the schemas $ jsonschema -F "{error.message}" -i hello-nginx.json 1.6.6-standalone/deployment.json u'template' is a required property
32. Validation is useful for - Catching typos - Fast feedback - Missing properties - Checking against multiple K8 versions
33. Introducing Kubeval
34. A nice CLI for validating K8S configurations $ kubeval --help Validate a Kubernetes YAML file against the relevant schema Usage: kubeval <file> [file...] [flags] Flags: -h, --help help for kubeval -v, --kubernetes-version string Version of Kubernetes to validate against --openshift Use OpenShift schemas instead of upstream Kubernetes --schema-location string Base URL used to download schemas. Can also be specified with the environment variable KUBEVAL_SCHEMA_LOCATION --version Display the kubeval version information and exit
35. Validate YAML files on the command line $ kubeval my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1 $ cat my-invalid-rc.yaml | kubeval The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ echo $? 1
36. Validate multiple resources in the same file $ kubeval multi.yaml The document fixtures/multi.yaml contains a valid Service The document fixtures/multi.yaml contains an invalid Deployment --> spec.template.spec.containers.0.env.0.value: Invalid type. Expected: string, given: integer The document fixtures/multi.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string The document fixtures/multi.json contains a valid Deployment The document fixtures/multi.yaml contains a valid ReplicationController
37. Validate against multiple versions of Kubernetes $ kubeval --v 1.7.9 my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string $ kubeval --v 1.8.1 my-invalid-rc.yaml The document my-invalid-rc.yaml contains an invalid ReplicationController --> spec.replicas: Invalid type. Expected: integer, given: string
38. Use as a library in other Go tools import ( "github.com/garethr/kubeval/kubeval" ) results, err := kubeval.Validate(fileContents, fileName)
39. Unit testing Custom business rules for configuration
40. Validation says something is valid, not that it’s what you intended
41. Lots of teams have a script to check certain properties of their K8S configurations against internal policies
42. Our internal tooling includes a linter that renders helm charts and validates the resources produced pass certain internal rules.
43. - Podspec needs resource requests - Ensure certain labels are set - Prevent usage of latest images - Prohibit privileged containers - Enforce naming conventions
44. Introducing kubetest
45. Run tests against your configurations $ kubetest rc.yaml --verbose INFO rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
46. Tests are written in Skylark
47. Skylark is a dialect of Python. It is an untyped dynamic language with high-level data types, first-class functions with lexical scope, and garbage collection
48. A Skylark interpreter is typically embedded within an application which may define additional domain-specific functions and data types beyond those provided by the core language
49. Example tests for kubetest #// vim: set ft=python: def test_for_latest_image(): if spec["kind"] == "ReplicationController": for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", "should not use latest images") def test_minimum_replicas(): if spec["kind"] == "ReplicationController": test = spec["spec"]["replicas"] >= 4 assert_true(test, "ReplicationController should have at least 4 replicas") test_for_latest_image() test_minimum_replicas()
50. Tests enforcing a team label #// vim: set ft=python: def test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
51. Linting Building common community assertions
52. Organisation-specific assertions are useful, but require you to write the tests yourself
53. Many assertions are common, they are the result of emerging community best-practice
54. I’d like to build an out-of-the-box experience for kubetest
55. Integration with K8 tools Demos and examples
56. Using kubeval with Helm $ git clone https://github.com/kubernetes/helm.git $ cd helm/docs/examples $ ls alpine nginx README.md $ helm template nginx | kubeval The document stdin contains a valid Secret The document stdin contains a valid ConfigMap The document stdin contains a valid Service The document stdin contains a valid Pod The document stdin contains a valid Deployment The document stdin contains a valid Job
57. Tests for our Helm Chart #// vim: set ft=python: def test_for_latest_image(): if spec["kind"] in ["Job", "Deployment"]: for container in spec["spec"]["template"]["spec"]["containers"]: tag = container["image"].split(":")[-1] assert_not_equal(tag, "latest", spec["kind"] + " should not use latest images") test_for_latest_image()
58. Using kubetest with Helm $ helm template nginx | kubetest --verbose INFO stdin Deployment should not use latest images INFO stdin Job should not use latest images
59. Using kubeval with ksonnet $ cat deployment.jsonnet local k = import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
60. Using kubeval with ksonnet $ cat deployment.jsonnet local k = import "ksonnet.beta.2/k.libsonnet"; local deployment = k.apps.v1beta1.deployment; local container = deployment.mixin.spec.template.spec.containersType; local containerPort = container.portsType; // Create nginx container with container port 80 open. local nginxContainer = container.new("nginx", "nginx:1.13.0") + container.ports(containerPort.newNamed("http", 80)); // Create default Deployment object from nginx container. deployment.new("nginx", 5, nginxContainer, {app: "nginx"}) $ jsonnet deployment.jsonnet | kubeval The document stdin contains a valid Deployment
61. Using kubetest with ksonnet $ jsonnet deployment.jsonnet | kubetest --verbose INFO stdin should not use latest images WARN stdin ReplicationController should have at least 7 replicas
62. Using kubeval with Puppet $ cat guestbook.pp kubernetes_service { 'frontend': ensure => 'present', metadata => { 'labels' => {'app' => 'guestbook', 'tier' => 'frontend'}, 'namespace' => 'default', }, spec => { 'type' => 'LoadBalancer', 'ports' => [ {'port' => 80, 'protocol' => 'TCP'} ], 'selector' => { 'app' => 'guestbook', 'tier' => 'frontend' }, },
63. Using kubeval with Puppet $ puppet kubernetes convert --manifest guestbook.pp | kubeval The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service The document stdin contains a valid ReplicationController The document stdin contains a valid Service
64. Using kubetest with Puppet $ puppet kubernetes convert --manifest guestbook.pp | kubetest --verbose INFO stdin should not use latest images INFO stdin should not use latest images INFO stdin should not use latest images
65. kubeval is a great tool to validate your Kubernetes configuration files as part of your CI pipeline
66. Using kubeval and kubetest in CI
67. Conclusions If all you remember is...
68. The community is still exploring ways to describe Kubernetes configuration in code
69. The Configuration Complexity Clock is real
70. Some of the current tools lend themselves to native testing, others are just data and templates
71. Having testing tools that work with different configuration approaches can make moving between tools easier
72. Lots more interesting opportunities around testing Kubernetes configs
73. And lots of inspiration we can take from other communities and tools
74. Any questions? And thanks for listening