Slide 1
Slide 1 text
Gareth Rushgrove
Developer tooling for
Kubernetes configuration
Tools for developer-friendly workflows
Slide 2
Slide 2 text
No content
Slide 3
Slide 3 text
@garethr
Slide 4
Slide 4 text
- Infrastructure as code
- Validation, linting and unit testing
- Demos and examples
Slide 5
Slide 5 text
Testing infrastructure
as code
A bit of useful history
Slide 6
Slide 6 text
Infrastructure as code is the process
of managing and provisioning
computers through machine-readable
definition files, rather than physical
hardware configuration or interactive
configuration tools
https://en.wikipedia.org/wiki/Infrastructure_as_Code
Slide 7
Slide 7 text
rspec-puppet - originally written in 2011
Slide 8
Slide 8 text
Examples span communities
ChefSpec, Puppet Lint, Puppet Syntax,
Food Critic, Cookstyle
Slide 9
Slide 9 text
Why test a declarative configuration?
Slide 10
Slide 10 text
Increasingly configuration:
- Contains logic
- Takes arguments
- Interfaces with other configuration
- Spans multiple versions of software
Slide 11
Slide 11 text
Those points from a Puppet talk from 4 years ago
Slide 12
Slide 12 text
As the community adopts higher-level
tools like Helm, ksonnet, Compose,
Kedge, Kapitan, etc. this becomes
more relevant to Kubernetes users
Slide 13
Slide 13 text
Take bitnami/kube-manifests as
an example. It contains 7000+ lines
of JSON in 159 files generated from
2000 lines of Jsonnet.
Slide 14
Slide 14 text
Sidenote
Brian Grant found 43 higher-levels
tools for managing K8S configurations
Slide 15
Slide 15 text
Relevant discussion in the App Def Working Group
Slide 16
Slide 16 text
I posit that the lessons learnt applying
testing practices to infrastructure as
code apply to Kubernetes configs
Slide 17
Slide 17 text
Widely adopted tools break down into
- Validation
- Linting
- Unit testing
- Acceptance testing
Slide 18
Slide 18 text
What would tools to address these
problems look like for Kubernetes?
Slide 19
Slide 19 text
Validation
Using the type schemas when writing configs
Slide 20
Slide 20 text
apiVersion: v1
kind: Service
metadata:
name: redis-master
labels:
app: redis
role: master
tier: backend
spec:
ports:
- port: 6379
targetPort: "6379"
selector:
app: redis
role: master1
tier: backend
Is this a valid Kubernetes configuration file?
Slide 21
Slide 21 text
Is this a valid Kubernetes configuration file?
apiVersion: v1
kind: ReplicationController
spec:
replicas: none
selector:
app: nginx
loadbalancer: lb-1
templates:
name: nginx
labels: backend
app: nginx
spec:
containers:
- name: nginx
image: nginx
Slide 22
Slide 22 text
Is this Helm template valid for Kubernetes?
apiVersion: v1
kind: Service
metadata:
name: {{ template "fullname" . }}
labels:
app: {{ template "fullname" . }}
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: "{{ .Release.Name }}"
heritage: "{{ .Release.Service }}"
spec:
ports:
- name: memcache
port: 11211
targetPort: memcache
selector:
app: {{ template "fullname" . }}
Slide 23
Slide 23 text
Is this Puppet code valid for Kubernetes?
kubernetes_pod { 'sample-pod':
ensure => present,
metadata => {
namespace => 'default',
},
spec => {
containers => [{
name => 'container-name',
image => 'nginx',
}]
},
}
Slide 24
Slide 24 text
Kubernetes has a well-defined set of
API primitives; Pods, Deployments,
Services, ReplicationControllers, etc.
Slide 25
Slide 25 text
Kubernetes uses OpenAPI to describe the API
Slide 26
Slide 26 text
OpenAPI uses JSON Schema internally
Slide 27
Slide 27 text
Kubernetes JSON Schema
Slide 28
Slide 28 text
That’s a lot of JSON
PS> Get-Content -Path swagger.json | Measure-Object -line).Lines
85340
PS> (Get-ChildItem -Path v*/*.json -Recurse | Measure-Object).Count
26181
PS> (Get-ChildItem -Path v*/*.json -Recurse | Get-Content | Measure-Object -line).Lines
7296392
Slide 29
Slide 29 text
OpenShift JSON Schema
Slide 30
Slide 30 text
Validate directly using the schemas
$ jsonschema -F "{error.message}" -i hello-nginx.json 1.6.6-standalone/deployment.json
u'template' is a required property
Slide 31
Slide 31 text
Validation is useful for
- Catching typos
- Fast feedback
- Missing properties
- Checking against multiple K8 versions
Slide 32
Slide 32 text
Introducing Kubeval
Slide 33
Slide 33 text
A nice CLI for validating K8S configurations
$ kubeval --help
Validate a Kubernetes YAML file against the relevant schema
Usage:
kubeval [file...] [flags]
Flags:
-h, --help help for kubeval
-v, --kubernetes-version string Version of Kubernetes to validate against
--openshift Use OpenShift schemas instead of upstream Kubernetes
--schema-location string Base URL used to download schemas. Can also be
specified with the environment variable
KUBEVAL_SCHEMA_LOCATION
--version Display the kubeval version information and exit
Slide 34
Slide 34 text
Validate YAML files on the command line
$ kubeval my-invalid-rc.yaml
The document my-invalid-rc.yaml contains an invalid ReplicationController
--> spec.replicas: Invalid type. Expected: integer, given: string
$ echo $?
1
$ cat my-invalid-rc.yaml | kubeval
The document my-invalid-rc.yaml contains an invalid ReplicationController
--> spec.replicas: Invalid type. Expected: integer, given: string
$ echo $?
1
Slide 35
Slide 35 text
Validate multiple resources in the same file
$ kubeval multi.yaml
The document fixtures/multi.yaml contains a valid Service
The document fixtures/multi.yaml contains an invalid Deployment
--> spec.template.spec.containers.0.env.0.value: Invalid type. Expected: string, given: integer
The document fixtures/multi.yaml contains an invalid ReplicationController
--> spec.replicas: Invalid type. Expected: integer, given: string
The document fixtures/multi.json contains a valid Deployment
The document fixtures/multi.yaml contains a valid ReplicationController
Slide 36
Slide 36 text
Validate against multiple versions of Kubernetes
$ kubeval --v 1.7.9 my-invalid-rc.yaml
The document my-invalid-rc.yaml contains an invalid ReplicationController
--> spec.replicas: Invalid type. Expected: integer, given: string
$ kubeval --v 1.8.1 my-invalid-rc.yaml
The document my-invalid-rc.yaml contains an invalid ReplicationController
--> spec.replicas: Invalid type. Expected: integer, given: string
Slide 37
Slide 37 text
Use as a library in other Go tools
import (
"github.com/garethr/kubeval/kubeval"
)
results, err := kubeval.Validate(fileContents, fileName)
Slide 38
Slide 38 text
Unit testing
Custom business rules for configuration
Slide 39
Slide 39 text
Validation says something is valid,
not that it’s what you intended
Slide 40
Slide 40 text
Lots of teams have a script to
check certain properties of their
K8S configurations against
internal policies
Slide 41
Slide 41 text
Our internal tooling includes a
linter that renders helm charts and
validates the resources produced
pass certain internal rules.
Slide 42
Slide 42 text
- Podspec needs resource requests
- Ensure certain labels are set
- Prevent usage of latest images
- Prohibit privileged containers
- Enforce naming conventions
Slide 43
Slide 43 text
Introducing kubetest
Slide 44
Slide 44 text
Run tests against your configurations
$ kubetest rc.yaml --verbose
INFO rc.yaml should not use latest images
WARN rc.yaml ReplicationController should have at least 4 replicas
Slide 45
Slide 45 text
Tests are written in Skylark
Slide 46
Slide 46 text
Skylark is a dialect of Python. It is
an untyped dynamic language
with high-level data types,
first-class functions with lexical
scope, and garbage collection
Slide 47
Slide 47 text
A Skylark interpreter is typically
embedded within an application
which may define additional
domain-specific functions and
data types beyond those provided
by the core language
Slide 48
Slide 48 text
Example tests for kubetest
#// vim: set ft=python:
def test_for_latest_image():
if spec["kind"] == "ReplicationController":
for container in spec["spec"]["template"]["spec"]["containers"]:
tag = container["image"].split(":")[-1]
assert_not_equal(tag, "latest", "should not use latest images")
def test_minimum_replicas():
if spec["kind"] == "ReplicationController":
test = spec["spec"]["replicas"] >= 4
assert_true(test, "ReplicationController should have at least 4 replicas")
test_for_latest_image()
test_minimum_replicas()
Slide 49
Slide 49 text
Tests enforcing a team label
#// vim: set ft=python:
def test_for_team_label():
if spec["kind"] == "Deployment":
labels = spec["spec"]["template"]["metadata"]["labels"]
assert_contains(labels, "team", "should indicate which team owns the deployment")
test_for_team_label()
Slide 50
Slide 50 text
Linting
Building common community assertions
Slide 51
Slide 51 text
Organisation-specific assertions
are useful, but require you to
write the tests yourself
Slide 52
Slide 52 text
Many assertions are common,
they are the result of emerging
community best-practice
Slide 53
Slide 53 text
I’d like to build an out-of-the-box
experience for kubetest
Slide 54
Slide 54 text
Integration with K8 tools
Demos and examples
Slide 55
Slide 55 text
Using kubeval with Helm
$ git clone https://github.com/kubernetes/helm.git
$ cd helm/docs/examples
$ ls
alpine nginx README.md
$ helm template nginx | kubeval
The document stdin contains a valid Secret
The document stdin contains a valid ConfigMap
The document stdin contains a valid Service
The document stdin contains a valid Pod
The document stdin contains a valid Deployment
The document stdin contains a valid Job
Slide 56
Slide 56 text
Tests for our Helm Chart
#// vim: set ft=python:
def test_for_latest_image():
if spec["kind"] in ["Job", "Deployment"]:
for container in spec["spec"]["template"]["spec"]["containers"]:
tag = container["image"].split(":")[-1]
assert_not_equal(tag, "latest", spec["kind"] + " should not use latest images")
test_for_latest_image()
Slide 57
Slide 57 text
Using kubetest with Helm
$ helm template nginx | kubetest --verbose
INFO stdin Deployment should not use latest images
INFO stdin Job should not use latest images
Slide 58
Slide 58 text
Using kubeval with ksonnet
$ cat deployment.jsonnet
local k = import "ksonnet.beta.2/k.libsonnet";
local deployment = k.apps.v1beta1.deployment;
local container = deployment.mixin.spec.template.spec.containersType;
local containerPort = container.portsType;
// Create nginx container with container port 80 open.
local nginxContainer =
container.new("nginx", "nginx:1.13.0") +
container.ports(containerPort.newNamed("http", 80));
// Create default Deployment object from nginx container.
deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
Slide 59
Slide 59 text
Using kubeval with ksonnet
$ cat deployment.jsonnet
local k = import "ksonnet.beta.2/k.libsonnet";
local deployment = k.apps.v1beta1.deployment;
local container = deployment.mixin.spec.template.spec.containersType;
local containerPort = container.portsType;
// Create nginx container with container port 80 open.
local nginxContainer =
container.new("nginx", "nginx:1.13.0") +
container.ports(containerPort.newNamed("http", 80));
// Create default Deployment object from nginx container.
deployment.new("nginx", 5, nginxContainer, {app: "nginx"})
$ jsonnet deployment.jsonnet | kubeval
The document stdin contains a valid Deployment
Slide 60
Slide 60 text
Using kubetest with ksonnet
$ jsonnet deployment.jsonnet | kubetest --verbose
INFO stdin should not use latest images
WARN stdin ReplicationController should have at least 7 replicas
Slide 61
Slide 61 text
Using kubeval with Puppet
$ cat guestbook.pp
kubernetes_service { 'frontend':
ensure => 'present',
metadata => {
'labels' => {'app' => 'guestbook', 'tier' => 'frontend'},
'namespace' => 'default',
},
spec => {
'type' => 'LoadBalancer',
'ports' => [
{'port' => 80, 'protocol' => 'TCP'}
],
'selector' => {
'app' => 'guestbook',
'tier' => 'frontend'
},
},
Slide 62
Slide 62 text
Using kubeval with Puppet
$ puppet kubernetes convert --manifest guestbook.pp | kubeval
The document stdin contains a valid ReplicationController
The document stdin contains a valid Service
The document stdin contains a valid ReplicationController
The document stdin contains a valid Service
The document stdin contains a valid ReplicationController
The document stdin contains a valid Service
Slide 63
Slide 63 text
Using kubetest with Puppet
$ puppet kubernetes convert --manifest guestbook.pp | kubetest --verbose
INFO stdin should not use latest images
INFO stdin should not use latest images
INFO stdin should not use latest images
Slide 64
Slide 64 text
kubeval is a great tool to validate your
Kubernetes configuration files as part
of your CI pipeline
Slide 65
Slide 65 text
Using kubeval and kubetest in CI
Slide 66
Slide 66 text
Conclusions
If all you remember is...
Slide 67
Slide 67 text
The community is still exploring
ways to describe Kubernetes
configuration in code
Slide 68
Slide 68 text
The Configuration Complexity Clock is real
Slide 69
Slide 69 text
Some of the current tools lend
themselves to native testing,
others are just data and templates
Slide 70
Slide 70 text
Having testing tools that work
with different configuration
approaches can make moving
between tools easier
Slide 71
Slide 71 text
Lots more interesting opportunities
around testing Kubernetes configs
Slide 72
Slide 72 text
And lots of inspiration we can take
from other communities and tools
Slide 73
Slide 73 text
Any questions?
And thanks for listening