A dark story of bug hunting

Slide 1

Slide 1 text

Gareth Rushgrove A dark story of bug hunting and the importance of specification

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

@garethr

Slide 4

Slide 4 text

- Act 1: Discovery - Act 2: Investigation - Act 3: Resolution

Slide 5

Slide 5 text

Setting the scene The background to our tale

Slide 6

Slide 6 text

apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis role: master tier: backend spec: ports: - port: 6379 targetPort: 6379 selector: app: redis role: master tier: backend Is this a valid Kubernetes configuration file?

Slide 7

Slide 7 text

Is this Puppet code valid for Kubernetes? $ cat example.pp kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, } $ puppet kubernetes compile --manifest example.pp

Slide 8

Slide 8 text

Is this Helm template valid for Kubernetes? apiVersion: v1 kind: Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}

Slide 9

Slide 9 text

Kubeval - validate Kubernetes configs with JSON schema

Slide 10

Slide 10 text

Like all software, kubeval has bugs. This is the story of one of them

Slide 11

Slide 11 text

Act 1 In which our bug is discovered

Slide 12

Slide 12 text

Initial bug report

Slide 13

Slide 13 text

The bug in action $ tail -n 6 valid-config.yaml spec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster $ kubeval valid-config.yaml The document valid-config.yaml is not a valid Service --> spec.ports.0.targetPort: Invalid type. Expected: string, given: integer

Slide 14

Slide 14 text

Act 2 In which our bug is hunted down

Slide 15

Slide 15 text

Look at the source code

Slide 16

Slide 16 text

Look at tools used to extract the schema

Slide 17

Slide 17 text

Look at the Kubernetes OpenAPI description

Slide 18

Slide 18 text

Look at the JSON Schema for targetPort { "type": "string", "format": "int-or-string" }

Slide 19

Slide 19 text

Narrow down the search to the OpenAPI spec

Slide 20

Slide 20 text

The format property is an open string-valued property, and can have any value to support documentation needs.

Slide 21

Slide 21 text

The bug here is that Kubernetes relies on format for parsing instructions

Slide 22

Slide 22 text

The reality of software rabbit holes

Slide 23

Slide 23 text

Act 3 In which our bug meets its fate

Slide 24

Slide 24 text

First attempt at fixing, a terrible idea to add complex string parsing to kubeval

Slide 25

Slide 25 text

Second attempt, fix the schema { "type": "object", "$schema": "http://json-schema.org/schema#", "oneOf": [ {"type": "string"}, {"type": "integer"} ] }

Slide 26

Slide 26 text

The end?

Slide 27

Slide 27 text

To-be-continued: fix upstream

Slide 28

Slide 28 text

Which relies on another upstream project...

Slide 29

Slide 29 text

Summary If all you remember is...

Slide 30

Slide 30 text

Software bugs often involve interactions between multiple projects, protocols and standards

Slide 31

Slide 31 text

Different layers of software, and associated standards, move at different speeds

Slide 32

Slide 32 text

Specification is powerful and important because it typically moves slowly

Slide 33

Slide 33 text

Bugs can be fixed in different places, often with different upfront and maintenance costs. Choose wisely

Slide 34

Slide 34 text

Thanks for listening Happy to answer questions later