A dark story of bug hunting

Talk from Software Circus, on how bugs in even small tools can require investigations which span projects, libraries, protocols and specifications.

Transcript

1. Gareth Rushgrove A dark story of bug hunting and the importance of specification
3. @garethr
4. - Act 1: Discovery - Act 2: Investigation - Act 3: Resolution
5. Setting the scene The background to our tale
6. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis role: master tier: backend spec: ports: - port: 6379 targetPort: 6379 selector: app: redis role: master tier: backend Is this a valid Kubernetes configuration file?
7. Is this Puppet code valid for Kubernetes? $ cat example.pp kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, } $ puppet kubernetes compile --manifest example.pp
8. Is this Helm template valid for Kubernetes? apiVersion: v1 kind: Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
9. Kubeval - validate Kubernetes configs with JSON schema
10. Like all software, kubeval has bugs. This is the story of one of them
11. Act 1 In which our bug is discovered
12. Initial bug report
13. The bug in action $ tail -n 6 valid-config.yaml spec: ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster $ kubeval valid-config.yaml The document valid-config.yaml is not a valid Service --> spec.ports.0.targetPort: Invalid type. Expected: string, given: integer
14. Act 2 In which our bug is hunted down
15. Look at the source code
16. Look at tools used to extract the schema
17. Look at the Kubernetes OpenAPI description
18. Look at the JSON Schema for targetPort { "type": "string", "format": "int-or-string" }
19. Narrow down the search to the OpenAPI spec
20. The format property is an open string-valued property, and can have any value to support documentation needs.
21. The bug here is that Kubernetes relies on format for parsing instructions
22. The reality of software rabbit holes
23. Act 3 In which our bug meets its fate
24. First attempt at fixing, a terrible idea to add complex string parsing to kubeval
25. Second attempt, fix the schema { "type": "object", "$schema": "http://json-schema.org/schema#", "oneOf": [ {"type": "string"}, {"type": "integer"} ] }
26. The end?
27. To-be-continued: fix upstream
28. Which relies on another upstream project...
29. Summary If all you remember is...
30. Software bugs often involve interactions between multiple projects, protocols and standards
31. Different layers of software, and associated standards, move at different speeds
32. Specification is powerful and important because it typically moves slowly
33. Bugs can be fixed in different places, often with different upfront and maintenance costs. Choose wisely
34. Thanks for listening Happy to answer questions later