Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A dark story of bug hunting

A dark story of bug hunting

Talk from Software Circus, on how bugs in even small tools can require investigations which span projects, libraries, protocols and specifications.


Gareth Rushgrove

September 07, 2017


  1. Gareth Rushgrove A dark story of bug hunting and the

    importance of specification
  2. None
  3. @garethr

  4. - Act 1: Discovery - Act 2: Investigation - Act

    3: Resolution
  5. Setting the scene The background to our tale

  6. apiVersion: v1 kind: Service metadata: name: redis-master labels: app: redis

    role: master tier: backend spec: ports: - port: 6379 targetPort: 6379 selector: app: redis role: master tier: backend Is this a valid Kubernetes configuration file?
  7. Is this Puppet code valid for Kubernetes? $ cat example.pp

    kubernetes_pod { 'sample-pod': ensure => present, metadata => { namespace => 'default', }, spec => { containers => [{ name => 'container-name', image => 'nginx', }] }, } $ puppet kubernetes compile --manifest example.pp
  8. Is this Helm template valid for Kubernetes? apiVersion: v1 kind:

    Service metadata: name: {{ template "fullname" . }} labels: app: {{ template "fullname" . }} chart: "{{ .Chart.Name }}-{{ .Chart.Version }}" release: "{{ .Release.Name }}" heritage: "{{ .Release.Service }}" spec: ports: - name: memcache port: 11211 targetPort: memcache selector: app: {{ template "fullname" . }}
  9. Kubeval - validate Kubernetes configs with JSON schema

  10. Like all software, kubeval has bugs. This is the story

    of one of them
  11. Act 1 In which our bug is discovered

  12. Initial bug report

  13. The bug in action $ tail -n 6 valid-config.yaml spec:

    ports: - port: 80 targetPort: 8082 selector: k8s-app: heapster $ kubeval valid-config.yaml The document valid-config.yaml is not a valid Service --> spec.ports.0.targetPort: Invalid type. Expected: string, given: integer
  14. Act 2 In which our bug is hunted down

  15. Look at the source code

  16. Look at tools used to extract the schema

  17. Look at the Kubernetes OpenAPI description

  18. Look at the JSON Schema for targetPort { "type": "string",

    "format": "int-or-string" }
  19. Narrow down the search to the OpenAPI spec

  20. The format property is an open string-valued property, and can

    have any value to support documentation needs.
  21. The bug here is that Kubernetes relies on format for

    parsing instructions
  22. The reality of software rabbit holes

  23. Act 3 In which our bug meets its fate

  24. First attempt at fixing, a terrible idea to add complex

    string parsing to kubeval
  25. Second attempt, fix the schema { "type": "object", "$schema": "http://json-schema.org/schema#",

    "oneOf": [ {"type": "string"}, {"type": "integer"} ] }
  26. The end?

  27. To-be-continued: fix upstream

  28. Which relies on another upstream project...

  29. Summary If all you remember is...

  30. Software bugs often involve interactions between multiple projects, protocols and

  31. Different layers of software, and associated standards, move at different

  32. Specification is powerful and important because it typically moves slowly

  33. Bugs can be fixed in different places, often with different

    upfront and maintenance costs. Choose wisely
  34. Thanks for listening Happy to answer questions later