A dark story of bug hunting

Talk from Software Circus, on how bugs in even small tools can require investigations which span projects, libraries, protocols and specifications.

Transcript

1. Gareth Rushgrove
   A dark story of bug hunting
   and the importance of
   specification
   

   View Slide

2. View Slide

3. @garethr
   

   View Slide

4. - Act 1: Discovery
   - Act 2: Investigation
   - Act 3: Resolution
   

   View Slide

5. Setting the scene
   The background to our tale
   

   View Slide

6. apiVersion: v1
   kind: Service
   metadata:
   name: redis-master
   labels:
   app: redis
   role: master
   tier: backend
   spec:
   ports:
   - port: 6379
   targetPort: 6379
   selector:
   app: redis
   role: master
   tier: backend
   Is this a valid Kubernetes configuration file?
   

   View Slide

7. Is this Puppet code valid for Kubernetes?
   $ cat example.pp
   kubernetes_pod { 'sample-pod':
   ensure => present,
   metadata => {
   namespace => 'default',
   },
   spec => {
   containers => [{
   name => 'container-name',
   image => 'nginx',
   }]
   },
   }
   $ puppet kubernetes compile --manifest example.pp
   

   View Slide

8. Is this Helm template valid for Kubernetes?
   apiVersion: v1
   kind: Service
   metadata:
   name: {{ template "fullname" . }}
   labels:
   app: {{ template "fullname" . }}
   chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
   release: "{{ .Release.Name }}"
   heritage: "{{ .Release.Service }}"
   spec:
   ports:
   - name: memcache
   port: 11211
   targetPort: memcache
   selector:
   app: {{ template "fullname" . }}
   

   View Slide

9. Kubeval - validate Kubernetes configs with JSON schema
   

   View Slide

10. Like all software, kubeval
    has bugs. This is the story
    of one of them
    

    View Slide

11. Act 1
    In which our bug is discovered
    

    View Slide

12. Initial bug report
    

    View Slide

13. The bug in action
    $ tail -n 6 valid-config.yaml
    spec:
    ports:
    - port: 80
    targetPort: 8082
    selector:
    k8s-app: heapster
    $ kubeval valid-config.yaml
    The document valid-config.yaml is not a valid Service
    --> spec.ports.0.targetPort: Invalid type. Expected: string, given: integer
    

    View Slide

14. Act 2
    In which our bug is hunted down
    

    View Slide

15. Look at the source code
    

    View Slide

16. Look at tools used to extract the schema
    

    View Slide

17. Look at the Kubernetes OpenAPI description
    

    View Slide

18. Look at the JSON Schema for targetPort
    {
    "type": "string",
    "format": "int-or-string"
    }
    

    View Slide

19. Narrow down the search to the OpenAPI spec
    

    View Slide

20. The format property is an open
    string-valued property, and can
    have any value to support
    documentation needs.
    

    View Slide

21. The bug here is that
    Kubernetes relies on format
    for parsing instructions
    

    View Slide

22. The reality of software rabbit holes
    

    View Slide

23. Act 3
    In which our bug meets its fate
    

    View Slide

24. First attempt at fixing, a terrible
    idea to add complex string
    parsing to kubeval
    

    View Slide

25. Second attempt, fix the schema
    {
    "type": "object",
    "$schema": "http://json-schema.org/schema#",
    "oneOf": [
    {"type": "string"},
    {"type": "integer"}
    ]
    }
    

    View Slide

26. The end?
    

    View Slide

27. To-be-continued: fix upstream
    

    View Slide

28. Which relies on another upstream project...
    

    View Slide

29. Summary
    If all you remember is...
    

    View Slide

30. Software bugs often involve
    interactions between multiple
    projects, protocols and standards
    

    View Slide

31. Different layers of software,
    and associated standards,
    move at different speeds
    

    View Slide

32. Specification is powerful and
    important because it typically
    moves slowly
    

    View Slide

33. Bugs can be fixed in different
    places, often with different
    upfront and maintenance costs.
    Choose wisely
    

    View Slide

34. Thanks for listening
    Happy to answer questions later
    

    View Slide