Kubernetes, Metadata and You

A talk from me and Liz Rice from Aqua Security, all about Kubernetes and Metadata. A bunch of background, details of relevant Kubernetes features, and examples of the kinds of things you can do and problems you can solve building on top of quality metadata.

Transcript

1. Liz Rice & Gareth Rushgrove
   Kubernetes, Metadata and You
   

   View Slide

2. @lizrice
   

   View Slide

3. @garethr
   

   View Slide

4. - What do we mean by metadata
   - Relevant Kubernetes features
   - Who is metadata useful for?
   - Tools, examples and demos
   

   View Slide

5. Metadata
   What is it and why do we care
   

   View Slide

6. metadata
   /ˈmɛtədeɪtə/
   noun
   data that provides information about other data
   

   View Slide

7. Understanding metadata from Jenn Riley
   

   View Slide

8. Descriptive metadata describes a
   resource for purposes such as
   discovery and identification
   Understanding Metadata, Jenn Riley
   

   View Slide

9. Structural metadata is metadata
   about a grouping of data and
   indicates how compound objects
   are put together
   Understanding Metadata, Jenn Riley
   

   View Slide

10. Administrative metadata
    provides information to help
    manage a resource, such as
    when and how it was created
    Understanding Metadata, Jenn Riley
    

    View Slide

11. Metadata use cases
    - Resource discovery
    - Organising resources
    - Facilitating interoperability
    - Identification
    - Archiving and preservation
    Understanding Metadata, Jenn Riley
    

    View Slide

12. Kubernetes and metadata
    Useful features for storing and using metadata
    

    View Slide

13. Labels
    

    View Slide

14. Labels are key/value pairs that are
    attached to objects, such as pods
    "labels": {
    "key1" : "value1",
    "key2" : "value2"
    }
    

    View Slide

15. Labels are intended to be used to
    specify identifying attributes of objects
    that are meaningful and relevant to
    users, but do not directly imply
    semantics to the core system
    

    View Slide

16. Filter objects with label selectors
    $ kubectl get pods -l environment=production,tier=frontend
    $ kubectl get pods -l 'environment in (production),tier in (frontend)'
    $ kubectl logs -l app=nginx
    

    View Slide

17. An item of metadata should be a label if
    - It is used by Kubernetes to
    identify this resource
    - It is useful to expose to operators for
    the purpose of querying the system
    From the Helm Chart best practices
    

    View Slide

18. Annotations
    

    View Slide

19. You can use annotations to
    attach arbitrary non-identifying
    metadata to objects
    

    View Slide

20. - Build, release or image information
    like timestamps or git branch
    - Links to logging, monitoring,
    analytics or audit tools
    - Support contact details
    

    View Slide

21. Example annotations on Ingress
    apiVersion: extensions/v1beta1
    kind: Ingress
    metadata:
    name: cafe-ingress-with-annotations
    annotations:
    nginx.org/proxy-connect-timeout: "30s"
    nginx.org/proxy-read-timeout: "20s"
    nginx.org/client-max-body-size: "4m"
    

    View Slide

22. Expose labels and annotations to containers
    ...
    volumeMounts:
    - name: podinfo
    mountPath: /etc
    readOnly: false
    volumes:
    - name: podinfo
    downwardAPI:
    items:
    - path: "labels"
    fieldRef:
    fieldPath: metadata.labels
    - path: "annotations"
    fieldRef:
    fieldPath: metadata.annotations
    

    View Slide

23. Docker images also have labels,
    which can be set at build time
    LABEL "com.example.vendor"="ACME Incorporated"
    LABEL com.example.label-with-value="foo"
    LABEL version="1.0"
    

    View Slide

24. OCI Image Spec defines annotations
    

    View Slide

25. What’s missing?
    Observations and gaps in metadata capabilities
    

    View Slide

26. Image labels / annotations are fixed
    at build time
    Kubernetes labels and annotations are
    associated with deployed software
    

    View Slide

27. Post-build / pre-deployment metadata
    use cases
    - Sign-off status
    - Test reports
    - Vulnerability scanning
    

    View Slide

28. Users
    Who is this metadata for?
    

    View Slide

29. There are different types of user for
    Kubernetes, but no widely agreed
    upon personas
    

    View Slide

30. View Slide

31. View Slide

32. View Slide

33. Kubernetes platform operators
    - Manages API Server, etcd and kubelet
    - Ensures a stable platform for other users
    - May manage underlying infrastructure
    - Might be a cloud provider or third party
    

    View Slide

34. Application developers
    - Build applications
    - Shouldn’t have to care about K8S at all
    - May also operate the application
    

    View Slide

35. Application operators
    - Manages tools used to access K8S
    - Looking at Helm, ksonnet and similar
    - Sets standards for others
    - May deploy apps or build pipelines
    

    View Slide

36. Examples and demos
    Building useful things with metadata
    

    View Slide

37. Demo 1
    Routing support issues using metadata
    

    View Slide

38. As an application operator
    So that failing apps get prompt support
    I want to alerts to reach the right
    support contact
    Without having to redeploy when
    support contact changes
    

    View Slide

39. manifesto - storing image metadata
    registry
    myorg/myrepo
    images
    data blobs metadata
    

    View Slide

40. manifesto - storing image metadata
    registry
    myorg/myrepo
    images
    data blobs
    _manifesto
    metadata
    

    View Slide

41. Demo - alert current contact on CrashLoopBackoff
    52.170.3.92
    Some really flaky
    app code
    

    View Slide

42. Health checks → restart failing pod
    CrashLoopBackoff → it needs attention
    Contact details associated with image
    

    View Slide

43. Alert current support contact if health check fails
    // If pod reaches CrashLoopBackoff, find the container image and call this function
    func contactAboutImage(image string) {
    // Get contact info from manifesto
    cmd := exec.Command("manifesto", "get", image, "contact")
    content, _ := cmd.Output()
    var c ContactFile
    json.Unmarshal(content, &c)
    message := "hey there, " + image + " needs some attention"
    send(message, c.Phone) // Sends message via Twilio
    }
    

    View Slide

44. Modify metadata without changing deployed code
    $ cat contact.file
    {
    "phone": "<--phone number goes here-->"
    }
    $ manifesto put lizrice/hello:healthcheck contact contact.file
    

    View Slide

45. Demo 2
    Being aware of security vulnerabilities
    

    View Slide

46. As an application operator
    So that I can keep my containers
    security patched
    I want to know which contain
    vulnerabilities
    

    View Slide

47. registry
    grafeas - storing & querying software metadata
    myorg/myrepo
    images
    Grafeas
    Occurrences Notes
    

    View Slide

48. grafeas - storing & querying software metadata
    registry
    myorg/myrepo
    images
    Grafeas
    Occurrences Notes
    Which images
    have vulnerability
    CVE-1234?
    CVE-1234
    CVE-3456
    

    View Slide

49. grafeas - storing & querying software metadata
    registry
    myorg/myrepo
    images
    Grafeas
    Occurrences Notes
    Which images
    have vulnerability
    CVE-1234?
    CVE-1234
    CVE-3456
    

    View Slide

50. Demo - vulnerability scan data stored in Grafeas
    

    View Slide

51. Grafeas Notes & Occurrences
    Demo - vulnerability scan data stored in Grafeas
    Aqua
    scanner
    Itay’s
    reformatter
    manifesto
    Grafeas
    Grafeas
    queries
    webhook
    

    View Slide

52. As an application operator
    So that I can maintain standards
    I want to automate
    pre-deployment checks
    

    View Slide

53. Admission control pattern
    Start
    deploy
    Is image
    OK?
    Run image
    Fail
    Check the metadata
    for the image
    ● Test results?
    ● Signed?
    ● Vulnerability policies?
    

    View Slide

54. Demo 3
    Using multiple sources of metadata
    

    View Slide

55. As an application operator
    So that I can help teams with audits
    I want to know what software
    packages different teams are using
    

    View Slide

56. Introducing Lumogon
    

    View Slide

57. Lumogon exports data about running containers
    $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan
    {
    "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1",
    "generated": "2017-05-09 07:59:24.287008012 +0000 UTC",
    "owner": "default",
    "group": [
    "default"
    ],
    "client_version": {
    "BuildVersion": "development",
    "BuildTime": "2017-05-09 06:56:22 UTC",
    "BuildSHA": "9e8f684432ff12b04b5b5d594caa0ebcce86b844"
    },
    "reportid": "c73a79dc-8612-4af8-8bd8-22e32ea11e38",
    "containers": {
    "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034": {
    "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1",
    "generated": "2017-05-09 07:59:03.513739277 +0000 UTC",
    "container_report_id": "8d17e541-11b3-4f25-b145-4ad9d3045995",
    "container_id": "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034",
    

    View Slide

58. Lumogon is a component part of Puppet Discovery
    

    View Slide

59. Kubernetes labels provide
    - A way to map services/pods to teams
    Lumogon provides
    - Metadata about packages in containers
    

    View Slide

60. Combining information from multiple sources
    $ ./collect_data_from_api_and_lumogon.py | ./output_package_table.py
    +------------------------+--------------------------+------------+----------------+
    | Package | Version | Occurrences | Teams |
    +------------------------+--------------------------+------------+----------------+
    | acl | 2.2.52-2 | 1 | team-humphrey |
    | adduser | 3.113+nmu3 | 3 | team-humphrey |
    | | | | team-shamu |
    | alpine-baselayout | 3.0.4-r0 | 2 | team-keiko |
    | alpine-keys | 2.1-r1 | 2 | team-keiko |
    | apk-tools | 2.7.3-r0 | 2 | team-keiko |
    | apt | 0.9.7.9+deb7u7 | 2 | team-shamu |
    | apt | 1.0.9.8.4 | 1 | team-humphrey |
    | base-files | 7.1wheezy8 | 2 | team-shamu |
    | base-files | 8+deb8u9 | 1 | team-humphrey |
    | base-passwd | 3.5.26 | 2 | team-shamu |
    | base-passwd | 3.5.37 | 1 | team-humphrey |
    | bash | 4.2+dfsg-0.1+deb7u3 | 2 | team-shamu |
    | bash | 4.3-11+deb8u1 | 1 | team-humphrey |
    | bsdutils | 1:2.25.2-6 | 1 | team-humphrey |
    | bsdutils | 1:2.20.1-5.3 | 2 | team-shamu |
    https://gist.github.com/garethr/dcdb5cd54b72bb80f422be95a2585bd3
    

    View Slide

61. Demo 4
    Enforcing metadata standards
    

    View Slide

62. As an application operator
    So that I can rely on metadata
    I want to enforce some standards
    around labels and annotations
    

    View Slide

63. A word document or markdown
    file is not a reliable way of
    ensuring standards are enforced
    

    View Slide

64. Introducing kubetest
    

    View Slide

65. Run tests against your configurations
    $ kubetest rc.yaml --verbose
    INFO rc.yaml should not use latest images
    WARN rc.yaml ReplicationController should have at least 4 replicas
    

    View Slide

66. Tests enforcing a team label
    #// vim: set ft=python:
    def test_for_team_label():
    if spec["kind"] == "Deployment":
    labels = spec["spec"]["template"]["metadata"]["labels"]
    assert_contains(labels, "team", "should indicate which team owns the deployment")
    test_for_team_label()
    

    View Slide

67. Conclusions
    If all you remember is...
    

    View Slide

68. As a Kubernetes operator
    Think about what metadata would
    make debugging a platform
    problem easier
    

    View Slide

69. As an application developer
    Add metadata to your applications,
    those operating it in production
    (maybe you) will thank you
    

    View Slide

70. As an application operator
    Think about schemas for metadata,
    and look at ways of encouraging or
    enforcing it’s usage
    

    View Slide

71. Metadata provides a flexible platform
    for building useful tools that make
    managing Kubernetes systems easier
    

    View Slide

72. Thanks for listening
    - aquasecurity/manifesto
    - puppet/lumogon
    - grafeas/grafeas
    A few useful GitHub projects
    

    View Slide