Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes, Metadata and You

Kubernetes, Metadata and You

A talk from me and Liz Rice from Aqua Security, all about Kubernetes and Metadata. A bunch of background, details of relevant Kubernetes features, and examples of the kinds of things you can do and problems you can solve building on top of quality metadata.


Gareth Rushgrove

December 06, 2017

More Decks by Gareth Rushgrove

Other Decks in Technology


  1. Liz Rice & Gareth Rushgrove Kubernetes, Metadata and You

  2. @lizrice

  3. @garethr

  4. - What do we mean by metadata - Relevant Kubernetes

    features - Who is metadata useful for? - Tools, examples and demos
  5. Metadata What is it and why do we care

  6. metadata /ˈmɛtədeɪtə/ noun data that provides information about other data

  7. Understanding metadata from Jenn Riley

  8. Descriptive metadata describes a resource for purposes such as discovery

    and identification Understanding Metadata, Jenn Riley
  9. Structural metadata is metadata about a grouping of data and

    indicates how compound objects are put together Understanding Metadata, Jenn Riley
  10. Administrative metadata provides information to help manage a resource, such

    as when and how it was created Understanding Metadata, Jenn Riley
  11. Metadata use cases - Resource discovery - Organising resources -

    Facilitating interoperability - Identification - Archiving and preservation Understanding Metadata, Jenn Riley
  12. Kubernetes and metadata Useful features for storing and using metadata

  13. Labels

  14. Labels are key/value pairs that are attached to objects, such

    as pods "labels": { "key1" : "value1", "key2" : "value2" }
  15. Labels are intended to be used to specify identifying attributes

    of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system
  16. Filter objects with label selectors $ kubectl get pods -l

    environment=production,tier=frontend $ kubectl get pods -l 'environment in (production),tier in (frontend)' $ kubectl logs -l app=nginx
  17. An item of metadata should be a label if -

    It is used by Kubernetes to identify this resource - It is useful to expose to operators for the purpose of querying the system From the Helm Chart best practices
  18. Annotations

  19. You can use annotations to attach arbitrary non-identifying metadata to

  20. - Build, release or image information like timestamps or git

    branch - Links to logging, monitoring, analytics or audit tools - Support contact details
  21. Example annotations on Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name:

    cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m"
  22. Expose labels and annotations to containers ... volumeMounts: - name:

    podinfo mountPath: /etc readOnly: false volumes: - name: podinfo downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "annotations" fieldRef: fieldPath: metadata.annotations
  23. Docker images also have labels, which can be set at

    build time LABEL "com.example.vendor"="ACME Incorporated" LABEL com.example.label-with-value="foo" LABEL version="1.0"
  24. OCI Image Spec defines annotations

  25. What’s missing? Observations and gaps in metadata capabilities

  26. Image labels / annotations are fixed at build time Kubernetes

    labels and annotations are associated with deployed software
  27. Post-build / pre-deployment metadata use cases - Sign-off status -

    Test reports - Vulnerability scanning
  28. Users Who is this metadata for?

  29. There are different types of user for Kubernetes, but no

    widely agreed upon personas
  30. None
  31. None
  32. None
  33. Kubernetes platform operators - Manages API Server, etcd and kubelet

    - Ensures a stable platform for other users - May manage underlying infrastructure - Might be a cloud provider or third party
  34. Application developers - Build applications - Shouldn’t have to care

    about K8S at all - May also operate the application
  35. Application operators - Manages tools used to access K8S -

    Looking at Helm, ksonnet and similar - Sets standards for others - May deploy apps or build pipelines
  36. Examples and demos Building useful things with metadata

  37. Demo 1 Routing support issues using metadata

  38. As an application operator So that failing apps get prompt

    support I want to alerts to reach the right support contact Without having to redeploy when support contact changes
  39. manifesto - storing image metadata registry myorg/myrepo images data blobs

  40. manifesto - storing image metadata registry myorg/myrepo images data blobs

    _manifesto metadata
  41. Demo - alert current contact on CrashLoopBackoff Some really

    flaky app code
  42. Health checks → restart failing pod CrashLoopBackoff → it needs

    attention Contact details associated with image
  43. Alert current support contact if health check fails // If

    pod reaches CrashLoopBackoff, find the container image and call this function func contactAboutImage(image string) { // Get contact info from manifesto cmd := exec.Command("manifesto", "get", image, "contact") content, _ := cmd.Output() var c ContactFile json.Unmarshal(content, &c) message := "hey there, " + image + " needs some attention" send(message, c.Phone) // Sends message via Twilio }
  44. Modify metadata without changing deployed code $ cat contact.file {

    "phone": "<--phone number goes here-->" } $ manifesto put lizrice/hello:healthcheck contact contact.file
  45. Demo 2 Being aware of security vulnerabilities

  46. As an application operator So that I can keep my

    containers security patched I want to know which contain vulnerabilities
  47. registry grafeas - storing & querying software metadata myorg/myrepo images

    Grafeas Occurrences Notes
  48. grafeas - storing & querying software metadata registry myorg/myrepo images

    Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
  49. grafeas - storing & querying software metadata registry myorg/myrepo images

    Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
  50. Demo - vulnerability scan data stored in Grafeas

  51. Grafeas Notes & Occurrences Demo - vulnerability scan data stored

    in Grafeas Aqua scanner Itay’s reformatter manifesto Grafeas Grafeas queries webhook
  52. As an application operator So that I can maintain standards

    I want to automate pre-deployment checks
  53. Admission control pattern Start deploy Is image OK? Run image

    Fail Check the metadata for the image • Test results? • Signed? • Vulnerability policies?
  54. Demo 3 Using multiple sources of metadata

  55. As an application operator So that I can help teams

    with audits I want to know what software packages different teams are using
  56. Introducing Lumogon

  57. Lumogon exports data about running containers $ docker run --rm

    -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan { "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1", "generated": "2017-05-09 07:59:24.287008012 +0000 UTC", "owner": "default", "group": [ "default" ], "client_version": { "BuildVersion": "development", "BuildTime": "2017-05-09 06:56:22 UTC", "BuildSHA": "9e8f684432ff12b04b5b5d594caa0ebcce86b844" }, "reportid": "c73a79dc-8612-4af8-8bd8-22e32ea11e38", "containers": { "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034": { "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1", "generated": "2017-05-09 07:59:03.513739277 +0000 UTC", "container_report_id": "8d17e541-11b3-4f25-b145-4ad9d3045995", "container_id": "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034",
  58. Lumogon is a component part of Puppet Discovery

  59. Kubernetes labels provide - A way to map services/pods to

    teams Lumogon provides - Metadata about packages in containers
  60. Combining information from multiple sources $ ./collect_data_from_api_and_lumogon.py | ./output_package_table.py +------------------------+--------------------------+------------+----------------+

    | Package | Version | Occurrences | Teams | +------------------------+--------------------------+------------+----------------+ | acl | 2.2.52-2 | 1 | team-humphrey | | adduser | 3.113+nmu3 | 3 | team-humphrey | | | | | team-shamu | | alpine-baselayout | 3.0.4-r0 | 2 | team-keiko | | alpine-keys | 2.1-r1 | 2 | team-keiko | | apk-tools | 2.7.3-r0 | 2 | team-keiko | | apt | | 2 | team-shamu | | apt | | 1 | team-humphrey | | base-files | 7.1wheezy8 | 2 | team-shamu | | base-files | 8+deb8u9 | 1 | team-humphrey | | base-passwd | 3.5.26 | 2 | team-shamu | | base-passwd | 3.5.37 | 1 | team-humphrey | | bash | 4.2+dfsg-0.1+deb7u3 | 2 | team-shamu | | bash | 4.3-11+deb8u1 | 1 | team-humphrey | | bsdutils | 1:2.25.2-6 | 1 | team-humphrey | | bsdutils | 1:2.20.1-5.3 | 2 | team-shamu | https://gist.github.com/garethr/dcdb5cd54b72bb80f422be95a2585bd3
  61. Demo 4 Enforcing metadata standards

  62. As an application operator So that I can rely on

    metadata I want to enforce some standards around labels and annotations
  63. A word document or markdown file is not a reliable

    way of ensuring standards are enforced
  64. Introducing kubetest

  65. Run tests against your configurations $ kubetest rc.yaml --verbose INFO

    rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
  66. Tests enforcing a team label #// vim: set ft=python: def

    test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
  67. Conclusions If all you remember is...

  68. As a Kubernetes operator Think about what metadata would make

    debugging a platform problem easier
  69. As an application developer Add metadata to your applications, those

    operating it in production (maybe you) will thank you
  70. As an application operator Think about schemas for metadata, and

    look at ways of encouraging or enforcing it’s usage
  71. Metadata provides a flexible platform for building useful tools that

    make managing Kubernetes systems easier
  72. Thanks for listening - aquasecurity/manifesto - puppet/lumogon - grafeas/grafeas A

    few useful GitHub projects