Kubernetes, Metadata and You

Kubernetes, Metadata and You

A talk from me and Liz Rice from Aqua Security, all about Kubernetes and Metadata. A bunch of background, details of relevant Kubernetes features, and examples of the kinds of things you can do and problems you can solve building on top of quality metadata.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

December 06, 2017
Tweet

Transcript

  1. 4.

    - What do we mean by metadata - Relevant Kubernetes

    features - Who is metadata useful for? - Tools, examples and demos
  2. 8.

    Descriptive metadata describes a resource for purposes such as discovery

    and identification Understanding Metadata, Jenn Riley
  3. 9.

    Structural metadata is metadata about a grouping of data and

    indicates how compound objects are put together Understanding Metadata, Jenn Riley
  4. 10.

    Administrative metadata provides information to help manage a resource, such

    as when and how it was created Understanding Metadata, Jenn Riley
  5. 11.

    Metadata use cases - Resource discovery - Organising resources -

    Facilitating interoperability - Identification - Archiving and preservation Understanding Metadata, Jenn Riley
  6. 13.
  7. 14.

    Labels are key/value pairs that are attached to objects, such

    as pods "labels": { "key1" : "value1", "key2" : "value2" }
  8. 15.

    Labels are intended to be used to specify identifying attributes

    of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system
  9. 16.

    Filter objects with label selectors $ kubectl get pods -l

    environment=production,tier=frontend $ kubectl get pods -l 'environment in (production),tier in (frontend)' $ kubectl logs -l app=nginx
  10. 17.

    An item of metadata should be a label if -

    It is used by Kubernetes to identify this resource - It is useful to expose to operators for the purpose of querying the system From the Helm Chart best practices
  11. 20.

    - Build, release or image information like timestamps or git

    branch - Links to logging, monitoring, analytics or audit tools - Support contact details
  12. 21.

    Example annotations on Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name:

    cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m"
  13. 22.

    Expose labels and annotations to containers ... volumeMounts: - name:

    podinfo mountPath: /etc readOnly: false volumes: - name: podinfo downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "annotations" fieldRef: fieldPath: metadata.annotations
  14. 23.

    Docker images also have labels, which can be set at

    build time LABEL "com.example.vendor"="ACME Incorporated" LABEL com.example.label-with-value="foo" LABEL version="1.0"
  15. 26.

    Image labels / annotations are fixed at build time Kubernetes

    labels and annotations are associated with deployed software
  16. 27.
  17. 30.
  18. 31.
  19. 32.
  20. 33.

    Kubernetes platform operators - Manages API Server, etcd and kubelet

    - Ensures a stable platform for other users - May manage underlying infrastructure - Might be a cloud provider or third party
  21. 34.

    Application developers - Build applications - Shouldn’t have to care

    about K8S at all - May also operate the application
  22. 35.

    Application operators - Manages tools used to access K8S -

    Looking at Helm, ksonnet and similar - Sets standards for others - May deploy apps or build pipelines
  23. 38.

    As an application operator So that failing apps get prompt

    support I want to alerts to reach the right support contact Without having to redeploy when support contact changes
  24. 42.

    Health checks → restart failing pod CrashLoopBackoff → it needs

    attention Contact details associated with image
  25. 43.

    Alert current support contact if health check fails // If

    pod reaches CrashLoopBackoff, find the container image and call this function func contactAboutImage(image string) { // Get contact info from manifesto cmd := exec.Command("manifesto", "get", image, "contact") content, _ := cmd.Output() var c ContactFile json.Unmarshal(content, &c) message := "hey there, " + image + " needs some attention" send(message, c.Phone) // Sends message via Twilio }
  26. 44.

    Modify metadata without changing deployed code $ cat contact.file {

    "phone": "<--phone number goes here-->" } $ manifesto put lizrice/hello:healthcheck contact contact.file
  27. 46.

    As an application operator So that I can keep my

    containers security patched I want to know which contain vulnerabilities
  28. 48.

    grafeas - storing & querying software metadata registry myorg/myrepo images

    Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
  29. 49.

    grafeas - storing & querying software metadata registry myorg/myrepo images

    Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
  30. 51.

    Grafeas Notes & Occurrences Demo - vulnerability scan data stored

    in Grafeas Aqua scanner Itay’s reformatter manifesto Grafeas Grafeas queries webhook
  31. 52.

    As an application operator So that I can maintain standards

    I want to automate pre-deployment checks
  32. 53.

    Admission control pattern Start deploy Is image OK? Run image

    Fail Check the metadata for the image • Test results? • Signed? • Vulnerability policies?
  33. 55.

    As an application operator So that I can help teams

    with audits I want to know what software packages different teams are using
  34. 57.

    Lumogon exports data about running containers $ docker run --rm

    -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan { "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1", "generated": "2017-05-09 07:59:24.287008012 +0000 UTC", "owner": "default", "group": [ "default" ], "client_version": { "BuildVersion": "development", "BuildTime": "2017-05-09 06:56:22 UTC", "BuildSHA": "9e8f684432ff12b04b5b5d594caa0ebcce86b844" }, "reportid": "c73a79dc-8612-4af8-8bd8-22e32ea11e38", "containers": { "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034": { "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1", "generated": "2017-05-09 07:59:03.513739277 +0000 UTC", "container_report_id": "8d17e541-11b3-4f25-b145-4ad9d3045995", "container_id": "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034",
  35. 59.

    Kubernetes labels provide - A way to map services/pods to

    teams Lumogon provides - Metadata about packages in containers
  36. 60.

    Combining information from multiple sources $ ./collect_data_from_api_and_lumogon.py | ./output_package_table.py +------------------------+--------------------------+------------+----------------+

    | Package | Version | Occurrences | Teams | +------------------------+--------------------------+------------+----------------+ | acl | 2.2.52-2 | 1 | team-humphrey | | adduser | 3.113+nmu3 | 3 | team-humphrey | | | | | team-shamu | | alpine-baselayout | 3.0.4-r0 | 2 | team-keiko | | alpine-keys | 2.1-r1 | 2 | team-keiko | | apk-tools | 2.7.3-r0 | 2 | team-keiko | | apt | 0.9.7.9+deb7u7 | 2 | team-shamu | | apt | 1.0.9.8.4 | 1 | team-humphrey | | base-files | 7.1wheezy8 | 2 | team-shamu | | base-files | 8+deb8u9 | 1 | team-humphrey | | base-passwd | 3.5.26 | 2 | team-shamu | | base-passwd | 3.5.37 | 1 | team-humphrey | | bash | 4.2+dfsg-0.1+deb7u3 | 2 | team-shamu | | bash | 4.3-11+deb8u1 | 1 | team-humphrey | | bsdutils | 1:2.25.2-6 | 1 | team-humphrey | | bsdutils | 1:2.20.1-5.3 | 2 | team-shamu | https://gist.github.com/garethr/dcdb5cd54b72bb80f422be95a2585bd3
  37. 62.

    As an application operator So that I can rely on

    metadata I want to enforce some standards around labels and annotations
  38. 63.

    A word document or markdown file is not a reliable

    way of ensuring standards are enforced
  39. 65.

    Run tests against your configurations $ kubetest rc.yaml --verbose INFO

    rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
  40. 66.

    Tests enforcing a team label #// vim: set ft=python: def

    test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
  41. 68.
  42. 69.

    As an application developer Add metadata to your applications, those

    operating it in production (maybe you) will thank you
  43. 70.

    As an application operator Think about schemas for metadata, and

    look at ways of encouraging or enforcing it’s usage
  44. 71.