Liz Rice & Gareth Rushgrove Kubernetes, Metadata and You

@lizrice

@garethr

- What do we mean by metadata - Relevant Kubernetes features - Who is metadata useful for? - Tools, examples and demos

Metadata What is it and why do we care

metadata /ˈmɛtədeɪtə/ noun data that provides information about other data

Understanding metadata from Jenn Riley

Descriptive metadata describes a resource for purposes such as discovery and identification Understanding Metadata, Jenn Riley

Structural metadata is metadata about a grouping of data and indicates how compound objects are put together Understanding Metadata, Jenn Riley

Administrative metadata provides information to help manage a resource, such as when and how it was created Understanding Metadata, Jenn Riley

Metadata use cases - Resource discovery - Organising resources - Facilitating interoperability - Identification - Archiving and preservation Understanding Metadata, Jenn Riley

Kubernetes and metadata Useful features for storing and using metadata

Labels

Labels are key/value pairs that are attached to objects, such as pods "labels": { "key1" : "value1", "key2" : "value2" }

Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system

Filter objects with label selectors $ kubectl get pods -l environment=production,tier=frontend $ kubectl get pods -l 'environment in (production),tier in (frontend)' $ kubectl logs -l app=nginx

An item of metadata should be a label if - It is used by Kubernetes to identify this resource - It is useful to expose to operators for the purpose of querying the system From the Helm Chart best practices

Annotations

You can use annotations to attach arbitrary non-identifying metadata to objects

- Build, release or image information like timestamps or git branch - Links to logging, monitoring, analytics or audit tools - Support contact details

Example annotations on Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m"

Expose labels and annotations to containers ... volumeMounts: - name: podinfo mountPath: /etc readOnly: false volumes: - name: podinfo downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "annotations" fieldRef: fieldPath: metadata.annotations

Docker images also have labels, which can be set at build time LABEL "com.example.vendor"="ACME Incorporated" LABEL com.example.label-with-value="foo" LABEL version="1.0"

OCI Image Spec defines annotations

What’s missing? Observations and gaps in metadata capabilities

Image labels / annotations are fixed at build time Kubernetes labels and annotations are associated with deployed software

Post-build / pre-deployment metadata use cases - Sign-off status - Test reports - Vulnerability scanning

Users Who is this metadata for?

There are different types of user for Kubernetes, but no widely agreed upon personas

No content

No content

No content

Kubernetes platform operators - Manages API Server, etcd and kubelet - Ensures a stable platform for other users - May manage underlying infrastructure - Might be a cloud provider or third party

Application developers - Build applications - Shouldn’t have to care about K8S at all - May also operate the application

Application operators - Manages tools used to access K8S - Looking at Helm, ksonnet and similar - Sets standards for others - May deploy apps or build pipelines

Examples and demos Building useful things with metadata

Demo 1 Routing support issues using metadata

As an application operator So that failing apps get prompt support I want to alerts to reach the right support contact Without having to redeploy when support contact changes

manifesto - storing image metadata registry myorg/myrepo images data blobs metadata

manifesto - storing image metadata registry myorg/myrepo images data blobs _manifesto metadata

Demo - alert current contact on CrashLoopBackoff 52.170.3.92 Some really flaky app code

Health checks → restart failing pod CrashLoopBackoff → it needs attention Contact details associated with image

Alert current support contact if health check fails // If pod reaches CrashLoopBackoff, find the container image and call this function func contactAboutImage(image string) { // Get contact info from manifesto cmd := exec.Command("manifesto", "get", image, "contact") content, _ := cmd.Output() var c ContactFile json.Unmarshal(content, &c) message := "hey there, " + image + " needs some attention" send(message, c.Phone) // Sends message via Twilio }

Modify metadata without changing deployed code $ cat contact.file { "phone": "<--phone number goes here-->" } $ manifesto put lizrice/hello:healthcheck contact contact.file

Demo 2 Being aware of security vulnerabilities

As an application operator So that I can keep my containers security patched I want to know which contain vulnerabilities

registry grafeas - storing & querying software metadata myorg/myrepo images Grafeas Occurrences Notes

grafeas - storing & querying software metadata registry myorg/myrepo images Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456

grafeas - storing & querying software metadata registry myorg/myrepo images Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456

Demo - vulnerability scan data stored in Grafeas

Grafeas Notes & Occurrences Demo - vulnerability scan data stored in Grafeas Aqua scanner Itay’s reformatter manifesto Grafeas Grafeas queries webhook

As an application operator So that I can maintain standards I want to automate pre-deployment checks

Admission control pattern Start deploy Is image OK? Run image Fail Check the metadata for the image ● Test results? ● Signed? ● Vulnerability policies?

Demo 3 Using multiple sources of metadata

As an application operator So that I can help teams with audits I want to know what software packages different teams are using

Introducing Lumogon

Lumogon exports data about running containers $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan { "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1", "generated": "2017-05-09 07:59:24.287008012 +0000 UTC", "owner": "default", "group": [ "default" ], "client_version": { "BuildVersion": "development", "BuildTime": "2017-05-09 06:56:22 UTC", "BuildSHA": "9e8f684432ff12b04b5b5d594caa0ebcce86b844" }, "reportid": "c73a79dc-8612-4af8-8bd8-22e32ea11e38", "containers": { "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034": { "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1", "generated": "2017-05-09 07:59:03.513739277 +0000 UTC", "container_report_id": "8d17e541-11b3-4f25-b145-4ad9d3045995", "container_id": "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034",

Lumogon is a component part of Puppet Discovery

Kubernetes labels provide - A way to map services/pods to teams Lumogon provides - Metadata about packages in containers

Combining information from multiple sources $ ./collect_data_from_api_and_lumogon.py | ./output_package_table.py +------------------------+--------------------------+------------+----------------+ | Package | Version | Occurrences | Teams | +------------------------+--------------------------+------------+----------------+ | acl | 2.2.52-2 | 1 | team-humphrey | | adduser | 3.113+nmu3 | 3 | team-humphrey | | | | | team-shamu | | alpine-baselayout | 3.0.4-r0 | 2 | team-keiko | | alpine-keys | 2.1-r1 | 2 | team-keiko | | apk-tools | 2.7.3-r0 | 2 | team-keiko | | apt | 0.9.7.9+deb7u7 | 2 | team-shamu | | apt | 1.0.9.8.4 | 1 | team-humphrey | | base-files | 7.1wheezy8 | 2 | team-shamu | | base-files | 8+deb8u9 | 1 | team-humphrey | | base-passwd | 3.5.26 | 2 | team-shamu | | base-passwd | 3.5.37 | 1 | team-humphrey | | bash | 4.2+dfsg-0.1+deb7u3 | 2 | team-shamu | | bash | 4.3-11+deb8u1 | 1 | team-humphrey | | bsdutils | 1:2.25.2-6 | 1 | team-humphrey | | bsdutils | 1:2.20.1-5.3 | 2 | team-shamu | https://gist.github.com/garethr/dcdb5cd54b72bb80f422be95a2585bd3

Demo 4 Enforcing metadata standards

As an application operator So that I can rely on metadata I want to enforce some standards around labels and annotations

A word document or markdown file is not a reliable way of ensuring standards are enforced

Introducing kubetest

Run tests against your configurations $ kubetest rc.yaml --verbose INFO rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas

Tests enforcing a team label #// vim: set ft=python: def test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()

Conclusions If all you remember is...

As a Kubernetes operator Think about what metadata would make debugging a platform problem easier

As an application developer Add metadata to your applications, those operating it in production (maybe you) will thank you

As an application operator Think about schemas for metadata, and look at ways of encouraging or enforcing it’s usage

Metadata provides a flexible platform for building useful tools that make managing Kubernetes systems easier

Thanks for listening - aquasecurity/manifesto - puppet/lumogon - grafeas/grafeas A few useful GitHub projects