Kubernetes, Metadata and You

A talk from me and Liz Rice from Aqua Security, all about Kubernetes and Metadata. A bunch of background, details of relevant Kubernetes features, and examples of the kinds of things you can do and problems you can solve building on top of quality metadata.

1. Liz Rice & Gareth Rushgrove Kubernetes, Metadata and You
2. @lizrice
3. @garethr
4. - What do we mean by metadata - Relevant Kubernetes features - Who is metadata useful for? - Tools, examples and demos
5. Metadata What is it and why do we care
6. metadata /ˈmɛtədeɪtə/ noun data that provides information about other data
7. Understanding metadata from Jenn Riley
8. Descriptive metadata describes a resource for purposes such as discovery and identification Understanding Metadata, Jenn Riley
9. Structural metadata is metadata about a grouping of data and indicates how compound objects are put together Understanding Metadata, Jenn Riley
10. Administrative metadata provides information to help manage a resource, such as when and how it was created Understanding Metadata, Jenn Riley
11. Metadata use cases - Resource discovery - Organising resources - Facilitating interoperability - Identification - Archiving and preservation Understanding Metadata, Jenn Riley
12. Kubernetes and metadata Useful features for storing and using metadata
13. Labels
14. Labels are key/value pairs that are attached to objects, such as pods "labels": { "key1" : "value1", "key2" : "value2" }
15. Labels are intended to be used to specify identifying attributes of objects that are meaningful and relevant to users, but do not directly imply semantics to the core system
16. Filter objects with label selectors $ kubectl get pods -l environment=production,tier=frontend $ kubectl get pods -l 'environment in (production),tier in (frontend)' $ kubectl logs -l app=nginx
17. An item of metadata should be a label if - It is used by Kubernetes to identify this resource - It is useful to expose to operators for the purpose of querying the system From the Helm Chart best practices
18. Annotations
19. You can use annotations to attach arbitrary non-identifying metadata to objects
20. - Build, release or image information like timestamps or git branch - Links to logging, monitoring, analytics or audit tools - Support contact details
21. Example annotations on Ingress apiVersion: extensions/v1beta1 kind: Ingress metadata: name: cafe-ingress-with-annotations annotations: nginx.org/proxy-connect-timeout: "30s" nginx.org/proxy-read-timeout: "20s" nginx.org/client-max-body-size: "4m"
22. Expose labels and annotations to containers ... volumeMounts: - name: podinfo mountPath: /etc readOnly: false volumes: - name: podinfo downwardAPI: items: - path: "labels" fieldRef: fieldPath: metadata.labels - path: "annotations" fieldRef: fieldPath: metadata.annotations
23. Docker images also have labels, which can be set at build time LABEL "com.example.vendor"="ACME Incorporated" LABEL com.example.label-with-value="foo" LABEL version="1.0"
24. OCI Image Spec defines annotations
25. What’s missing? Observations and gaps in metadata capabilities
26. Image labels / annotations are fixed at build time Kubernetes labels and annotations are associated with deployed software
27. Post-build / pre-deployment metadata use cases - Sign-off status - Test reports - Vulnerability scanning
28. Users Who is this metadata for?
29. There are different types of user for Kubernetes, but no widely agreed upon personas
33. Kubernetes platform operators - Manages API Server, etcd and kubelet - Ensures a stable platform for other users - May manage underlying infrastructure - Might be a cloud provider or third party
34. Application developers - Build applications - Shouldn’t have to care about K8S at all - May also operate the application
35. Application operators - Manages tools used to access K8S - Looking at Helm, ksonnet and similar - Sets standards for others - May deploy apps or build pipelines
36. Examples and demos Building useful things with metadata
37. Demo 1 Routing support issues using metadata
38. As an application operator So that failing apps get prompt support I want to alerts to reach the right support contact Without having to redeploy when support contact changes
39. manifesto - storing image metadata registry myorg/myrepo images data blobs metadata
40. manifesto - storing image metadata registry myorg/myrepo images data blobs _manifesto metadata
41. Demo - alert current contact on CrashLoopBackoff 52.170.3.92 Some really flaky app code
42. Health checks → restart failing pod CrashLoopBackoff → it needs attention Contact details associated with image
43. Alert current support contact if health check fails // If pod reaches CrashLoopBackoff, find the container image and call this function func contactAboutImage(image string) { // Get contact info from manifesto cmd := exec.Command("manifesto", "get", image, "contact") content, _ := cmd.Output() var c ContactFile json.Unmarshal(content, &c) message := "hey there, " + image + " needs some attention" send(message, c.Phone) // Sends message via Twilio }
44. Modify metadata without changing deployed code $ cat contact.file { "phone": "<--phone number goes here-->" } $ manifesto put lizrice/hello:healthcheck contact contact.file
45. Demo 2 Being aware of security vulnerabilities
46. As an application operator So that I can keep my containers security patched I want to know which contain vulnerabilities
47. registry grafeas - storing & querying software metadata myorg/myrepo images Grafeas Occurrences Notes
48. grafeas - storing & querying software metadata registry myorg/myrepo images Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
49. grafeas - storing & querying software metadata registry myorg/myrepo images Grafeas Occurrences Notes Which images have vulnerability CVE-1234? CVE-1234 CVE-3456
50. Demo - vulnerability scan data stored in Grafeas
51. Grafeas Notes & Occurrences Demo - vulnerability scan data stored in Grafeas Aqua scanner Itay’s reformatter manifesto Grafeas Grafeas queries webhook
52. As an application operator So that I can maintain standards I want to automate pre-deployment checks
53. Admission control pattern Start deploy Is image OK? Run image Fail Check the metadata for the image • Test results? • Signed? • Vulnerability policies?
54. Demo 3 Using multiple sources of metadata
55. As an application operator So that I can help teams with audits I want to know what software packages different teams are using
56. Introducing Lumogon
57. Lumogon exports data about running containers $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock puppet/lumogon scan { "$schema": "http://puppet.com/lumogon/core/draft-01/schema#1", "generated": "2017-05-09 07:59:24.287008012 +0000 UTC", "owner": "default", "group": [ "default" ], "client_version": { "BuildVersion": "development", "BuildTime": "2017-05-09 06:56:22 UTC", "BuildSHA": "9e8f684432ff12b04b5b5d594caa0ebcce86b844" }, "reportid": "c73a79dc-8612-4af8-8bd8-22e32ea11e38", "containers": { "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034": { "$schema": "http://puppet.com/lumogon/containerreport/draft-01/schema#1", "generated": "2017-05-09 07:59:03.513739277 +0000 UTC", "container_report_id": "8d17e541-11b3-4f25-b145-4ad9d3045995", "container_id": "5982d3f16bbbf9530ae09915b22a0d189044e3b953e5e417e2783b90de579034",
58. Lumogon is a component part of Puppet Discovery
59. Kubernetes labels provide - A way to map services/pods to teams Lumogon provides - Metadata about packages in containers
60. Combining information from multiple sources $ ./collect_data_from_api_and_lumogon.py | ./output_package_table.py +------------------------+--------------------------+------------+----------------+ | Package | Version | Occurrences | Teams | +------------------------+--------------------------+------------+----------------+ | acl | 2.2.52-2 | 1 | team-humphrey | | adduser | 3.113+nmu3 | 3 | team-humphrey | | | | | team-shamu | | alpine-baselayout | 3.0.4-r0 | 2 | team-keiko | | alpine-keys | 2.1-r1 | 2 | team-keiko | | apk-tools | 2.7.3-r0 | 2 | team-keiko | | apt | 0.9.7.9+deb7u7 | 2 | team-shamu | | apt | 1.0.9.8.4 | 1 | team-humphrey | | base-files | 7.1wheezy8 | 2 | team-shamu | | base-files | 8+deb8u9 | 1 | team-humphrey | | base-passwd | 3.5.26 | 2 | team-shamu | | base-passwd | 3.5.37 | 1 | team-humphrey | | bash | 4.2+dfsg-0.1+deb7u3 | 2 | team-shamu | | bash | 4.3-11+deb8u1 | 1 | team-humphrey | | bsdutils | 1:2.25.2-6 | 1 | team-humphrey | | bsdutils | 1:2.20.1-5.3 | 2 | team-shamu | https://gist.github.com/garethr/dcdb5cd54b72bb80f422be95a2585bd3
61. Demo 4 Enforcing metadata standards
62. As an application operator So that I can rely on metadata I want to enforce some standards around labels and annotations
63. A word document or markdown file is not a reliable way of ensuring standards are enforced
64. Introducing kubetest
65. Run tests against your configurations $ kubetest rc.yaml --verbose INFO rc.yaml should not use latest images WARN rc.yaml ReplicationController should have at least 4 replicas
66. Tests enforcing a team label #// vim: set ft=python: def test_for_team_label(): if spec["kind"] == "Deployment": labels = spec["spec"]["template"]["metadata"]["labels"] assert_contains(labels, "team", "should indicate which team owns the deployment") test_for_team_label()
67. Conclusions If all you remember is...
68. As a Kubernetes operator Think about what metadata would make debugging a platform problem easier
69. As an application developer Add metadata to your applications, those operating it in production (maybe you) will thank you
70. As an application operator Think about schemas for metadata, and look at ways of encouraging or enforcing it’s usage
71. Metadata provides a flexible platform for building useful tools that make managing Kubernetes systems easier
72. Thanks for listening - aquasecurity/manifesto - puppet/lumogon - grafeas/grafeas A few useful GitHub projects