Managing Container Configuration with Metadata

Keynote talk at Configuration Management Camp, covering the power of metadata for building operations focused tools around containers. Talk of standards, container APIs and metadata.

Transcript

1. Managing Container Configuration with Metadata Puppet Labs Gareth Rushgrove Working towards standard interfaces
2. Gareth Rushgrove @garethr
3. Gareth Rushgrove
4. This Talk The Introduction
5. Shipping containers are cool Gareth Rushgrove
6. But nothing without all the paper work Gareth Rushgrove
7. A manifest or ship's manifest is a document listing the cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials. Gareth Rushgrove
8. A bill of lading is a document issued by a carrier which details a shipment of merchandise and gives title of that shipment to a specified party. Gareth Rushgrove
9. Lets apply the same principles to a different type of container Gareth Rushgrove
10. Container Labels What have we got to work with?
11. Labels on Docker Engines Gareth Rushgrove
12. $ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd" Provide information about the host Gareth Rushgrove
13. Labels to guide Swarm scheduling Gareth Rushgrove $ docker run -d -P \ -e constraint:storage==ssd --name db mysql
14. Labels on Docker images Gareth Rushgrove
15. Dockerfile Label instruction Gareth Rushgrove LABEL [<namespace>.]<key>[=<value>] ...
16. LABEL vendor=ACME\ Incorporated LABEL com.example.version.is-beta LABEL com.example.version="0.0.1-beta" LABEL com.example.release-date="2015-02-12" Don’t do this - new layer per label Gareth Rushgrove
17. LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12" Better - only one layer Gareth Rushgrove
18. $ docker inspect 4fa6e0f0c678 ... "Labels": { "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" } ... Access labels via inspect Gareth Rushgrove
19. Containers can have additional labels too Gareth Rushgrove
20. $ docker run \ -d \ --label com.example.group="webservers" \ --label com.example.environment="production" \ busybox \ top Add labels at docker runtime Gareth Rushgrove
21. Query based on labels with filters Gareth Rushgrove
22. $ docker images --filter "label=com.example.is-beta" Filter images by label Gareth Rushgrove
23. $ docker ps --filter "label=com.example.is-beta" Filter containers by label Gareth Rushgrove
24. Gareth Rushgrove Kubernetes labels Gareth Rushgrove
25. "labels": { "key1" : "value1", "key2" : "value2" } Apply arbitrary metadata to objects Gareth Rushgrove
26. $ kubectl get pods -l 'environment in (production, qa)’ Query using sets or equalities Gareth Rushgrove
27. Gareth Rushgrove Application container specification Gareth Rushgrove
28. "annotations": [ { "name": "authors", "value": "Carly Container <carly@example.com>" }, { "name": "created", "value": "2014-10-27T19:32:27.67021798Z" }, { "name": "documentation", "value": “https://example.com/docs" }, { "name": "homepage", "value": "https://example.com" } ] Appc defines annotations and labels Gareth Rushgrove
29. But what metadata to store? Gareth Rushgrove
30. Package Managers A quick aside
31. Gareth Rushgrove I like system packages Gareth Rushgrove
32. The power of system packages lies not in the file format but in the metadata Gareth Rushgrove
33. DPKG and RPM Gareth Rushgrove
34. Gareth Rushgrove Debian New Maintainer’s Guide Gareth Rushgrove
35. Gareth Rushgrove Fedora Packaging Guidelines Gareth Rushgrove
36. Summary: A CD player app that rocks! Name: cdplayer Version: 1.0 Release: 1 Copyright: GPL Group: Applications/Sound Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer-1.0.tgz URL: http://www.gnomovision.com/cdplayer/cdplayer.html Distribution: WSS Linux Vendor: White Socks Software, Inc. Packager: Santa Claus <sclaus@northpole.com> %description It slices! It dices! It's a CD player app that can't be beat. By using the resonant frequency of the CD itself, it is able to simulate 20X Example RPM spec file Gareth Rushgrove
37. Given standard metadata what can we do? Gareth Rushgrove
38. $ dpkg -L lynx /. /usr /usr/share /usr/share/doc /usr/share/doc/lynx /usr/share/doc/lynx/copyright /usr/share/doc/lynx/changelog.gz /usr/share/doc/lynx/changelog.Debian.gz List files from packages Gareth Rushgrove
39. $ rpm -qf /usr/bin/mysqlaccess MySQL-client-3.23.57-1 What installed that file? Gareth Rushgrove
40. $ apt-cache unmet Package libdataobjects-sqlite3-ruby1.9.1 version 0.10.1.1-1 has an unmet dep: Depends: libdataobjects-ruby1.9 Find unmet dependencies Gareth Rushgrove
41. $ rpm -qdf /usr/bin/mysqlaccess /usr/share/man/man1/mysql.1.gz /usr/share/man/man1/mysqlaccess.1.gz /usr/share/man/man1/mysqladmin.1.gz /usr/share/man/man1/mysqldump.1.gz /usr/share/man/man1/mysqlshow.1.gz Find documentation Gareth Rushgrove
42. Standards The power of agreement
43. Gareth Rushgrove Docker official label guidance Gareth Rushgrove
44. All (third-party) tools should prefix their keys with the reverse DNS notation of a domain controlled by the author. For example, com.example.some-label. Gareth Rushgrove
45. The com.docker.*, io.docker.* and org.dockerproject.* namespaces are reserved for Docker’s internal use. Gareth Rushgrove
46. Keys should only consist of lower- cased alphanumeric characters, dots and dashes (for example, [a- z0-9-.]). Gareth Rushgrove
47. Keys should start and end with an alpha numeric character. Gareth Rushgrove
48. Keys may not contain consecutive dots or dashes. Gareth Rushgrove
49. Keys without namespace (dots) are reserved for CLI use. Gareth Rushgrove
50. How widely adhered to? Gareth Rushgrove
51. Gareth Rushgrove < 20% from a small sample Gareth Rushgrove
52. Without complete metadata we can’t trust the tools built on top Gareth Rushgrove
53. Gareth Rushgrove Hadolint, Dockerfile Linter Gareth Rushgrove
54. DL3006 Always tag the version of an image explicitely. DL4000 Specify a maintainer of the Dockerfile FROM debian SC2154 node_verion is referenced but not assigned (did you mean 'node_version'?). DL3009 Delete the apt-get lists after installing something DL3015 Avoid additional packages by specifying `—no-install-recommends` RUN export node_version="0.10" \ && apt-get update && apt-get -y install nodejs="$node_verion" Includes common issues and shellcheck linting of bash Gareth Rushgrove
55. Gareth Rushgrove Docker Label Inspector Gareth Rushgrove
56. $ dli lint ========> Check all labels have namespaces [WARN] Label 'vendor' should use a namespace based on reverse DNS notation ========> Check labels don't use reserved namespaces ========> Check labels only use valid characters ========> Check labels start and end with alpanumeric characters ========> Check labels for double dots and dashes Check against Docker guidelines Gareth Rushgrove
57. $ dli validate ========> Check labels based on schema in 'schema.json' [ERROR] u'com.example.is-beta' is a required property Check against a schema Gareth Rushgrove
58. { "title": "Dockerfile schema", "type": "object", "properties": { "com.example.release-date": { "type": "string" }, "com.example.is-beta": { "type": "string" }, "com.example.version": { "description": "Version", "type": "integer", "minimum": 0 } }, "required": ["com.example.is-beta", "com.example.version"] } Define labels in JSON Schema Gareth Rushgrove
59. DEMO
60. Runtime Metadata A missing piece, and some ideas
61. What temperature is a refrigerated shipping containers at? Gareth Rushgrove
62. docker exec as an API Gareth Rushgrove
63. FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile / Dockerfile example Gareth Rushgrove
64. $ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq { "net.morethanseven.dockerfile": "/Dockerfile", “com.containermetadata.exec.packages”: "apk info -vv" } Discover our API Gareth Rushgrove
65. $ docker run -i -t garethr/alpine cat /Dockerfile FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile / Read the Dockerfile Gareth Rushgrove
66. $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2 - the musl c library (libc) implementation busybox-1.23.2-r0 - Size optimized toolbox of many common UNIX utilities alpine-baselayout-2.3.2-r0 - Alpine base dir structure and init scripts openrc-0.15.1-r3 - OpenRC manages the services, startup and shutdown of alpine-conf-3.2.1-r6 - Alpine configuration management scripts List installed packages Gareth Rushgrove
67. DEMO
68. Gareth Rushgrove More thoughts from R.I.Pienaar Gareth Rushgrove
69. Tooling What could we build atop our metadata?
70. Documentation discovery Gareth Rushgrove
71. License verification Gareth Rushgrove
72. Links to source code or release notes Gareth Rushgrove
73. Automatically generated interfaces Gareth Rushgrove
74. Package search Gareth Rushgrove
75. DEMO
76. Conclusions If all you remember is…
77. Gareth Rushgrove Step 1 Step 2 Step 3 Metadata! Something… Profit
78. Share schemas and namespaces Gareth Rushgrove
79. Build agreement Gareth Rushgrove
80. Build tooling Gareth Rushgrove
81. Extract standards Gareth Rushgrove
82. Questions? And thanks for listening