Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Talk from #dockercon EU in Barcelona, all about Docker labels and building higher level tools on top of metadata

Transcript

1. Shipping Manifests, Bill of Lading and Docker Metadata and Container Gareth Rushgrove Senior Software Engineer, Puppet Labs
2. @garethr
3. Senior Engineer at Puppet Labs
4. Creator of the Puppet Docker module
5. This Talk The introduction
6. Shipping containers are cool
7. But nothing without all the paper work
8. A manifest or ship's manifest is a document listing the cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials.
9. A bill of lading is a document issued by a carrier which details a shipment of merchandise and gives title of that shipment to a specified party.
10. State of the Software Supply Chain
11. —State of the Software Supply Chain 2015 A once safe component may be found to be vulnerable at any time” “
12. —State of the Software Supply Chain 2015 Defective components… end up in our software largely unnoticed” “
13. Lets apply the same principles to a different type of container
14. Docker Labels Docker builtin metadata capabilities
15. Added in 1.6 One meta data to rule them all
16. Labels on Docker Engines
17. Provide information about the host $ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd"
18. Labels to guide Swarm scheduling $ docker run -d -P \ -e constraint:storage==ssd --name db mysql
19. Labels on Docker images
20. LABEL [<namespace>.]<key>[=<value>] ... Dockerfile Label instruction
21. Don’t do this - new layer per label LABEL vendor=ACME\ Incorporated LABEL com.example.version.is-beta LABEL com.example.version="0.0.1-beta" LABEL com.example.release-date="2015-02-12"
22. Better - only one layer LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"
23. $ docker inspect 4fa6e0f0c678 ... "Labels": { "vendor": "ACME Incorporated", "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" } ... Access labels via inspect
24. Containers can have additional labels too
25. Add labels at docker run time $ docker run \ -d \ --label com.example.group="webservers" \ --label com.example.environment="production" \ busybox \ top
26. Query based on labels with filters
27. Filter images by label $ docker images --filter "label=com.example.is-beta"
28. Filter containers by label $ docker ps --filter "label=com.example.is-beta"
29. Added in v1.17 Lots of label support in the API
30. But what to store in Labels?
31. A quick aside Package Managers
32. I like system packages
33. The power of system packages lies not in the file format but in the metadata
34. DPKG and RPM
35. Debian New Maintainers’ Guide
36. Fedora packaging guidelines
37. Summary: A CD player app that rocks! Name: cdplayer Version: 1.0 Release: 1 Copyright: GPL Group: Applications/Sound Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer URL: http://www.gnomovision.com/cdplayer/cdplayer.html Distribution: WSS Linux Vendor: White Socks Software, Inc. Packager: Santa Claus <sclaus@northpole.com> %description It slices! It dices! It's a CD player app that can't be beat. By using the resonant frequency of the CD itself, it is able to simulate 20X Example RPM spec file
38. Given standard metadata what can we do?
39. $ dpkg -L lynx /. /usr /usr/share /usr/share/doc /usr/share/doc/lynx /usr/share/doc/lynx/copyright /usr/share/doc/lynx/changelog.gz /usr/share/doc/lynx/changelog.Debian.gz List files from package
40. What installed that file? $ rpm -qf /usr/bin/mysqlaccess MySQL-client-3.23.57-1
41. Find unmet dependencies $ apt-cache unmet Package libdataobjects-sqlite3-ruby1.9.1 version 0.10.1.1-1 has an unmet dep: Depends: libdataobjects-ruby1.9
42. $ rpm -qdf /usr/bin/mysqlaccess /usr/share/man/man1/mysql.1.gz /usr/share/man/man1/mysqlaccess.1.gz /usr/share/man/man1/mysqladmin.1.gz /usr/share/man/man1/mysqldump.1.gz /usr/share/man/man1/mysqlshow.1.gz Find documentation
43. $ apticron The following packages are currently pending an upgrade: xfree86-common 4.3.0.dfsg.1-14sarge3 libice6 4.3.0.dfsg.1-14sarge3 libsm6 4.3.0.dfsg.1-14sarge3 xlibs-data 4.3.0.dfsg.1-14sarge3 libx11-6 4.3.0.dfsg.1-14sarge3 libxext6 4.3.0.dfsg.1-14sarge3 libxpm4 4.3.0.dfsg.1-14sarge3 Find outdated packages
44. Standards The power of agreement
45. Docker official label guidance
46. All (third-party) tools should prefix their keys with the reverse DNS notation of a domain controlled by the author. For example, com.example.some-label. 1
47. The com.docker.*, io.docker.* and org.dockerproject.* namespaces are reserved for Docker’s internal use. 2
48. Keys should only consist of lower-cased alphanumeric characters, dots and dashes (for example, [a-z0-9-.]). 3
49. Keys should start and end with an alpha numeric character. 4
50. Keys may not contain consecutive dots or dashes. 5
51. Keys without namespace (dots) are reserved for CLI use. 6
52. How widely adhered to?
53. < 20% from a small sample
54. But some folks care
55. Unify the labels for Docker images
56. Merged
57. The problem with inconsistent metadata
58. Without complete metadata we can’t trust the tools built on top
59. Docker Label Inspector
60. Check against Docker guidelines $ dli lint ========> Check all labels have namespaces [WARN] Label 'vendor' should use a namespace based on reverse DNS notation ========> Check labels don't use reserved namespaces ========> Check labels only use valid characters ========> Check labels start and end with alpanumeric characters ========> Check labels for double dots and dashes
61. $ dli validate ========> Check labels based on schema in 'schema.json' [ERROR] u'com.example.is-beta' is a required property Check against a schema
62. { "title": "Dockerfile schema", "type": "object", "properties": { "com.example.release-date": { "type": "string" }, "com.example.is-beta": { "type": "string" }, "com.example.version": { "description": "Version", "type": "integer", "minimum": 0 } }, "required": ["com.example.is-beta", "com.example.version"] } Define labels in JSON schema
63. Stand well back LIVE DEMO
64. Runtime Metadata A missing piece, and some ideas
65. What temperature is a refrigerated shipping containers at? *Not all shipping containers are refrigerated
66. docker exec as an API
67. Dockerfile example FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /
68. Discover out API $ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \ | jq { "net.morethanseven.dockerfile": "/Dockerfile", "net.morethanseven.exec.packages": "apk info -vv" }
69. Read the Dockerfile $ docker run -i -t garethr/alpine cat /Dockerfile FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /
70. $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2 - the musl c library (libc) implementati busybox-1.23.2-r0 - Size optimized toolbox of many comm alpine-baselayout-2.3.2-r0 - Alpine base dir structure openrc-0.15.1-r3 - OpenRC manages the services, startup alpine-conf-3.2.1-r6 - Alpine configuration management List installed packages
71. Second time lucky LIVE DEMO
72. R.I.Pienaar
73. Tooling What could we build atop our metadata?
74. Documentation discovery
75. License verification
76. Links to source code or release notes
77. Automatically generated interfaces
78. Package search
79. Tempting fate LIVE DEMO
80. Conclusions If all you remember is…
81. Step 1 Step 2 Step 3 Metadata! Something… Profit
82. Share schemas and namespaces
83. Build agreement
84. Build tooling
85. Extract standards
86. Thank you! Gareth Rushgrove @garethr gareth@puppetlabs.com