$30 off During Our Annual Pro Sale. View Details »

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Talk from #dockercon EU in Barcelona, all about Docker labels and building higher level tools on top of metadata

Gareth Rushgrove

November 16, 2015

More Decks by Gareth Rushgrove

Other Decks in Technology


  1. Shipping Manifests, Bill of Lading and Docker Metadata and Container

    Gareth Rushgrove Senior Software Engineer, Puppet Labs
  2. @garethr

  3. Senior Engineer at Puppet Labs

  4. Creator of the Puppet Docker module

  5. This Talk The introduction

  6. Shipping containers are cool

  7. But nothing without all the paper work

  8. A manifest or ship's manifest is a document listing the

    cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials.
  9. A bill of lading is a document issued by a

    carrier which details a shipment of merchandise and gives title of that shipment to a specified party.
  10. State of the Software Supply Chain

  11. —State of the Software Supply Chain 2015 A once safe

    component may be found to be vulnerable at any time” “
  12. —State of the Software Supply Chain 2015 Defective components… end

    up in our software largely unnoticed” “
  13. Lets apply the same principles to a different type of

  14. Docker Labels Docker builtin metadata capabilities

  15. Added in 1.6 One meta data to rule them all

  16. Labels on Docker Engines

  17. Provide information about the host $ docker daemon \ --label

    com.example.environment="production" \ --label com.example.storage="ssd"
  18. Labels to guide Swarm scheduling $ docker run -d -P

    \ -e constraint:storage==ssd --name db mysql
  19. Labels on Docker images

  20. LABEL [<namespace>.]<key>[=<value>] ... Dockerfile Label instruction

  21. Don’t do this - new layer per label LABEL vendor=ACME\

    Incorporated LABEL com.example.version.is-beta LABEL com.example.version="0.0.1-beta" LABEL com.example.release-date="2015-02-12"
  22. Better - only one layer LABEL vendor="ACME\ Incorporated" \ com.example.is-beta

    \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12"
  23. $ docker inspect 4fa6e0f0c678 ... "Labels": { "vendor": "ACME Incorporated",

    "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" } ... Access labels via inspect
  24. Containers can have additional labels too

  25. Add labels at docker run time $ docker run \

    -d \ --label com.example.group="webservers" \ --label com.example.environment="production" \ busybox \ top
  26. Query based on labels with filters

  27. Filter images by label $ docker images --filter "label=com.example.is-beta"

  28. Filter containers by label $ docker ps --filter "label=com.example.is-beta"

  29. Added in v1.17 Lots of label support in the API

  30. But what to store in Labels?

  31. A quick aside Package Managers

  32. I like system packages

  33. The power of system packages lies not in the file

    format but in the metadata
  34. DPKG and RPM

  35. Debian New Maintainers’ Guide

  36. Fedora packaging guidelines

  37. Summary: A CD player app that rocks! Name: cdplayer Version:

    1.0 Release: 1 Copyright: GPL Group: Applications/Sound Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer URL: http://www.gnomovision.com/cdplayer/cdplayer.html Distribution: WSS Linux Vendor: White Socks Software, Inc. Packager: Santa Claus <sclaus@northpole.com> %description It slices! It dices! It's a CD player app that can't be beat. By using the resonant frequency of the CD itself, it is able to simulate 20X Example RPM spec file
  38. Given standard metadata what can we do?

  39. $ dpkg -L lynx /. /usr /usr/share /usr/share/doc /usr/share/doc/lynx /usr/share/doc/lynx/copyright

    /usr/share/doc/lynx/changelog.gz /usr/share/doc/lynx/changelog.Debian.gz List files from package
  40. What installed that file? $ rpm -qf /usr/bin/mysqlaccess MySQL-client-3.23.57-1

  41. Find unmet dependencies $ apt-cache unmet Package libdataobjects-sqlite3-ruby1.9.1 version

    has an unmet dep: Depends: libdataobjects-ruby1.9
  42. $ rpm -qdf /usr/bin/mysqlaccess /usr/share/man/man1/mysql.1.gz /usr/share/man/man1/mysqlaccess.1.gz /usr/share/man/man1/mysqladmin.1.gz /usr/share/man/man1/mysqldump.1.gz /usr/share/man/man1/mysqlshow.1.gz Find

  43. $ apticron The following packages are currently pending an upgrade:

    xfree86-common 4.3.0.dfsg.1-14sarge3 libice6 4.3.0.dfsg.1-14sarge3 libsm6 4.3.0.dfsg.1-14sarge3 xlibs-data 4.3.0.dfsg.1-14sarge3 libx11-6 4.3.0.dfsg.1-14sarge3 libxext6 4.3.0.dfsg.1-14sarge3 libxpm4 4.3.0.dfsg.1-14sarge3 Find outdated packages
  44. Standards The power of agreement

  45. Docker official label guidance

  46. All (third-party) tools should prefix their keys with the reverse

    DNS notation of a domain controlled by the author. For example, com.example.some-label. 1
  47. The com.docker.*, io.docker.* and org.dockerproject.* namespaces are reserved for Docker’s

    internal use. 2
  48. Keys should only consist of lower-cased alphanumeric characters, dots and

    dashes (for example, [a-z0-9-.]). 3
  49. Keys should start and end with an alpha numeric character.

  50. Keys may not contain consecutive dots or dashes. 5

  51. Keys without namespace (dots) are reserved for CLI use. 6

  52. How widely adhered to?

  53. < 20% from a small sample

  54. But some folks care

  55. Unify the labels for Docker images

  56. Merged

  57. The problem with inconsistent metadata

  58. Without complete metadata we can’t trust the tools built on

  59. Docker Label Inspector

  60. Check against Docker guidelines $ dli lint ========> Check all

    labels have namespaces [WARN] Label 'vendor' should use a namespace based on reverse DNS notation ========> Check labels don't use reserved namespaces ========> Check labels only use valid characters ========> Check labels start and end with alpanumeric characters ========> Check labels for double dots and dashes
  61. $ dli validate ========> Check labels based on schema in

    'schema.json' [ERROR] u'com.example.is-beta' is a required property Check against a schema
  62. { "title": "Dockerfile schema", "type": "object", "properties": { "com.example.release-date": {

    "type": "string" }, "com.example.is-beta": { "type": "string" }, "com.example.version": { "description": "Version", "type": "integer", "minimum": 0 } }, "required": ["com.example.is-beta", "com.example.version"] } Define labels in JSON schema
  63. Stand well back LIVE DEMO

  64. Runtime Metadata A missing piece, and some ideas

  65. What temperature is a refrigerated shipping containers at? *Not all

    shipping containers are refrigerated
  66. docker exec as an API

  67. Dockerfile example FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv"

    RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /
  68. Discover out API $ docker inspect -f "{{json .Config.Labels }}"

    \ garethr/alpine \ | jq { "net.morethanseven.dockerfile": "/Dockerfile", "net.morethanseven.exec.packages": "apk info -vv" }
  69. Read the Dockerfile $ docker run -i -t garethr/alpine cat

    /Dockerfile FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile /
  70. $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2

    - the musl c library (libc) implementati busybox-1.23.2-r0 - Size optimized toolbox of many comm alpine-baselayout-2.3.2-r0 - Alpine base dir structure openrc-0.15.1-r3 - OpenRC manages the services, startup alpine-conf-3.2.1-r6 - Alpine configuration management List installed packages
  71. Second time lucky LIVE DEMO

  72. R.I.Pienaar

  73. Tooling What could we build atop our metadata?

  74. Documentation discovery

  75. License verification

  76. Links to source code or release notes

  77. Automatically generated interfaces

  78. Package search

  79. Tempting fate LIVE DEMO

  80. Conclusions If all you remember is…

  81. Step 1 Step 2 Step 3 Metadata! Something… Profit

  82. Share schemas and namespaces

  83. Build agreement

  84. Build tooling

  85. Extract standards

  86. Thank you! Gareth Rushgrove @garethr gareth@puppetlabs.com