Managing Container Configuration with Metadata

Managing Container Configuration with Metadata

Keynote talk at Configuration Management Camp, covering the power of metadata for building operations focused tools around containers. Talk of standards, container APIs and metadata.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

February 02, 2016
Tweet

Transcript

  1. Managing Container Configuration with Metadata Puppet Labs Gareth Rushgrove Working

    towards standard interfaces
  2. Gareth Rushgrove @garethr

  3. Gareth Rushgrove

  4. This Talk The Introduction

  5. Shipping containers are cool Gareth Rushgrove

  6. But nothing without all the paper work Gareth Rushgrove

  7. A manifest or ship's manifest is a document listing the

    cargo, passengers, and crew of a ship, aircraft, or vehicle, for the use of customs and other officials. Gareth Rushgrove
  8. A bill of lading is a document issued by a

    carrier which details a shipment of merchandise and gives title of that shipment to a specified party. Gareth Rushgrove
  9. Lets apply the same principles to a different type of

    container Gareth Rushgrove
  10. Container Labels What have we got to work with?

  11. Labels on Docker Engines Gareth Rushgrove

  12. $ docker daemon \ --label com.example.environment="production" \ --label com.example.storage="ssd" Provide

    information about the host Gareth Rushgrove
  13. Labels to guide Swarm scheduling Gareth Rushgrove $ docker run

    -d -P \ -e constraint:storage==ssd --name db mysql
  14. Labels on Docker images Gareth Rushgrove

  15. Dockerfile Label instruction Gareth Rushgrove LABEL [<namespace>.]<key>[=<value>] ...

  16. LABEL vendor=ACME\ Incorporated LABEL com.example.version.is-beta LABEL com.example.version="0.0.1-beta" LABEL com.example.release-date="2015-02-12" Don’t

    do this - new layer per label Gareth Rushgrove
  17. LABEL vendor="ACME\ Incorporated" \ com.example.is-beta \ com.example.version="0.0.1-beta" \ com.example.release-date="2015-02-12" Better

    - only one layer Gareth Rushgrove
  18. $ docker inspect 4fa6e0f0c678 ... "Labels": { "vendor": "ACME Incorporated",

    "com.example.is-beta": "", "com.example.version": "0.0.1-beta", "com.example.release-date": "2015-02-12" } ... Access labels via inspect Gareth Rushgrove
  19. Containers can have additional labels too Gareth Rushgrove

  20. $ docker run \ -d \ --label com.example.group="webservers" \ --label

    com.example.environment="production" \ busybox \ top Add labels at docker runtime Gareth Rushgrove
  21. Query based on labels with filters Gareth Rushgrove

  22. $ docker images --filter "label=com.example.is-beta" Filter images by label Gareth

    Rushgrove
  23. $ docker ps --filter "label=com.example.is-beta" Filter containers by label Gareth

    Rushgrove
  24. Gareth Rushgrove Kubernetes labels Gareth Rushgrove

  25. "labels": { "key1" : "value1", "key2" : "value2" } Apply

    arbitrary metadata to objects Gareth Rushgrove
  26. $ kubectl get pods -l 'environment in (production, qa)’ Query

    using sets or equalities Gareth Rushgrove
  27. Gareth Rushgrove Application container specification Gareth Rushgrove

  28. "annotations": [ { "name": "authors", "value": "Carly Container <carly@example.com>" },

    { "name": "created", "value": "2014-10-27T19:32:27.67021798Z" }, { "name": "documentation", "value": “https://example.com/docs" }, { "name": "homepage", "value": "https://example.com" } ] Appc defines annotations and labels Gareth Rushgrove
  29. But what metadata to store? Gareth Rushgrove

  30. Package Managers A quick aside

  31. Gareth Rushgrove I like system packages Gareth Rushgrove

  32. The power of system packages lies not in the file

    format but in the metadata Gareth Rushgrove
  33. DPKG and RPM Gareth Rushgrove

  34. Gareth Rushgrove Debian New Maintainer’s Guide Gareth Rushgrove

  35. Gareth Rushgrove Fedora Packaging Guidelines Gareth Rushgrove

  36. Summary: A CD player app that rocks! Name: cdplayer Version:

    1.0 Release: 1 Copyright: GPL Group: Applications/Sound Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer-1.0.tgz URL: http://www.gnomovision.com/cdplayer/cdplayer.html Distribution: WSS Linux Vendor: White Socks Software, Inc. Packager: Santa Claus <sclaus@northpole.com> %description It slices! It dices! It's a CD player app that can't be beat. By using the resonant frequency of the CD itself, it is able to simulate 20X Example RPM spec file Gareth Rushgrove
  37. Given standard metadata what can we do? Gareth Rushgrove

  38. $ dpkg -L lynx /. /usr /usr/share /usr/share/doc /usr/share/doc/lynx /usr/share/doc/lynx/copyright

    /usr/share/doc/lynx/changelog.gz /usr/share/doc/lynx/changelog.Debian.gz List files from packages Gareth Rushgrove
  39. $ rpm -qf /usr/bin/mysqlaccess MySQL-client-3.23.57-1 What installed that file? Gareth

    Rushgrove
  40. $ apt-cache unmet Package libdataobjects-sqlite3-ruby1.9.1 version 0.10.1.1-1 has an unmet

    dep: Depends: libdataobjects-ruby1.9 Find unmet dependencies Gareth Rushgrove
  41. $ rpm -qdf /usr/bin/mysqlaccess /usr/share/man/man1/mysql.1.gz /usr/share/man/man1/mysqlaccess.1.gz /usr/share/man/man1/mysqladmin.1.gz /usr/share/man/man1/mysqldump.1.gz /usr/share/man/man1/mysqlshow.1.gz Find

    documentation Gareth Rushgrove
  42. Standards The power of agreement

  43. Gareth Rushgrove Docker official label guidance Gareth Rushgrove

  44. All (third-party) tools should prefix their keys with the reverse

    DNS notation of a domain controlled by the author. For example, com.example.some-label. Gareth Rushgrove
  45. The com.docker.*, io.docker.* and org.dockerproject.* namespaces are reserved for Docker’s

    internal use. Gareth Rushgrove
  46. Keys should only consist of lower- cased alphanumeric characters, dots

    and dashes (for example, [a- z0-9-.]). Gareth Rushgrove
  47. Keys should start and end with an alpha numeric character.

    Gareth Rushgrove
  48. Keys may not contain consecutive dots or dashes. Gareth Rushgrove

  49. Keys without namespace (dots) are reserved for CLI use. Gareth

    Rushgrove
  50. How widely adhered to? Gareth Rushgrove

  51. Gareth Rushgrove < 20% from a small sample Gareth Rushgrove

  52. Without complete metadata we can’t trust the tools built on

    top Gareth Rushgrove
  53. Gareth Rushgrove Hadolint, Dockerfile Linter Gareth Rushgrove

  54. DL3006 Always tag the version of an image explicitely. DL4000

    Specify a maintainer of the Dockerfile FROM debian SC2154 node_verion is referenced but not assigned (did you mean 'node_version'?). DL3009 Delete the apt-get lists after installing something DL3015 Avoid additional packages by specifying `—no-install-recommends` RUN export node_version="0.10" \ && apt-get update && apt-get -y install nodejs="$node_verion" Includes common issues and shellcheck linting of bash Gareth Rushgrove
  55. Gareth Rushgrove Docker Label Inspector Gareth Rushgrove

  56. $ dli lint ========> Check all labels have namespaces [WARN]

    Label 'vendor' should use a namespace based on reverse DNS notation ========> Check labels don't use reserved namespaces ========> Check labels only use valid characters ========> Check labels start and end with alpanumeric characters ========> Check labels for double dots and dashes Check against Docker guidelines Gareth Rushgrove
  57. $ dli validate ========> Check labels based on schema in

    'schema.json' [ERROR] u'com.example.is-beta' is a required property Check against a schema Gareth Rushgrove
  58. { "title": "Dockerfile schema", "type": "object", "properties": { "com.example.release-date": {

    "type": "string" }, "com.example.is-beta": { "type": "string" }, "com.example.version": { "description": "Version", "type": "integer", "minimum": 0 } }, "required": ["com.example.is-beta", "com.example.version"] } Define labels in JSON Schema Gareth Rushgrove
  59. DEMO

  60. Runtime Metadata A missing piece, and some ideas

  61. What temperature is a refrigerated shipping containers at? Gareth Rushgrove

  62. docker exec as an API Gareth Rushgrove

  63. FROM alpine LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk

    add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile / Dockerfile example Gareth Rushgrove
  64. $ docker inspect -f "{{json .Config.Labels }}" \ garethr/alpine \

    | jq { "net.morethanseven.dockerfile": "/Dockerfile", “com.containermetadata.exec.packages”: "apk info -vv" } Discover our API Gareth Rushgrove
  65. $ docker run -i -t garethr/alpine cat /Dockerfile FROM alpine

    LABEL net.morethanseven.dockerfile="/Dockerfile" \ net.morethanseven.exec.packages="apk info -vv" RUN apk add --update bash && rm -rf /var/cache/apk/* COPY Dockerfile / Read the Dockerfile Gareth Rushgrove
  66. $ docker run -i -t garethr/alpine apk info -vv musl-1.1.11-r2

    - the musl c library (libc) implementation busybox-1.23.2-r0 - Size optimized toolbox of many common UNIX utilities alpine-baselayout-2.3.2-r0 - Alpine base dir structure and init scripts openrc-0.15.1-r3 - OpenRC manages the services, startup and shutdown of alpine-conf-3.2.1-r6 - Alpine configuration management scripts List installed packages Gareth Rushgrove
  67. DEMO

  68. Gareth Rushgrove More thoughts from R.I.Pienaar Gareth Rushgrove

  69. Tooling What could we build atop our metadata?

  70. Documentation discovery Gareth Rushgrove

  71. License verification Gareth Rushgrove

  72. Links to source code or release notes Gareth Rushgrove

  73. Automatically generated interfaces Gareth Rushgrove

  74. Package search Gareth Rushgrove

  75. DEMO

  76. Conclusions If all you remember is…

  77. Gareth Rushgrove Step 1 Step 2 Step 3 Metadata! Something…

    Profit
  78. Share schemas and namespaces Gareth Rushgrove

  79. Build agreement Gareth Rushgrove

  80. Build tooling Gareth Rushgrove

  81. Extract standards Gareth Rushgrove

  82. Questions? And thanks for listening