Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Talk from #dockercon EU in Barcelona, all about Docker labels and building higher level tools on top of metadata

Gareth Rushgrove

November 16, 2015
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Shipping Manifests, Bill of Lading and Docker
    Metadata and Container
    Gareth Rushgrove
    Senior Software Engineer, Puppet Labs

    View full-size slide

  2. Senior Engineer at Puppet Labs

    View full-size slide

  3. Creator of the Puppet Docker module

    View full-size slide

  4. This Talk
    The introduction

    View full-size slide

  5. Shipping containers are cool

    View full-size slide

  6. But nothing without all the
    paper work

    View full-size slide

  7. A manifest or ship's manifest
    is a document listing the
    cargo, passengers, and crew
    of a ship, aircraft, or vehicle,
    for the use of customs and
    other officials.

    View full-size slide

  8. A bill of lading is a
    document issued by a carrier
    which details a shipment of
    merchandise and gives title
    of that shipment to a
    specified party.

    View full-size slide

  9. State of the Software Supply Chain

    View full-size slide

  10. —State of the Software
    Supply Chain 2015
    A once safe component
    may be found to be
    vulnerable at any time”

    View full-size slide

  11. —State of the Software
    Supply Chain 2015
    Defective components…
    end up in our software
    largely unnoticed”

    View full-size slide

  12. Lets apply the same
    principles to a different type
    of container

    View full-size slide

  13. Docker Labels
    Docker builtin metadata capabilities

    View full-size slide

  14. Added in 1.6
    One meta data to rule them all

    View full-size slide

  15. Labels on Docker
    Engines

    View full-size slide

  16. Provide information about the host
    $ docker daemon \
    --label com.example.environment="production" \
    --label com.example.storage="ssd"

    View full-size slide

  17. Labels to guide Swarm scheduling
    $ docker run -d -P \
    -e constraint:storage==ssd --name db mysql

    View full-size slide

  18. Labels on Docker
    images

    View full-size slide

  19. LABEL [.][=] ...
    Dockerfile Label instruction

    View full-size slide

  20. Don’t do this - new layer per label
    LABEL vendor=ACME\ Incorporated
    LABEL com.example.version.is-beta
    LABEL com.example.version="0.0.1-beta"
    LABEL com.example.release-date="2015-02-12"

    View full-size slide

  21. Better - only one layer
    LABEL vendor="ACME\ Incorporated" \
    com.example.is-beta \
    com.example.version="0.0.1-beta" \
    com.example.release-date="2015-02-12"

    View full-size slide

  22. $ docker inspect 4fa6e0f0c678
    ...
    "Labels": {
    "vendor": "ACME Incorporated",
    "com.example.is-beta": "",
    "com.example.version": "0.0.1-beta",
    "com.example.release-date": "2015-02-12"
    }
    ... Access labels via inspect

    View full-size slide

  23. Containers can have
    additional labels too

    View full-size slide

  24. Add labels at docker run time
    $ docker run \
    -d \
    --label com.example.group="webservers" \
    --label com.example.environment="production" \
    busybox \
    top

    View full-size slide

  25. Query based on labels
    with filters

    View full-size slide

  26. Filter images by label
    $ docker images --filter "label=com.example.is-beta"

    View full-size slide

  27. Filter containers by label
    $ docker ps --filter "label=com.example.is-beta"

    View full-size slide

  28. Added in v1.17
    Lots of label support in the API

    View full-size slide

  29. But what to store in
    Labels?

    View full-size slide

  30. A quick aside
    Package Managers

    View full-size slide

  31. I like system packages

    View full-size slide

  32. The power of system
    packages lies not in the
    file format but in the
    metadata

    View full-size slide

  33. DPKG and RPM

    View full-size slide

  34. Debian New Maintainers’ Guide

    View full-size slide

  35. Fedora packaging guidelines

    View full-size slide

  36. Summary: A CD player app that rocks!
    Name: cdplayer
    Version: 1.0
    Release: 1
    Copyright: GPL
    Group: Applications/Sound
    Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer
    URL: http://www.gnomovision.com/cdplayer/cdplayer.html
    Distribution: WSS Linux
    Vendor: White Socks Software, Inc.
    Packager: Santa Claus
    %description
    It slices! It dices! It's a CD player app that
    can't be beat. By using the resonant frequency
    of the CD itself, it is able to simulate 20X
    Example RPM spec file

    View full-size slide

  37. Given standard
    metadata what
    can we do?

    View full-size slide

  38. $ dpkg -L lynx
    /.
    /usr
    /usr/share
    /usr/share/doc
    /usr/share/doc/lynx
    /usr/share/doc/lynx/copyright
    /usr/share/doc/lynx/changelog.gz
    /usr/share/doc/lynx/changelog.Debian.gz
    List files from package

    View full-size slide

  39. What installed that file?
    $ rpm -qf /usr/bin/mysqlaccess
    MySQL-client-3.23.57-1

    View full-size slide

  40. Find unmet dependencies
    $ apt-cache unmet
    Package libdataobjects-sqlite3-ruby1.9.1 version
    0.10.1.1-1 has an unmet dep:
    Depends: libdataobjects-ruby1.9

    View full-size slide

  41. $ rpm -qdf /usr/bin/mysqlaccess
    /usr/share/man/man1/mysql.1.gz
    /usr/share/man/man1/mysqlaccess.1.gz
    /usr/share/man/man1/mysqladmin.1.gz
    /usr/share/man/man1/mysqldump.1.gz
    /usr/share/man/man1/mysqlshow.1.gz
    Find documentation

    View full-size slide

  42. $ apticron
    The following packages are currently pending an
    upgrade:
    xfree86-common 4.3.0.dfsg.1-14sarge3
    libice6 4.3.0.dfsg.1-14sarge3
    libsm6 4.3.0.dfsg.1-14sarge3
    xlibs-data 4.3.0.dfsg.1-14sarge3
    libx11-6 4.3.0.dfsg.1-14sarge3
    libxext6 4.3.0.dfsg.1-14sarge3
    libxpm4 4.3.0.dfsg.1-14sarge3
    Find outdated packages

    View full-size slide

  43. Standards
    The power of agreement

    View full-size slide

  44. Docker official label guidance

    View full-size slide

  45. All (third-party) tools should
    prefix their keys with the
    reverse DNS notation of a
    domain controlled by the
    author. For example,
    com.example.some-label.
    1

    View full-size slide

  46. The com.docker.*, io.docker.*
    and org.dockerproject.*
    namespaces are reserved for
    Docker’s internal use. 2

    View full-size slide

  47. Keys should only consist of
    lower-cased alphanumeric
    characters, dots and dashes
    (for example, [a-z0-9-.]). 3

    View full-size slide

  48. Keys should start and end
    with an alpha numeric
    character.
    4

    View full-size slide

  49. Keys may not contain
    consecutive dots or dashes.
    5

    View full-size slide

  50. Keys without namespace
    (dots) are reserved for CLI use.
    6

    View full-size slide

  51. How widely adhered to?

    View full-size slide

  52. < 20% from a small sample

    View full-size slide

  53. But some folks care

    View full-size slide

  54. Unify the labels for Docker images

    View full-size slide

  55. The problem with
    inconsistent metadata

    View full-size slide

  56. Without complete
    metadata we can’t trust
    the tools built on top

    View full-size slide

  57. Docker Label Inspector

    View full-size slide

  58. Check against Docker guidelines
    $ dli lint
    ========> Check all labels have namespaces
    [WARN] Label 'vendor' should use a namespace based
    on reverse DNS notation
    ========> Check labels don't use reserved namespaces
    ========> Check labels only use valid characters
    ========> Check labels start and end with alpanumeric
    characters
    ========> Check labels for double dots and dashes

    View full-size slide

  59. $ dli validate
    ========> Check labels based on schema in 'schema.json'
    [ERROR] u'com.example.is-beta' is a required property
    Check against a schema

    View full-size slide

  60. {
    "title": "Dockerfile schema",
    "type": "object",
    "properties": {
    "com.example.release-date": {
    "type": "string"
    },
    "com.example.is-beta": {
    "type": "string"
    },
    "com.example.version": {
    "description": "Version",
    "type": "integer",
    "minimum": 0
    }
    },
    "required": ["com.example.is-beta",
    "com.example.version"]
    }
    Define labels in JSON schema

    View full-size slide

  61. Stand well back
    LIVE DEMO

    View full-size slide

  62. Runtime Metadata
    A missing piece, and some ideas

    View full-size slide

  63. What temperature is a
    refrigerated shipping
    containers at?
    *Not all shipping containers are refrigerated

    View full-size slide

  64. docker exec as an API

    View full-size slide

  65. Dockerfile example
    FROM alpine
    LABEL net.morethanseven.dockerfile="/Dockerfile" \
    net.morethanseven.exec.packages="apk info -vv"
    RUN apk add --update bash && rm -rf /var/cache/apk/*
    COPY Dockerfile /

    View full-size slide

  66. Discover out API
    $ docker inspect -f "{{json .Config.Labels }}" \
    garethr/alpine \
    | jq
    {
    "net.morethanseven.dockerfile": "/Dockerfile",
    "net.morethanseven.exec.packages": "apk info -vv"
    }

    View full-size slide

  67. Read the Dockerfile
    $ docker run -i -t garethr/alpine cat /Dockerfile
    FROM alpine
    LABEL net.morethanseven.dockerfile="/Dockerfile" \
    net.morethanseven.exec.packages="apk info -vv"
    RUN apk add --update bash && rm -rf /var/cache/apk/*
    COPY Dockerfile /

    View full-size slide

  68. $ docker run -i -t garethr/alpine apk info -vv
    musl-1.1.11-r2 - the musl c library (libc) implementati
    busybox-1.23.2-r0 - Size optimized toolbox of many comm
    alpine-baselayout-2.3.2-r0 - Alpine base dir structure
    openrc-0.15.1-r3 - OpenRC manages the services, startup
    alpine-conf-3.2.1-r6 - Alpine configuration management
    List installed packages

    View full-size slide

  69. Second time lucky
    LIVE DEMO

    View full-size slide

  70. Tooling
    What could we build atop our metadata?

    View full-size slide

  71. Documentation
    discovery

    View full-size slide

  72. License verification

    View full-size slide

  73. Links to source code or
    release notes

    View full-size slide

  74. Automatically
    generated interfaces

    View full-size slide

  75. Package search

    View full-size slide

  76. Tempting fate
    LIVE DEMO

    View full-size slide

  77. Conclusions
    If all you remember is…

    View full-size slide

  78. Step 1
    Step 2
    Step 3
    Metadata!
    Something…
    Profit

    View full-size slide

  79. Share schemas and
    namespaces

    View full-size slide

  80. Build agreement

    View full-size slide

  81. Build tooling

    View full-size slide

  82. Extract standards

    View full-size slide

  83. Thank you!
    Gareth Rushgrove
    @garethr
    [email protected]

    View full-size slide