Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Shipping Manifests, Bill of Lading and Docker - Metadata and Container

Talk from #dockercon EU in Barcelona, all about Docker labels and building higher level tools on top of metadata

Gareth Rushgrove

November 16, 2015
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Shipping Manifests, Bill of Lading and Docker
    Metadata and Container
    Gareth Rushgrove
    Senior Software Engineer, Puppet Labs

    View Slide

  2. @garethr

    View Slide

  3. Senior Engineer at Puppet Labs

    View Slide

  4. Creator of the Puppet Docker module

    View Slide

  5. This Talk
    The introduction

    View Slide

  6. Shipping containers are cool

    View Slide

  7. But nothing without all the
    paper work

    View Slide

  8. A manifest or ship's manifest
    is a document listing the
    cargo, passengers, and crew
    of a ship, aircraft, or vehicle,
    for the use of customs and
    other officials.

    View Slide

  9. A bill of lading is a
    document issued by a carrier
    which details a shipment of
    merchandise and gives title
    of that shipment to a
    specified party.

    View Slide

  10. State of the Software Supply Chain

    View Slide

  11. —State of the Software
    Supply Chain 2015
    A once safe component
    may be found to be
    vulnerable at any time”

    View Slide

  12. —State of the Software
    Supply Chain 2015
    Defective components…
    end up in our software
    largely unnoticed”

    View Slide

  13. Lets apply the same
    principles to a different type
    of container

    View Slide

  14. Docker Labels
    Docker builtin metadata capabilities

    View Slide

  15. Added in 1.6
    One meta data to rule them all

    View Slide

  16. Labels on Docker
    Engines

    View Slide

  17. Provide information about the host
    $ docker daemon \
    --label com.example.environment="production" \
    --label com.example.storage="ssd"

    View Slide

  18. Labels to guide Swarm scheduling
    $ docker run -d -P \
    -e constraint:storage==ssd --name db mysql

    View Slide

  19. Labels on Docker
    images

    View Slide

  20. LABEL [.][=] ...
    Dockerfile Label instruction

    View Slide

  21. Don’t do this - new layer per label
    LABEL vendor=ACME\ Incorporated
    LABEL com.example.version.is-beta
    LABEL com.example.version="0.0.1-beta"
    LABEL com.example.release-date="2015-02-12"

    View Slide

  22. Better - only one layer
    LABEL vendor="ACME\ Incorporated" \
    com.example.is-beta \
    com.example.version="0.0.1-beta" \
    com.example.release-date="2015-02-12"

    View Slide

  23. $ docker inspect 4fa6e0f0c678
    ...
    "Labels": {
    "vendor": "ACME Incorporated",
    "com.example.is-beta": "",
    "com.example.version": "0.0.1-beta",
    "com.example.release-date": "2015-02-12"
    }
    ... Access labels via inspect

    View Slide

  24. Containers can have
    additional labels too

    View Slide

  25. Add labels at docker run time
    $ docker run \
    -d \
    --label com.example.group="webservers" \
    --label com.example.environment="production" \
    busybox \
    top

    View Slide

  26. Query based on labels
    with filters

    View Slide

  27. Filter images by label
    $ docker images --filter "label=com.example.is-beta"

    View Slide

  28. Filter containers by label
    $ docker ps --filter "label=com.example.is-beta"

    View Slide

  29. Added in v1.17
    Lots of label support in the API

    View Slide

  30. But what to store in
    Labels?

    View Slide

  31. A quick aside
    Package Managers

    View Slide

  32. I like system packages

    View Slide

  33. The power of system
    packages lies not in the
    file format but in the
    metadata

    View Slide

  34. DPKG and RPM

    View Slide

  35. Debian New Maintainers’ Guide

    View Slide

  36. Fedora packaging guidelines

    View Slide

  37. Summary: A CD player app that rocks!
    Name: cdplayer
    Version: 1.0
    Release: 1
    Copyright: GPL
    Group: Applications/Sound
    Source: ftp://ftp.gnomovision.com/pub/cdplayer/cdplayer
    URL: http://www.gnomovision.com/cdplayer/cdplayer.html
    Distribution: WSS Linux
    Vendor: White Socks Software, Inc.
    Packager: Santa Claus
    %description
    It slices! It dices! It's a CD player app that
    can't be beat. By using the resonant frequency
    of the CD itself, it is able to simulate 20X
    Example RPM spec file

    View Slide

  38. Given standard
    metadata what
    can we do?

    View Slide

  39. $ dpkg -L lynx
    /.
    /usr
    /usr/share
    /usr/share/doc
    /usr/share/doc/lynx
    /usr/share/doc/lynx/copyright
    /usr/share/doc/lynx/changelog.gz
    /usr/share/doc/lynx/changelog.Debian.gz
    List files from package

    View Slide

  40. What installed that file?
    $ rpm -qf /usr/bin/mysqlaccess
    MySQL-client-3.23.57-1

    View Slide

  41. Find unmet dependencies
    $ apt-cache unmet
    Package libdataobjects-sqlite3-ruby1.9.1 version
    0.10.1.1-1 has an unmet dep:
    Depends: libdataobjects-ruby1.9

    View Slide

  42. $ rpm -qdf /usr/bin/mysqlaccess
    /usr/share/man/man1/mysql.1.gz
    /usr/share/man/man1/mysqlaccess.1.gz
    /usr/share/man/man1/mysqladmin.1.gz
    /usr/share/man/man1/mysqldump.1.gz
    /usr/share/man/man1/mysqlshow.1.gz
    Find documentation

    View Slide

  43. $ apticron
    The following packages are currently pending an
    upgrade:
    xfree86-common 4.3.0.dfsg.1-14sarge3
    libice6 4.3.0.dfsg.1-14sarge3
    libsm6 4.3.0.dfsg.1-14sarge3
    xlibs-data 4.3.0.dfsg.1-14sarge3
    libx11-6 4.3.0.dfsg.1-14sarge3
    libxext6 4.3.0.dfsg.1-14sarge3
    libxpm4 4.3.0.dfsg.1-14sarge3
    Find outdated packages

    View Slide

  44. Standards
    The power of agreement

    View Slide

  45. Docker official label guidance

    View Slide

  46. All (third-party) tools should
    prefix their keys with the
    reverse DNS notation of a
    domain controlled by the
    author. For example,
    com.example.some-label.
    1

    View Slide

  47. The com.docker.*, io.docker.*
    and org.dockerproject.*
    namespaces are reserved for
    Docker’s internal use. 2

    View Slide

  48. Keys should only consist of
    lower-cased alphanumeric
    characters, dots and dashes
    (for example, [a-z0-9-.]). 3

    View Slide

  49. Keys should start and end
    with an alpha numeric
    character.
    4

    View Slide

  50. Keys may not contain
    consecutive dots or dashes.
    5

    View Slide

  51. Keys without namespace
    (dots) are reserved for CLI use.
    6

    View Slide

  52. How widely adhered to?

    View Slide

  53. < 20% from a small sample

    View Slide

  54. But some folks care

    View Slide

  55. Unify the labels for Docker images

    View Slide

  56. Merged

    View Slide

  57. The problem with
    inconsistent metadata

    View Slide

  58. Without complete
    metadata we can’t trust
    the tools built on top

    View Slide

  59. Docker Label Inspector

    View Slide

  60. Check against Docker guidelines
    $ dli lint
    ========> Check all labels have namespaces
    [WARN] Label 'vendor' should use a namespace based
    on reverse DNS notation
    ========> Check labels don't use reserved namespaces
    ========> Check labels only use valid characters
    ========> Check labels start and end with alpanumeric
    characters
    ========> Check labels for double dots and dashes

    View Slide

  61. $ dli validate
    ========> Check labels based on schema in 'schema.json'
    [ERROR] u'com.example.is-beta' is a required property
    Check against a schema

    View Slide

  62. {
    "title": "Dockerfile schema",
    "type": "object",
    "properties": {
    "com.example.release-date": {
    "type": "string"
    },
    "com.example.is-beta": {
    "type": "string"
    },
    "com.example.version": {
    "description": "Version",
    "type": "integer",
    "minimum": 0
    }
    },
    "required": ["com.example.is-beta",
    "com.example.version"]
    }
    Define labels in JSON schema

    View Slide

  63. Stand well back
    LIVE DEMO

    View Slide

  64. Runtime Metadata
    A missing piece, and some ideas

    View Slide

  65. What temperature is a
    refrigerated shipping
    containers at?
    *Not all shipping containers are refrigerated

    View Slide

  66. docker exec as an API

    View Slide

  67. Dockerfile example
    FROM alpine
    LABEL net.morethanseven.dockerfile="/Dockerfile" \
    net.morethanseven.exec.packages="apk info -vv"
    RUN apk add --update bash && rm -rf /var/cache/apk/*
    COPY Dockerfile /

    View Slide

  68. Discover out API
    $ docker inspect -f "{{json .Config.Labels }}" \
    garethr/alpine \
    | jq
    {
    "net.morethanseven.dockerfile": "/Dockerfile",
    "net.morethanseven.exec.packages": "apk info -vv"
    }

    View Slide

  69. Read the Dockerfile
    $ docker run -i -t garethr/alpine cat /Dockerfile
    FROM alpine
    LABEL net.morethanseven.dockerfile="/Dockerfile" \
    net.morethanseven.exec.packages="apk info -vv"
    RUN apk add --update bash && rm -rf /var/cache/apk/*
    COPY Dockerfile /

    View Slide

  70. $ docker run -i -t garethr/alpine apk info -vv
    musl-1.1.11-r2 - the musl c library (libc) implementati
    busybox-1.23.2-r0 - Size optimized toolbox of many comm
    alpine-baselayout-2.3.2-r0 - Alpine base dir structure
    openrc-0.15.1-r3 - OpenRC manages the services, startup
    alpine-conf-3.2.1-r6 - Alpine configuration management
    List installed packages

    View Slide

  71. Second time lucky
    LIVE DEMO

    View Slide

  72. R.I.Pienaar

    View Slide

  73. Tooling
    What could we build atop our metadata?

    View Slide

  74. Documentation
    discovery

    View Slide

  75. License verification

    View Slide

  76. Links to source code or
    release notes

    View Slide

  77. Automatically
    generated interfaces

    View Slide

  78. Package search

    View Slide

  79. Tempting fate
    LIVE DEMO

    View Slide

  80. Conclusions
    If all you remember is…

    View Slide

  81. Step 1
    Step 2
    Step 3
    Metadata!
    Something…
    Profit

    View Slide

  82. Share schemas and
    namespaces

    View Slide

  83. Build agreement

    View Slide

  84. Build tooling

    View Slide

  85. Extract standards

    View Slide

  86. Thank you!
    Gareth Rushgrove
    @garethr
    [email protected]

    View Slide