Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's Inside That Container?

What's Inside That Container?

My talk from Configuration Management Camp 2017. A look at the reality of container usage in the wild, what problems that leads to in operations, and maybe a few ideas of things we can do or build to help.

Gareth Rushgrove

February 06, 2017
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. (without introducing more risk)
    What's Inside That
    Container?
    Puppet
    Gareth Rushgrove
    Containers and config management in the real world

    View Slide

  2. (without introducing more risk)
    @garethr

    View Slide

  3. (without introducing more risk)
    Gareth Rushgrove

    View Slide

  4. (without introducing more risk)
    What we’ll cover
    This talk

    View Slide

  5. - What is configuration management?
    - Docker base image usage
    - The problem with containers as black boxes
    - Ideas and demos
    Gareth Rushgrove

    View Slide

  6. (without introducing more risk)
    Useful background
    What is Configuration
    Management?

    View Slide

  7. - 1950s research
    - 1960s 480 series
    - 1991 MIL-HDBK-61
    - 1998 ANSI-EIA-649
    Gareth Rushgrove

    View Slide

  8. - Identification
    - Control
    - Status accounting
    - Verification and audit
    Gareth Rushgrove
    Military Handbook Configuration Management Guidance MIL-HDBK-61B

    View Slide

  9. Configuration management verifies
    that a system is identified and
    documented in sufficient detail
    Gareth Rushgrove
    National Consensus Standard for Configuration Management EIA-649

    View Slide

  10. Configuration management verifies
    that a system performs as intended
    Gareth Rushgrove
    National Consensus Standard for Configuration Management EIA-649

    View Slide

  11. NOTE: Not a tool, a practice
    Gareth Rushgrove

    View Slide

  12. (without introducing more risk)
    And where to find them
    Docker Base Images

    View Slide

  13. (without introducing more risk)
    Gareth Rushgrove
    Docker Hub API
    $ curl -s https://registry.hub.docker.com/v2/...
    ...repositories/library/ubuntu/ | jq .pull_count

    View Slide

  14. Gareth Rushgrove
    Image downloads

    View Slide

  15. What about other popular official
    images? Node (32,523,647), Java
    (16,635,049), etc. Currently based
    on Debian too
    Gareth Rushgrove

    View Slide

  16. Gareth Rushgrove
    Image size

    View Slide

  17. (without introducing more risk)
    Gareth Rushgrove
    Count files
    $ find -maxdepth 1 -type d | while read -r
    dir; do printf "%s:\t" "$dir"; find "$dir"
    -type f | wc -l; done

    View Slide

  18. (without introducing more risk)
    Gareth Rushgrove
    Count packages
    $ dpkg -l | grep ^ii | wc -l
    $ dnf list installed
    $ rpm -qa | wc -l
    $ apk info | wc -l

    View Slide

  19. Gareth Rushgrove
    Image contents

    View Slide

  20. (without introducing more risk)
    Thanks David Gageot

    View Slide

  21. (without introducing more risk)
    Gareth Rushgrove
    Popular on GitHub
    SELECT RTRIM(LTRIM(SUBSTR(line, 5))) AS line_group.base_image
    FROM (
    SELECT SPLIT(dockerfile, '\n') AS line
    FROM [github.dockerfiles_content]
    HAVING LEFT(line, 5) = 'FROM '
    )
    SELECT image, count(*) AS count
    FROM (
    SELECT FIRST(SPLIT(line_group.base_image, ':')) AS image
    FROM [github.images]
    )
    GROUP BY image
    ORDER BY count DESC

    View Slide

  22. Gareth Rushgrove
    Image GitHub popularity

    View Slide

  23. Gareth Rushgrove
    Usage growth

    View Slide

  24. (without introducing more risk)
    Thanks to Microbadger

    View Slide

  25. Gareth Rushgrove
    Hub image sample

    View Slide

  26. Gareth Rushgrove

    View Slide

  27. The majority of people using Docker
    are using images containing an entire
    operating system filesystem
    Gareth Rushgrove

    View Slide

  28. Alpine usage is growing more rapidly
    than others, but starting from a much
    smaller install base
    Gareth Rushgrove

    View Slide

  29. Scratch, or other approaches like Nix,
    appear to occupy a small niche
    Gareth Rushgrove

    View Slide

  30. Windows and Windows Nano images
    will undoubtedly become more
    common over the next year or so
    Gareth Rushgrove

    View Slide

  31. Debian derivatives account for the
    majority of images today
    Gareth Rushgrove

    View Slide

  32. (without introducing more risk)
    What does this all mean to me?
    Problems

    View Slide

  33. (without introducing more risk)
    You don’t know, and that’s a problem

    View Slide

  34. (without introducing more risk)
    Visibility and control are critical

    View Slide

  35. (without introducing more risk)
    What you don’t know can hurt you

    View Slide

  36. (without introducing more risk)
    Vulnerable images on Docker Hub

    View Slide

  37. (without introducing more risk)
    Not quite so simple

    View Slide

  38. Can you tell me all the versions of
    OpenSSL you have in production
    right now?
    Gareth Rushgrove

    View Slide

  39. Containers are a black box
    Gareth Rushgrove

    View Slide

  40. Containers are a black box from the
    point of view of the scheduler
    Gareth Rushgrove

    View Slide

  41. Containers are NOT a black box from
    the point of view of the operator
    Gareth Rushgrove

    View Slide

  42. These are all generally configuration
    management problems; identification,
    control, status accounting, verification
    and audit
    Gareth Rushgrove

    View Slide

  43. (without introducing more risk)
    Experiments and proof-of-concept work
    Ideas

    View Slide

  44. Immutability means we need to know
    what we put inside the box
    Gareth Rushgrove

    View Slide

  45. (without introducing more risk)
    Tangent about immutability

    View Slide

  46. It’s not enough to say ubuntu:16.04
    Gareth Rushgrove

    View Slide

  47. (without introducing more risk)
    LIVE DEMOS

    View Slide

  48. (without introducing more risk)
    Gareth Rushgrove
    Embed inventory in image
    $ add-container-inventory centos garethr/centos-inventory
    ----> Using existing puppet-inventory volume
    ----> Generating inventory for image centos
    ----> Saving inventory to temporary image inventory-21042
    ----> Committing new image to garethr/centos-inventory
    sha256:f84b3655252c946dfb888de2d74348afed97ef89d817e469adad3a
    ----> Cleaning up

    View Slide

  49. (without introducing more risk)
    https://gist.github.com/garethr/
    922f6374015b59e0f6cd007f8b34eedf

    View Slide

  50. (without introducing more risk)
    Gareth Rushgrove
    Read image inventory data
    $ docker run --rm garethr/centos-inventory cat /inventory.json |
    {
    {
    "title": "rpm",
    "resource": "package",
    "provider": “yum",
    "versions": [
    "4.11.3-17.el7"
    ]
    },
    {
    "title": “libuser",
    "resource": "package",
    "provider": “yum",

    View Slide

  51. (without introducing more risk)
    Gareth Rushgrove
    Runtime and multi-OS
    $ docker exec an-opensuse-container cat /inventory.json | jq
    {
    {
    "title": "netcfg",
    "resource": "package",
    "provider": "zypper",
    "versions": [
    "11.5-27.2"
    ]
    },
    {
    "title": "shadow",
    "resource": "package",
    "provider": "zypper",

    View Slide

  52. GIVEN all my containers now have an inventory
    WHAT things can I do with it?
    Gareth Rushgrove

    View Slide

  53. Asking the black box what OS
    it contains?
    Gareth Rushgrove

    View Slide

  54. (without introducing more risk)
    Gareth Rushgrove
    Use jq to query inventory
    $ docker run --rm garethr/centos-inventory \
    cat /inventory.json |
    jq '.facts.operatingsystem'
    "CentOS"

    View Slide

  55. What about a package search engine?
    Gareth Rushgrove

    View Slide

  56. (without introducing more risk)
    Gareth Rushgrove
    Package search
    $ search-inventory rpm
    festive_edison rpm 4.11.3-17.el7 yum
    optimistic_shockley rpm 4.11.3-17.el7 yum
    condescending_swartz rpm 4.11.2-10.1 zypper
    festive_edison rpm-libs 4.11.3-17.el7 yum
    festive_edison rpm-python 4.11.3-17.el7 yum
    optimistic_shockley rpm-libs 4.11.3-17.el7 yum
    optimistic_shockley rpm-python 4.11.3-17.el7 yum
    festive_edison rpm-build-libs 4.11.3-17.el7 yum
    optimistic_shockley rpm-build-libs 4.11.3-17.el7 yum
    $ search-inventory package:openssl,provider:yum
    festive_edison openssl-libs 1:1.0.1e-51.el7_2.5 yum
    optimistic_shockley openssl-libs 1:1.0.1e-51.el7_2.5 yum

    View Slide

  57. (without introducing more risk)
    Search packages across containers

    View Slide

  58. (without introducing more risk)
    Gareth Rushgrove
    Search in BigQuery
    SELECT
    resources.title AS package,
    resources.versions AS version,
    facts.hostname AS hostname,
    facts.operatingsystem AS operatingsystem
    FROM
    inventory.sample
    WHERE
    resources.resource="package"
    AND resources.title="openssl"

    View Slide

  59. checking containers for CVEs?
    Gareth Rushgrove

    View Slide

  60. (without introducing more risk)
    Red Hat Security Data API

    View Slide

  61. (without introducing more risk)
    Gareth Rushgrove
    CVE scanner
    $ scan-containers-for-cves python
    ---> Scanning package: python
    ---> CVEs found for python-2.7.5-34.el7 in festive_edison
    +---------------+----------+------------+-------------------+
    | CVE | Severity | CVSS score | Date |
    +---------------+----------+------------+-------------------+
    | CVE-2014-4650 | moderate | 5.0 | 23 June 2014 |
    | CVE-2013-1752 | moderate | 4.3 | 25 September 2012 |
    | CVE-2013-1753 | moderate | 4.3 | 25 September 2012 |
    | CVE-2014-7185 | low | 4.0 | 23 June 2014 |
    | CVE-2014-4616 | moderate | 4.0 | 19 May 2014 |
    +---------------+----------+------------+-------------------+
    ---> Scanning package: python
    ---> CVEs found for python-2.7.5-34.el7 in optimistic_shockley
    +---------------+----------+------------+-------------------+

    View Slide

  62. (without introducing more risk)
    https://gist.github.com/garethr/
    d4f6e23ff19939e4877f441434ede8da

    View Slide

  63. (without introducing more risk)
    If all you remember is
    Conclusions

    View Slide

  64. Operators treating containers as black
    boxes are going to have a bad time
    Gareth Rushgrove

    View Slide

  65. Beware the difference between
    the purity of linux containers and
    the pragmatic reality of Docker
    base images
    Gareth Rushgrove

    View Slide

  66. When using containers we
    need to reconsider solutions to
    previously solved configuration
    management problems
    Gareth Rushgrove

    View Slide

  67. (without introducing more risk)
    Questions?
    And thanks for listening

    View Slide