What's Inside That Container?

What's Inside That Container?

My talk from Configuration Management Camp 2017. A look at the reality of container usage in the wild, what problems that leads to in operations, and maybe a few ideas of things we can do or build to help.

98234c645fe8c935edc0fec0186d28b8?s=128

Gareth Rushgrove

February 06, 2017
Tweet

Transcript

  1. (without introducing more risk) What's Inside That Container? Puppet Gareth

    Rushgrove Containers and config management in the real world
  2. (without introducing more risk) @garethr

  3. (without introducing more risk) Gareth Rushgrove

  4. (without introducing more risk) What we’ll cover This talk

  5. - What is configuration management? - Docker base image usage

    - The problem with containers as black boxes - Ideas and demos Gareth Rushgrove
  6. (without introducing more risk) Useful background What is Configuration Management?

  7. - 1950s research - 1960s 480 series - 1991 MIL-HDBK-61

    - 1998 ANSI-EIA-649 Gareth Rushgrove
  8. - Identification - Control - Status accounting - Verification and

    audit Gareth Rushgrove Military Handbook Configuration Management Guidance MIL-HDBK-61B
  9. Configuration management verifies that a system is identified and documented

    in sufficient detail Gareth Rushgrove National Consensus Standard for Configuration Management EIA-649
  10. Configuration management verifies that a system performs as intended Gareth

    Rushgrove National Consensus Standard for Configuration Management EIA-649
  11. NOTE: Not a tool, a practice Gareth Rushgrove

  12. (without introducing more risk) And where to find them Docker

    Base Images
  13. (without introducing more risk) Gareth Rushgrove Docker Hub API $

    curl -s https://registry.hub.docker.com/v2/... ...repositories/library/ubuntu/ | jq .pull_count
  14. Gareth Rushgrove Image downloads

  15. What about other popular official images? Node (32,523,647), Java (16,635,049),

    etc. Currently based on Debian too Gareth Rushgrove
  16. Gareth Rushgrove Image size

  17. (without introducing more risk) Gareth Rushgrove Count files $ find

    -maxdepth 1 -type d | while read -r dir; do printf "%s:\t" "$dir"; find "$dir" -type f | wc -l; done
  18. (without introducing more risk) Gareth Rushgrove Count packages $ dpkg

    -l | grep ^ii | wc -l $ dnf list installed $ rpm -qa | wc -l $ apk info | wc -l
  19. Gareth Rushgrove Image contents

  20. (without introducing more risk) Thanks David Gageot

  21. (without introducing more risk) Gareth Rushgrove Popular on GitHub SELECT

    RTRIM(LTRIM(SUBSTR(line, 5))) AS line_group.base_image FROM ( SELECT SPLIT(dockerfile, '\n') AS line FROM [github.dockerfiles_content] HAVING LEFT(line, 5) = 'FROM ' ) SELECT image, count(*) AS count FROM ( SELECT FIRST(SPLIT(line_group.base_image, ':')) AS image FROM [github.images] ) GROUP BY image ORDER BY count DESC
  22. Gareth Rushgrove Image GitHub popularity

  23. Gareth Rushgrove Usage growth

  24. (without introducing more risk) Thanks to Microbadger

  25. Gareth Rushgrove Hub image sample

  26. Gareth Rushgrove

  27. The majority of people using Docker are using images containing

    an entire operating system filesystem Gareth Rushgrove
  28. Alpine usage is growing more rapidly than others, but starting

    from a much smaller install base Gareth Rushgrove
  29. Scratch, or other approaches like Nix, appear to occupy a

    small niche Gareth Rushgrove
  30. Windows and Windows Nano images will undoubtedly become more common

    over the next year or so Gareth Rushgrove
  31. Debian derivatives account for the majority of images today Gareth

    Rushgrove
  32. (without introducing more risk) What does this all mean to

    me? Problems
  33. (without introducing more risk) You don’t know, and that’s a

    problem
  34. (without introducing more risk) Visibility and control are critical

  35. (without introducing more risk) What you don’t know can hurt

    you
  36. (without introducing more risk) Vulnerable images on Docker Hub

  37. (without introducing more risk) Not quite so simple

  38. Can you tell me all the versions of OpenSSL you

    have in production right now? Gareth Rushgrove
  39. Containers are a black box Gareth Rushgrove

  40. Containers are a black box from the point of view

    of the scheduler Gareth Rushgrove
  41. Containers are NOT a black box from the point of

    view of the operator Gareth Rushgrove
  42. These are all generally configuration management problems; identification, control, status

    accounting, verification and audit Gareth Rushgrove
  43. (without introducing more risk) Experiments and proof-of-concept work Ideas

  44. Immutability means we need to know what we put inside

    the box Gareth Rushgrove
  45. (without introducing more risk) Tangent about immutability

  46. It’s not enough to say ubuntu:16.04 Gareth Rushgrove

  47. (without introducing more risk) LIVE DEMOS

  48. (without introducing more risk) Gareth Rushgrove Embed inventory in image

    $ add-container-inventory centos garethr/centos-inventory ----> Using existing puppet-inventory volume ----> Generating inventory for image centos ----> Saving inventory to temporary image inventory-21042 ----> Committing new image to garethr/centos-inventory sha256:f84b3655252c946dfb888de2d74348afed97ef89d817e469adad3a ----> Cleaning up
  49. (without introducing more risk) https://gist.github.com/garethr/ 922f6374015b59e0f6cd007f8b34eedf

  50. (without introducing more risk) Gareth Rushgrove Read image inventory data

    $ docker run --rm garethr/centos-inventory cat /inventory.json | { { "title": "rpm", "resource": "package", "provider": “yum", "versions": [ "4.11.3-17.el7" ] }, { "title": “libuser", "resource": "package", "provider": “yum",
  51. (without introducing more risk) Gareth Rushgrove Runtime and multi-OS $

    docker exec an-opensuse-container cat /inventory.json | jq { { "title": "netcfg", "resource": "package", "provider": "zypper", "versions": [ "11.5-27.2" ] }, { "title": "shadow", "resource": "package", "provider": "zypper",
  52. GIVEN all my containers now have an inventory WHAT things

    can I do with it? Gareth Rushgrove
  53. Asking the black box what OS it contains? Gareth Rushgrove

  54. (without introducing more risk) Gareth Rushgrove Use jq to query

    inventory $ docker run --rm garethr/centos-inventory \ cat /inventory.json | jq '.facts.operatingsystem' "CentOS"
  55. What about a package search engine? Gareth Rushgrove

  56. (without introducing more risk) Gareth Rushgrove Package search $ search-inventory

    rpm festive_edison rpm 4.11.3-17.el7 yum optimistic_shockley rpm 4.11.3-17.el7 yum condescending_swartz rpm 4.11.2-10.1 zypper festive_edison rpm-libs 4.11.3-17.el7 yum festive_edison rpm-python 4.11.3-17.el7 yum optimistic_shockley rpm-libs 4.11.3-17.el7 yum optimistic_shockley rpm-python 4.11.3-17.el7 yum festive_edison rpm-build-libs 4.11.3-17.el7 yum optimistic_shockley rpm-build-libs 4.11.3-17.el7 yum $ search-inventory package:openssl,provider:yum festive_edison openssl-libs 1:1.0.1e-51.el7_2.5 yum optimistic_shockley openssl-libs 1:1.0.1e-51.el7_2.5 yum
  57. (without introducing more risk) Search packages across containers

  58. (without introducing more risk) Gareth Rushgrove Search in BigQuery SELECT

    resources.title AS package, resources.versions AS version, facts.hostname AS hostname, facts.operatingsystem AS operatingsystem FROM inventory.sample WHERE resources.resource="package" AND resources.title="openssl"
  59. checking containers for CVEs? Gareth Rushgrove

  60. (without introducing more risk) Red Hat Security Data API

  61. (without introducing more risk) Gareth Rushgrove CVE scanner $ scan-containers-for-cves

    python ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in festive_edison +---------------+----------+------------+-------------------+ | CVE | Severity | CVSS score | Date | +---------------+----------+------------+-------------------+ | CVE-2014-4650 | moderate | 5.0 | 23 June 2014 | | CVE-2013-1752 | moderate | 4.3 | 25 September 2012 | | CVE-2013-1753 | moderate | 4.3 | 25 September 2012 | | CVE-2014-7185 | low | 4.0 | 23 June 2014 | | CVE-2014-4616 | moderate | 4.0 | 19 May 2014 | +---------------+----------+------------+-------------------+ ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in optimistic_shockley +---------------+----------+------------+-------------------+
  62. (without introducing more risk) https://gist.github.com/garethr/ d4f6e23ff19939e4877f441434ede8da

  63. (without introducing more risk) If all you remember is Conclusions

  64. Operators treating containers as black boxes are going to have

    a bad time Gareth Rushgrove
  65. Beware the difference between the purity of linux containers and

    the pragmatic reality of Docker base images Gareth Rushgrove
  66. When using containers we need to reconsider solutions to previously

    solved configuration management problems Gareth Rushgrove
  67. (without introducing more risk) Questions? And thanks for listening