(without introducing more risk) What's Inside That Container? Puppet Gareth Rushgrove Containers and config management in the real world

(without introducing more risk) @garethr

(without introducing more risk) Gareth Rushgrove

(without introducing more risk) What we’ll cover This talk

- What is configuration management? - Docker base image usage - The problem with containers as black boxes - Ideas and demos Gareth Rushgrove

(without introducing more risk) Useful background What is Configuration Management?

- 1950s research - 1960s 480 series - 1991 MIL-HDBK-61 - 1998 ANSI-EIA-649 Gareth Rushgrove

- Identification - Control - Status accounting - Verification and audit Gareth Rushgrove Military Handbook Configuration Management Guidance MIL-HDBK-61B

Configuration management verifies that a system is identified and documented in sufficient detail Gareth Rushgrove National Consensus Standard for Configuration Management EIA-649

Configuration management verifies that a system performs as intended Gareth Rushgrove National Consensus Standard for Configuration Management EIA-649

NOTE: Not a tool, a practice Gareth Rushgrove

(without introducing more risk) And where to find them Docker Base Images

(without introducing more risk) Gareth Rushgrove Docker Hub API $ curl -s https://registry.hub.docker.com/v2/... ...repositories/library/ubuntu/ | jq .pull_count

Gareth Rushgrove Image downloads

What about other popular official images? Node (32,523,647), Java (16,635,049), etc. Currently based on Debian too Gareth Rushgrove

Gareth Rushgrove Image size

(without introducing more risk) Gareth Rushgrove Count files $ find -maxdepth 1 -type d | while read -r dir; do printf "%s:\t" "$dir"; find "$dir" -type f | wc -l; done

(without introducing more risk) Gareth Rushgrove Count packages $ dpkg -l | grep ^ii | wc -l $ dnf list installed $ rpm -qa | wc -l $ apk info | wc -l

Gareth Rushgrove Image contents

(without introducing more risk) Thanks David Gageot

(without introducing more risk) Gareth Rushgrove Popular on GitHub SELECT RTRIM(LTRIM(SUBSTR(line, 5))) AS line_group.base_image FROM ( SELECT SPLIT(dockerfile, '\n') AS line FROM [github.dockerfiles_content] HAVING LEFT(line, 5) = 'FROM ' ) SELECT image, count(*) AS count FROM ( SELECT FIRST(SPLIT(line_group.base_image, ':')) AS image FROM [github.images] ) GROUP BY image ORDER BY count DESC

Gareth Rushgrove Image GitHub popularity

Gareth Rushgrove Usage growth

(without introducing more risk) Thanks to Microbadger

Gareth Rushgrove Hub image sample

Gareth Rushgrove

The majority of people using Docker are using images containing an entire operating system filesystem Gareth Rushgrove

Alpine usage is growing more rapidly than others, but starting from a much smaller install base Gareth Rushgrove

Scratch, or other approaches like Nix, appear to occupy a small niche Gareth Rushgrove

Windows and Windows Nano images will undoubtedly become more common over the next year or so Gareth Rushgrove

Debian derivatives account for the majority of images today Gareth Rushgrove

(without introducing more risk) What does this all mean to me? Problems

(without introducing more risk) You don’t know, and that’s a problem

(without introducing more risk) Visibility and control are critical

(without introducing more risk) What you don’t know can hurt you

(without introducing more risk) Vulnerable images on Docker Hub

(without introducing more risk) Not quite so simple

Can you tell me all the versions of OpenSSL you have in production right now? Gareth Rushgrove

Containers are a black box Gareth Rushgrove

Containers are a black box from the point of view of the scheduler Gareth Rushgrove

Containers are NOT a black box from the point of view of the operator Gareth Rushgrove

These are all generally configuration management problems; identification, control, status accounting, verification and audit Gareth Rushgrove

(without introducing more risk) Experiments and proof-of-concept work Ideas

Immutability means we need to know what we put inside the box Gareth Rushgrove

(without introducing more risk) Tangent about immutability

It’s not enough to say ubuntu:16.04 Gareth Rushgrove

(without introducing more risk) LIVE DEMOS

(without introducing more risk) Gareth Rushgrove Embed inventory in image $ add-container-inventory centos garethr/centos-inventory ----> Using existing puppet-inventory volume ----> Generating inventory for image centos ----> Saving inventory to temporary image inventory-21042 ----> Committing new image to garethr/centos-inventory sha256:f84b3655252c946dfb888de2d74348afed97ef89d817e469adad3a ----> Cleaning up

(without introducing more risk) https://gist.github.com/garethr/ 922f6374015b59e0f6cd007f8b34eedf

(without introducing more risk) Gareth Rushgrove Read image inventory data $ docker run --rm garethr/centos-inventory cat /inventory.json | { { "title": "rpm", "resource": "package", "provider": “yum", "versions": [ "4.11.3-17.el7" ] }, { "title": “libuser", "resource": "package", "provider": “yum",

(without introducing more risk) Gareth Rushgrove Runtime and multi-OS $ docker exec an-opensuse-container cat /inventory.json | jq { { "title": "netcfg", "resource": "package", "provider": "zypper", "versions": [ "11.5-27.2" ] }, { "title": "shadow", "resource": "package", "provider": "zypper",

GIVEN all my containers now have an inventory WHAT things can I do with it? Gareth Rushgrove

Asking the black box what OS it contains? Gareth Rushgrove

(without introducing more risk) Gareth Rushgrove Use jq to query inventory $ docker run --rm garethr/centos-inventory \ cat /inventory.json | jq '.facts.operatingsystem' "CentOS"

What about a package search engine? Gareth Rushgrove

(without introducing more risk) Gareth Rushgrove Package search $ search-inventory rpm festive_edison rpm 4.11.3-17.el7 yum optimistic_shockley rpm 4.11.3-17.el7 yum condescending_swartz rpm 4.11.2-10.1 zypper festive_edison rpm-libs 4.11.3-17.el7 yum festive_edison rpm-python 4.11.3-17.el7 yum optimistic_shockley rpm-libs 4.11.3-17.el7 yum optimistic_shockley rpm-python 4.11.3-17.el7 yum festive_edison rpm-build-libs 4.11.3-17.el7 yum optimistic_shockley rpm-build-libs 4.11.3-17.el7 yum $ search-inventory package:openssl,provider:yum festive_edison openssl-libs 1:1.0.1e-51.el7_2.5 yum optimistic_shockley openssl-libs 1:1.0.1e-51.el7_2.5 yum

(without introducing more risk) Search packages across containers

(without introducing more risk) Gareth Rushgrove Search in BigQuery SELECT resources.title AS package, resources.versions AS version, facts.hostname AS hostname, facts.operatingsystem AS operatingsystem FROM inventory.sample WHERE resources.resource="package" AND resources.title="openssl"

checking containers for CVEs? Gareth Rushgrove

(without introducing more risk) Red Hat Security Data API

(without introducing more risk) Gareth Rushgrove CVE scanner $ scan-containers-for-cves python ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in festive_edison +---------------+----------+------------+-------------------+ | CVE | Severity | CVSS score | Date | +---------------+----------+------------+-------------------+ | CVE-2014-4650 | moderate | 5.0 | 23 June 2014 | | CVE-2013-1752 | moderate | 4.3 | 25 September 2012 | | CVE-2013-1753 | moderate | 4.3 | 25 September 2012 | | CVE-2014-7185 | low | 4.0 | 23 June 2014 | | CVE-2014-4616 | moderate | 4.0 | 19 May 2014 | +---------------+----------+------------+-------------------+ ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in optimistic_shockley +---------------+----------+------------+-------------------+

(without introducing more risk) https://gist.github.com/garethr/ d4f6e23ff19939e4877f441434ede8da

(without introducing more risk) If all you remember is Conclusions

Operators treating containers as black boxes are going to have a bad time Gareth Rushgrove

Beware the difference between the purity of linux containers and the pragmatic reality of Docker base images Gareth Rushgrove

When using containers we need to reconsider solutions to previously solved configuration management problems Gareth Rushgrove

(without introducing more risk) Questions? And thanks for listening