What's Inside That Container?

My talk from Configuration Management Camp 2017. A look at the reality of container usage in the wild, what problems that leads to in operations, and maybe a few ideas of things we can do or build to help.

Transcript

1. (without introducing more risk) What's Inside That Container? Puppet Gareth Rushgrove Containers and config management in the real world
2. (without introducing more risk) @garethr
3. (without introducing more risk) Gareth Rushgrove
4. (without introducing more risk) What we’ll cover This talk
5. - What is configuration management? - Docker base image usage - The problem with containers as black boxes - Ideas and demos Gareth Rushgrove
6. (without introducing more risk) Useful background What is Configuration Management?
7. - 1950s research - 1960s 480 series - 1991 MIL-HDBK-61 - 1998 ANSI-EIA-649 Gareth Rushgrove
8. - Identification - Control - Status accounting - Verification and audit Gareth Rushgrove Military Handbook Configuration Management Guidance MIL-HDBK-61B
9. Configuration management verifies that a system is identified and documented in sufficient detail Gareth Rushgrove National Consensus Standard for Configuration Management EIA-649
10. Configuration management verifies that a system performs as intended Gareth Rushgrove National Consensus Standard for Configuration Management EIA-649
11. NOTE: Not a tool, a practice Gareth Rushgrove
12. (without introducing more risk) And where to find them Docker Base Images
13. (without introducing more risk) Gareth Rushgrove Docker Hub API $ curl -s https://registry.hub.docker.com/v2/... ...repositories/library/ubuntu/ | jq .pull_count
14. Gareth Rushgrove Image downloads
15. What about other popular official images? Node (32,523,647), Java (16,635,049), etc. Currently based on Debian too Gareth Rushgrove
16. Gareth Rushgrove Image size
17. (without introducing more risk) Gareth Rushgrove Count files $ find -maxdepth 1 -type d | while read -r dir; do printf "%s:\t" "$dir"; find "$dir" -type f | wc -l; done
18. (without introducing more risk) Gareth Rushgrove Count packages $ dpkg -l | grep ^ii | wc -l $ dnf list installed $ rpm -qa | wc -l $ apk info | wc -l
19. Gareth Rushgrove Image contents
20. (without introducing more risk) Thanks David Gageot
21. (without introducing more risk) Gareth Rushgrove Popular on GitHub SELECT RTRIM(LTRIM(SUBSTR(line, 5))) AS line_group.base_image FROM ( SELECT SPLIT(dockerfile, '\n') AS line FROM [github.dockerfiles_content] HAVING LEFT(line, 5) = 'FROM ' ) SELECT image, count(*) AS count FROM ( SELECT FIRST(SPLIT(line_group.base_image, ':')) AS image FROM [github.images] ) GROUP BY image ORDER BY count DESC
22. Gareth Rushgrove Image GitHub popularity
23. Gareth Rushgrove Usage growth
24. (without introducing more risk) Thanks to Microbadger
25. Gareth Rushgrove Hub image sample
26. Gareth Rushgrove
27. The majority of people using Docker are using images containing an entire operating system filesystem Gareth Rushgrove
28. Alpine usage is growing more rapidly than others, but starting from a much smaller install base Gareth Rushgrove
29. Scratch, or other approaches like Nix, appear to occupy a small niche Gareth Rushgrove
30. Windows and Windows Nano images will undoubtedly become more common over the next year or so Gareth Rushgrove
31. Debian derivatives account for the majority of images today Gareth Rushgrove
32. (without introducing more risk) What does this all mean to me? Problems
33. (without introducing more risk) You don’t know, and that’s a problem
34. (without introducing more risk) Visibility and control are critical
35. (without introducing more risk) What you don’t know can hurt you
36. (without introducing more risk) Vulnerable images on Docker Hub
37. (without introducing more risk) Not quite so simple
38. Can you tell me all the versions of OpenSSL you have in production right now? Gareth Rushgrove
39. Containers are a black box Gareth Rushgrove
40. Containers are a black box from the point of view of the scheduler Gareth Rushgrove
41. Containers are NOT a black box from the point of view of the operator Gareth Rushgrove
42. These are all generally configuration management problems; identification, control, status accounting, verification and audit Gareth Rushgrove
43. (without introducing more risk) Experiments and proof-of-concept work Ideas
44. Immutability means we need to know what we put inside the box Gareth Rushgrove
45. (without introducing more risk) Tangent about immutability
46. It’s not enough to say ubuntu:16.04 Gareth Rushgrove
47. (without introducing more risk) LIVE DEMOS
48. (without introducing more risk) Gareth Rushgrove Embed inventory in image $ add-container-inventory centos garethr/centos-inventory ----> Using existing puppet-inventory volume ----> Generating inventory for image centos ----> Saving inventory to temporary image inventory-21042 ----> Committing new image to garethr/centos-inventory sha256:f84b3655252c946dfb888de2d74348afed97ef89d817e469adad3a ----> Cleaning up
49. (without introducing more risk) https://gist.github.com/garethr/ 922f6374015b59e0f6cd007f8b34eedf
50. (without introducing more risk) Gareth Rushgrove Read image inventory data $ docker run --rm garethr/centos-inventory cat /inventory.json | { { "title": "rpm", "resource": "package", "provider": “yum", "versions": [ "4.11.3-17.el7" ] }, { "title": “libuser", "resource": "package", "provider": “yum",
51. (without introducing more risk) Gareth Rushgrove Runtime and multi-OS $ docker exec an-opensuse-container cat /inventory.json | jq { { "title": "netcfg", "resource": "package", "provider": "zypper", "versions": [ "11.5-27.2" ] }, { "title": "shadow", "resource": "package", "provider": "zypper",
52. GIVEN all my containers now have an inventory WHAT things can I do with it? Gareth Rushgrove
53. Asking the black box what OS it contains? Gareth Rushgrove
54. (without introducing more risk) Gareth Rushgrove Use jq to query inventory $ docker run --rm garethr/centos-inventory \ cat /inventory.json | jq '.facts.operatingsystem' "CentOS"
55. What about a package search engine? Gareth Rushgrove
56. (without introducing more risk) Gareth Rushgrove Package search $ search-inventory rpm festive_edison rpm 4.11.3-17.el7 yum optimistic_shockley rpm 4.11.3-17.el7 yum condescending_swartz rpm 4.11.2-10.1 zypper festive_edison rpm-libs 4.11.3-17.el7 yum festive_edison rpm-python 4.11.3-17.el7 yum optimistic_shockley rpm-libs 4.11.3-17.el7 yum optimistic_shockley rpm-python 4.11.3-17.el7 yum festive_edison rpm-build-libs 4.11.3-17.el7 yum optimistic_shockley rpm-build-libs 4.11.3-17.el7 yum $ search-inventory package:openssl,provider:yum festive_edison openssl-libs 1:1.0.1e-51.el7_2.5 yum optimistic_shockley openssl-libs 1:1.0.1e-51.el7_2.5 yum
57. (without introducing more risk) Search packages across containers
58. (without introducing more risk) Gareth Rushgrove Search in BigQuery SELECT resources.title AS package, resources.versions AS version, facts.hostname AS hostname, facts.operatingsystem AS operatingsystem FROM inventory.sample WHERE resources.resource="package" AND resources.title="openssl"
59. checking containers for CVEs? Gareth Rushgrove
60. (without introducing more risk) Red Hat Security Data API
61. (without introducing more risk) Gareth Rushgrove CVE scanner $ scan-containers-for-cves python ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in festive_edison +---------------+----------+------------+-------------------+ | CVE | Severity | CVSS score | Date | +---------------+----------+------------+-------------------+ | CVE-2014-4650 | moderate | 5.0 | 23 June 2014 | | CVE-2013-1752 | moderate | 4.3 | 25 September 2012 | | CVE-2013-1753 | moderate | 4.3 | 25 September 2012 | | CVE-2014-7185 | low | 4.0 | 23 June 2014 | | CVE-2014-4616 | moderate | 4.0 | 19 May 2014 | +---------------+----------+------------+-------------------+ ---> Scanning package: python ---> CVEs found for python-2.7.5-34.el7 in optimistic_shockley +---------------+----------+------------+-------------------+
62. (without introducing more risk) https://gist.github.com/garethr/ d4f6e23ff19939e4877f441434ede8da
63. (without introducing more risk) If all you remember is Conclusions
64. Operators treating containers as black boxes are going to have a bad time Gareth Rushgrove
65. Beware the difference between the purity of linux containers and the pragmatic reality of Docker base images Gareth Rushgrove
66. When using containers we need to reconsider solutions to previously solved configuration management problems Gareth Rushgrove
67. (without introducing more risk) Questions? And thanks for listening