What's Inside That Container?

My talk from Configuration Management Camp 2017. A look at the reality of container usage in the wild, what problems that leads to in operations, and maybe a few ideas of things we can do or build to help.

Transcript

1. (without introducing more risk)
   What's Inside That
   Container?
   Puppet
   Gareth Rushgrove
   Containers and config management in the real world
   

   View Slide

2. (without introducing more risk)
   @garethr
   

   View Slide

3. (without introducing more risk)
   Gareth Rushgrove
   

   View Slide

4. (without introducing more risk)
   What we’ll cover
   This talk
   

   View Slide

5. - What is configuration management?
   - Docker base image usage
   - The problem with containers as black boxes
   - Ideas and demos
   Gareth Rushgrove
   

   View Slide

6. (without introducing more risk)
   Useful background
   What is Configuration
   Management?
   

   View Slide

7. - 1950s research
   - 1960s 480 series
   - 1991 MIL-HDBK-61
   - 1998 ANSI-EIA-649
   Gareth Rushgrove
   

   View Slide

8. - Identification
   - Control
   - Status accounting
   - Verification and audit
   Gareth Rushgrove
   Military Handbook Configuration Management Guidance MIL-HDBK-61B
   

   View Slide

9. Configuration management verifies
   that a system is identified and
   documented in sufficient detail
   Gareth Rushgrove
   National Consensus Standard for Configuration Management EIA-649
   

   View Slide

10. Configuration management verifies
    that a system performs as intended
    Gareth Rushgrove
    National Consensus Standard for Configuration Management EIA-649
    

    View Slide

11. NOTE: Not a tool, a practice
    Gareth Rushgrove
    

    View Slide

12. (without introducing more risk)
    And where to find them
    Docker Base Images
    

    View Slide

13. (without introducing more risk)
    Gareth Rushgrove
    Docker Hub API
    $ curl -s https://registry.hub.docker.com/v2/...
    ...repositories/library/ubuntu/ | jq .pull_count
    

    View Slide

14. Gareth Rushgrove
    Image downloads
    

    View Slide

15. What about other popular official
    images? Node (32,523,647), Java
    (16,635,049), etc. Currently based
    on Debian too
    Gareth Rushgrove
    

    View Slide

16. Gareth Rushgrove
    Image size
    

    View Slide

17. (without introducing more risk)
    Gareth Rushgrove
    Count files
    $ find -maxdepth 1 -type d | while read -r
    dir; do printf "%s:\t" "$dir"; find "$dir"
    -type f | wc -l; done
    

    View Slide

18. (without introducing more risk)
    Gareth Rushgrove
    Count packages
    $ dpkg -l | grep ^ii | wc -l
    $ dnf list installed
    $ rpm -qa | wc -l
    $ apk info | wc -l
    

    View Slide

19. Gareth Rushgrove
    Image contents
    

    View Slide

20. (without introducing more risk)
    Thanks David Gageot
    

    View Slide

21. (without introducing more risk)
    Gareth Rushgrove
    Popular on GitHub
    SELECT RTRIM(LTRIM(SUBSTR(line, 5))) AS line_group.base_image
    FROM (
    SELECT SPLIT(dockerfile, '\n') AS line
    FROM [github.dockerfiles_content]
    HAVING LEFT(line, 5) = 'FROM '
    )
    SELECT image, count(*) AS count
    FROM (
    SELECT FIRST(SPLIT(line_group.base_image, ':')) AS image
    FROM [github.images]
    )
    GROUP BY image
    ORDER BY count DESC
    

    View Slide

22. Gareth Rushgrove
    Image GitHub popularity
    

    View Slide

23. Gareth Rushgrove
    Usage growth
    

    View Slide

24. (without introducing more risk)
    Thanks to Microbadger
    

    View Slide

25. Gareth Rushgrove
    Hub image sample
    

    View Slide

26. Gareth Rushgrove
    

    View Slide

27. The majority of people using Docker
    are using images containing an entire
    operating system filesystem
    Gareth Rushgrove
    

    View Slide

28. Alpine usage is growing more rapidly
    than others, but starting from a much
    smaller install base
    Gareth Rushgrove
    

    View Slide

29. Scratch, or other approaches like Nix,
    appear to occupy a small niche
    Gareth Rushgrove
    

    View Slide

30. Windows and Windows Nano images
    will undoubtedly become more
    common over the next year or so
    Gareth Rushgrove
    

    View Slide

31. Debian derivatives account for the
    majority of images today
    Gareth Rushgrove
    

    View Slide

32. (without introducing more risk)
    What does this all mean to me?
    Problems
    

    View Slide

33. (without introducing more risk)
    You don’t know, and that’s a problem
    

    View Slide

34. (without introducing more risk)
    Visibility and control are critical
    

    View Slide

35. (without introducing more risk)
    What you don’t know can hurt you
    

    View Slide

36. (without introducing more risk)
    Vulnerable images on Docker Hub
    

    View Slide

37. (without introducing more risk)
    Not quite so simple
    

    View Slide

38. Can you tell me all the versions of
    OpenSSL you have in production
    right now?
    Gareth Rushgrove
    

    View Slide

39. Containers are a black box
    Gareth Rushgrove
    

    View Slide

40. Containers are a black box from the
    point of view of the scheduler
    Gareth Rushgrove
    

    View Slide

41. Containers are NOT a black box from
    the point of view of the operator
    Gareth Rushgrove
    

    View Slide

42. These are all generally configuration
    management problems; identification,
    control, status accounting, verification
    and audit
    Gareth Rushgrove
    

    View Slide

43. (without introducing more risk)
    Experiments and proof-of-concept work
    Ideas
    

    View Slide

44. Immutability means we need to know
    what we put inside the box
    Gareth Rushgrove
    

    View Slide

45. (without introducing more risk)
    Tangent about immutability
    

    View Slide

46. It’s not enough to say ubuntu:16.04
    Gareth Rushgrove
    

    View Slide

47. (without introducing more risk)
    LIVE DEMOS
    

    View Slide

48. (without introducing more risk)
    Gareth Rushgrove
    Embed inventory in image
    $ add-container-inventory centos garethr/centos-inventory
    ----> Using existing puppet-inventory volume
    ----> Generating inventory for image centos
    ----> Saving inventory to temporary image inventory-21042
    ----> Committing new image to garethr/centos-inventory
    sha256:f84b3655252c946dfb888de2d74348afed97ef89d817e469adad3a
    ----> Cleaning up
    

    View Slide

49. (without introducing more risk)
    https://gist.github.com/garethr/
    922f6374015b59e0f6cd007f8b34eedf
    

    View Slide

50. (without introducing more risk)
    Gareth Rushgrove
    Read image inventory data
    $ docker run --rm garethr/centos-inventory cat /inventory.json |
    {
    {
    "title": "rpm",
    "resource": "package",
    "provider": “yum",
    "versions": [
    "4.11.3-17.el7"
    ]
    },
    {
    "title": “libuser",
    "resource": "package",
    "provider": “yum",
    

    View Slide

51. (without introducing more risk)
    Gareth Rushgrove
    Runtime and multi-OS
    $ docker exec an-opensuse-container cat /inventory.json | jq
    {
    {
    "title": "netcfg",
    "resource": "package",
    "provider": "zypper",
    "versions": [
    "11.5-27.2"
    ]
    },
    {
    "title": "shadow",
    "resource": "package",
    "provider": "zypper",
    

    View Slide

52. GIVEN all my containers now have an inventory
    WHAT things can I do with it?
    Gareth Rushgrove
    

    View Slide

53. Asking the black box what OS
    it contains?
    Gareth Rushgrove
    

    View Slide

54. (without introducing more risk)
    Gareth Rushgrove
    Use jq to query inventory
    $ docker run --rm garethr/centos-inventory \
    cat /inventory.json |
    jq '.facts.operatingsystem'
    "CentOS"
    

    View Slide

55. What about a package search engine?
    Gareth Rushgrove
    

    View Slide

56. (without introducing more risk)
    Gareth Rushgrove
    Package search
    $ search-inventory rpm
    festive_edison rpm 4.11.3-17.el7 yum
    optimistic_shockley rpm 4.11.3-17.el7 yum
    condescending_swartz rpm 4.11.2-10.1 zypper
    festive_edison rpm-libs 4.11.3-17.el7 yum
    festive_edison rpm-python 4.11.3-17.el7 yum
    optimistic_shockley rpm-libs 4.11.3-17.el7 yum
    optimistic_shockley rpm-python 4.11.3-17.el7 yum
    festive_edison rpm-build-libs 4.11.3-17.el7 yum
    optimistic_shockley rpm-build-libs 4.11.3-17.el7 yum
    $ search-inventory package:openssl,provider:yum
    festive_edison openssl-libs 1:1.0.1e-51.el7_2.5 yum
    optimistic_shockley openssl-libs 1:1.0.1e-51.el7_2.5 yum
    

    View Slide

57. (without introducing more risk)
    Search packages across containers
    

    View Slide

58. (without introducing more risk)
    Gareth Rushgrove
    Search in BigQuery
    SELECT
    resources.title AS package,
    resources.versions AS version,
    facts.hostname AS hostname,
    facts.operatingsystem AS operatingsystem
    FROM
    inventory.sample
    WHERE
    resources.resource="package"
    AND resources.title="openssl"
    

    View Slide

59. checking containers for CVEs?
    Gareth Rushgrove
    

    View Slide

60. (without introducing more risk)
    Red Hat Security Data API
    

    View Slide

61. (without introducing more risk)
    Gareth Rushgrove
    CVE scanner
    $ scan-containers-for-cves python
    ---> Scanning package: python
    ---> CVEs found for python-2.7.5-34.el7 in festive_edison
    +---------------+----------+------------+-------------------+
    | CVE | Severity | CVSS score | Date |
    +---------------+----------+------------+-------------------+
    | CVE-2014-4650 | moderate | 5.0 | 23 June 2014 |
    | CVE-2013-1752 | moderate | 4.3 | 25 September 2012 |
    | CVE-2013-1753 | moderate | 4.3 | 25 September 2012 |
    | CVE-2014-7185 | low | 4.0 | 23 June 2014 |
    | CVE-2014-4616 | moderate | 4.0 | 19 May 2014 |
    +---------------+----------+------------+-------------------+
    ---> Scanning package: python
    ---> CVEs found for python-2.7.5-34.el7 in optimistic_shockley
    +---------------+----------+------------+-------------------+
    

    View Slide

62. (without introducing more risk)
    https://gist.github.com/garethr/
    d4f6e23ff19939e4877f441434ede8da
    

    View Slide

63. (without introducing more risk)
    If all you remember is
    Conclusions
    

    View Slide

64. Operators treating containers as black
    boxes are going to have a bad time
    Gareth Rushgrove
    

    View Slide

65. Beware the difference between
    the purity of linux containers and
    the pragmatic reality of Docker
    base images
    Gareth Rushgrove
    

    View Slide

66. When using containers we
    need to reconsider solutions to
    previously solved configuration
    management problems
    Gareth Rushgrove
    

    View Slide

67. (without introducing more risk)
    Questions?
    And thanks for listening
    

    View Slide