Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Patterns for secure container base image management

Patterns for secure container base image management

Presentation from SnykCon 2020, all about the people, process and tools needed to manage container base images. Thoughts about team organisation, trade-offs and examples of how to use Snyk to solve this problem.

Gareth Rushgrove

October 22, 2020
Tweet

More Decks by Gareth Rushgrove

Other Decks in Technology

Transcript

  1. Gareth Rushgrove | Director, Product Management | @garethr
    Patterns for secure container
    base image management

    View full-size slide

  2. Gareth Rushgrove
    Director, Product Management
    @garethr

    View full-size slide

  3. Agenda The user problem
    01
    What are base images?
    02
    Teams and responsibility
    03
    Setting some rules
    04
    Automation and workflow
    05

    View full-size slide

  4. User problem
    Vulnerability overload

    View full-size slide

  5. Don’t panic!
    $ snyk container test ghcr.io/garethr/snykt/app
    ...
    Tested 127 dependencies for known issues, found 180 issues.
    Debian 8 is no longer supported by the Debian maintainers. Vulnerability detection may
    be affected by a lack of security updates.
    You test an image for vulnerabilities and find LOTS of issues
    100s of vulnerabilities! It’s using an out-of-date operating system! Who owns this image? Who is
    responsible for fixing vulnerabilities? How many other images like this do we have? Help!

    View full-size slide

  6. Base images
    What and why (and why not)

    View full-size slide

  7. Using base images
    FROM ubuntu:latest
    Standing on the shoulders of software giants
    It’s common with container images to start
    building on top of an existing base image that
    already has software you want.
    This might be an operating system like ubuntu,
    alpine or debian or it could be a language like
    python, ruby, node or really anything else.
    OK, technically often a parent image but hey.
    Libraries and underlying software
    provided by someone else
    Your software

    View full-size slide

  8. We do quite a lot of that. Some are just lightly customized OS
    images, others have extensive configuration and packages
    installed.

    View full-size slide

  9. Distinct responsibilities
    Hardening, common configuration
    Your organization might have some common hardening
    or configuration changes or maybe metadata it wants to
    apply to all images in use by other teams. This is often
    intended to be common for all images used. Maybe you
    have myorg/base
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application

    View full-size slide

  10. Distinct responsibilities
    Common software
    Some organizations provide a layer of common software
    or middleware. This might be language or framework
    specific, say a separate image for Java (myorg/java) and
    another for Python (myorg/python).
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application

    View full-size slide

  11. Distinct responsibilities
    Your application
    Finally the specifics of your application, whether in source
    or binary form. And metadata specific to the application.
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application

    View full-size slide

  12. The idea is to offload that work from the individual engineers and
    have it come more automagically.

    View full-size slide

  13. Can you fix vulnerabilities once?
    Base
    image
    Scale vulnerability management
    When considering container
    vulnerabilities, you want to be able
    to reason about vulnerabilities in
    images you’re running, but also
    understand the overlap and source
    of those vulnerabilities.
    Can you address a vulnerability once,
    and have it resolved everywhere?

    View full-size slide

  14. Teams and responsibility
    A worked example

    View full-size slide

  15. Organizing into teams
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application
    Libraries and underlying software
    provided by someone else
    Hardening, common configuration
    Common software
    Your application
    One team to rule them all
    This could be the case when
    teams are completely
    independent, or when you
    have one central image team.
    A base image team
    A team which provides a
    standard set of approved
    base images for application
    teams to consume.
    Separate base/security teams
    Larger organizations might have
    teams with more distinct
    responsibilities, potentially with
    even more layers.

    View full-size slide

  16. Pros and cons
    One team to rule them all
    PROS
    Simple to understand
    responsibilities.
    CONS
    Potential for chaos if every
    team can do their own thing.
    One central team for ALL
    images likely to become a
    bottleneck.
    A base image team
    PROS
    Able to build strong domain
    expertise in the center,
    ideally fix/triage issues once.
    CONS
    Needs some level of
    governance in order to
    ensure applications teams
    benefit from central
    expertise.
    Separate base/security teams
    PROS
    Same as having a base image
    team, with the added advantage
    of deeper specialisms.
    CONS
    Coordination between
    additional teams can slow down
    the process.

    View full-size slide

  17. Start with your own base image
    $ cat base/Dockerfile
    FROM python:3.6.0-slim
    $ docker build -t ghcr.io/garethr/snykt/base -f base/Dockerfile

    View full-size slide

  18. Establishing a baseline
    $ snyk container test ghcr.io/garethr/snykt/base --file=base/Dockerfile
    ...
    Introduced by your base image (python:3.6.0-slim)
    Fixed in: 5.28.1-6+deb10u1
    ✗ High severity vulnerability found in gnutls28/libgnutls30
    Description: Out-of-bounds Write
    Info: https://snyk.io/vuln/SNYK-DEBIAN10-GNUTLS28-609778
    Introduced through: gnutls28/[email protected]+deb10u4, [email protected]
    From: gnutls28/[email protected]+deb10u4
    From: [email protected] > gnutls28/[email protected]+deb10u4
    Introduced by your base image (python:3.6.0-slim)
    Tested 111 dependencies for known issues, found 178 issues.

    View full-size slide

  19. Watch out for new vulnerabilities
    $ snyk container monitor ghcr.io/garethr/snykt/base --file=base/Dockerfile
    Updated when new vulnerabilities are disclosed
    Snyk will send you alerts via email or Slack as well when new
    vulnerabilities are discovered in the packages you’re using.

    View full-size slide

  20. Actionable recommendations

    View full-size slide

  21. A middleware image
    $ cat middleware/Dockerfile
    FROM ghcr.io/garethr/snykt/base
    RUN apt-get update && apt-get install -y \
    gunicorn \
    sqlite3 \
    && rm -rf /var/lib/apt/lists/
    $ docker build -t ghcr.io/garethr/snykt/middleware -f middleware/Dockerfile

    View full-size slide

  22. The sum of all vulnerabilities
    $ snyk container test ghcr.io/garethr/snykt/middleware --file=middleware/Dockerfile
    ...
    Tested 127 dependencies for known issues, found 180 issues.
    Vulnerabilities from the base image and the new instructions
    But a different team is responsible for some of these, so let’s reason about those separately.

    View full-size slide

  23. Where did the issue come from?
    $ snyk container test ghcr.io/garethr/snykt/middleware --file=middleware/Dockerfile --json | snykout
    +------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Found 180 unique vulnerabilities for ghcr.io/garethr/snykt/middleware |
    +----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+
    | Package | Severity | ID | Issue | Installed | Fixed in |
    +----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+
    | ncurses/libncurses5 | HIGH | CVE-2017-10684 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
    | ncurses/libncurses5 | HIGH | CVE-2017-10685 | Improper Input Validation | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
    | sqlite3/libsqlite3-0 | HIGH | CVE-2020-9794 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | |
    | sqlite3/libsqlite3-0 | HIGH | CVE-2019-8457 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | |
    | ncurses/libncurses5 | MEDIUM | CVE-2017-16879 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u3 |
    | ncurses/libncurses5 | MEDIUM | CVE-2017-13729 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 |
    ...
    +------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Base image vulnerabilities from ghcr.io/garethr/snykt/base |
    +---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+
    | Package | Severity | ID | Issue | Installed | Fixed in |
    +---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+
    | apt/libapt-pkg4.12 | HIGH | CVE-2019-3462 | Arbitrary Code Injection | 1.0.9.8.4 | 1.0.9.8.5 |
    | bzip2/libbz2-1.0 | HIGH | CVE-2019-12900 | Out-of-bounds Write | 1.0.6-7+b3 | 1.0.6-7+deb8u1 |
    | glibc/libc-bin | HIGH | CVE-2018-1000001 | Out-of-Bounds | 2.19-18+deb8u7 | |
    | glibc/libc-bin | HIGH | CVE-2014-9761 | Out-of-Bounds | 2.19-18+deb8u7 | |

    View full-size slide

  24. My responsibility
    $ snyk container monitor ghcr.io/garethr/snykt/middleware \
    --file=middleware/Dockerfile \
    --policy-path=ignores/base.snyk
    Vulnerabilities not in the base image
    We’re monitoring only the unique vulnerabilities added in this image.

    View full-size slide

  25. Clear reason for ignoring

    View full-size slide

  26. Our application images
    $ cat app/Dockerfile
    FROM ghcr.io/garethr/snykt/middleware
    EXPOSE 8080
    WORKDIR /app
    ADD app.py .
    CMD ["gunicorn", "-w", "4", "app:app"]
    $ docker build -t ghcr.io/garethr/snykt/app -f app/Dockerfile

    View full-size slide

  27. Visualising ownership
    $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | snykout
    +-----------------------------------------------------------------------------------------+
    | Found 180 unique vulnerabilities for ghcr.io/garethr/snykt/app |
    +--------------+--------------+--------------+--------------+--------------+--------------+
    | Package | Severity | ID | Issue | Installed | Fixed in |
    +--------------+--------------+--------------+--------------+--------------+--------------+
    +------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Base image vulnerabilities from ghcr.io/garethr/snykt/middleware |
    +----------------------------------+--------------+------------------+------------------------------+----------------------+---------------------------+
    | Package | Severity | ID | Issue | Installed | Fixed in |
    +----------------------------------+--------------+------------------+------------------------------+----------------------+---------------------------+
    | apt/libapt-pkg4.12 | HIGH | CVE-2019-3462 | Arbitrary Code Injection | 1.0.9.8.4 | 1.0.9.8.5 |
    | bzip2/libbz2-1.0 | HIGH | CVE-2019-12900 | Out-of-bounds Write | 1.0.6-7+b3 | 1.0.6-7+deb8u1 |
    | glibc/libc-bin | HIGH | CVE-2018-1000001 | Out-of-Bounds | 2.19-18+deb8u7 | |
    ...

    View full-size slide

  28. Someone else’s responsibility
    $ snyk container monitor ghcr.io/garethr/snykt/app \
    --file=app/Dockerfile \
    --policy-path=ignores/middleware.snyk
    We can still see all the vulnerabilities in the image
    Understand the risk, while acknowledging fixing is someone else’s responsibility.

    View full-size slide

  29. Fix issues with pull requests
    COMING SOON

    View full-size slide

  30. Setting some rules
    Enforcing policies on downstream

    View full-size slide

  31. Good practices/enforcing rules
    Responsibility at the edges
    Not every team plays by the rules.
    There is a useful balance between
    centrally enforced policies and
    encouraging best practices.
    How do you make it as easy as possible
    for consumers of base images to do the
    right thing?

    View full-size slide

  32. Open Policy Agent
    Service
    OPA
    .rego
    Query
    (any JSON value)
    Decision
    (any JSON value)
    Data
    (JSON)
    Policy
    (Rego)
    Request, Event, etc.
    Declarative
    Express policy in a high-level, declarative language
    that promotes safe, performant, fine-grained controls.
    Use a language purpose-built for policy in a world
    where JSON is pervasive.
    Context-aware
    Leverage external information to write the policies
    you really care about. Write logic that adapts to the
    world around it and attach that logic to the systems
    that need it.

    View full-size slide

  33. Prohibiting high severity issues
    package main
    deny[msg] {
    issue = input.vulnerabilities[index]
    issue.severity = "high"
    msg = sprintf("High severity issue found. package: %v issue: %v", [issue.name, issue.title])
    }
    A Rego example
    This is the Open Policy Agent language for describing policies. Here we’re saying we want to prohibit
    any images with known high-severity vulnerabilities.
    REGO

    View full-size slide

  34. Watching out for crypto issues
    package main
    deny[msg] {
    issue = input.vulnerabilities[index]
    issue.identifiers["CWE"][_] = "CWE-327"
    issue.severity = "high"
    msg = sprintf("High severity cryptography issue (CWE-327). package: %v", [issue.name])
    }
    warn[msg] {
    issue = input.vulnerabilities[index]
    issue.identifiers["CWE"][_] = "CWE-327"
    issue.severity != "high"
    msg = sprintf("Crypto issue (CWE-327). package: %v severity: %v", [issue.name, issue.severity])
    }
    REGO

    View full-size slide

  35. Test an image against the policy
    $ snyk container test ghcr.io/garethr/snykt/base --file=base/Dockerfile --json | conftest test -
    WARN - Cryptography issue (CWE-327). package: gnupg severity: low
    WARN - Cryptography issue (CWE-327). package: openssl/libssl1.0.0 severity: medium
    WARN - Cryptography issue (CWE-327). package: openssl/libssl1.0.0 severity: low
    WARN - Cryptography issue (CWE-327). package: openssl severity: medium
    WARN - High severity issue found. package: apt/libapt-pkg4.12 issue: Arbitrary Code Injection
    FAIL - High severity issue found. package: apt issue: Arbitrary Code Injection
    FAIL - High severity issue found. package: bzip2/libbz2-1.0 issue: Out-of-bounds Write
    FAIL - High severity issue found. package: glibc/libc-bin issue: Out-of-bounds Read
    ...
    Conftest
    Conftest is an open source command line tool, part of the Open Policy Agent project.
    It’s useful to testing things locally, and works in CI environments too.

    View full-size slide

  36. Allow only trusted base images
    package main
    allowed := {
    "python:slim"
    }
    deny[msg] {
    not valid(input.docker.baseImage)
    msg = sprintf("Not using a permitted base image: %v", [input.docker.baseImage])
    }
    valid(image) {
    allowed[image]
    }
    REGO

    View full-size slide

  37. Catch prohibited base images
    $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | conftest test -
    FAIL - Not using a permitted base image: ghcr.io/garethr/snykt/middleware
    4 tests, 3 passed, 0 warnings, 1 failure, 0 exceptions

    View full-size slide

  38. Allow images from our registry
    package main
    allowed := {
    "python:slim"
    }
    deny[msg] {
    not valid(input.docker.baseImage)
    msg = sprintf("Not using a permitted base image: %v", [input.docker.baseImage])
    }
    valid(image) {
    allowed[image]
    }
    valid(image) {
    startswith(image, "ghcr.io/garethr/snykt/")
    }
    REGO

    View full-size slide

  39. Ensure images pass the policy
    $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | conftest test -
    4 tests, 4 passed, 0 warnings, 0 failure, 0 exceptions
    To enforce or not to enforce?
    Over time it’s best to move towards enforcing policies automatically, but to start with you’ll probably
    want to roll out slowly and carefully. You might already have these policies written down, but unless
    you’re automatically checking or enforcing you’ll likely have a long tail of images in breach.
    The advantage of Snyk is you can help developer test early in the SDLC, on the local machine or in CI,
    when it’s fast and cheap to fix the problem.

    View full-size slide

  40. Automation and workflow
    Demos and ideas

    View full-size slide

  41. Automate in your CI pipeline
    Embrace CI for building agreement
    between teams
    Using Snyk in your CI pipeline allows for
    automating this or other patterns for
    managing container images securely.
    Investing in good practices around CI
    should help security, build, operations
    and development teams work more
    closely together.

    View full-size slide

  42. Use the Snyk API
    Integrate Snyk into your existing tools
    Different organizations, and different teams,
    might manage base images differently. The
    Snyk API makes it easy to build custom
    dashboards or use the data about base
    images for reporting.
    snyk.docs.apiary.io

    View full-size slide

  43. Conclusions
    If all you remember is...

    View full-size slide

  44. Observe
    How are your teams using base
    images today?

    View full-size slide

  45. Orient
    Who actually fixes vulnerabilities?
    Do all teams bear the burden of
    fixing the same issues?

    View full-size slide

  46. Decide
    How can you shift the
    responsibilities of teams to make
    base image management easier?

    View full-size slide

  47. Act
    Put in place the automation to scale.

    View full-size slide

  48. Thanks for listening
    Try Snyk for free at snyk.io/signup

    View full-size slide