Tweet
Tweet

Slide 1

Slide 1 text

Gareth Rushgrove | Director, Product Management | @garethr Patterns for secure container base image management

Slide 2

Slide 2 text

Gareth Rushgrove Director, Product Management @garethr

Slide 3

Slide 3 text

Agenda The user problem 01 What are base images? 02 Teams and responsibility 03 Setting some rules 04 Automation and workflow 05

Slide 4

Slide 4 text

User problem Vulnerability overload

Slide 5

Slide 5 text

Don’t panic! $ snyk container test ghcr.io/garethr/snykt/app ... Tested 127 dependencies for known issues, found 180 issues. Debian 8 is no longer supported by the Debian maintainers. Vulnerability detection may be affected by a lack of security updates. You test an image for vulnerabilities and find LOTS of issues 100s of vulnerabilities! It’s using an out-of-date operating system! Who owns this image? Who is responsible for fixing vulnerabilities? How many other images like this do we have? Help!

Slide 6

Slide 6 text

Base images What and why (and why not)

Slide 7

Slide 7 text

Using base images FROM ubuntu:latest Standing on the shoulders of software giants It’s common with container images to start building on top of an existing base image that already has software you want. This might be an operating system like ubuntu, alpine or debian or it could be a language like python, ruby, node or really anything else. OK, technically often a parent image but hey. Libraries and underlying software provided by someone else Your software

Slide 8

Slide 8 text

We do quite a lot of that. Some are just lightly customized OS images, others have extensive configuration and packages installed.

Slide 9

Slide 9 text

Distinct responsibilities Hardening, common configuration Your organization might have some common hardening or configuration changes or maybe metadata it wants to apply to all images in use by other teams. This is often intended to be common for all images used. Maybe you have myorg/base Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application

Slide 10

Slide 10 text

Distinct responsibilities Common software Some organizations provide a layer of common software or middleware. This might be language or framework specific, say a separate image for Java (myorg/java) and another for Python (myorg/python). Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application

Slide 11

Slide 11 text

Distinct responsibilities Your application Finally the specifics of your application, whether in source or binary form. And metadata specific to the application. Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application

Slide 12

Slide 12 text

The idea is to offload that work from the individual engineers and have it come more automagically.

Slide 13

Slide 13 text

Can you fix vulnerabilities once? Base image Scale vulnerability management When considering container vulnerabilities, you want to be able to reason about vulnerabilities in images you’re running, but also understand the overlap and source of those vulnerabilities. Can you address a vulnerability once, and have it resolved everywhere?

Slide 14

Slide 14 text

Teams and responsibility A worked example

Slide 15

Slide 15 text

Organizing into teams Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application Libraries and underlying software provided by someone else Hardening, common configuration Common software Your application One team to rule them all This could be the case when teams are completely independent, or when you have one central image team. A base image team A team which provides a standard set of approved base images for application teams to consume. Separate base/security teams Larger organizations might have teams with more distinct responsibilities, potentially with even more layers.

Slide 16

Slide 16 text

Pros and cons One team to rule them all PROS Simple to understand responsibilities. CONS Potential for chaos if every team can do their own thing. One central team for ALL images likely to become a bottleneck. A base image team PROS Able to build strong domain expertise in the center, ideally fix/triage issues once. CONS Needs some level of governance in order to ensure applications teams benefit from central expertise. Separate base/security teams PROS Same as having a base image team, with the added advantage of deeper specialisms. CONS Coordination between additional teams can slow down the process.

Slide 17

Slide 17 text

Start with your own base image $ cat base/Dockerfile FROM python:3.6.0-slim $ docker build -t ghcr.io/garethr/snykt/base -f base/Dockerfile

Slide 18

Slide 18 text

Establishing a baseline $ snyk container test ghcr.io/garethr/snykt/base --file=base/Dockerfile ... Introduced by your base image (python:3.6.0-slim) Fixed in: 5.28.1-6+deb10u1 ✗ High severity vulnerability found in gnutls28/libgnutls30 Description: Out-of-bounds Write Info: https://snyk.io/vuln/SNYK-DEBIAN10-GNUTLS28-609778 Introduced through: gnutls28/[email protected]+deb10u4, [email protected] From: gnutls28/[email protected]+deb10u4 From: [email protected] > gnutls28/[email protected]+deb10u4 Introduced by your base image (python:3.6.0-slim) Tested 111 dependencies for known issues, found 178 issues.

Slide 19

Slide 19 text

Watch out for new vulnerabilities $ snyk container monitor ghcr.io/garethr/snykt/base --file=base/Dockerfile Updated when new vulnerabilities are disclosed Snyk will send you alerts via email or Slack as well when new vulnerabilities are discovered in the packages you’re using.

Slide 20

Slide 20 text

Actionable recommendations

Slide 21

Slide 21 text

A middleware image $ cat middleware/Dockerfile FROM ghcr.io/garethr/snykt/base RUN apt-get update && apt-get install -y \ gunicorn \ sqlite3 \ && rm -rf /var/lib/apt/lists/ $ docker build -t ghcr.io/garethr/snykt/middleware -f middleware/Dockerfile

Slide 22

Slide 22 text

The sum of all vulnerabilities $ snyk container test ghcr.io/garethr/snykt/middleware --file=middleware/Dockerfile ... Tested 127 dependencies for known issues, found 180 issues. Vulnerabilities from the base image and the new instructions But a different team is responsible for some of these, so let’s reason about those separately.

Slide 23

Slide 23 text

Where did the issue come from? $ snyk container test ghcr.io/garethr/snykt/middleware --file=middleware/Dockerfile --json | snykout +------------------------------------------------------------------------------------------------------------------------------------------------------+ | Found 180 unique vulnerabilities for ghcr.io/garethr/snykt/middleware | +----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+ | Package | Severity | ID | Issue | Installed | Fixed in | +----------------------------------+---------------+------------------+------------------------------------+-------------------+-----------------------+ | ncurses/libncurses5 | HIGH | CVE-2017-10684 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 | | ncurses/libncurses5 | HIGH | CVE-2017-10685 | Improper Input Validation | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 | | sqlite3/libsqlite3-0 | HIGH | CVE-2020-9794 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | | | sqlite3/libsqlite3-0 | HIGH | CVE-2019-8457 | Out-of-bounds Read | 3.8.7.1-1+deb8u6 | | | ncurses/libncurses5 | MEDIUM | CVE-2017-16879 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u3 | | ncurses/libncurses5 | MEDIUM | CVE-2017-13729 | Out-of-Bounds | 5.9+20140913-1+b1 | 5.9+20140913-1+deb8u1 | ... +------------------------------------------------------------------------------------------------------------------------------------------------------+ | Base image vulnerabilities from ghcr.io/garethr/snykt/base | +---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+ | Package | Severity | ID | Issue | Installed | Fixed in | +---------------------------+-------------+------------------+--------------------------------------+----------------------+---------------------------+ | apt/libapt-pkg4.12 | HIGH | CVE-2019-3462 | Arbitrary Code Injection | 1.0.9.8.4 | 1.0.9.8.5 | | bzip2/libbz2-1.0 | HIGH | CVE-2019-12900 | Out-of-bounds Write | 1.0.6-7+b3 | 1.0.6-7+deb8u1 | | glibc/libc-bin | HIGH | CVE-2018-1000001 | Out-of-Bounds | 2.19-18+deb8u7 | | | glibc/libc-bin | HIGH | CVE-2014-9761 | Out-of-Bounds | 2.19-18+deb8u7 | |

Slide 24

Slide 24 text

My responsibility $ snyk container monitor ghcr.io/garethr/snykt/middleware \ --file=middleware/Dockerfile \ --policy-path=ignores/base.snyk Vulnerabilities not in the base image We’re monitoring only the unique vulnerabilities added in this image.

Slide 25

Slide 25 text

Clear reason for ignoring

Slide 26

Slide 26 text

Our application images $ cat app/Dockerfile FROM ghcr.io/garethr/snykt/middleware EXPOSE 8080 WORKDIR /app ADD app.py . CMD ["gunicorn", "-w", "4", "app:app"] $ docker build -t ghcr.io/garethr/snykt/app -f app/Dockerfile

Slide 27

Slide 27 text

Visualising ownership $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | snykout +-----------------------------------------------------------------------------------------+ | Found 180 unique vulnerabilities for ghcr.io/garethr/snykt/app | +--------------+--------------+--------------+--------------+--------------+--------------+ | Package | Severity | ID | Issue | Installed | Fixed in | +--------------+--------------+--------------+--------------+--------------+--------------+ +------------------------------------------------------------------------------------------------------------------------------------------------------+ | Base image vulnerabilities from ghcr.io/garethr/snykt/middleware | +----------------------------------+--------------+------------------+------------------------------+----------------------+---------------------------+ | Package | Severity | ID | Issue | Installed | Fixed in | +----------------------------------+--------------+------------------+------------------------------+----------------------+---------------------------+ | apt/libapt-pkg4.12 | HIGH | CVE-2019-3462 | Arbitrary Code Injection | 1.0.9.8.4 | 1.0.9.8.5 | | bzip2/libbz2-1.0 | HIGH | CVE-2019-12900 | Out-of-bounds Write | 1.0.6-7+b3 | 1.0.6-7+deb8u1 | | glibc/libc-bin | HIGH | CVE-2018-1000001 | Out-of-Bounds | 2.19-18+deb8u7 | | ...

Slide 28

Slide 28 text

Someone else’s responsibility $ snyk container monitor ghcr.io/garethr/snykt/app \ --file=app/Dockerfile \ --policy-path=ignores/middleware.snyk We can still see all the vulnerabilities in the image Understand the risk, while acknowledging fixing is someone else’s responsibility.

Slide 29

Slide 29 text

Fix issues with pull requests COMING SOON

Slide 30

Slide 30 text

Setting some rules Enforcing policies on downstream

Slide 31

Slide 31 text

Good practices/enforcing rules Responsibility at the edges Not every team plays by the rules. There is a useful balance between centrally enforced policies and encouraging best practices. How do you make it as easy as possible for consumers of base images to do the right thing?

Slide 32

Slide 32 text

Open Policy Agent Service OPA .rego Query (any JSON value) Decision (any JSON value) Data (JSON) Policy (Rego) Request, Event, etc. Declarative Express policy in a high-level, declarative language that promotes safe, performant, fine-grained controls. Use a language purpose-built for policy in a world where JSON is pervasive. Context-aware Leverage external information to write the policies you really care about. Write logic that adapts to the world around it and attach that logic to the systems that need it.

Slide 33

Slide 33 text

Prohibiting high severity issues package main deny[msg] { issue = input.vulnerabilities[index] issue.severity = "high" msg = sprintf("High severity issue found. package: %v issue: %v", [issue.name, issue.title]) } A Rego example This is the Open Policy Agent language for describing policies. Here we’re saying we want to prohibit any images with known high-severity vulnerabilities. REGO

Slide 34

Slide 34 text

Watching out for crypto issues package main deny[msg] { issue = input.vulnerabilities[index] issue.identifiers["CWE"][_] = "CWE-327" issue.severity = "high" msg = sprintf("High severity cryptography issue (CWE-327). package: %v", [issue.name]) } warn[msg] { issue = input.vulnerabilities[index] issue.identifiers["CWE"][_] = "CWE-327" issue.severity != "high" msg = sprintf("Crypto issue (CWE-327). package: %v severity: %v", [issue.name, issue.severity]) } REGO

Slide 35

Slide 35 text

Test an image against the policy $ snyk container test ghcr.io/garethr/snykt/base --file=base/Dockerfile --json | conftest test - WARN - Cryptography issue (CWE-327). package: gnupg severity: low WARN - Cryptography issue (CWE-327). package: openssl/libssl1.0.0 severity: medium WARN - Cryptography issue (CWE-327). package: openssl/libssl1.0.0 severity: low WARN - Cryptography issue (CWE-327). package: openssl severity: medium WARN - High severity issue found. package: apt/libapt-pkg4.12 issue: Arbitrary Code Injection FAIL - High severity issue found. package: apt issue: Arbitrary Code Injection FAIL - High severity issue found. package: bzip2/libbz2-1.0 issue: Out-of-bounds Write FAIL - High severity issue found. package: glibc/libc-bin issue: Out-of-bounds Read ... Conftest Conftest is an open source command line tool, part of the Open Policy Agent project. It’s useful to testing things locally, and works in CI environments too.

Slide 36

Slide 36 text

Allow only trusted base images package main allowed := { "python:slim" } deny[msg] { not valid(input.docker.baseImage) msg = sprintf("Not using a permitted base image: %v", [input.docker.baseImage]) } valid(image) { allowed[image] } REGO

Slide 37

Slide 37 text

Catch prohibited base images $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | conftest test - FAIL - Not using a permitted base image: ghcr.io/garethr/snykt/middleware 4 tests, 3 passed, 0 warnings, 1 failure, 0 exceptions

Slide 38

Slide 38 text

Allow images from our registry package main allowed := { "python:slim" } deny[msg] { not valid(input.docker.baseImage) msg = sprintf("Not using a permitted base image: %v", [input.docker.baseImage]) } valid(image) { allowed[image] } valid(image) { startswith(image, "ghcr.io/garethr/snykt/") } REGO

Slide 39

Slide 39 text

Ensure images pass the policy $ snyk container test ghcr.io/garethr/snykt/app --file=app/Dockerfile --json | conftest test - 4 tests, 4 passed, 0 warnings, 0 failure, 0 exceptions To enforce or not to enforce? Over time it’s best to move towards enforcing policies automatically, but to start with you’ll probably want to roll out slowly and carefully. You might already have these policies written down, but unless you’re automatically checking or enforcing you’ll likely have a long tail of images in breach. The advantage of Snyk is you can help developer test early in the SDLC, on the local machine or in CI, when it’s fast and cheap to fix the problem.

Slide 40

Slide 40 text

Demo

Slide 41

Slide 41 text

Automation and workflow Demos and ideas

Slide 42

Slide 42 text

Automate in your CI pipeline Embrace CI for building agreement between teams Using Snyk in your CI pipeline allows for automating this or other patterns for managing container images securely. Investing in good practices around CI should help security, build, operations and development teams work more closely together.

Slide 43

Slide 43 text

Use the Snyk API Integrate Snyk into your existing tools Different organizations, and different teams, might manage base images differently. The Snyk API makes it easy to build custom dashboards or use the data about base images for reporting. snyk.docs.apiary.io

Slide 44

Slide 44 text

Conclusions If all you remember is...

Slide 45

Slide 45 text

Observe How are your teams using base images today?

Slide 46

Slide 46 text

Orient Who actually fixes vulnerabilities? Do all teams bear the burden of fixing the same issues?

Slide 47

Slide 47 text

Decide How can you shift the responsibilities of teams to make base image management easier?

Slide 48

Slide 48 text

Act Put in place the automation to scale.

Slide 49

Slide 49 text

Thanks for listening Try Snyk for free at snyk.io/signup