Patterns for secure container base image management
Presentation from SnykCon 2020, all about the people, process and tools needed to manage container base images. Thoughts about team organisation, trade-offs and examples of how to use Snyk to solve this problem.
dependencies for known issues, found 180 issues. Debian 8 is no longer supported by the Debian maintainers. Vulnerability detection may be affected by a lack of security updates. You test an image for vulnerabilities and ﬁnd LOTS of issues 100s of vulnerabilities! It’s using an out-of-date operating system! Who owns this image? Who is responsible for ﬁxing vulnerabilities? How many other images like this do we have? Help!
software giants It’s common with container images to start building on top of an existing base image that already has software you want. This might be an operating system like ubuntu, alpine or debian or it could be a language like python, ruby, node or really anything else. OK, technically often a parent image but hey. Libraries and underlying software provided by someone else Your software
common hardening or conﬁguration changes or maybe metadata it wants to apply to all images in use by other teams. This is often intended to be common for all images used. Maybe you have myorg/base Libraries and underlying software provided by someone else Hardening, common conﬁguration Common software Your application
common software or middleware. This might be language or framework speciﬁc, say a separate image for Java (myorg/java) and another for Python (myorg/python). Libraries and underlying software provided by someone else Hardening, common conﬁguration Common software Your application
When considering container vulnerabilities, you want to be able to reason about vulnerabilities in images you’re running, but also understand the overlap and source of those vulnerabilities. Can you address a vulnerability once, and have it resolved everywhere?
else Hardening, common conﬁguration Common software Your application Libraries and underlying software provided by someone else Hardening, common conﬁguration Common software Your application Libraries and underlying software provided by someone else Hardening, common conﬁguration Common software Your application One team to rule them all This could be the case when teams are completely independent, or when you have one central image team. A base image team A team which provides a standard set of approved base images for application teams to consume. Separate base/security teams Larger organizations might have teams with more distinct responsibilities, potentially with even more layers.
Simple to understand responsibilities. CONS Potential for chaos if every team can do their own thing. One central team for ALL images likely to become a bottleneck. A base image team PROS Able to build strong domain expertise in the center, ideally ﬁx/triage issues once. CONS Needs some level of governance in order to ensure applications teams beneﬁt from central expertise. Separate base/security teams PROS Same as having a base image team, with the added advantage of deeper specialisms. CONS Coordination between additional teams can slow down the process.
Introduced by your base image (python:3.6.0-slim) Fixed in: 5.28.1-6+deb10u1 ✗ High severity vulnerability found in gnutls28/libgnutls30 Description: Out-of-bounds Write Info: https://snyk.io/vuln/SNYK-DEBIAN10-GNUTLS28-609778 Introduced through: email@example.com+deb10u4, firstname.lastname@example.org From: email@example.com+deb10u4 From: firstname.lastname@example.org > email@example.com+deb10u4 Introduced by your base image (python:3.6.0-slim) Tested 111 dependencies for known issues, found 178 issues.
--file=middleware/Dockerfile ... Tested 127 dependencies for known issues, found 180 issues. Vulnerabilities from the base image and the new instructions But a diﬀerent team is responsible for some of these, so let’s reason about those separately.
plays by the rules. There is a useful balance between centrally enforced policies and encouraging best practices. How do you make it as easy as possible for consumers of base images to do the right thing?
Decision (any JSON value) Data (JSON) Policy (Rego) Request, Event, etc. Declarative Express policy in a high-level, declarative language that promotes safe, performant, ﬁne-grained controls. Use a language purpose-built for policy in a world where JSON is pervasive. Context-aware Leverage external information to write the policies you really care about. Write logic that adapts to the world around it and attach that logic to the systems that need it.
input.vulnerabilities[index] issue.severity = "high" msg = sprintf("High severity issue found. package: %v issue: %v", [issue.name, issue.title]) } A Rego example This is the Open Policy Agent language for describing policies. Here we’re saying we want to prohibit any images with known high-severity vulnerabilities. REGO
--file=app/Dockerfile --json | conftest test - 4 tests, 4 passed, 0 warnings, 0 failure, 0 exceptions To enforce or not to enforce? Over time it’s best to move towards enforcing policies automatically, but to start with you’ll probably want to roll out slowly and carefully. You might already have these policies written down, but unless you’re automatically checking or enforcing you’ll likely have a long tail of images in breach. The advantage of Snyk is you can help developer test early in the SDLC, on the local machine or in CI, when it’s fast and cheap to ﬁx the problem.
between teams Using Snyk in your CI pipeline allows for automating this or other patterns for managing container images securely. Investing in good practices around CI should help security, build, operations and development teams work more closely together.
Diﬀerent organizations, and diﬀerent teams, might manage base images diﬀerently. The Snyk API makes it easy to build custom dashboards or use the data about base images for reporting. snyk.docs.apiary.io