Introduction to Software Supply Chain Security & Tooling

Supply Chain
Security Tooling

View Slide

Welcome!
Dependencies, Tutorials, Links to Slides:
https://github.com/jeefy/sigstore-intro-tutorial

View Slide

$ whoami - Bob
Bob Killen
[email protected]
OSS Program Manager @ Google
Github: @mrbobbytables
Twitter: @mrbobbytables
Site: mrbobbytabl.es

View Slide

$ whoami - Jeff
Jeffrey Sica
[email protected]
DevEx @ CNCF
Github: @jeefy
Twitter: @jeefy
Site: jeefy.dev

View Slide

Before We Begin
https://docs.sigstore.dev/cosign/installation/

View Slide

Before We Begin Pt. 2
Docker will be required to follow along with the
demos.
Docker is not required to use the tooling shown.
https://docs.docker.com/engine/install/#desktop

View Slide

Before We Begin Pt. 3
KinD (Kubernetes in Docker) and Helm will be
used for later demos.
https://kind.sigs.k8s.io/docs/user/quick-start/#installation
https://helm.sh/docs/intro/install/

View Slide

Disclaimer
You should have some knowledge of
containers and Kubernetes

View Slide

Containers: Galaxy-scale overview
● Package an application with all of its dependencies
(container image)
● Run instances of the packaged application (container)
● Think “Virtual Machine” but sharing kernel space and OS
libraries (cgroups)

View Slide

Kubernetes: Galaxy-scale overview
● “Container Orchestration” – It takes an entire orchestra
to play a symphony
● Manages containers and their dependencies (Storage,
Networking, third-party resources) in a declarative way
● 100% API Driven

View Slide

So… What Is Supply Chain Security?
Sourcing Inventory Production
Planning Transportation

View Slide

The “Software Supply Chain”
Code Artifacts Distribution
Dependencies Execution

View Slide

Current State?

View Slide

Current State
https://www.bike-eu.com/market/nieuws/2022/01/imbalance-in-shipping-container-supply-chain-is-far-from-over-10142078

View Slide

Current State
https://www.cnbc.com/2021/03/26/satellite-images-of-ship-ever-given-in-suez-canal-shows-work-underway.html

View Slide

Current State
https://www.freightwaves.com/news/insurers-call-for-action-to-prevent-containership-fires

View Slide

https://sonatype.com/resources/white-paper-2021-state-of-the-software-supply-chain-report-2021

View Slide

View Slide

Solarwinds Attack
● SolarWinds Orion network management tool’s build servers
compromised
● Hack injected a malicious DLL at build time that was signed
and distributed to clients automatically when they updated
● 18,000 SolarWinds customers including many Fortune 500
Companies, Government Agencies and Tier 1 Network
Providers were impacted
Project Trebuchet: How SolarWinds
is Using Open Source to Secure
Their Supply Chain in the Wake of
the Sunburst Hack

View Slide

Solarwinds Attack
● SolarWinds Orion network management tool’s build servers
compromised
● Hack injected a malicious DLL at build time that was signed
and distributed to clients automatically when they updated
● 18,000 SolarWinds customers including many Fortune 500
Companies, Government Agencies and Tier 1 Network
Providers were impacted
Project Trebuchet: How SolarWinds
is Using Open Source to Secure
Their Supply Chain in the Wake of
the Sunburst Hack
NO source code compromised

View Slide

https://sonatype.com/resources/white-paper-2021-state-of-the-software-supply-chain-report-2021

View Slide

US Gov Policy Changes
Executive Order 14028
- (Sec. 2) Remove barriers to threat information sharing between
Government and the Private Sector
- (Sec. 3) Modernize and implement stronger cybersecurity
standards in the Federal Government
- (Sec. 4) Improve Software Supply Chain Security
- (Sec. 5) Establish a Cybersecurity Safety Review Board
- (Sec. 6) Create a standard playbook for responding to cyber
incidents
- (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal
Government Networks
- (Sec. 8) Improve Investigative and Remediation Capabilities

View Slide

- (Sec. 2) Remove barriers to threat information sharing
between the Government and the Private Sector
incidents
Government Networks

View Slide

- (Sec. 2) Remove barriers to threat information sharing between the
incidents
Government Networks

View Slide

incidents
Government Networks
Work on OSS? Want a security key?
Find me after and get a free Titan key.

View Slide

incidents
Government Networks

View Slide

- (Sec. 6) Create a standard playbook for responding to
cyber incidents
Government Networks

View Slide

incidents
- (Sec. 7) Improve Detection of Cybersecurity Incidents on
Federal Government Networks

View Slide

incidents
Government Networks
- (Sec. 8) Improve Investigative and Remediation
Capabilities

View Slide

The Problem In A Nutshell

View Slide

Supply Chain Attacks are on the rise and
the Software Development Lifecycle has
become a popular vector for attacks.

View Slide

How can we trust the software we
choose to run?

View Slide

View Slide

A Reasonable Solution

View Slide

SLSA

View Slide

Supply chain
Levels for
Software
Artifacts 💃
slsa.dev

View Slide

Source Integrity Build Integrity
Source Build Package
Dev
Dependencies
Users
Software Supply Chain
Artifact
Process

View Slide

Dev
Inject bad
code (A)
Compromise
source control (B)
Build from modiﬁed
sources (C)
Dependencies
Compromise
build system (D)
Bypass CI/CD, inject
bad artifact (F)
Compromise
package repository (G)
Use bad package (H)
Users
Software Supply Chain:
Vulnerability Points
Artifact
Process
Inject bad or
vulnerable
dependency (E)

View Slide

Dev
Inject bad
code (A)
Compromise
source control (B)
Build from modiﬁed
sources (C)
Dependencies
Compromise
build system (D)
Bypass CI/CD, inject
bad artifact (F)
Compromise
package repository (G)
Use bad package (H)
Users
Software Supply Chain:
Trust Boundaries
Artifact
Process
Inject bad or
vulnerable
dependency (E)

View Slide

SLSA Levels
Automation & Provenance
Build must be fully scripted/automated
and generate provenance
Version Control & Signed
Provenance
Requires using version control and hosted
build service that generates authenticated
provenance
Non-falsiﬁable, Ephemeral
Builds are fully trustworthy, with identity
attestations of underlying build
infrastructure/hardware. Ephemeral
builds leave nothing behind.
Hermetic Builds, Review
All build inputs/dependencies are speciﬁed
upfront with no internet egress during the build.
Two-party reviews.
https://slsa.dev/spec/v0.1/requirements

View Slide

Build Provenance
Set of metadata that describes a software
artifact and how it was built.
SLSA spec ﬁelds*
- subject - artifact(s) information (name+digest)
- builder - the entity that produced the artifact(s)
- invocation - execution command / information
- buildConﬁg - record of steps executed during build
- materials - other artifacts/dependencies used
during build

View Slide

{
"_type": "https://in-toto.io/Statement/v0.1",
"subject": [{ ... }],
"predicateType": "https://slsa.dev/provenance/v0.2",
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"digest": { /* DigestSet */ },
"entryPoint": ""
},
"parameters": { /* object */ },
"environment": { /* object */ }
},
…
…
"buildConfig": { /* object */ },
"metadata": {
"buildInvocationId": "",
"buildStartedOn": "",
"buildFinishedOn": "",
"completeness": {
"parameters": true/false,
"environment": true/false,
"materials": true/false
},
"reproducible": true/false
},
"materials": [
{
"uri": "",
"digest": { /* DigestSet */ }
}
]
}}

View Slide

{
"subject": [{ ... }],
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
},
},
…
…
"metadata": {
"completeness": {
},
},
"materials": [
{
"uri": "",
}
]
}}
builder - Entity that produced the artifact
Examples:
"builder": { "id": "mailto:[email protected]" }
"builder": {
"id": "https://github.com/Attestations/GitHubHostedActions@v1"
}
"builder": {
"id": "https://gitlab.com/foo/bar/-/runners/12345678"
}

View Slide

{
"subject": [{ ... }],
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
},
},
…
…
"metadata": {
"completeness": {
},
},
"materials": [
{
"uri": "",
}
]
}}
invocation - Execution command / information
Example:
"invocation": {
"configSource": {
"uri": "git+https://github.com/foo/bar.git,
"digest": { "sha1": "1234..."}, //git commit hash
"entryPoint": "build.yaml:build"
},
"parameters": {"inputs": {}},
"environment": {
“arch”: “amd64”,
"env": {
"GITHUB_RUN_ID": "1234",
"GITHUB_RUN_NUMBER": "5678",
"GITHUB_EVENT_NAME": "push"
}
}
}

View Slide

{
"subject": [{ ... }],
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
},
},
…
…
"metadata": {
"completeness": {
},
},
"materials": [
{
"uri": "",
}
]
}}
materials - other artifacts/dependencies
used during build
Example:
"materials": [{
"uri": "https://example.com/example-1.2.3.tar.gz",
"digest": {"sha256": "1234..."}
}]
"materials": [{
"uri": "git+https://github.com/foo/bar.git",
"digest": {"sha1": "abc..."}
}]

View Slide

{
"subject": [{ ... }],
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
},
},
…
…
"metadata": {
"completeness": {
},
},
"materials": [
{
"uri": "",
}
]
}}
buildConfig - record of steps executed
during build
Example:
"buildConfig": {
"steps": [
{
"image": "pkg:docker/make@sha256:244fd47e07d1004f0aed9c",
"arguments": ["build"]
}
]
}
"buildConfig": {
"commands": [
"./configure --enable-some-feature",
"make foo.zip"
],
"shell": "bash"
}

View Slide

{
"subject": [{ ... }],
"predicate": {
"builder": {
"id": ""
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
},
},
…
…
"metadata": {
"completeness": {
},
},
"materials": [
{
"uri": "",
}
]
}}
subject - Artifact(s) information (name+digest)
Examples:
"subject": [{"name": "_", "digest": {"sha256": "5678..."}}]
"subject": [ {
"name": "herpderp.exe",
"digest": {"sha256": "1234..."}
}]

View Slide

Software Attestation
A software attestation is a signed set metadata
(provenance) about one or more software artifacts.
{
"payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...",
"payloadType": "application/vnd.in-toto+json",
"signatures": [{
"keyid": "my-awesome-builder-key", //optional
"sig": "Re4ya66MyFyc9Y..."
}]
}

View Slide

{
"signatures": [{
"keyid": "my-awesome-builder",
}]
}
{
"subject": [{
"name": "_",
"digest": {"sha256": "5678..."}
}],
"predicate": {
"builder": {
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
}
…

View Slide

{
"signatures": [{
"keyid": "my-awesome-builder",
}]
}
{
"subject": [{
"name": "_",
"digest": {"sha256": "5678..."}
}],
"predicate": {
"builder": {
},
"buildType": "",
"invocation": {
"configSource": {
"uri": "",
"entryPoint": ""
}
…
Thanks SLSA!

View Slide

Is There A Reasonable Technical
Solution?

View Slide

sigstore
https://sigstore.dev

View Slide

Community Driven

View Slide

Let’s Encrypt?
Let’s Sign.

View Slide

View Slide

An Overview Of sigstore
● fulcio
● rekor
● cosign

View Slide

An Overview Of sigstore
● fulcio
○ Identity
● rekor
○ Immutable log with artifact metadata
● cosign
○ Utility to automate combining fulcio, rekor, and
OCI-compliant artifacts
○ Utilizes underlying sigstore libraries

View Slide

fulcio
Free Root-CA for code signing certs - issuing certiﬁcates based on an
OIDC email address
fulcio only signs short-lived certiﬁcates that are valid for under 20
minutes
https://docs.sigstore.dev/fulcio/overview
https://github.com/sigstore/fulcio

View Slide

fulcio
Free Root-CA for code signing certs - issuing certiﬁcates based on an
OIDC email address.
fulcio only signs short-lived certiﬁcates that are valid for under 20
minutes.
https://twitter.com/jacques_chester/status/1506332697387491335

View Slide

rekor
Greek for “Record”
rekor is a transparency log where anyone can ﬁnd and verify signatures,
and check whether someone’s changed the source code, the build
platform or the artifact repository.
https://docs.sigstore.dev/rekor/overview
https://github.com/sigstore/rekor

View Slide

cosign
cosign: Container Signing, veriﬁcation and storage in an OCI registry
“Make signatures invisible infrastructure”
Pluggable components
https://docs.sigstore.dev/cosign/overview
https://github.com/sigstore/cosign

View Slide

How it works!
https://www.sigstore.dev/how-it-works

View Slide

The. Demo.

View Slide

Demo Recap
● Built a container
● Auth’d using cosign
● Signed a container using cosign
● Inspected the container’s signature using
cosign

View Slide

Cool I have a signed container.
Now what?

View Slide

Kubernetes Admission Webhooks
Lightweight extensible way to “do something”
with API Server requests
Two kinds:
- Validating Admission Webhook
- Mutating Admission Webhook

View Slide

…
webhooks:
- name: "pod-policy.example.com"
rules:
- apiGroups: [""]
apiVersions: ["v1"]
operations: ["CREATE"]
resources: ["pods"]
scope: "Namespaced"

View Slide

Lightweight extensible way to “do something”
with API Server requests
Two kinds:
- Validating Admission Webhook
- Mutating Admission Webhook

View Slide

sigstore Policy Controller
…
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: image-policy
spec:
images:
- glob: "**"

View Slide

sigstore Policy Controller
…
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
name: image-policy
spec:
images:
- glob: "**"
Beta

View Slide

Kubernetes native policy engine
Entirely CRD-based
https://kyverno.io

View Slide

Demo: Part Deux

View Slide

Demo Recap
● Installed Kyverno
● Attempted to create a Pod with an unsigned
container image
● Successfully created a Pod with a signed
container image Task Failed
Successfully!

View Slide

Demo Recap
● Installed the Kyverno
● Attempted to create a Pod with an unsigned
container image
● Successfully created a Pod with a signed
container image Task Failed
Successfully!

View Slide

kyverno is
https://kyverno.io
powerful
configurable
complicated

View Slide

Let’s Talk Automation

View Slide

https://github.blog/2021-12-06-safeguard-container-signing-capability-actions/

View Slide

GitHub Action
jobs:
test_cosign_action:
runs-on: ubuntu-latest
permissions: {}
name: Install Cosign and test presence in path
steps:
- name: Install Cosign
uses: sigstore/cosign-installer@main
- name: Check install!
run: cosign version
https://github.com/sigstore/cosign-installer

View Slide

GitHub Action
…
- name: Sign image with a key
run: |
cosign sign --key env://COSIGN_PRIVATE_KEY ${TAGS}
env:
TAGS: ${{ steps.docker_meta.outputs.tags }}
COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
https://github.com/sigstore/cosign-installer

View Slide

Batteries included support for cosign
Create a secret called signing-secrets with
the following structure:
● cosign.key
● cosign.password
cosign generate-key-pair k8s://tekton-chains/signing-secrets
Tekton (Chains)

View Slide

“Demo”: The Third

View Slide

Demo Recap
● Set up GitHub Action
● Use cosign to sign the image
● Use SLSA tooling to generate attestation
● User cosign to attach attestation in rekor
● Inspected and veriﬁed the image and SLSA
payload

View Slide

sigstore: Not Just Containers

View Slide

Rust Crate Signing

View Slide

PyPi Signing
https://pypi.org/project/sigstore/

View Slide

sigstore: Not Just Artifacts

View Slide

gitsign
Keyless Git signing with Sigstore!
Homebrew:
brew install sigstore/tap/gitsign
Go:
go install github.com/sigstore/gitsign@latest

View Slide

gitsign
# Sign all commits
git config --global commit.gpgsign true
# Sign all tags
git config --global tag.gpgsign true
# Use gitsign for signing
git config --global gpg.x509.program gitsign
# gitsign expects x509 args
git config --global gpg.format x509
https://github.com/sigstore/gitsign

View Slide

Demo #4

View Slide

Demo Recap
● Signed a git commit with gitsign
● Inspected the rekor log of the commit

View Slide

Tutorial Recap
● Supply chain security is an increasing
concern when building, shipping, and running
code.
● The SLSA framework can be used to enable
or improve trust in the software that we build
and run.

View Slide

Tutorial Recap
● sigstore looks to solve supply chain security
with an end to end solution based on linking
Identity and TLS certiﬁcates
● Automating SLSA provenance generation
and artifact signing lowers the mental load
on developers shipping software

View Slide

Tutorial Recap
● Starts with Containers / OCI-compliant
artifacts
● Already moving further down the stack to
packages

View Slide

Secure Distribution
By Default

View Slide

Thanks + Q&A
A @mrbobbytables + @jeefy Production

View Slide

Graveyard

View Slide

View Slide

SLSA Framework
Build Integrity
● Modiﬁcation of code after source control
● Compromised build platforms
● Bypassing CI/CD
Source Integrity
● Available change history
● Code review
● Compromised source control systems
Dependencies
● Applying SLSA checks recursively to dependencies
● Dependency confusion

View Slide

Introduction to Software Supply Chain Security & Tooling

Introduction to Software Supply Chain Security & Tooling

Bob Killen

More Decks by Bob Killen

Other Decks in Technology

Featured

Transcript