Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Introduction to Software Supply Chain Security ...

Bob Killen
August 05, 2022

Introduction to Software Supply Chain Security & Tooling

You’re familiar with containers and Kubernetes, but suddenly people are talking about supply chains, and you’re pretty sure they don’t mean the Suez Canal. Join Jeffrey Sica and Bob Killen for an in-depth look at the nuts, bolts, and metaphorical shipping containers that make up modern container supply chain security. They’ll demo sigstore and show how container signing automation works with Kubernetes. You’ll leave prepared to make decisions that will keep your org unstuck.

Bob Killen

August 05, 2022
Tweet

More Decks by Bob Killen

Other Decks in Technology

Transcript

  1. $ whoami - Bob Bob Killen [email protected] OSS Program Manager

    @ Google Github: @mrbobbytables Twitter: @mrbobbytables Site: mrbobbytabl.es
  2. $ whoami - Jeff Jeffrey Sica [email protected] DevEx @ CNCF

    Github: @jeefy Twitter: @jeefy Site: jeefy.dev
  3. Before We Begin Pt. 2 Docker will be required to

    follow along with the demos. Docker is not required to use the tooling shown. https://docs.docker.com/engine/install/#desktop
  4. Before We Begin Pt. 3 KinD (Kubernetes in Docker) and

    Helm will be used for later demos. https://kind.sigs.k8s.io/docs/user/quick-start/#installation https://helm.sh/docs/intro/install/
  5. Containers: Galaxy-scale overview • Package an application with all of

    its dependencies (container image) • Run instances of the packaged application (container) • Think “Virtual Machine” but sharing kernel space and OS libraries (cgroups)
  6. Kubernetes: Galaxy-scale overview • “Container Orchestration” – It takes an

    entire orchestra to play a symphony • Manages containers and their dependencies (Storage, Networking, third-party resources) in a declarative way • 100% API Driven
  7. Solarwinds Attack • SolarWinds Orion network management tool’s build servers

    compromised • Hack injected a malicious DLL at build time that was signed and distributed to clients automatically when they updated • 18,000 SolarWinds customers including many Fortune 500 Companies, Government Agencies and Tier 1 Network Providers were impacted Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supply Chain in the Wake of the Sunburst Hack
  8. Solarwinds Attack • SolarWinds Orion network management tool’s build servers

    compromised • Hack injected a malicious DLL at build time that was signed and distributed to clients automatically when they updated • 18,000 SolarWinds customers including many Fortune 500 Companies, Government Agencies and Tier 1 Network Providers were impacted Project Trebuchet: How SolarWinds is Using Open Source to Secure Their Supply Chain in the Wake of the Sunburst Hack NO source code compromised
  9. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  10. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  11. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  12. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities Work on OSS? Want a security key? Find me after and get a free Titan key.
  13. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between the Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  14. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  15. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  16. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  17. US Gov Policy Changes Executive Order 14028 - (Sec. 2)

    Remove barriers to threat information sharing between Government and the Private Sector - (Sec. 3) Modernize and implement stronger cybersecurity standards in the Federal Government - (Sec. 4) Improve Software Supply Chain Security - (Sec. 5) Establish a Cybersecurity Safety Review Board - (Sec. 6) Create a standard playbook for responding to cyber incidents - (Sec. 7) Improve Detection of Cybersecurity Incidents on Federal Government Networks - (Sec. 8) Improve Investigative and Remediation Capabilities
  18. Supply Chain Attacks are on the rise and the Software

    Development Lifecycle has become a popular vector for attacks.
  19. Source Integrity Build Integrity Source Build Package Dev Inject bad

    code (A) Compromise source control (B) Build from modified sources (C) Dependencies Compromise build system (D) Bypass CI/CD, inject bad artifact (F) Compromise package repository (G) Use bad package (H) Users Software Supply Chain: Vulnerability Points Artifact Process Inject bad or vulnerable dependency (E)
  20. Source Integrity Build Integrity Source Build Package Dev Inject bad

    code (A) Compromise source control (B) Build from modified sources (C) Dependencies Compromise build system (D) Bypass CI/CD, inject bad artifact (F) Compromise package repository (G) Use bad package (H) Users Software Supply Chain: Trust Boundaries Artifact Process Inject bad or vulnerable dependency (E)
  21. SLSA Levels Automation & Provenance Build must be fully scripted/automated

    and generate provenance Version Control & Signed Provenance Requires using version control and hosted build service that generates authenticated provenance Non-falsifiable, Ephemeral Builds are fully trustworthy, with identity attestations of underlying build infrastructure/hardware. Ephemeral builds leave nothing behind. Hermetic Builds, Review All build inputs/dependencies are specified upfront with no internet egress during the build. Two-party reviews. https://slsa.dev/spec/v0.1/requirements
  22. Build Provenance Set of metadata that describes a software artifact

    and how it was built. SLSA spec fields* - subject - artifact(s) information (name+digest) - builder - the entity that produced the artifact(s) - invocation - execution command / information - buildConfig - record of steps executed during build - materials - other artifacts/dependencies used during build
  23. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }}
  24. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }}
  25. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} builder - Entity that produced the artifact Examples: "builder": { "id": "mailto:[email protected]" } "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" } "builder": { "id": "https://gitlab.com/foo/bar/-/runners/12345678" }
  26. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} invocation - Execution command / information Example: "invocation": { "configSource": { "uri": "git+https://github.com/foo/bar.git, "digest": { "sha1": "1234..."}, //git commit hash "entryPoint": "build.yaml:build" }, "parameters": {"inputs": {}}, "environment": { “arch”: “amd64”, "env": { "GITHUB_RUN_ID": "1234", "GITHUB_RUN_NUMBER": "5678", "GITHUB_EVENT_NAME": "push" } } }
  27. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} materials - other artifacts/dependencies used during build Example: "materials": [{ "uri": "https://example.com/example-1.2.3.tar.gz", "digest": {"sha256": "1234..."} }] "materials": [{ "uri": "git+https://github.com/foo/bar.git", "digest": {"sha1": "abc..."} }]
  28. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} buildConfig - record of steps executed during build Example: "buildConfig": { "steps": [ { "image": "pkg:docker/make@sha256:244fd47e07d1004f0aed9c", "arguments": ["build"] } ] } "buildConfig": { "commands": [ "./configure --enable-some-feature", "make foo.zip" ], "shell": "bash" }
  29. { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ ... }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate":

    { "builder": { "id": "<URI>" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" }, "parameters": { /* object */ }, "environment": { /* object */ } }, … … "buildConfig": { /* object */ }, "metadata": { "buildInvocationId": "<STRING>", "buildStartedOn": "<TIMESTAMP>", "buildFinishedOn": "<TIMESTAMP>", "completeness": { "parameters": true/false, "environment": true/false, "materials": true/false }, "reproducible": true/false }, "materials": [ { "uri": "<URI>", "digest": { /* DigestSet */ } } ] }} subject - Artifact(s) information (name+digest) Examples: "subject": [{"name": "_", "digest": {"sha256": "5678..."}}] "subject": [ { "name": "herpderp.exe", "digest": {"sha256": "1234..."} }]
  30. Software Attestation A software attestation is a signed set metadata

    (provenance) about one or more software artifacts. { "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder-key", //optional "sig": "Re4ya66MyFyc9Y..." }] }
  31. { "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder", "sig":

    "Re4ya66MyFyc9Y..." }] } { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ "name": "_", "digest": {"sha256": "5678..."} }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate": { "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" } …
  32. { "payload": "Gew9gICJzdWJqZWN0IjogWwogICAg...", "payloadType": "application/vnd.in-toto+json", "signatures": [{ "keyid": "my-awesome-builder", "sig":

    "Re4ya66MyFyc9Y..." }] } { "_type": "https://in-toto.io/Statement/v0.1", "subject": [{ "name": "_", "digest": {"sha256": "5678..."} }], "predicateType": "https://slsa.dev/provenance/v0.2", "predicate": { "builder": { "id": "https://github.com/Attestations/GitHubHostedActions@v1" }, "buildType": "<URI>", "invocation": { "configSource": { "uri": "<URI>", "digest": { /* DigestSet */ }, "entryPoint": "<STRING>" } … Thanks SLSA!
  33. An Overview Of sigstore • fulcio ◦ Identity • rekor

    ◦ Immutable log with artifact metadata • cosign ◦ Utility to automate combining fulcio, rekor, and OCI-compliant artifacts ◦ Utilizes underlying sigstore libraries
  34. fulcio Free Root-CA for code signing certs - issuing certificates

    based on an OIDC email address fulcio only signs short-lived certificates that are valid for under 20 minutes https://docs.sigstore.dev/fulcio/overview https://github.com/sigstore/fulcio
  35. fulcio Free Root-CA for code signing certs - issuing certificates

    based on an OIDC email address. fulcio only signs short-lived certificates that are valid for under 20 minutes. https://twitter.com/jacques_chester/status/1506332697387491335
  36. rekor Greek for “Record” rekor is a transparency log where

    anyone can find and verify signatures, and check whether someone’s changed the source code, the build platform or the artifact repository. https://docs.sigstore.dev/rekor/overview https://github.com/sigstore/rekor
  37. cosign cosign: Container Signing, verification and storage in an OCI

    registry “Make signatures invisible infrastructure” Pluggable components https://docs.sigstore.dev/cosign/overview https://github.com/sigstore/cosign
  38. Demo Recap • Built a container • Auth’d using cosign

    • Signed a container using cosign • Inspected the container’s signature using cosign
  39. Kubernetes Admission Webhooks Lightweight extensible way to “do something” with

    API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook
  40. Kubernetes Admission Webhooks Lightweight extensible way to “do something” with

    API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook
  41. Kubernetes Admission Webhooks … webhooks: - name: "pod-policy.example.com" rules: -

    apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE"] resources: ["pods"] scope: "Namespaced"
  42. Kubernetes Admission Webhooks Lightweight extensible way to “do something” with

    API Server requests Two kinds: - Validating Admission Webhook - Mutating Admission Webhook
  43. Demo Recap • Installed Kyverno • Attempted to create a

    Pod with an unsigned container image • Successfully created a Pod with a signed container image Task Failed Successfully!
  44. Demo Recap • Installed the Kyverno • Attempted to create

    a Pod with an unsigned container image • Successfully created a Pod with a signed container image Task Failed Successfully!
  45. GitHub Action jobs: test_cosign_action: runs-on: ubuntu-latest permissions: {} name: Install

    Cosign and test presence in path steps: - name: Install Cosign uses: sigstore/cosign-installer@main - name: Check install! run: cosign version https://github.com/sigstore/cosign-installer
  46. GitHub Action … - name: Sign image with a key

    run: | cosign sign --key env://COSIGN_PRIVATE_KEY ${TAGS} env: TAGS: ${{ steps.docker_meta.outputs.tags }} COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}} COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}} https://github.com/sigstore/cosign-installer
  47. Batteries included support for cosign Create a secret called signing-secrets

    with the following structure: • cosign.key • cosign.password cosign generate-key-pair k8s://tekton-chains/signing-secrets Tekton (Chains)
  48. Demo Recap • Set up GitHub Action • Use cosign

    to sign the image • Use SLSA tooling to generate attestation • User cosign to attach attestation in rekor • Inspected and verified the image and SLSA payload
  49. gitsign # Sign all commits git config --global commit.gpgsign true

    # Sign all tags git config --global tag.gpgsign true # Use gitsign for signing git config --global gpg.x509.program gitsign # gitsign expects x509 args git config --global gpg.format x509 https://github.com/sigstore/gitsign
  50. Demo Recap • Signed a git commit with gitsign •

    Inspected the rekor log of the commit
  51. Tutorial Recap • Supply chain security is an increasing concern

    when building, shipping, and running code. • The SLSA framework can be used to enable or improve trust in the software that we build and run.
  52. Tutorial Recap • sigstore looks to solve supply chain security

    with an end to end solution based on linking Identity and TLS certificates • Automating SLSA provenance generation and artifact signing lowers the mental load on developers shipping software
  53. Tutorial Recap • Starts with Containers / OCI-compliant artifacts •

    Already moving further down the stack to packages
  54. SLSA Framework Build Integrity • Modification of code after source

    control • Compromised build platforms • Bypassing CI/CD Source Integrity • Available change history • Code review • Compromised source control systems Dependencies • Applying SLSA checks recursively to dependencies • Dependency confusion