Shifting left in security: Detecting security threats earlier to minimize risk

The cost of fixing security vulnerabilities and the damage they can do rises exponentially with the time of discovery and mitigation. In this talk Tamas will highlight the importance of including security early in the software development life cycle and will highlight Google's offerings (binary authorization, container scanning and others) to establish trust in the build process.

Transcript

2. ABOUT THE SPEAKER The journey to the cloud: • Mechatronics B.Sc. 🤖, Automotive Software Engineering M.Sc 🚗. • Software developer & Product owner at a startup in München • Started as Java backend engineer. • Transitioned to DevOps and Cloud ☁
3. AGENDA • Mitigating threats in the software supply chain • “Shifting left on security” • Securing artifacts and enforcing policies at deploy time • Wrap up
4. WHAT IS THE SOFTWARE SUPPLY CHAIN? • “A software supply chain is composed of the components, libraries, tools, and processes used to develop, build, and publish a software artifact.” (Wikipedia) (Google Cloud Events)
5. BYPASSED CODE REVIEW (Software supply chain threats – Google Docs)
6. BYPASSED CODE REVIEW • “Hypocrite commits” – (experiment by Qiushi Wu and Kangjie Lu University of Minnesota) • ”How hard it is to sneak a vulnerability into Linux Kernel code?” • Vulnerability is formed by multiple conditions, located at different parts of the code base • A hypocrite commit fixes a seemingly minor issue, but in reality is the missing condition to form a vulnerability
7. ENFORCING CODE REVIEWS • Four eyes principle 👀 • Protect master/main branch • Specify a list of users who act as code owners (CODEOWNERS file) and require their approval before work can merge.
8. COMPROMISED SOURCE CONTROL (Software supply chain threats – Google Docs)
9. COMPROMISED SOURCE CONTROL • Securing source control: private network, authentication (+2FA) • Signed commits • Create your GPG key, upload your public key to GitLab/GitHub • Enforce signed commits • Start signing (git commit –S –m “…”)
10. SECURING SOURCE CODE • Source code analysis • Secret scanning 🔒 • Gitleaks (OOS) • git-hooks • Client-side hooks (pre-commit) • Server-side hooks (pre-receive) • GitHub Action • Safe Docker base images • Multi-stage Docker builds • Use distroless images (gcr.io/distroless)
11. • Protect your source repository • Not accessible from public network • 2FA authentication enforced (UI) • Stay up to date • Developer access over SSH keys => rotate regularly • Machine/service access via ephemeral credentials • Repository best practices • Four-eyes principle • Use (secret) scanners in pre-commit/receive hooks • Define CODEOWNERS • Source Code Integrity • Signed Commits • Integrity: an unbroken chain of trust during build and deployment time • Imlemented using cryptographic signatures. MITIGATING SOURCE THREATS
12. COMPROMISED DEPENDENCY (Software supply chain threats – Google Docs)
13. COMPROMISED DEPENDENCY • NPM: eventstreams • Given a package with 1.9mio weekly downloads. Maintainer is busy, no meaningful updates in years • A contributor volunteers and takes over. • Contributor takes ownership on npm • Adds support for flat-map • A week later removes dependency and adds built-in implementation. Bumps major version making sure many are left behind using the previous infected versin. • The flat-map library • LGTM on GitHub, although 1 contributor • However, the version published to npm snuck some additional code into the minified file • In there there is an additional test/data.js. This is encrypted, but the npm_package_description as the AES256 key is used to decrypt it. • For the vast majority of parent packages, this will result in an error (which the malicious code silently catches and ignores), since their package description won’t be the correct AES256 key and the output will be nonsense. • targeted package: copay-dash, a bitcoin wallet platform. Its description, "A Secure Bitcoin Wallet", successfully decrypts the test/data.js • Then another round of decryption is used to finally steal your bitcoin wallet…
14. COMPROMISED DEPENDENCY • Integrity of dependencies
15. COMPROMISED DEPENDENCY • Verifying checksums: • Transitive dependencies
16. COMPROMISED DEPENDENCY • Verifying signatures: • Gradle only supports verification of signatures published on remote repositories as ASCII-armored PGP files • Not all artifacts are published with signatures • Gradle will try to download the corresponding .asc file. If present, downloads the keys required to perform verification and then verifies the signature.
17. COMPROMISED DEPENDENCY • Dependency analysis • Dependency scanning • (open-source) OWASP dependency check (“DependencyCheck”) • Maven/Gradle plugin • (commercial) Snyk • Dependency upgrade • Dependabot, renovate • Open Source Insights (https://deps.dev/) • A web site that provides information about known direct and indirect dependencies, known vulnerabilities, and license information for open source software. • Data available as a Google Cloud Dataset => Use BigQuery to explore and analyse data. • Open Source Vulnerabilities database (https://osv.dev/) • A searchable vulnerability database
18. UPLOAD MODIFIED PACKAGE (Software supply chain threats – Google Docs)
19. UPLOAD MODIFIED PACKAGE • CodeCov, the The Leading Code Coverage Solution • Product: code analysis tool, which users integrate into their CI/CD • Root cause: • The attacker was able to extract a key for a Google Cloud Storage service account from an intermediate layer in our public Codecov Self-Hosted Docker image. • The attacker used this key to modify a shell script in Google Cloud Storage, which was served directly to end-users with the malicious changes in place.
20. UPLOAD MODIFIED PACKAGE
21. UPLOAD MODIFIED PACKAGE • CodeCov, the The Leading Code Coverage Solution • Product: code analysis tool, which users integrate into their CI/CD • It happens once, that hackers have placed a single malicious line to a 1k+ line bash script. • Ran for 2.5+ months…
22. • Verify integrity of dependencies / tools • Verify integrity of source code (tag signature) • Secure buld environment: • Hermetic builds: network perimeter to prevent exfiltration (VPC Service Controls), dependencies from trusted, private registry • Build services should generate provenance • i.e. an image signature claiming that „Built by CloudBuild®“ MITIGATING BUILD THREATS (SLSA - Provenance)
23. MITIGATING BUILD THREATS
24. SUPPLY-CHAIN LEVELS FOR SOFTWARE ARTIFACTS (SLSA) • It’s a security framework, a check-list of standards and controls to prevent tampering, improve integrity, and secure packages and infrastructure in your projects, businesses or enterprises.
25. SUPPLY-CHAIN LEVELS FOR SOFTWARE ARTIFACTS (SLSA)
26. “SHIFTING LEFT ON SECUIRTY” • Benefits of catching security defects early? 2021 DevOps Report (“Dora”)
27. CONTAINER/ARTIFACT REGISTRY • Google’s solutions: • Container Registry • For Docker images only • Artifact Registry • For Docker images, application dependencies and libraries. • Container Analysis • Store metadata • Container Analysis is built on top of Grafeas, an open source component metadata API • Automatic image scanning for OS vulnerabilities / On-demand scanning
28. CONTAINER/ARTIFACT REGISTRY • Container Analysis:
29. ENFORCING ATTESTATIONS USING BINARY AUTHORIZATION • Binary Authorization (BA) is a deploy-time security control for GKE or CloudRun. Enforces that only signed and attested images are deployed. • BA uses a policy that is a set of rules that govern the deployment and validation of container images • Rules define required attestations that are applied on cluster/namespace/SA level.
30. KEY CONCEPTS • Note (CA API): piece of metadata • Attestor (BA API): Google Cloud resource that Binary Authorization uses to verify the attestation at image deploy time. • Attestation (CA API): digital document that certifies an image. References: • NoteId • PubKeyId of attestor
31. FROM VULNERABILITIES TO ENFORCEMENT Source Repository Cloud Build Container Registry GKE Container Analysis Binary Authorization
32. CREATING ATTESTATIONS: GCLOUD CLI • Create Note • Create attestor (referencing the note) • Add public key to attestor • Create attestation • Sources: • https://cloud.google.com/binary-authorization/docs/creating-attestors-cli • https://cloud.google.com/binary-authorization/docs/making-attestations
33. CREATING ATTESTATIONS: CLOUD BUILD • Allows you to build an image that has a helper shell script. Supports: • KMS Key-based Signing • PGP Key-based Signing • Sources: • https://github.com/GoogleCloudPlatform/cloud-builders-community/tree/master/binauthz- attestation
34. CREATING ATTESTATIONS: KRITIS SIGNER • Kritis Signer can create attestations based on identified software vulnerabilities in your container images. • Steps: • Create Note • Create KMS key • Create GSA for Kritis, grant necessary rights • Define policy • Run Kritis Signer
35. CREATING ATTESTATIONS: KRITIS SIGNER • No attestor created by Kritis Signer • But the attestation can be retrieved for our Note:
36. BINARY AUTHORIZATION: BREAKGLASS • Breakglass • Breakglass event is automatically logged to Cloud Audit Logs => useful to trigger notifications / alerts based on these logs!
37. TAKEWAY • Secure base images • Distroless images for applications • Bases of helper/build images: • Check regularly for updates • Vulnerability scanners (Container Analysis) • Secure your source code: • Enforce code reviews • Enforce signed commits • Code scanners • Leaked secrets (server side pre-receive hooks) • Dependency scanners (Vulnerability, Updates) • Start building up integrity: • When doing release: sign the tag, verify tag signature in build pipeline • Signed Containers => Binary Authorization • Apply SLSA step by step (L1 => L2 => L3 => L4)