Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Let’s Secure a CI/CD Pipeline

Let’s Secure a CI/CD Pipeline

Originally given at O'Reilly Media Cloud Superstream: Cloud Security on June 8, 2022.

Your team uses a CI/CD pipeline to deploy infrastructure and applications to production. However, your security team warns you that your pipeline might be vulnerable. Can you improve it? Join Rosemary Wang to discover how to assess and secure your CI/CD pipeline and protect it from supply chain attacks. You’ll learn some patterns and tools that can help and find out how to scale the practices across your system.

Be8b596c46f4c9a1aec6a7586af33134?s=128

Rosemary Wang

June 08, 2022
Tweet

More Decks by Rosemary Wang

Other Decks in Technology

Transcript

  1. Rosemary Wang | June 8, 2022 Let’s Secure a CI/CD

    Pipeline 1
  2. jenkins.io/security/advisory/2022-02-15/ about.codecov.io/security-update/ msrc-blog.microsoft.com/2020/12/31/microsoft-internal-solorigate-investigation-update/ 2

  3. 🤔 + 😬 = 😱 3

  4. Rosemary Wang (she/her) Developer Advocate at HashiCorp @joatmon08 joatmon08.github.io 4

  5. How to secure better… 5

  6. How to secure better… and remediate faster. 6

  7. Securing CI/CD Pipelines An Overview ✓ Access Control ✓ Secrets

    ✓ Runners ✓ Dependencies ✓ Configuration 7
  8. Access Control 🚥 8

  9. limit service’s access rights to minimum required least privilege 9

  10. Pipelines need access. Code Repository accesses… Build infrastructure. Infrastructure Provider

    Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Run integration tests. Store for User Data 10
  11. Choose your access. Code Repository Checkout code. Read repositories. 11

  12. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout

    code. Read repositories. Write speci fi c services. 12
  13. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout

    code. Deploy application. Application Platform / Release Repository Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. 13
  14. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout

    code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. 14
  15. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout

    code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. Read test results. 15
  16. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout

    code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. Read test results. Run integration tests. Store for User Data Read table in testing environment. 16
  17. The challenge of least privilege Solutions ★ Limit access from

    pipeline early in development ★ Limit write access to repositories early in development 17
  18. The challenge of least privilege Solutions ★ Limit access from

    pipeline early in development ★ Limit write access to repositories early in development ★ Offer self-service to refine access ★ Offer templates of secure policies 18
  19. The challenge of least privilege Solutions ★ Limit access from

    pipeline early in development ★ Limit write access to repositories early in development ★ Offer self-service to refine access ★ Offer templates of secure policies ★ Review access on a regular cadence ★ Audit pipeline runs 19
  20. Secrets 🔒 20

  21. sensitive information linked to access to a system or service

    secrets 21
  22. vault_database_secret_backend_connection.post gres will be created • resource "vault_database_secret_backend_connection" "postgres" {


    ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST] 22
  23. vault_database_secret_backend_connection.post gres will be created • resource "vault_database_secret_backend_connection" "postgres" {


    ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST] 23 😱
  24. Plan R Remediate the secret • Regret • Revoke •

    Rotate • Reference • Replace • Re-run 24
  25. Plan R Remediate the secret • Regret • Revoke •

    Rotate • Reference • Replace • Re-run 25 100 pipelines later… 😓
  26. Pipelines use secrets. 26 Certi fi cates Access Usernames &

    Passwords Testing User Data Tokens SSH Keys Encryption Keys
  27. Pipelines create secrets. 27 Con fi guration Usernames & Passwords

    SSH Keys Tokens
  28. The challenge of secrets Solutions ★ Mask or omit in

    pipeline output ★ Use a secrets manager 28
  29. The challenge of secrets Solutions ★ Mask or omit in

    pipeline output ★ Use a secrets manager ★ Issue new credentials per pipeline run ★ Audit secrets usage 29
  30. Runners 👟 30

  31. resources that run pipeline stages or tasks runners 31

  32. 32 Virtual Machine 🤔 1. Someone accesses CI/CD runner (e.g.,

    SSH). 2. Access other infrastructure. Database
  33. 33 Container 🤔 1. Someone accesses CI/CD runner (e.g., SSH).

    Virtual Machine $ mount /dev/<id> /mnt $ chroot /mnt 2. Container can access host fi lesystem. 3. Access code or fi les for other jobs. Other Jobs on Virtual Machine
  34. 34 Infrastructure Provider Runner Managed Service Engineer Must be authorized

    user. Must be authorized account. Secrets Manager
  35. 35 Infrastructure Provider Runner Managed Service Engineer Must be authorized

    user. Must be authorized account. Secrets Manager Allow IP address over VPN. Allow IP addresses for CI framework.
  36. The challenge of securing runners Solutions ★ Use trusted /

    verified images ★ Scan for OS vulnerabilities 36
  37. The challenge of securing runners Solutions ★ Use trusted /

    verified images ★ Scan for OS vulnerabilities ★ Define network policy ★ Run as a non-root 37
  38. The challenge of securing runners Solutions ★ Use trusted /

    verified images ★ Scan for OS vulnerabilities ★ Define network policy ★ Run as a non-root ★ Use ephemeral secrets ★ Audit remote access to runner 38
  39. Dependencies 🖇 39

  40. third-party code used for pipeline stages or tasks dependencies 40

  41. 41 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name:

    Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist Downloaded from trusted source? Veri fi ed code? Correct plugin?
  42. 42 public class UnverifiedPlugin { protected static void getFiles(AbstractBuild b,

    FilePath workspace) { // code to replace project files or metadata // code to gather information // code to siphon credentials } }
  43. The challenge of securing dependencies Solutions ★ Scan for vulnerabilities

    ★ Verify checksums and signatures 43
  44. The challenge of securing dependencies Solutions ★ Scan for vulnerabilities

    ★ Verify checksums and signatures ★ Use verified registry ★ Pin versions 44
  45. Configuration 📄 45

  46. de fi ne delivery pipelines through source code pipeline as

    code 46
  47. 47 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name:

    Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist test_plugin_checkout_has_fetch_depth_of_1 test_plugin_go_uses_secure_version_1.14 test_plugin_release_includes_signature
  48. The challenge of securing configuration Solutions ★ Apply immutability to

    pipeline configuration ★ Offer pipeline templates with secure defaults 48
  49. The challenge of securing configuration Solutions ★ Apply immutability to

    pipeline configuration ★ Offer pipeline templates with secure defaults ★ Test pipelines as code ★ Secure dependencies that allow arbitrary code / command 49
  50. The challenge of securing configuration Solutions ★ Apply immutability to

    pipeline configuration ★ Offer pipeline templates with secure defaults ★ Test pipelines as code ★ Secure dependencies that allow arbitrary code / command ★ Audit changes to pipeline configuration 50
  51. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets

    ✓ Runners ✓ Dependencies ✓ Configuration 51
  52. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets

    ✓ Runners ✓ Dependencies ✓ Configuration 52 Favor immutability.
  53. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets

    ✓ Runners ✓ Dependencies ✓ Configuration 53 Favor immutability. Limit blast radius.
  54. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets

    ✓ Runners ✓ Dependencies ✓ Configuration 54 Favor immutability. Limit blast radius. Automate to reduce friction.
  55. Rosemary Wang @joatmon08 joatmon08.github.io thank you! 55