Minimum Secure Pipeline

Originally presented at HashiTalks: Secure 2023.

Whether you are a developer, operator, or other kind of engineer, you will need to secure your continuous integration frameworks and deployment pipelines. Rosemary Wang provides a list of practices and patterns to build a minimum secure pipeline. From access control to pipelines as code, you can apply these basics to improving your CI/CD pipelines and educating other engineers, no matter what you're deploying.

Transcript

1. Rosemary Wang | May 3, 2023 Minimum Secure Pipeline 1
2. jenkins.io/security/advisory/2022-02-15/ about.codecov.io/security-update/ msrc-blog.microsoft.com/2020/12/31/microsoft-internal-solorigate-investigation-update/ 2
3. Rosemary Wang (she/her) Developer Advocate at HashiCorp @joatmon08 joatmon08.github.io 3
4. How to secure better… and remediate faster. 4
5. Securing CI/CD Pipelines An Overview ✓ Access Control ✓ Secrets ✓ Runners ✓ Dependencies ✓ Configuration 5
6. Access Control 🚥 6
7. limit service’s access rights to minimum required least privilege 7
8. Choose your access. Code Repository Build infrastructure. Infrastructure Provider Checkout code. Deploy application. Application Platform / Release Repository Test for security. Security Tool Check code quality. Quality Assurance Tool Read repositories. Write speci fi c services. Write speci fi c nodes or namespaces. Read test results. Read test results. Run integration tests. Store for User Data Read table in testing environment. 8
9. The challenge of least privilege Solutions ★ Limit access from pipeline early in development ★ Limit write access to repositories early in development 9
10. The challenge of least privilege Solutions ★ Limit access from pipeline early in development ★ Limit write access to repositories early in development ★ Offer self-service to refine access ★ Offer templates of secure policies 10
11. The challenge of least privilege Solutions ★ Limit access from pipeline early in development ★ Limit write access to repositories early in development ★ Offer self-service to refine access ★ Offer templates of secure policies ★ Review access on a regular cadence ★ Audit pipeline runs 11
12. Secrets 🔒 12
13. sensitive information linked to access to a system or service secrets 13
14. vault_database_secret_backend_connection.post gres will be created • resource "vault_database_secret_backend_connection" "postgres" {
 ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST] 14 😱
15. Plan R Remediate the secret • Regret • Revoke • Rotate • Reference • Replace • Re-run 15 100 pipelines later… 😓
16. Pipelines use secrets. 16 Certi fi cates Access Usernames & Passwords Testing User Data Tokens SSH Keys Encryption Keys
17. Pipelines create secrets. 17 Con fi guration Usernames & Passwords SSH Keys Tokens
18. The challenge of secrets Solutions ★ Mask or omit in pipeline output ★ Use a secrets manager 18
19. The challenge of secrets Solutions ★ Mask or omit in pipeline output ★ Use a secrets manager ★ Issue new credentials per pipeline run ★ Audit secrets usage 19
20. Runners 👟 20
21. resources that run pipeline stages or tasks runners 21
22. 22 Virtual Machine 🤔 1. Someone accesses CI/CD runner (e.g., SSH). 2. Access other infrastructure. Database
23. 23 Container 🤔 1. Someone accesses CI/CD runner (e.g., SSH). Virtual Machine $ mount /dev/<id> /mnt $ chroot /mnt 2. Container can access host fi lesystem. 3. Access code or fi les for other jobs. Other Jobs on Virtual Machine
24. 24 Infrastructure Provider Runner Managed Service Engineer Must be authorized user. Must be authorized account. Secrets Manager Allow IP address over VPN. Allow IP addresses for CI framework.
25. The challenge of securing runners Solutions ★ Use trusted / verified images ★ Scan for OS vulnerabilities 25
26. The challenge of securing runners Solutions ★ Use trusted / verified images ★ Scan for OS vulnerabilities ★ Define network policy ★ Run as a non-root 26
27. The challenge of securing runners Solutions ★ Use trusted / verified images ★ Scan for OS vulnerabilities ★ Define network policy ★ Run as a non-root ★ Use ephemeral secrets ★ Audit remote access to runner 27
28. Dependencies 🖇 28
29. third-party code used for pipeline stages or tasks dependencies 29
30. 30 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist Downloaded from trusted source? Veri fi ed code? Correct plugin?
31. 31 public class UnverifiedPlugin { protected static void getFiles(AbstractBuild b, FilePath workspace) { // code to replace project files or metadata // code to gather information // code to siphon credentials } }
32. The challenge of securing dependencies Solutions ★ Scan for vulnerabilities ★ Verify checksums and signatures 32
33. The challenge of securing dependencies Solutions ★ Scan for vulnerabilities ★ Verify checksums and signatures ★ Use verified registry ★ Pin versions 33
34. Configuration 📄 34
35. de fi ne delivery pipelines through source code pipeline as code 35
36. 36 name: release jobs: goreleaser: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v2 with: fetch-depth: 0 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.14 - name: Run GoReleaser uses: goreleaser/goreleaser-action@v2 with: version: latest args: release --rm-dist test_plugin_checkout_has_fetch_depth_of_1 test_plugin_go_uses_secure_version_1.14 test_plugin_release_includes_signature
37. The challenge of securing configuration Solutions ★ Apply immutability to pipeline configuration ★ Offer pipeline templates with secure defaults 37
38. The challenge of securing configuration Solutions ★ Apply immutability to pipeline configuration ★ Offer pipeline templates with secure defaults ★ Test pipelines as code ★ Secure dependencies that allow arbitrary code / command 38
39. The challenge of securing configuration Solutions ★ Apply immutability to pipeline configuration ★ Offer pipeline templates with secure defaults ★ Test pipelines as code ★ Secure dependencies that allow arbitrary code / command ★ Audit changes to pipeline configuration 39
40. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets ✓ Runners ✓ Dependencies ✓ Configuration 40
41. Securing CI/CD Pipelines In Summary ✓ Access Control ✓ Secrets ✓ Runners ✓ Dependencies ✓ Configuration 41 Favor immutability. Limit blast radius. Automate to reduce friction.
42. Rosemary Wang @joatmon08 joatmon08.github.io thank you! 42