I Spy: An Insecure Delivery Pipeline

Your team uses a delivery pipeline to deploy infrastructure and applications to production. However, your security team warns you that your pipeline might be vulnerable. Can you spy some improvements you can make to it? In this session, let’s investigate the different ways a pipeline authenticates, authorizes, and audits people, machines, and services. We’ll also apply some tools and techniques to solve the security riddle of an example pipeline using Amazon Web Services, Terraform, and GitHub Actions. You’ll spy some practices for injecting and managing secrets, securing pipeline runner access control and script execution, and auditing pipeline outputs, scripts, and stages to prevent different attacks.

Key Takeaways:

- You’ll learn practices to secure an infrastructure delivery pipeline by...
- Injecting and managing secrets and credentials using a secrets manager
- Securing access control to the pipeline runner
- Auditing pipeline outputs, scripts, and stages for compliance

Transcript

1. DevSecCon June 2021 
 
 @joatmon08 I spy: an insecure delivery pipeline
2. Let’s play a game 
 of “I spy”.
3. Developer Advocate at HashiCorp 
 she/her 
 joatmon08.github.io 
 @joatmon08 Rosemary Wang
4. [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
5. I spy a… Password Public endpoint Possible misconfiguration
6. variable "db_username" { type = string description = "username for PostgreSQL database" } variable "db_password" { type = string description = "password for PostgreSQL database" } INPUT VARIABLES [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
7. vault_database_secret_backend_connection.post gres will be create d • resource "vault_database_secret_backend_connection" "postgres" {
 ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
8. vault_database_secret_backend_connection.post gres will be create d • resource "vault_database_secret_backend_connection" "postgres" {
 ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS 😱 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
9. Plan R Remediate credentials 1. Regret 2. Revoke 3. Rotate 4. Reference 5. Replace 6. Re-run 😱
10. variable "db_username" { type = string description = "username for PostgreSQL database" sensitive = true } variable "db_password" { type = string description = "password for PostgreSQL database" sensitive = true } INPUT VARIABLES [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
11. vault_database_secret_backend_connection.po stgres will be create d • resource "vault_database_secret_backend_connecti on" "postgres" {
 ◦ postgresql { ▪ connection_url = (sensitive) PIPELINE LOGS 😀 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
12. Plan R Remediate credentials 1. Regret 2. Revoke 3. Rotate 4. Reference 5. Replace 6. Re-run Use a secrets manager ROTATE ROOT PASSWORD $ vault write -force \ infrastructure-pipeline/database/rotate-root/postgre s 😀
13. RETRIEVE SECRETS AUTHENTICATE TO SECRETS MANAGER PIPELINE CONFIGURATION - name: Import Secret s uses: hashicorp/vault-action@v2.1. 0 with : url: ${{ secrets.VAULT_ADDR } } exportToken: tru e secrets: | infrastructure-pipeline/aws/creds/pipeline access_key | AWS_ACCESS_KEY_ID ; infrastructure-pipeline/aws/creds/pipeline secret_key | AWS_SECRET_ACCESS_KEY ; infrastructure-pipeline/aws/creds/pipeline security_token | AWS_SESSION_TOKEN ; infrastructure-pipeline/terraform/creds/pipeline token | TFC_TOKE N 😀 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
14. I spy…secrets management in this pipeline. 1. Issue credentials for each pipeline run. 2. Manage access to secrets. 3. Audit usage of credentials. Note: I might spy secrets in application logs but probably not their delivery pipelines.
15. I spy a… Password Public endpoint Possible misconfiguration
16. RETRIEVE SECRETS AUTHENTICATE TO SECRETS MANAGER PIPELINE CONFIGURATION - name: Import Secret s uses: hashicorp/vault-action@v2.1. 0 with : url: ${{ secrets.VAULT_ADDR }} [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]
17. MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS PUBLIC SUBNETS 😱 VAULT SERVER PEERING CONNECTION
18. Public Endpoints Possible solutions 1. Shorten time to live for secrets 2. Only allow IP addresses from continuous integration framework 3. Use self-hosted runners Developer productivity? Least privilege? Management?
19. MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS PEERING CONNECTION BOUNDARY CONTROLLER BOUNDARY WORKER Least privilege?
20. MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS PEERING CONNECTION 3. Retrieve secrets BOUNDARY CONTROLLER BOUNDARY WORKER 1. Authenticate to secure access management tool 2. Proxy to different endpoints 🙂 Least privilege?
21. MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS SELF-HOSTED RUNNER VAULT AGENT PEERING CONNECTION 2. Authenticate to secrets manager 3. Retrieve secrets 🙂 Management? 4. Make changes to infrastructure. 1. Use self-hosted runner to start job.
22. I spy…access control in this pipeline 1. Proxy and audit access to endpoints with secure access management. 2. Manage and control your own runners.
23. I spy a… Password Public endpoint Possible misconfiguration
24. CODE EDITOR on : push : branches : - mai n paths-ignore : - 'vault/** ' - 'infrastructure/** ' pull_request : jobs : steps : - name: Import Secret s uses: hashicorp/vault-action@v2.1. 0 - name: Setup Terrafor m uses: hashicorp/setup-terraform@v 1 - name: Terraform Pla n id: pla n if: github.event_name == ‘pull_request ' - name: Terraform Plan Statu s if: steps.plan.outcome == 'failure ' run: exit 1 - name: Terraform Appl y if: github.ref == 'refs/heads/main' && github.event_name == ‘push ' run: terraform apply -auto-approve -var-file=prod.tfvars 😱 What happens if someone removes the “if” statements?
25. CODE EDITOR on : push : branches : - mai n paths-ignore : - 'vault/** ' - 'infrastructure/** ' pull_request : jobs : steps : - name: Import Secret s uses: hashicorp/vault-action@v2.1. 0 - name: Setup Terrafor m uses: hashicorp/setup-terraform@v 1 - name: Terraform Pla n id: pla n if: github.event_name == ‘pull_request ' - name: Terraform Plan Statu s if: steps.plan.outcome == 'failure ' run: exit 1 - name: Terraform Appl y if: github.ref == 'refs/heads/main' && github.event_name == ‘push ' run: terraform apply -auto-approve -var-file=prod.tfvars How do you know that these GitHub 
 Actions are secure? 😱
26. Insecure pipeline configuration Multiple exploits 1. Mutable pipeline configuration 2. Remote code execution 3. Unverified plugins (supply chain attacks)
27. I spy…policy as code in this pipeline. Automate policies to check pipelines and their outputs. 1. Use pipeline as code to enforce stages. 2. Audit your pipeline logs for possible issues. 3. Verify plugin checksums. 😀
28. I spy a… Password Public endpoint Possible misconfiguration
29. Access control Proxy and audit access to endpoints with secure access management. Manage and control your own runners. Secrets management Issue credentials for each pipeline run. Manage access to secrets. Audit usage of credentials. Policy as code Use pipeline as code to enforce stages. Audit your pipeline logs for possible issues. Verify plugin checksums. I spy…a more secure delivery pipeline!
30. joatmon08.github.io 
 Rosemary Wang 
 @joatmon08 Thank you!