DevSecCon June 2021 
 
 @joatmon08 I spy: an insecure delivery pipeline

Let’s play a game 
 of “I spy”.

Developer Advocate at HashiCorp 
 she/her 
 joatmon08.github.io 
 @joatmon08 Rosemary Wang

[UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

I spy a… Password Public endpoint Possible misconfiguration

variable "db_username" { type = string description = "username for PostgreSQL database" } variable "db_password" { type = string description = "password for PostgreSQL database" } INPUT VARIABLES [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

vault_database_secret_backend_connection.post gres will be create d • resource "vault_database_secret_backend_connection" "postgres" {
 ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

vault_database_secret_backend_connection.post gres will be create d • resource "vault_database_secret_backend_connection" "postgres" {
 ◦ postgresql { ▪ connection_url = "postgres:// hcpvault:ZWtW62okZyJh@terraform-2020113 0215226595400000001.cho1mmdxhp1z.us- west-2.rds.amazonaws.com:5432/prod" PIPELINE LOGS 😱 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

Plan R Remediate credentials 1. Regret 2. Revoke 3. Rotate 4. Reference 5. Replace 6. Re-run 😱

variable "db_username" { type = string description = "username for PostgreSQL database" sensitive = true } variable "db_password" { type = string description = "password for PostgreSQL database" sensitive = true } INPUT VARIABLES [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

vault_database_secret_backend_connection.po stgres will be create d • resource "vault_database_secret_backend_connecti on" "postgres" {
 ◦ postgresql { ▪ connection_url = (sensitive) PIPELINE LOGS 😀 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

Plan R Remediate credentials 1. Regret 2. Revoke 3. Rotate 4. Reference 5. Replace 6. Re-run Use a secrets manager ROTATE ROOT PASSWORD $ vault write -force \ infrastructure-pipeline/database/rotate-root/postgre s 😀

RETRIEVE SECRETS AUTHENTICATE TO SECRETS MANAGER PIPELINE CONFIGURATION - name: Import Secret s uses: hashicorp/[email protected]. 0 with : url: ${{ secrets.VAULT_ADDR } } exportToken: tru e secrets: | infrastructure-pipeline/aws/creds/pipeline access_key | AWS_ACCESS_KEY_ID ; infrastructure-pipeline/aws/creds/pipeline secret_key | AWS_SECRET_ACCESS_KEY ; infrastructure-pipeline/aws/creds/pipeline security_token | AWS_SESSION_TOKEN ; infrastructure-pipeline/terraform/creds/pipeline token | TFC_TOKE N 😀 [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

I spy…secrets management in this pipeline. 1. Issue credentials for each pipeline run. 2. Manage access to secrets. 3. Audit usage of credentials. Note: I might spy secrets in application logs but probably not their delivery pipelines.

I spy a… Password Public endpoint Possible misconfiguration

RETRIEVE SECRETS AUTHENTICATE TO SECRETS MANAGER PIPELINE CONFIGURATION - name: Import Secret s uses: hashicorp/[email protected]. 0 with : url: ${{ secrets.VAULT_ADDR }} [UNIT TEST] TERRAFORM FMT [BUILD] TERRAFORM INIT [DEPLOY] TERRAFORM PLAN [RELEASE] TERRAFORM APPLY [TEST]

MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS PUBLIC SUBNETS 😱 VAULT SERVER PEERING CONNECTION

Public Endpoints Possible solutions 1. Shorten time to live for secrets 2. Only allow IP addresses from continuous integration framework 3. Use self-hosted runners Developer productivity? Least privilege? Management?

MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS PEERING CONNECTION BOUNDARY CONTROLLER BOUNDARY WORKER Least privilege?

MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS PEERING CONNECTION 3. Retrieve secrets BOUNDARY CONTROLLER BOUNDARY WORKER 1. Authenticate to secure access management tool 2. Proxy to different endpoints 🙂 Least privilege?

MY VPC 10.0.0.0/16 HASHICORP VIRTUAL NETWORK 172.25.16.0/20 PRIVATE SUBNETS GITHUB ACTIONS VAULT SERVER PUBLIC SUBNETS SELF-HOSTED RUNNER VAULT AGENT PEERING CONNECTION 2. Authenticate to secrets manager 3. Retrieve secrets 🙂 Management? 4. Make changes to infrastructure. 1. Use self-hosted runner to start job.

I spy…access control in this pipeline 1. Proxy and audit access to endpoints with secure access management. 2. Manage and control your own runners.

I spy a… Password Public endpoint Possible misconfiguration

CODE EDITOR on : push : branches : - mai n paths-ignore : - 'vault/** ' - 'infrastructure/** ' pull_request : jobs : steps : - name: Import Secret s uses: hashicorp/[email protected]. 0 - name: Setup Terrafor m uses: hashicorp/setup-terraform@v 1 - name: Terraform Pla n id: pla n if: github.event_name == ‘pull_request ' - name: Terraform Plan Statu s if: steps.plan.outcome == 'failure ' run: exit 1 - name: Terraform Appl y if: github.ref == 'refs/heads/main' && github.event_name == ‘push ' run: terraform apply -auto-approve -var-file=prod.tfvars 😱 What happens if someone removes the “if” statements?

CODE EDITOR on : push : branches : - mai n paths-ignore : - 'vault/** ' - 'infrastructure/** ' pull_request : jobs : steps : - name: Import Secret s uses: hashicorp/[email protected]. 0 - name: Setup Terrafor m uses: hashicorp/setup-terraform@v 1 - name: Terraform Pla n id: pla n if: github.event_name == ‘pull_request ' - name: Terraform Plan Statu s if: steps.plan.outcome == 'failure ' run: exit 1 - name: Terraform Appl y if: github.ref == 'refs/heads/main' && github.event_name == ‘push ' run: terraform apply -auto-approve -var-file=prod.tfvars How do you know that these GitHub 
 Actions are secure? 😱

Insecure pipeline configuration Multiple exploits 1. Mutable pipeline configuration 2. Remote code execution 3. Unverified plugins (supply chain attacks)

I spy…policy as code in this pipeline. Automate policies to check pipelines and their outputs. 1. Use pipeline as code to enforce stages. 2. Audit your pipeline logs for possible issues. 3. Verify plugin checksums. 😀

I spy a… Password Public endpoint Possible misconfiguration

Access control Proxy and audit access to endpoints with secure access management. Manage and control your own runners. Secrets management Issue credentials for each pipeline run. Manage access to secrets. Audit usage of credentials. Policy as code Use pipeline as code to enforce stages. Audit your pipeline logs for possible issues. Verify plugin checksums. I spy…a more secure delivery pipeline!

joatmon08.github.io 
 Rosemary Wang 
 @joatmon08 Thank you!