Can your CI/CD pipeline keep a secret?

It's hard to keep secrets, harder to ensure they stay safe. There are countless horror stories of data breaches due to exposed secrets or disgruntled employees with unrestricted access to secrets causing harm. With CI/CD becoming the default practice for software delivery, secrets are usually stored as environment variables These are difficult to manage especially in a large organization with multiple projects across several teams. Encryption, secure storage, and frequent rotation of secrets are the panaceas but only a few tools are available. HashiCorp’s Vault provides encrypted, secure and access-controlled secret management. In this talk, I will be sharing common ways secrets are exposed in CI/CD pipelines and how Vault can be used to securely use secrets in CI jobs.
At the end of the session, viewers will learn about vulnerabilities in secrets management, why they need to use secret management tools i.e. Vault and they will see a demo of how they can secure their secrets in a CI tool like GitLab CI.

Video of session is at https://bit.ly/hashitalks-africa20-secrets

Transcript

1. Can your CI/CD pipeline keep a secret? Abubakar Siddiq Ango Technical Evangelism Program Manager at GitLab abuango.me | twitter.com/sarki247
2. It’s Hard to keep a secret! abuango.me | twitter.com/sarki247
3. In this talk ▪ What’s a secret and why is it so hard to keep? ▪ How most CI/CD pipelines currently handle Secrets and its vulnerabilities ▪ Using Vault with CI/CD with JWT Auth method ▪ Demo using GitLab CI with Vault abuango.me | twitter.com/sarki247
4. ▪ Tokens ▪ API Keys ▪ Passwords ▪ Certificates What is a secret? abuango.me | twitter.com/sarki247
5. ...but it’s hard to keep safe. It’s not if but when the secret will be discovered. abuango.me | twitter.com/sarki247
6. But why difficult? ▪ Managing hundreds of unique secrets for various uses ▪ Need for multiple people to have access to the secrets ▪ Complex passwords are prefered and not memorable, you are not even supposed to ▪ Rotating passwords for any reason is hard work abuango.me | twitter.com/sarki247
7. CI/CD & Secrets abuango.me | twitter.com/sarki247
8. Continuous Integration / Delivery We love automation! A tonne of tools are available now to help you build, test and deploy your code, but they need secrets to do their work and most often, they take advantage of environment variables. abuango.me | twitter.com/sarki247
9. What could possibly go wrong? Exposed secrets in Job logs abuango.me | twitter.com/sarki247 ZDNet News: CI build logs continue to expose company secrets Source: https://www.zdnet.com/article/ci-b uild-logs-continue-to-expose-com pany-secrets/
10. What could possibly go wrong? Privilege Escalation abuango.me | twitter.com/sarki247
11. The best kept secrets don’t exist. The best kept secrets are constantly rotated with little possibility of someone keeping it for too long. abuango.me | twitter.com/sarki247
12. Secrets Management Secure management & Rotation of Secrets and Data Encryption abuango.me | twitter.com/sarki247
13. abuango.me | twitter.com/sarki247
14. Using Vault abuango.me | twitter.com/sarki247
15. Vault Auth Methods Auth methods are the components in Vault that perform authentication and are responsible for assigning identity and a set of policies to a user. ▪ Userpass ▪ AWS ▪ GitHub ▪ RADIUS ▪ OKTA ▪ JWT/OIDC ▪ etc. abuango.me | twitter.com/sarki247
16. Vault Auth Methods - JWT The JWT auth method can be used to authenticate with Vault using OIDC (OpenID Connect) or by providing a JWT. JWT is suitable for Automation as no user intervention is required. abuango.me | twitter.com/sarki247
17. Vault Auth Methods - JWT JWT signatures will be verified against public keys from the issuer, which is possible in 3 ways: ▪ Static Keys ▪ A JSON Web Key Set (JWKS) URL ▪ OIDC Discovery URL Bound Claims can be used to restrict access, except they match specific criterias. abuango.me | twitter.com/sarki247
18. Vault Auth Methods - JWT CI Send JWT to Vault Verify JW T and Claim s with GitLab JW KS Send Token to GitLab CI once verified abuango.me | twitter.com/sarki247
19. Demo abuango.me | twitter.com/sarki247
20. CODE EDITOR { "jti": "c82eeb0c-5c6f-4a33-abf5-4c474b92b558", # Unique identifier for this token "iss": "gitlab.com", # Issuer, the domain of your GitLab "iat": 1585710286, # Issued at "nbf": 1585798372, # Not valid before "exp": 1585713886, # Expire at "sub": "job_1212", # Subject (job id) "namespace_id": "1", "namespace_path": "mygroup", "project_id": "22", "project_path": "abuango/vault-demo", "user_id": "42", "user_login": "myuser", "user_email": "myuser@example.com" "pipeline_id": "1212", "job_id": "1212", "ref": "master", # Git ref for this job "ref_type": "branch", # Git ref type, branch or tag "ref_protected": "true" # true if this git ref is protected } GitLab.com project with CI JWT Payload: $CI_JOB_JWT
21. Enable JWT Authentication CODE EDITOR $ vault auth enable jwt Success! Enabled jwt auth method at: jwt/ $ vault write auth/jwt/config \ jwks_url="https://gitlab..com/-/jwks" \ bound_issuer="gitlab.com" abuango.me | twitter.com/sarki247
22. Access Policies CODE EDITOR $ vault policy write vault-demo-staging - <<EOF path "kv/data/vault-demo/staging/*" { capabilities = [ "read" ] } EOF Success! Uploaded policy: vault-demo-staging $ vault policy write vault-demo-production - <<EOF path "kv/data/vault-demo/production/*" { capabilities = [ "read" ] } EOF Success! Uploaded policy: vault-demo-production abuango.me | twitter.com/sarki247
23. Roles linking JWT to Access Policies CODE EDITOR $ vault write auth/jwt/role/vault-demo-staging - <<EOF { "role_type": "jwt", "policies": ["vault-demo-staging"], "token_explicit_max_ttl": 60, "user_claim": "user_email", "bound_claims": { "project_id": "18461410", "ref": "master", "ref_type": "branch" } } EOF $ vault write auth/jwt/role/vault-demo-production - <<EOF { "role_type": "jwt", "policies": ["vault-demo-production"], "token_explicit_max_ttl": 60, "user_claim": "user_email", "bound_claims": { "project_id": "18461410", "ref": "production", "ref_type": "branch" } } EOF abuango.me | twitter.com/sarki247
24. GitLab CI Job Production CODE EDITOR image: vault:latest read_secrets: only: refs: - production script: # Check job's ref name - echo $CI_COMMIT_REF_NAME # Vault's address can be provided here or as CI variable - export VAULT_ADDR=https://vault.abuango.me:8200 # Authenticate and get token. Token expiry time and other properties can be configured when configuring JWT Auth - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=vault-demo-production jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in an environment variable - export PASSWORD="$(vault kv get -field=key kv/vault-demo/production/api)" # Use the secret - echo $PASSWORD # This will fail because the role vault-demo-production can not read secrets from kv/vault-demo/staging/* - export PASSWORD="$(vault kv get -field=key kv/vault-demo/staging/api)" # Use the secret - echo $PASSWORD abuango.me | twitter.com/sarki247
25. GitLab CI Job Staging CODE EDITOR image: vault:latest read_secrets: only: refs: - master script: # Check job's ref name - echo $CI_COMMIT_REF_NAME # Vault's address can be provided here or as CI variable - export VAULT_ADDR=https://vault.abuango.me:8200 # Authenticate and get token. Token expiry time and other properties can be configured when configuring JWT Auth - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=vault-demo-staging jwt=$CI_JOB_JWT)" # Now use the VAULT_TOKEN to read the secret and store it in an environment variable - export PASSWORD="$(vault kv get -field=key kv/vault-demo/staging/api)" # Use the secret - echo $PASSWORD # This will fail because the role vault-demo-staging can not read secrets from kv/vault-demo/production/* - export PASSWORD="$(vault kv get -field=key kv/vault-demo/production/api)" # Use the secret - echo $PASSWORD
26. Thank You! 26 abuango.me | twitter.com/sarki247 abuango.me | twitter.com/sarki247