Your (container) secret's safe with me - 2018 version

An updated version of this talk, concentrating on Kubernetes (I've previously published a version covering secrets in several container orchestrators)

Transcript

1. Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved. Your (container) secret’s safe with me Liz Rice @LizRice | @AquaSecTeam
2. 2 Secrets @LizRice | @AquaSecTeam
3. 3 Secrets @LizRice | @AquaSecTeam Secret store Cluster
4. Desirable attributes for secrets management
5. 5 Secrets @LizRice | @AquaSecTeam Secrets photo: Katie Tegtmeyer ▪ Encrypted ▪ At rest and in transit ▪ Only decrypted in memory
6. 6 Secrets @LizRice | @AquaSecTeam Secrets photo: James Case ▪ Access control ▪ Only accessible by containers that need them ▪ And users ▪ Write-only access
7. 7 Secrets @LizRice | @AquaSecTeam Secrets photo: Irena Jackson ▪ Life-cycle ▪ Risk of leak increases over time ▪ Rotation, revocation, audit logging
8. Passing secrets to containers
9. 9 Bad places for secrets @LizRice | @AquaSecTeam ▪ Source code ▪ Dockerfiles / images
10. 10 docker run -e VARNAME=secret ... Environment variables @LizRice | @AquaSecTeam
11. 11 ▪ docker inspect ▪ Leaky logs ▪ docker exec ▪ /proc directory Environment variables @LizRice | @AquaSecTeam
12. 12 docker run -v /hostsecrets:/secrets ... Mounted volume @LizRice | @AquaSecTeam
13. 13 ▪ docker inspect ▪ Leaky logs ▪ docker exec ▪ /proc directory Mounted volume @LizRice | @AquaSecTeam
14. Kubernetes support for secrets
15. 15 Bad places for secrets @LizRice | @AquaSecTeam ▪ Source code ▪ Dockerfiles / images ▪ In plain text in YAML files
16. 16 Kubernetes secrets @LizRice | @AquaSecTeam ▪ Secrets are Kubernetes objects ▪ Refer to secret in pod YAML as environment variable: ... env: - name: MYSECRET valueFrom: secretKeyRef: name: mysecret key: secret_key
17. 17 Kubernetes secrets @LizRice | @AquaSecTeam ▪ ...or as a file in a volume mount: ... volumeMounts: - name: secret mountPath: /.secrets readOnly: true volumes: - name: secret secret: secretName: mysecret
18. Encrypting secrets
19. 19 Kubernetes secrets @LizRice | @AquaSecTeam ▪ Stored in etcd ▪ Make sure secrets are encrypted! ▪ --experimental-encryption-provider-config on API Server
20. 20 Encrypting etcd @LizRice | @AquaSecTeam kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: bXlWZXJ5U2VjcmV0RW5jcnlwdGlvbktleQo= - identity: {}
21. 21 Secrets all the way down @LizRice | @AquaSecTeam ▪ EncryptionConfig holds a secret key... xkcd.com/1416
22. 22 External key stores @LizRice | @AquaSecTeam ▪ Secret storage in 3rd party backend ▪ Hashicorp Vault, Amazon KMS, Azure Key Vault, CyberArk Vault… ▪ Kubernetes adding Key Management Service plugin support
23. Access control
24. 24 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: secret-reader rules: - apiGroups: [“”] resources: [“secrets”] verbs: [“get”, “list”, “watch”] Kubernetes RBAC for secrets @LizRice | @AquaSecTeam ▪ Role for read-only access to secrets
25. 25 Kubernetes RBAC for secrets @LizRice | @AquaSecTeam kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets namespace: development subjects: - kind: User name: dave apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io ▪ Let “dave” read secrets in the “development” namespace
26. 26 RBAC secrets best practices @LizRice | @AquaSecTeam kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: my-secret-reader rules: - apiGroups: [“”] resources: [“secrets”] resourceNames: [“my-secret”] verbs: [“get”] ▪ Be careful with list & watch ▪ Limit to get where possible ▪ Limit access to only the secret(s) an app needs
27. 27 Least privileges @LizRice | @AquaSecTeam pod ▪ Read-only mount ▪ Split to separate container with simple behaviour
28. Lifecycle
29. 29 Kubernetes secret rotation @LizRice | @AquaSecTeam ▪ Files support updating secret values ▪ Need to restart pod to get new env var value
30. 30 Audit logging secrets access @LizRice | @AquaSecTeam apiVersion: audit.k8s.io/v1beta1 kind: Policy rules: # Log secret changes at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets"]
31. Secret updates demo
32. 32 Commercial secrets solutions @LizRice | @AquaSecTeam ▪ File system & env var support ▪ Update secrets without container restart ▪ No env var leak through inspect or /proc ▪ Full auditing of secret usage ▪ User & container access control ▪ 3rd party storage ▪ Any orchestrator Encrypted ✓ Access control ✓ Life-cycle ✓
33. Summary
34. 34 Secrets @LizRice | @AquaSecTeam Secrets photo: Iain Merchant ▪ Turn on encryption ▪ Access secrets at runtime ▪ Not built in ▪ Rotate secrets
35. Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved. The Ultimate Guide to Secrets Management in Containers tiny.cc/secrets @LizRice | @AquaSecTeam
36. Orchestrator support for secrets
37. 37 Docker @LizRice | @AquaSecTeam ▪ Secrets support built in for Docker Swarm services ▪ Not standalone containers ▪ Encrypted transmission with mutual authentication ▪ Secret accessible when exposed to service ▪ Mounted to a temporary fs (not env vars) ▪ RBAC in Enterprise Edition
38. 38 Docker @LizRice | @AquaSecTeam ▪ Encrypted in Raft log ▪ Lock your Swarm!! ▪ Shared to Swarm managers ▪ Audit log with events ▪ Rotation requires container restart & secret dance Encrypted ✓ Access control ✓ Life-cycle ?
39. 39 DC/OS @LizRice | @AquaSecTeam ▪ Enterprise DC/OS ▪ Plug-ins for Mesos/Marathon ▪ Encrypted in ZooKeeper ▪ Env vars ▪ Access control by service path ▪ Restart service to update value Encrypted ✓ Access control ✓ Life-cycle ?
40. 40 Nomad @LizRice | @AquaSecTeam ▪ Integrated with Vault ▪ Use production mode ▪ Encryption & security primitives
41. 41 Nomad @LizRice | @AquaSecTeam ▪ Secrets passed as files ▪ Nomad takes care of interactions with Vault ▪ Tasks get tokens so they can retrieve values ▪ Poll for changed values ▪ Access control ▪ Audit logging Encrypted ✓ Access control ✓ Life-cycle ✓