Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved.
Your (container) secret’s safe with me
Liz Rice
@LizRice | @AquaSecTeam
Slide 2
Slide 2 text
2
Secrets
@LizRice | @AquaSecTeam
Slide 3
Slide 3 text
3
Secrets
@LizRice | @AquaSecTeam
Secret
store
Cluster
Slide 4
Slide 4 text
Desirable attributes for secrets management
Slide 5
Slide 5 text
5
Secrets
@LizRice | @AquaSecTeam
Secrets
photo: Katie Tegtmeyer
■ Encrypted
■ At rest and in transit
■ Only decrypted in
memory
Slide 6
Slide 6 text
6
Secrets
@LizRice | @AquaSecTeam
Secrets
photo: James Case
■ Access control
■ Only accessible by containers
that need them
■ And users
■ Write-only access
Slide 7
Slide 7 text
7
Secrets
@LizRice | @AquaSecTeam
Secrets
photo: Irena Jackson
■ Life-cycle
■ Risk of leak increases over time
■ Rotation, revocation, audit logging
Slide 8
Slide 8 text
Passing secrets to containers
Slide 9
Slide 9 text
9
Bad places for secrets
@LizRice | @AquaSecTeam
■ Source code
■ Dockerfiles / images
15
Bad places for secrets
@LizRice | @AquaSecTeam
■ Source code
■ Dockerfiles / images
■ In plain text in YAML files
Slide 16
Slide 16 text
16
Kubernetes secrets
@LizRice | @AquaSecTeam
■ Secrets are Kubernetes objects
■ Refer to secret in pod YAML as environment variable:
...
env:
- name: MYSECRET
valueFrom:
secretKeyRef:
name: mysecret
key: secret_key
Slide 17
Slide 17 text
17
Kubernetes secrets
@LizRice | @AquaSecTeam
■ ...or as a file in a volume mount:
...
volumeMounts:
- name: secret
mountPath: /.secrets
readOnly: true
volumes:
- name: secret
secret:
secretName: mysecret
Slide 18
Slide 18 text
Encrypting secrets
Slide 19
Slide 19 text
19
Kubernetes secrets
@LizRice | @AquaSecTeam
■ Stored in etcd
■ Make sure secrets are encrypted!
■ --experimental-encryption-provider-config on API
Server
21
Secrets all the way down
@LizRice | @AquaSecTeam
■ EncryptionConfig holds a secret key...
xkcd.com/1416
Slide 22
Slide 22 text
22
External key stores
@LizRice | @AquaSecTeam
■ Secret storage in 3rd party backend
■ Hashicorp Vault, Amazon KMS, Azure Key
Vault, CyberArk Vault…
■ Kubernetes adding Key Management
Service plugin support
Slide 23
Slide 23 text
Access control
Slide 24
Slide 24 text
24
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: secret-reader
rules:
- apiGroups: [“”]
resources: [“secrets”]
verbs: [“get”, “list”, “watch”]
Kubernetes RBAC for secrets
@LizRice | @AquaSecTeam
■ Role for read-only
access to secrets
Slide 25
Slide 25 text
25
Kubernetes RBAC for secrets
@LizRice | @AquaSecTeam
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-secrets
namespace: development
subjects:
- kind: User
name: dave
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: secret-reader
apiGroup: rbac.authorization.k8s.io
■ Let “dave” read secrets
in the “development”
namespace
Slide 26
Slide 26 text
26
RBAC secrets best practices
@LizRice | @AquaSecTeam
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: my-secret-reader
rules:
- apiGroups: [“”]
resources: [“secrets”]
resourceNames: [“my-secret”]
verbs: [“get”]
■ Be careful with list & watch
■ Limit to get where possible
■ Limit access to only the
secret(s) an app needs
Slide 27
Slide 27 text
27
Least privileges
@LizRice | @AquaSecTeam
pod
■ Read-only mount
■ Split to separate
container with
simple behaviour
Slide 28
Slide 28 text
Lifecycle
Slide 29
Slide 29 text
29
Kubernetes secret rotation
@LizRice | @AquaSecTeam
■ Files support updating secret values
■ Need to restart pod to get new env var value
Slide 30
Slide 30 text
30
Audit logging secrets access
@LizRice | @AquaSecTeam
apiVersion: audit.k8s.io/v1beta1
kind: Policy
rules:
# Log secret changes at the Metadata level.
- level: Metadata
resources:
- group: "" # core API group
resources: ["secrets"]
Slide 31
Slide 31 text
Secret updates demo
Slide 32
Slide 32 text
32
Commercial secrets solutions
@LizRice | @AquaSecTeam
■ File system & env var support
■ Update secrets without container restart
■ No env var leak through inspect or /proc
■ Full auditing of secret usage
■ User & container access control
■ 3rd party storage
■ Any orchestrator
Encrypted ✓ Access control ✓ Life-cycle ✓
Slide 33
Slide 33 text
Summary
Slide 34
Slide 34 text
34
Secrets
@LizRice | @AquaSecTeam
Secrets
photo: Iain Merchant
■ Turn on encryption
■ Access secrets at runtime
■ Not built in
■ Rotate secrets
Slide 35
Slide 35 text
Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved.
The Ultimate Guide to Secrets Management in Containers
tiny.cc/secrets
@LizRice | @AquaSecTeam
Slide 36
Slide 36 text
Orchestrator support for secrets
Slide 37
Slide 37 text
37
Docker
@LizRice | @AquaSecTeam
■ Secrets support built in for Docker Swarm services
■ Not standalone containers
■ Encrypted transmission with mutual
authentication
■ Secret accessible when exposed to service
■ Mounted to a temporary fs (not env vars)
■ RBAC in Enterprise Edition
Slide 38
Slide 38 text
38
Docker
@LizRice | @AquaSecTeam
■ Encrypted in Raft log
■ Lock your Swarm!!
■ Shared to Swarm managers
■ Audit log with events
■ Rotation requires container restart
& secret dance
Encrypted ✓ Access control ✓ Life-cycle ?
Slide 39
Slide 39 text
39
DC/OS
@LizRice | @AquaSecTeam
■ Enterprise DC/OS
■ Plug-ins for Mesos/Marathon
■ Encrypted in ZooKeeper
■ Env vars
■ Access control by service path
■ Restart service to update value
Encrypted ✓ Access control ✓ Life-cycle ?
Slide 40
Slide 40 text
40
Nomad
@LizRice | @AquaSecTeam
■ Integrated with Vault
■ Use production mode
■ Encryption & security primitives
Slide 41
Slide 41 text
41
Nomad
@LizRice | @AquaSecTeam
■ Secrets passed as files
■ Nomad takes care of interactions with Vault
■ Tasks get tokens so they can retrieve values
■ Poll for changed values
■ Access control
■ Audit logging
Encrypted ✓ Access control ✓ Life-cycle ✓