Slide 1

Slide 1 text

Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved. Your (container) secret’s safe with me Liz Rice @LizRice | @AquaSecTeam

Slide 2

Slide 2 text

2 Secrets @LizRice | @AquaSecTeam

Slide 3

Slide 3 text

3 Secrets @LizRice | @AquaSecTeam Secret store Cluster

Slide 4

Slide 4 text

Desirable attributes for secrets management

Slide 5

Slide 5 text

5 Secrets @LizRice | @AquaSecTeam Secrets photo: Katie Tegtmeyer ■ Encrypted ■ At rest and in transit ■ Only decrypted in memory

Slide 6

Slide 6 text

6 Secrets @LizRice | @AquaSecTeam Secrets photo: James Case ■ Access control ■ Only accessible by containers that need them ■ And users ■ Write-only access

Slide 7

Slide 7 text

7 Secrets @LizRice | @AquaSecTeam Secrets photo: Irena Jackson ■ Life-cycle ■ Risk of leak increases over time ■ Rotation, revocation, audit logging

Slide 8

Slide 8 text

Passing secrets to containers

Slide 9

Slide 9 text

9 Bad places for secrets @LizRice | @AquaSecTeam ■ Source code ■ Dockerfiles / images

Slide 10

Slide 10 text

10 docker run -e VARNAME=secret ... Environment variables @LizRice | @AquaSecTeam

Slide 11

Slide 11 text

11 ■ docker inspect ■ Leaky logs ■ docker exec ■ /proc directory Environment variables @LizRice | @AquaSecTeam

Slide 12

Slide 12 text

12 docker run -v /hostsecrets:/secrets ... Mounted volume @LizRice | @AquaSecTeam

Slide 13

Slide 13 text

13 ■ docker inspect ■ Leaky logs ■ docker exec ■ /proc directory Mounted volume @LizRice | @AquaSecTeam

Slide 14

Slide 14 text

Kubernetes support for secrets

Slide 15

Slide 15 text

15 Bad places for secrets @LizRice | @AquaSecTeam ■ Source code ■ Dockerfiles / images ■ In plain text in YAML files

Slide 16

Slide 16 text

16 Kubernetes secrets @LizRice | @AquaSecTeam ■ Secrets are Kubernetes objects ■ Refer to secret in pod YAML as environment variable: ... env: - name: MYSECRET valueFrom: secretKeyRef: name: mysecret key: secret_key

Slide 17

Slide 17 text

17 Kubernetes secrets @LizRice | @AquaSecTeam ■ ...or as a file in a volume mount: ... volumeMounts: - name: secret mountPath: /.secrets readOnly: true volumes: - name: secret secret: secretName: mysecret

Slide 18

Slide 18 text

Encrypting secrets

Slide 19

Slide 19 text

19 Kubernetes secrets @LizRice | @AquaSecTeam ■ Stored in etcd ■ Make sure secrets are encrypted! ■ --experimental-encryption-provider-config on API Server

Slide 20

Slide 20 text

20 Encrypting etcd @LizRice | @AquaSecTeam kind: EncryptionConfig apiVersion: v1 resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: bXlWZXJ5U2VjcmV0RW5jcnlwdGlvbktleQo= - identity: {}

Slide 21

Slide 21 text

21 Secrets all the way down @LizRice | @AquaSecTeam ■ EncryptionConfig holds a secret key... xkcd.com/1416

Slide 22

Slide 22 text

22 External key stores @LizRice | @AquaSecTeam ■ Secret storage in 3rd party backend ■ Hashicorp Vault, Amazon KMS, Azure Key Vault, CyberArk Vault… ■ Kubernetes adding Key Management Service plugin support

Slide 23

Slide 23 text

Access control

Slide 24

Slide 24 text

24 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: secret-reader rules: - apiGroups: [“”] resources: [“secrets”] verbs: [“get”, “list”, “watch”] Kubernetes RBAC for secrets @LizRice | @AquaSecTeam ■ Role for read-only access to secrets

Slide 25

Slide 25 text

25 Kubernetes RBAC for secrets @LizRice | @AquaSecTeam kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets namespace: development subjects: - kind: User name: dave apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io ■ Let “dave” read secrets in the “development” namespace

Slide 26

Slide 26 text

26 RBAC secrets best practices @LizRice | @AquaSecTeam kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: my-secret-reader rules: - apiGroups: [“”] resources: [“secrets”] resourceNames: [“my-secret”] verbs: [“get”] ■ Be careful with list & watch ■ Limit to get where possible ■ Limit access to only the secret(s) an app needs

Slide 27

Slide 27 text

27 Least privileges @LizRice | @AquaSecTeam pod ■ Read-only mount ■ Split to separate container with simple behaviour

Slide 28

Slide 28 text

Lifecycle

Slide 29

Slide 29 text

29 Kubernetes secret rotation @LizRice | @AquaSecTeam ■ Files support updating secret values ■ Need to restart pod to get new env var value

Slide 30

Slide 30 text

30 Audit logging secrets access @LizRice | @AquaSecTeam apiVersion: audit.k8s.io/v1beta1 kind: Policy rules: # Log secret changes at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets"]

Slide 31

Slide 31 text

Secret updates demo

Slide 32

Slide 32 text

32 Commercial secrets solutions @LizRice | @AquaSecTeam ■ File system & env var support ■ Update secrets without container restart ■ No env var leak through inspect or /proc ■ Full auditing of secret usage ■ User & container access control ■ 3rd party storage ■ Any orchestrator Encrypted ✓ Access control ✓ Life-cycle ✓

Slide 33

Slide 33 text

Summary

Slide 34

Slide 34 text

34 Secrets @LizRice | @AquaSecTeam Secrets photo: Iain Merchant ■ Turn on encryption ■ Access secrets at runtime ■ Not built in ■ Rotate secrets

Slide 35

Slide 35 text

Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved. The Ultimate Guide to Secrets Management in Containers tiny.cc/secrets @LizRice | @AquaSecTeam

Slide 36

Slide 36 text

Orchestrator support for secrets

Slide 37

Slide 37 text

37 Docker @LizRice | @AquaSecTeam ■ Secrets support built in for Docker Swarm services ■ Not standalone containers ■ Encrypted transmission with mutual authentication ■ Secret accessible when exposed to service ■ Mounted to a temporary fs (not env vars) ■ RBAC in Enterprise Edition

Slide 38

Slide 38 text

38 Docker @LizRice | @AquaSecTeam ■ Encrypted in Raft log ■ Lock your Swarm!! ■ Shared to Swarm managers ■ Audit log with events ■ Rotation requires container restart & secret dance Encrypted ✓ Access control ✓ Life-cycle ?

Slide 39

Slide 39 text

39 DC/OS @LizRice | @AquaSecTeam ■ Enterprise DC/OS ■ Plug-ins for Mesos/Marathon ■ Encrypted in ZooKeeper ■ Env vars ■ Access control by service path ■ Restart service to update value Encrypted ✓ Access control ✓ Life-cycle ?

Slide 40

Slide 40 text

40 Nomad @LizRice | @AquaSecTeam ■ Integrated with Vault ■ Use production mode ■ Encryption & security primitives

Slide 41

Slide 41 text

41 Nomad @LizRice | @AquaSecTeam ■ Secrets passed as files ■ Nomad takes care of interactions with Vault ■ Tasks get tokens so they can retrieve values ■ Poll for changed values ■ Access control ■ Audit logging Encrypted ✓ Access control ✓ Life-cycle ✓