Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Your (container) secret's safe with me - 2018 version

Liz Rice
March 21, 2018

Your (container) secret's safe with me - 2018 version

An updated version of this talk, concentrating on Kubernetes (I've previously published a version covering secrets in several container orchestrators)

Liz Rice

March 21, 2018
Tweet

More Decks by Liz Rice

Other Decks in Technology

Transcript

  1. Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved.

    Your (container) secret’s safe with me Liz Rice @LizRice | @AquaSecTeam
  2. 5 Secrets @LizRice | @AquaSecTeam Secrets photo: Katie Tegtmeyer ▪

    Encrypted ▪ At rest and in transit ▪ Only decrypted in memory
  3. 6 Secrets @LizRice | @AquaSecTeam Secrets photo: James Case ▪

    Access control ▪ Only accessible by containers that need them ▪ And users ▪ Write-only access
  4. 7 Secrets @LizRice | @AquaSecTeam Secrets photo: Irena Jackson ▪

    Life-cycle ▪ Risk of leak increases over time ▪ Rotation, revocation, audit logging
  5. 11 ▪ docker inspect ▪ Leaky logs ▪ docker exec

    ▪ /proc directory Environment variables @LizRice | @AquaSecTeam
  6. 13 ▪ docker inspect ▪ Leaky logs ▪ docker exec

    ▪ /proc directory Mounted volume @LizRice | @AquaSecTeam
  7. 15 Bad places for secrets @LizRice | @AquaSecTeam ▪ Source

    code ▪ Dockerfiles / images ▪ In plain text in YAML files
  8. 16 Kubernetes secrets @LizRice | @AquaSecTeam ▪ Secrets are Kubernetes

    objects ▪ Refer to secret in pod YAML as environment variable: ... env: - name: MYSECRET valueFrom: secretKeyRef: name: mysecret key: secret_key
  9. 17 Kubernetes secrets @LizRice | @AquaSecTeam ▪ ...or as a

    file in a volume mount: ... volumeMounts: - name: secret mountPath: /.secrets readOnly: true volumes: - name: secret secret: secretName: mysecret
  10. 19 Kubernetes secrets @LizRice | @AquaSecTeam ▪ Stored in etcd

    ▪ Make sure secrets are encrypted! ▪ --experimental-encryption-provider-config on API Server
  11. 20 Encrypting etcd @LizRice | @AquaSecTeam kind: EncryptionConfig apiVersion: v1

    resources: - resources: - secrets providers: - aescbc: keys: - name: key1 secret: bXlWZXJ5U2VjcmV0RW5jcnlwdGlvbktleQo= - identity: {}
  12. 21 Secrets all the way down @LizRice | @AquaSecTeam ▪

    EncryptionConfig holds a secret key... xkcd.com/1416
  13. 22 External key stores @LizRice | @AquaSecTeam ▪ Secret storage

    in 3rd party backend ▪ Hashicorp Vault, Amazon KMS, Azure Key Vault, CyberArk Vault… ▪ Kubernetes adding Key Management Service plugin support
  14. 24 kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: secret-reader rules: -

    apiGroups: [“”] resources: [“secrets”] verbs: [“get”, “list”, “watch”] Kubernetes RBAC for secrets @LizRice | @AquaSecTeam ▪ Role for read-only access to secrets
  15. 25 Kubernetes RBAC for secrets @LizRice | @AquaSecTeam kind: RoleBinding

    apiVersion: rbac.authorization.k8s.io/v1 metadata: name: read-secrets namespace: development subjects: - kind: User name: dave apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io ▪ Let “dave” read secrets in the “development” namespace
  16. 26 RBAC secrets best practices @LizRice | @AquaSecTeam kind: ClusterRole

    apiVersion: rbac.authorization.k8s.io/v1 metadata: name: my-secret-reader rules: - apiGroups: [“”] resources: [“secrets”] resourceNames: [“my-secret”] verbs: [“get”] ▪ Be careful with list & watch ▪ Limit to get where possible ▪ Limit access to only the secret(s) an app needs
  17. 27 Least privileges @LizRice | @AquaSecTeam pod ▪ Read-only mount

    ▪ Split to separate container with simple behaviour
  18. 29 Kubernetes secret rotation @LizRice | @AquaSecTeam ▪ Files support

    updating secret values ▪ Need to restart pod to get new env var value
  19. 30 Audit logging secrets access @LizRice | @AquaSecTeam apiVersion: audit.k8s.io/v1beta1

    kind: Policy rules: # Log secret changes at the Metadata level. - level: Metadata resources: - group: "" # core API group resources: ["secrets"]
  20. 32 Commercial secrets solutions @LizRice | @AquaSecTeam ▪ File system

    & env var support ▪ Update secrets without container restart ▪ No env var leak through inspect or /proc ▪ Full auditing of secret usage ▪ User & container access control ▪ 3rd party storage ▪ Any orchestrator Encrypted ✓ Access control ✓ Life-cycle ✓
  21. 34 Secrets @LizRice | @AquaSecTeam Secrets photo: Iain Merchant ▪

    Turn on encryption ▪ Access secrets at runtime ▪ Not built in ▪ Rotate secrets
  22. Copyright @ 2017 Aqua Security Software Ltd. All Rights Reserved.

    The Ultimate Guide to Secrets Management in Containers tiny.cc/secrets @LizRice | @AquaSecTeam
  23. 37 Docker @LizRice | @AquaSecTeam ▪ Secrets support built in

    for Docker Swarm services ▪ Not standalone containers ▪ Encrypted transmission with mutual authentication ▪ Secret accessible when exposed to service ▪ Mounted to a temporary fs (not env vars) ▪ RBAC in Enterprise Edition
  24. 38 Docker @LizRice | @AquaSecTeam ▪ Encrypted in Raft log

    ▪ Lock your Swarm!! ▪ Shared to Swarm managers ▪ Audit log with events ▪ Rotation requires container restart & secret dance Encrypted ✓ Access control ✓ Life-cycle ?
  25. 39 DC/OS @LizRice | @AquaSecTeam ▪ Enterprise DC/OS ▪ Plug-ins

    for Mesos/Marathon ▪ Encrypted in ZooKeeper ▪ Env vars ▪ Access control by service path ▪ Restart service to update value Encrypted ✓ Access control ✓ Life-cycle ?
  26. 40 Nomad @LizRice | @AquaSecTeam ▪ Integrated with Vault ▪

    Use production mode ▪ Encryption & security primitives
  27. 41 Nomad @LizRice | @AquaSecTeam ▪ Secrets passed as files

    ▪ Nomad takes care of interactions with Vault ▪ Tasks get tokens so they can retrieve values ▪ Poll for changed values ▪ Access control ▪ Audit logging Encrypted ✓ Access control ✓ Life-cycle ✓