Sharing Secrets in a Container World (ConFoo Montreal 2017)

https://confoo.ca/en/yul2017/session/sharing-secrets-in-a-containerized-world

Transcript

1. Sharing Secrets in a Container World ConFoo Montreal 2017 Christian Heimes cheimes@redhat.com @ChristianHeimes
2. Sharing secrets in a Container World, ConFoo Montreal 2017 Who am I? • from Hamburg Germany • ~ 20 years Linux • Python Core Developer – maintainer of ssl, hashlib – Coverity Scan – PEP 370, 452, 456, 543 – security team
3. Sharing secrets in a Container World, ConFoo Montreal 2017 Professional life • Senior Software Engineer at Red Hat • Security Engineering & Identity Management – FreeIPA IdM – Dogtag PKI – Custodia Secrets Management
4. Sharing secrets in a Container World, ConFoo Montreal 2017 DevOp Professionally paranoid security engineer
5. Sharing secrets in a Container World, ConFoo Montreal 2017 Agenda • What are “containers” and “secrets”? • Best practices for secrets management • Quick look at existing solutions • A different approach • Q & A
6. Sharing secrets in a Container World, ConFoo Montreal 2017 No silver bullet
7. Sharing secrets in a Container World, ConFoo Montreal 2017
8. Sharing secrets in a Container World, ConFoo Montreal 2017 Development • Make it work • Make it fast • Make it secure... • … if there is some spare time
9. Containers
10. Sharing secrets in a Container World, ConFoo Montreal 2017 Containers • chroot on steroids private view of and limited access to system • packaging solution “reproducible” build • deployment mechanism • mostly stateless (insects) • sometimes stateful (elephants)
11. Sharing secrets in a Container World, ConFoo Montreal 2017 Linux containers?
12. Sharing secrets in a Container World, ConFoo Montreal 2017 Containers are an illusion!
13. Sharing secrets in a Container World, ConFoo Montreal 2017 Linux Kernel namespaces, seccomp Container run-time Docker, rkt, OCI runC, ... Linux Kernel namespaces, seccomp, SELinux, ... Orchestration Kubernetes, Mesos, Docker Swarm, ...
14. Secrets
15. Sharing secrets in a Container World, ConFoo Montreal 2017 Definition (Noah Kantrowitz) • small • required • radioactive “Behind Closed Doors, Managing Passwords in a Dangerous World”
16. Sharing secrets in a Container World, ConFoo Montreal 2017 Types of Secrets • Passwords • Tokens (API keys) • Private / public key pairs (certificates) • One-time-pads • Kerberos tickets • ...
17. Sharing secrets in a Container World, ConFoo Montreal 2017 Temperature / Speed • Hot / Online used a lot, automated • Cold / Offline used rarely, human interaction • Fast / Dynamic regular change, automatic rotation • Slow / Static change is “big”
18. Sharing secrets in a Container World, ConFoo Montreal 2017 Hot & Fast
19. Sharing secrets in a Container World, ConFoo Montreal 2017 Secrets authentication • send secret to peer password, token • symmetric proof of knowledge digest auth, challenge response, TOTP / HOTP • asymmetric proof X.509 certs, JOSE • proof of access HSM, TPM, smart cards
20. Secrets Management
21. Sharing secrets in a Container World, ConFoo Montreal 2017 Infrastructure pre-conditions • data center is protected • hardware is safe (CPU UEFI, firmwares, …) TPM remote attestation • Kernel and OS are secure SecureBoot, CoreBoot (Heads), dm-verity • container run-time and orchestration are secure • PKI with trusted CA
22. Sharing secrets in a Container World, ConFoo Montreal 2017 Best practices • encrypt at rest (encrypted storage) • encrypt in transit • audit log • access control (RBAC, HBAC) • identity • rollover / renewal • least amount of privileges • no cleartext secrets on the file system
23. Secrets Management Solutions
24. Sharing secrets in a Container World, ConFoo Montreal 2017 Un-solution • back secrets into container image • put secrets in container config # Dockerfile ENV SECRET=”not so secret” $ docker run -e SECRET=bad ...
25. Sharing secrets in a Container World, ConFoo Montreal 2017 Kubernetes secrets • etcd database – no encryption at rest – optional TLS • no ACLs • controlled by kubelet – files in tmpfs – ENV vars
26. Sharing secrets in a Container World, ConFoo Montreal 2017 Docker Swarm secrets • encrypted storage • identities with PKI • secrets in tmpfs
27. Sharing secrets in a Container World, ConFoo Montreal 2017 Square KeyWhiz • encrypted storage with HSM • strong focus on PKI • LDAP for authentication / authorization • secrets in KeyWhizFS (FUSE file system)
28. Sharing secrets in a Container World, ConFoo Montreal 2017 More • HashiCorp Vault • EnvConsule • Rancher and Cattle • … Jeff Nickoloff: “Secrets and LIE-abilities: The State of Modern Secret Management (2017)”
29. Common design problem
30. Sharing secrets in a Container World, ConFoo Montreal 2017 env vars file system
31. Sharing secrets in a Container World, ConFoo Montreal 2017 File system attacks • accidental backup • directory traversal attack GET /download?file=../../secrets/key
32. Sharing secrets in a Container World, ConFoo Montreal 2017 env vars • directory traversal attack • debug tools • other means GET /download?file=../proc/self/environ GET /download?file=../proc/self/environ SELECT getenv('secret');
33. Sharing secrets in a Container World, ConFoo Montreal 2017 Get secrets into container Let application request secret
34. Meet Custodia
35. Sharing secrets in a Container World, ConFoo Montreal 2017 Custodia Definition according to wikitionary Latin • protection, safekeeping • custody, guardianship English • Care (concern, responsibility) • pyx (box) ◦ housing of a ship’s compass ◦ container for consecrated bread
36. Sharing secrets in a Container World, ConFoo Montreal 2017 What is Custodia? • Technology (not yet a product) • Secrets-as-a-Service API ◦ Existing solutions push secrets to client ◦ Actively request secrets from a service • Secrets transport and routing layer definition • Pluggable authentication and authorization • Reference implementation in Python
37. Sharing secrets in a Container World, ConFoo Montreal 2017 What is Custodia not? • Custodia is not intended to replace CoreOS etcd, OpenStack, Barbican, Hashicorp Vault etc. • Custodia is not a secrets storage server ◦ adapter to existing solutions • Custodia is not limited to containers • Custodia is not just a proof of concept ◦ FreeIPA / Red Hat IdM
38. Design
39. Sharing secrets in a Container World, ConFoo Montreal 2017 Threat analysis results • No secrets on the file system • Strong encryption at rest and in transit • End-to-end encryption • Reduce access and exposure • Audit logs • Key rotation for stateful containers PCI-DSS compliance
40. Sharing secrets in a Container World, ConFoo Montreal 2017 Design principles • Applications request secrets (active pull) • Use standard protocols and data formats • Make it easy to write clients, servers and integration • Flexible (authentication, authentication, storage) • Extensible (secrets on demand, HSM) • Dynamic secrets
41. Sharing secrets in a Container World, ConFoo Montreal 2017 Building blocks • HTTP REST API HTTPS, HTTP over Unix socket • JSON optionally JOSE for key encapsulation • Storage layer abstraction • Pluggable authentication and authorization • Transparent routing and forwarding built-in Load balancing, separation of tenants
42. Sharing secrets in a Container World, ConFoo Montreal 2017 Benefits • Simple interface for developers (API) • Decoupled layers give flexibility • Configuration and policies are moved to infrastructure Let ops take care of secrets • Vendor agnostic management and storage Standardization in progress!
43. Custodia for containers
44. Sharing secrets in a Container World, ConFoo Montreal 2017 Authentication?
45. Sharing secrets in a Container World, ConFoo Montreal 2017 CURL your secrets $ curl -H 'Accept: text/plain' --unix-socket /var/run/custodia.sock \ http://localhost/secrets/container/key secret value
46. Sharing secrets in a Container World, ConFoo Montreal 2017 CURL your secrets $ curl -H 'Accept: text/plain' --unix-socket /var/run/custodia.sock \ http://localhost/secrets/container/key secret value
47. Sharing secrets in a Container World, ConFoo Montreal 2017 Unix sockets to the rescue! • Run Custodia in host PID namespace • Mount Unix socket into containers • getsockopt() SO_PEERCRED PID, effective UID, effective GID • getsockopt() SO_PEERSEC SELinux process label (sVirt) • /proc/PID/cgroup Docker container ID (HACK)
48. Sharing secrets in a Container World, ConFoo Montreal 2017 Kubernetes master Architecture diagram Container Custodia Application Application Custodia Custodia Key Store Key Store Key Store Key Store Key Store Integration on the system over a local socket • Kubernetes node • VM
49. Integration
50. Sharing secrets in a Container World, ConFoo Montreal 2017 Config parser integration • INI parser • YAML parser user = client password = ${CUSTODIA:container/key} user: client password: !custodia container/key
51. Sharing secrets in a Container World, ConFoo Montreal 2017 Docker credential store docker command docker-store-custodia Custodia daemon calls connects over Unix socket $ cat > ~/.docker/config.json << EOF {"credsStore": "custodia"} EOF Fetch secret
52. Sharing secrets in a Container World, ConFoo Montreal 2017 Delegation – ‘software HSM’ • digest auth • PKCS#11 • ssh-agent • gpg-agent
53. Sharing secrets in a Container World, ConFoo Montreal 2017 Custodia integration plans • Java Keystore provider (java.security.KeyStore) • SecretService / libsecret provider Gnome / KDE keyring API for system DBUS on servers • OpenStack oslo.config • PKCS #11 / PKCS #12 provider Private keys for OpenSSL (Apache mod_ssl), SSH... • Ansible Tower • CloudForms
54. Sharing secrets in a Container World, ConFoo Montreal 2017 FreeIPA KDC LDAP PKI DNS CLI/UI
55. Sharing secrets in a Container World, ConFoo Montreal 2017 Storage: FreeIPA Vault & Dogtag PKI • Dogtag PKI KRA (key recovery agent) • LDAP storage with 389-DS master-master replication and replication topology • Encrypted in transit and on disk optional support for HSM (hardware security module) • GSSAPI / Kerberos authentication • Central access control • Signed, tamper-proof audit log
56. Sharing secrets in a Container World, ConFoo Montreal 2017 Links • https://github.com/latchset/custodia • https://github.com/latchset (demos) • Docker Hub: latchset/custodia • https://custodia.readthedocs.io/ • FreeIPA https://www.freeipa.org • Ansible + Vagrant demo https://github.com/tiran/pki-vagans/ Christian Heimes <cheimes@redhat.com> @ChristianHeimes
57. Sharing secrets in a Container World, ConFoo Montreal 2017 Questions?
58. THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVide os facebook.com/redhatinc twitter.com/RedHatNews