Developing an Ecosystem for More Secure Kubernetes Secrets Management

Katsuya Yamaguchi (Yahoo! JAPAN / Security & Developer PF Division, System Management Group, Technology Group / Software Engineer)

https://tech-verse.me/ja/sessions/190
https://tech-verse.me/en/sessions/190
https://tech-verse.me/ko/sessions/190

Transcript

1. Developing an Ecosystem for More Secure Kubernetes Secrets Management Katsuya Yamaguchi / Yahoo! JAPAN
2. © Yahoo Japan Speaker Katsuya Yamaguchi - Joined Yahoo! JAPAN as a new grad in 2017 - Software Engineer - Data Protection Technology Team
3. © Yahoo Japan About this session - Secrets Store CSI Driver as an ecosystem for our platforms - What is Secrets Store CSI Driver? - What are the advantages for developer? Secrets Store CSI Driver => SSCD Kubernetes => K8s
4. © Yahoo Japan - Background - What is Secrets Store CSI Driver? - Overview of the ecosystem Agenda
5. © Yahoo Japan Background
6. © Yahoo Japan Our in-house Secrets Manager store Revoke generation 1 2 3 Version control - High confidentiality, integrity, and availability - Centralized secrets management - Database password - API key - Encryption Key - Services suitable for each platform
7. © Yahoo Japan Our in-house computing platforms Secrets Manager Computing platforms Kubernetes Pod FaaS Function PaaS Application … Developer
8. © Yahoo Japan Our in-house computing platforms Secrets Manager Computing platforms Kubernetes Pod FaaS Function PaaS Application Access Control … Authentication platform Developer
9. © Yahoo Japan Our in-house computing platforms Secrets Manager Computing platforms Kubernetes Pod FaaS Function PaaS Application Access Control … Authentication platform Developer
10. © Yahoo Japan Responsibilities of Secrets Management Developer Secrets Manager Application Computing platforms - Conveying secret data to platforms - Keeping secret data safe until you need to use it - Managing secret data on platforms
11. © Yahoo Japan How do we manage secret data on K8s? - Kubernetes Secret Resource - Using sidecar for Secrets Manager - Using client library for Secrets Manager
12. © Yahoo Japan How do we manage secret data on K8s? - Kubernetes Secret Resource - Using sidecar for Secrets Manager - Using client library for Secrets Manager
13. © Yahoo Japan Kubernetes Secret Resource CI/CD Secrets Manager Application Container Volume Developer
14. © Yahoo Japan Problems: Kubernetes Secret Resource - Complex additional steps required to acquire secret data - Possibility of secret data leakage - Key management for etcd
15. © Yahoo Japan How do we manage secret data on K8s? - Kubernetes Secret Resource - Using sidecar for Secrets Manager - Using client library for Secrets Manager
16. © Yahoo Japan Using sidecar for Secrets Manager Secrets Manager Application Container Sidecar Container Security credentials for Athenz Access Control
17. © Yahoo Japan Problems: Using sidecar for Secrets Manager - Additional implementation for getting secret data - Same problems as Kubernetes Secret Resource - Wastage of resources of the sidecar
18. © Yahoo Japan How do we manage secret data on K8s? - Kubernetes Secret Resource - Using sidecar for Secrets Manager - Using client library for Secrets Manager
19. © Yahoo Japan Using client library for Secrets Manager Secrets Manager Access Control Application that uses the client library
20. © Yahoo Japan Problems: Using client library for Secrets Manager - Implementation cost - Additional dependency of this library - Learning cost for using this library
21. © Yahoo Japan Ideal Secrets Management for developers Should not take much for key management Should not use special libraries or modules Confidentiality of secret data is not developer dependent
22. © Yahoo Japan External Secrets Operator Sealed Secrets HashiCorp Vault Easy integration with internal platform Easy to use Popularity Installation costs Comparison of application as an ecosystem Inconsistent with internal security measures Enterprise edition is required Secrets Store CSI Driver
23. © Yahoo Japan External Secrets Operator Sealed Secrets HashiCorp Vault Easy integration with internal platform Easy to use Popularity Installation costs Comparison of application as an ecosystem Inconsistent with internal security measures Enterprise edition is required Secrets Store CSI Driver
24. © Yahoo Japan External Secrets Operator Sealed Secrets HashiCorp Vault Easy integration with internal platform Easy to use Popularity Installation costs Comparison of application as an ecosystem Inconsistent with internal security measures Enterprise edition is required Secrets Store CSI Driver
25. © Yahoo Japan External Secrets Operator Sealed Secrets HashiCorp Vault Easy integration with internal platform Easy to use Popularity Installation costs Comparison of application as an ecosystem Inconsistent with internal security measures Enterprise edition is required Secrets Store CSI Driver ✓
26. © Yahoo Japan What is the Secrets Store CSI Driver (SSCD)?
27. © Yahoo Japan Container Storage Interface (CSI) External storage Container Orchestration CSI Driver
28. © Yahoo Japan Secrets Store CSI Driver (SSCD) Azure Key Vault AWS Secrets Manager GCP Secret Manager AWS Provider GCP Provider Azure Provider SSCD
29. © Yahoo Japan Advantage: Easy to introduce Provider SSCD Secrets Manager https://github.com/kubernetes-sigs/secrets-store-csi-driver/blob/main/provider/fake/fake_server.go Enable work just by developing a provider
30. © Yahoo Japan Control Plane Node Worker Nodes SSCD Provider Secrets Manager How Secrets Store CSI Volume works
31. © Yahoo Japan Control Plane Node Worker Nodes SSCD Provider Secrets Manager How Secrets Store CSI Volume works
32. © Yahoo Japan How are the secrets mounted to a container? Application Container tmpfs mount: e.g., /mnt/secrets/<secret name>.txt - Provider writes secret data to files - Application can read secret data from the files Secret data as files: (Provider fetches secret data from exernal secrets manager)
33. © Yahoo Japan Secret data is not written to the disk on Host Unnecessary key management for etcd Only container can access secret data Advantage: High Confidentiality
34. © Yahoo Japan Usage SecretProviderClass Application YAML Provider name Information of secret data stored in an external Secrets Manager
35. © Yahoo Japan Known Limitation: Mounted Secret Data is not Updated Automatically SecretProviderClass Application Container Secrets Manager Updated secret data is not reflected in the pod mount
36. © Yahoo Japan Overview of the ecosystem
37. © Yahoo Japan Secrets Manager K8s private cloud Authentication platform
38. © Yahoo Japan Provider for In-house Secrets Manager 1. Get a security credential for Athenz 2. Fetch secret data using the security credential 3. Write secret data to the volume
39. © Yahoo Japan Security credentials of Athenz X.509 certificate for K8s service account X.509 certificate (security credentials)
40. © Yahoo Japan Workings of provider SSCD Provider Secrets Manager Security credential for a service
41. © Yahoo Japan SSCD as an ecosystem for our platforms SSCD Provider Multi-cluster K8s as a service SSCD Provider SSCD Provider Secrets Manager
42. © Yahoo Japan Future work Improving familiarity - How to use, benefits, system linking Expansion of this ecosystem - Introducing this ecosystem to other platforms Additional Functions - Considering to enable “Sync as Kubernetes Secret” feature - Mounted secret data and Kubernetes Secret are not updated automatically
43. © Yahoo Japan Conclusion - Secrets Store CSI Driver as an ecosystem for our Kubernetes platforms - More secure secrets management for developer - Increased productivity for developers
44. © Yahoo Japan Thank you