Building a Container Platform: Best Practices for Enterprise-Grade Kubernetes Security

What do autonomous vehicles and container platforms at Cruise have in common? For starters, we view safety and security as the number one top priority. These slides discuss 5 critical security topics that intersect with container platforms and explore how Cruise tackled their challenges to enable a "Secure by Default" PaaS environment for Self-Driving Cars.

Topics include: Identity, Authentication, Authorization, Secrets Management, & Encryption.

Transcript

1. @mf_ruth Building a Container Platform Best Practices for Enterprise-Grade Kubernetes Security Mike Ruth Staff Security Engineer @ Cruise
2. We’re building the world’s most advanced self-driving vehicles to safely connect people with the places, things, and experiences they care about. https://cruise-automation.github.io/webviz/worldview/ https://getcruise.com/
3. @mf_ruth [C]
4. @mf_ruth Security Domains • Identity • Authentication • Authorization • Secrets • Encryption • And more!
5. @mf_ruth Identity • Representation of an entity interacting with a system • Take one of two forms: Users, Services • Contain unique identifiers (metadata) and credentials
6. @mf_ruth Establishing User Identity • Identity Provider (IdP) User • Cloud IAM Users • Kubernetes Users?
7. @mf_ruth Establishing Service Identity • Cloud (GCP) Service Accounts • Kubernetes Service Accounts
8. @mf_ruth Identity Management Cruise • Identity Providers enable a single management location for User identity (and sometimes Service Identity). • Use of Single Sign On for federated identity across clouds and platforms • Google domain. Leverage Okta for IdP. Duo for MFA. • Can begin to see how IdP Users + GCP & K8s Service Accounts provides identity for all of the people and services that need to interact with Cruise PaaS
9. @mf_ruth Authentication • Authentication (AuthN) is the confirmation that an entity is whom they claim to be, by verifying one’s identity • Kubernetes verifies requests through the K8s API Server • Support for multiple credential types ◦ X.509 Certs, Bearer Tokens, and Basic Auth (but pls no)
10. @mf_ruth Authenticating Users • Users typically leverage ◦ A config file (kubeconfig) ◦ Kubernetes CLI - kubectl (don’t ask how to pronounce this) • Kubeconfig contains some credential associated with a K8s cluster ◦ Often an X.509 cert ◦ Often the admin cert that gets provisioned during cluster creation • Cruise PaaS ◦ Users are authenticated based on their IdP domain identity ◦ This uses OIDC bearer tokens
11. @mf_ruth • User-Managed Kubernetes Cluster • Kubeconfig contains a long-lived client certificate and key Authenticating Users
12. @mf_ruth • Google-Managed GKE Cluster • Kubeconfig contains short lived access token • config-helper uses OIDC refresh token to regenerate access token Authenticating Users
13. @mf_ruth Authenticating Services • K8s SAs bind to Pod ◦ Services in pod implicitly capable of authenticating to API Server • GCP SAs can use their: ◦ JSON keyfile with google SDK ◦ or craft an access token (JWT) similar to user flow
14. @mf_ruth Authorization • Authorization (AuthZ) is the confirmation that an entity can perform the action they’re attempting to perform • AuthZ is enforced through the K8s API Server ◦ Multiple access control types (RBAC, ABAC, etc) • Group membership + Rolebindings + Permissions makes up the majority of authorization flows in K8s
15. @mf_ruth Role Based Access Control (RBAC)
16. @mf_ruth RBAC in Kubernetes • RBAC binds Kubernetes identities against Kubernetes resources ◦ Provides CRUD-like permissions ◦ Enables granular authZ against those resources • Two role-binding types ◦ Namespace ◦ Cluster • Notable Limitations ◦ Kubernetes has no default context of IdP identities ◦ Binding IdP users or GCP users/SAs programmatically is needed
17. @mf_ruth Binding Identities in K8s (RBACSync) https://github.com/cruise-automation/rbacsync
18. @mf_ruth Secrets • Anything sensitive, but typically these are credentials or key material ◦ tokens, passwords, certificates, encryption keys, etc • Kubernetes Secrets are the default mechanism for storing secrets
19. @mf_ruth Kubernetes Secrets • Persisted in state storage (etcd) • Referenced natively in manifests. Convenient to use! ◦ Injected into env vars, in-memory tmpfs volumes Limitations & Security Concerns • State storage not encrypted by default (GKE notwithstanding) • Strict tenancy by namespace is frequently overlooked, meaning over-authorization to Kubernetes Secrets is common • Incredibly difficult to manage in GitOps flows ◦ Avoid encrypting secrets and storing them in VCS!
20. @mf_ruth Secrets management is hard
21. @mf_ruth Centralized Secrets Management Enter Hashicorp Vault
22. @mf_ruth Vault Auth Backends
23. Group Permissions Path Team Admin read, create, update, delete, list secret/<prefix>/<namespace>/* Team Contractor list secret/<prefix>/<namespace>/* App Service Account list, read secret/<prefix>/<namespace>/<env>/<app>/* Vault Pathing
24. @mf_ruth Enter Daytona Secrets fast and secure https://github.com/cruise-automation/daytona
25. @mf_ruth Daytona • Automate Vault authentication, fetching of secrets, and Vault token renewal • Deployed as a sidecar, initContainer, or entrypoint ◦ Inject secrets into tmpfs or env vars • Support for AWS, GCP, Kubernetes Auth Backends
26. In-Memory Volume App Container Kubernetes Pod Secrets Vault Auth Vault Integration - Kubernetes @mf_ruth Vault Login Kubernetes service accounts used for Vault authentication. Secrets Injection Daytona side-loads secrets Short-lived Credentials Vault generates temporary AWS & GCP credentials on-demand
27. @mf_ruth Encryption • Broad topic, but we can break it down into two categories: in transit & at rest • Large variance between Kubernetes defaults and GKE defaults
28. @mf_ruth Encryption - In Transit • Kubernetes uses TLS to perform encryption in transit. This requires: ◦ Kubernetes sub-components (i.e. kubelet, kube-proxy) to have certificates for mTLS ◦ Certificates and keys to be available on worker nodes ◦ Certificates to be minted by a trusted CA chain. Often internal Kubernetes CA. • This is the responsibility of the cluster owner to configure and manage
29. @mf_ruth Encryption - In Transit List of some of the services in K8s that need TLS: • Kubernetes State Storage (etcd) API* • Kubernetes API* • Kubelet API* • Workload Ingress *GKE manages by default.
30. @mf_ruth Encryption - At Rest List of some of the services in K8s that need to be persisted at rest: • Kubernetes State Storage* • Kubernetes Node Disks* • Workload Volumes* • Kubernetes Service Account Credentials • Workload Secrets *GKE manages by default
31. @mf_ruth And more! More than just the domains we talked about, other topics to consider: • Auditing (Compliance, Threat Detection, Alerting) • Platform Hardening (SecurityContext, Node Metadata Protection, Network Policies, Pod Security Policies, Policy Enforcement)* • Secure Supply Chains (Trusted Image Building, Vulnerability Scanning, Attestation) • Patch Management • Zero Trust Networking *See k-rail for workload policy enforcement: https://github.com/cruise-automation/k-rail
32. @mf_ruth Thank You Questions?