Hierarchical Namespaces now with less hierarchy

Canadian CNCF Meetup - Feb 16 2021

Video: https://youtu.be/NSAyBFZzkyk

Transcript

1. Public Public Hierarchical Namespaces: now with less hierarchy! Eastern Canadian CNCF Meetup - Feb 16 2021 Adrian Ludwin (aludwin@google.com) Ginny Ji (ginnyji@google.com)
2. Public Public Introduction to multitenancy
3. Public Kubernetes at a glance control plane user CLI/API/UI kubelet kubelet kubelet NODES
4. Public control plane tenant CLI/API/UI kubelet kubelet kubelet NODES One tenant, one cluster
5. Public CP 1 Tenant 1 kubelet CLI/API/UI kubelet Cluster 1 CP 2 Tenant 2 kubelet CLI/API/UI kubelet kubelet Cluster 2 Multiple tenants, multiple clusters? kubelet
6. Public kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster kubelet kubelet kubelet CP Tenant CLI/API/UI Cluster Kubesprawl: how does this scale?
7. Public CP Tenant ns3-pod1 CLI/API/UI ns2-pod1 ns2-pod2 Cluster Tenant Tenant Tenant ns1-pod1 ns1-pod2 ns3-pod2 ns3-pod3 namespace Namespace 2 Alternative: many tenants, one cluster Namespace 1 Namespace 3
8. Public Regionalization So... should I ever use multiple clusters? Blast radius Scalability Cluster snowflakes … mainly CRDs, webhooks, or cluster-scoped operators See also: “The myth of the monocluster”
9. Public About wg-multitenancy The Multitenancy Working Group was formed to categorize and solve multitenancy problems in the Kubernetes ecosystem. Current projects include HNC (this presentation), Virtual Clusters and the multitenancy benchmark project. There’s more at the end of this presentation, but TL;DR: github.com/kubernetes-sigs/multi-tenancy
10. Public Public Namespaces: the foundation of multitenancy
11. Public Namespaces Namespaces are the primary unit of tenancy in Kubernetes. By themselves, they don’t do much except organize other objects - but almost all policies support namespaces by default.
12. Public Some security features require namespaces Service accounts and Secrets are freely usable within a namespace • Anyone with permission to deploy a pod in a namespace can use any Secret or run as any SA • This is why it’s best practice to segregate workloads and teams in different namespaces if their secrets/SAs are sensitive Note: namespaces only isolate the control plane, not the data plane • A malicious workload that escapes its container can attack anything else in the cluster • Use sandboxing (e.g. gVisor, Kata) to defend the data plane
13. Public Other features provide support for namespaces RBAC works best at the namespace level: • Only way to scope object creation • Least brittle way to scope other operations Also applies to most other policies: • Resource quotas and limit ranges only apply to namespaces • Network policies can be more finely targeted but use namespace boundaries by default ◦ Caveat: requires labels, which are not secure
14. Public What if you want policies across namespaces? Usually, you need a tool and source-of-truth outside of Kubernetes: • Flux, Argo, GKE Config Sync, Anthos Config Management Alternatively, some in-cluster solutions add “accounts” or “tenants” • Kiosk or the Tenant CRD (another wg-multitenancy project) We felt there was a need for a solution that: • Was fully Kubernetes-native (i.e. no dependencies on Git) • Extended existing concepts rather than add new ones
15. Public Hierarchical namespaces An incubating OSS standard to express ownership, which allows for admin delegation and cascading policies. Hierarchical Namespaces are provided by the Hierarchical Namespace Controller (HNC). org 1 org 2 team A team B svc 1 svc 2 team C subteam C2 snowflake team
16. Public Properties of hierarchical namespaces Entirely Kubernetes-native, but compatible with existing Gitops tools (e.g. Flux). Builds on regular Kubernetes namespaces, plus: • Delegated subnamespace creation without cluster privileges • Cascading policies, secrets, configmaps, etc. • Trusted labels for policy application (e.g. Network Policies) • Easy to extend and integrate But sometimes, a strict hierarchy is also too limiting...
17. Public Public Introducing Exceptions
18. Public Example Team A Eng Department Team B Team C Playground
19. Public Team A real-secret Eng Department real-secret Team B real-secret Team C real-secret Playground Secret exceptions and exemptions No secret here!
20. Public Team A common-quota Eng Department common-quota Team B Team C common-quota Playground common-quota Quota exceptions team-b-rq team-b-rq
21. Public Solution Annotations that allow you to limit the propagation of an object to descendant namespaces.
22. Public Demo
23. Public To find more about Exceptions: • Visit HNC User Guide in our Github repo ◦ https://github.com/.../hnc/docs/user-guide • Google for `HNC v0.7.0 Exceptions`
24. Public Public HNC: the path to v1.0
25. Public What’s needed to get to v1.0? • We don’t currently intend to add new features to v1.0 ◦ This doesn’t mean we’re refusing to add more features ◦ Just that none are required for the initial stable release • What’s left: ◦ Ensure we’re using the latest up-to-date APIs (e.g. CRD v1beta1 -> v1) to support the latest Kubernetes releases ◦ Improve stability and (if necessary) usability as required ◦ (Possibly) Move to our own repo under kubernetes-sigs ▪ We’ve received initial approval from sig-auth ▪ We’ve completed our API review; still need an internal code review
26. Public How can you help? • Try using HNC! ◦ The more usage and bug reports we get, the faster we can progress to v1.0! ◦ On any cluster (v1.16 or later): install from Github (latest is v0.7.0) ▪ Check out our documentation, especially our quickstarts! ▪ Get the kubectl plugin from Krew ◦ On GKE: try Hierarchy Controller (it’s free). Includes HNC plus “enterprise” features like: ▪ Gitops via Config Sync unstructured repos ▪ Hierarchical observability to filter logs and usage ▪ Hierarchical resource quota (expected release date: eoFeb) ▪ Multi-cluster installation and management (ETA: Q1; requires ACM subscription) • Contribute to HNC: ◦ Check out any “help-wanted” issues in our hnc-v0.8 or hnc-backlog milestones on Github
27. Public Learning more • More presentations: ◦ CNCF Kubecon EU (Aug 2020): Multitenant clusters with Hierarchical Namespaces ◦ CNCF webinar (Jun 2020): Better walls make better tenants ◦ VMWare TGIK (Sep 2020): Let’s try hierarchical namespaces ▪ … maybe watch that one at double speed • Join the multitenancy working group ◦ We meet every second Tuesday, plus there’s a Slack channel and mailing list ◦ https://github.com/kubernetes-sigs/multi-tenancy
28. Public Thanks!