»Consistent user authentication in multi-cloud hosted Kubernetes clusters«

As hosted Kubernetes solutions mature, it becomes ever more compelling to operate clusters across multiple cloud providers. A general point of friction can often be the differences in how you are able to authenticate to those clusters. Cloud providers tend to integrate their own proprietary solutions and hosted control planes lack the flexibility to use authentication providers and audit sinks.

During this talk I will show how a reverse proxy in front of the Kubernetes API can implement uniform OIDC authentication across hosted Kubernetes solutions.

Transcript

1. jetstack.io Kube-OIDC-Proxy Consistent user authentication across clusters hosted in multiple clouds Presented by Christian Simon
2. jetstack.io WHO AM I? Christian Simon • Using Kubernetes for 4+ years • Started kube-lego • Working on Tarmak simonswine simonswine
3. jetstack.io WHO IS JOSH? Josh Van Leeuwen • Joined as intern during his final year of university • Now working full time as Solutions Engineer • Developer of kube-oidc-proxy joshvanl
4. jetstack.io WHO ARE JETSTACK? Jetstack • Cert-Manager, Tarmak, … • Professional services: Consulting, Training, ... • UK-based, remote friendly company • Nice place to work: We are hiring! jetstackhq jetstack jetstackhq
5. jetstack.io AGENDA • Introduction • Foundations ◦ OAuth2 ◦ JWT ◦ OpenID Connect (OIDC) ◦ Authn/z in • Kube-OIDC-Proxy • Demo
6. jetstack.io INTRODUCTION Authentication on Managed Kubernetes • Bespoke IAM • Lock-in • No integration with existing systems (on-premises etc.) • Inconsistent RBAC across multiple providers/accounts • Lack of customisation
7. Authentication Foundations jetstack.io
8. jetstack.io FOUNDATIONS JSON Web Tokens (JWTs) • Standardised signed token, to assert certain metadata (RFC7519) • Defined on top of JSON Web Signature (RFC7515) and JSON Web Encryption (RFC7516) • JWS: Consists of three base64 encoded parts separated by a dot: <Header>.<Payload>.<Signature> • JWT defines fields on the payload (iss, aud, exp, iat, sub)
9. jetstack.io FOUNDATIONS JSON Web Tokens (JWTs) - Example eyJhbGciOiJSUzI1NiIsImtpZCI6IjkyNDU0ZDAzZD M1NTg0NmE3ODYxYjBlOWVlYTFlZDY4MTJjYTZmMmUi fQ . eyJpc3MiOiJodHRwczovL2RleC5jaHJpc3RpYW4tZ2 NwLmpldHN0YWNrLm5ldCIsInN1YiI6IkNnWXlNak13 TkRnU0JtZHBkR2gxWWciLCJhdWQiOiJXWlpHUUpjRU N2VVE2anByMVI5Z05CZDAiLCJleHAiOjE1NTQ1NjA1 MjUsImlhdCI6MTU1NDQ3NDEyNSwiYXRfaGFzaCI6Ik 13OFhtLWJqaFduN0NIVG1LT0x4RGciLCJlbWFpbCI6 InNpbW9uQHN3aW5lLmRlIiwiZW1haWxfdmVyaWZpZW QiOnRydWUsImdyb3VwcyI6WyJqZXRzdGFjazpqZXRz dGFjay1kZXZzIl0sIm5hbWUiOiJDaHJpc3RpYW4gU2 ltb24ifQ . <Signature>
10. jetstack.io FOUNDATIONS JSON Web Tokens (JWTs) - Example { “alg":"RS256", "Kid":"92454d03d355846a7861b0e9eea1ed6812ca6f2e" } . { "iss":"https://dex.christian-gcp.jetstack.net", "sub":"CgYyMjMwNDgSBmdpdGh1Yg", "aud":"WZZGQJcECvUQ6jpr1R9gNBd0", "exp":1554560525, "iat":1554474125, "at_hash":"Mw8Xm-bjhWn7CHTmKOLxDg", "email":"simon@swine.de", "email_verified":true, "groups":["jetstack:jetstack-devs"], "name":"Christian Simon"} . <Signature>
11. jetstack.io FOUNDATIONS OAuth2 • Industry standard for authentication through a third party • Defines different grant types, which are different flows of handling token exchange with provider • During the token exchange a long lived refresh_token can be issued, to renew expired access_tokens (called offline access)
12. jetstack.io FOUNDATIONS OAuth2 - Naming • Resource Owner: End user that logs into an application • Client: Application that you log into • Auth provider: (e.g. Google, GitHub,...). Trusted provider that can check the identity of the resource owners • Authorization Server: Server issuing access_tokens on the resource owners’ behalf (typically run by auth provider) • Resource Server: Hosts profile information about resource owners (typically run by auth provider)
13. jetstack.io FOUNDATIONS OAuth2 - Authorization Code Grant Resource Owner Client Authorization Server Resource Server Authorization code request Consent and login Request resource with access token Respond with resource Token exchange Auth Provider
14. jetstack.io FOUNDATIONS OpenID Connect • Identity standard on top of OAuth2 (Spec 1.0) • Specifies a format for a JWT which is called id_token • Defines standard claims, which define fields that are gonna be returned in the JWT payload • Some providers implement OIDC: Google, Azure, Auth0, Okta • Dex is an open source implementation of OIDC, which can connect to other providers otherwise not supporting it • Provides a discovery mechanism for API endpoints
15. jetstack.io FOUNDATIONS OpenID Connect (OIDC) Resource Owner Kubernetes API server Authorization Server Resource Server Consent and login Request resource with ID token Respond JSON Web Token (Resource + Signature) Do discovery for metadata and public keys Response Auth Provider
16. jetstack.io FOUNDATIONS Kubernetes - Authentication & Authorization Authorization Mutating/Validating Webhooks Can they? (RBAC) Authentication ... kube-apiserver Authorization Admission Control User / Service »Who is the client?« Service accounts/Mutual TLS/Basic Auth/Token/OIDC »Is the client allowed to do this?» RBAC/Impersonation »Validation of payload» Mutation/Validation hooks, policies, quota ....
17. jetstack.io FOUNDATIONS Kubernetes - Impersonation • Impersonation allows to mimic another user in the Kubernetes API • Permission is controllable via RBAC statements • Headers for users Impersonate-User: jane.doe@example.com • Header for groups, can be repeated: Impersonate-Group: developers Impersonate-Group: admins
18. Kube-OIDC-Proxy jetstack.io
19. jetstack.io KUBE-OIDC-PROXY Authentication Authorization Mutating/Validating Webhooks Can they? (RBAC) Authentication ... kube-apiserver Authorization Admission Control kube-oidc-proxy »Who is the client?« Service account of kube-oidc-proxy »Is the client allowed to do this?» Impersonation allowed Action allowed for chosen user »Validation of payload» Mutation/Validation hooks, policies, quota .... User / Service »Who is the client?« OIDC auth
20. jetstack.io KUBE-OIDC-PROXY Authentication failed • OIDC validation failed: 401 Unauthorized • No claim username existing in the JWT: 403 Forbidden • Request to kube-oidc-proxy contains impersonation headers: 403 Forbidden
21. jetstack.io KUBE-OIDC-PROXY Reverse Proxy on Managed Kubernetes • Consistent IAM for all the providers ✓ • No Lock in ✓ • Integration with existing systems ✓ • Consistent RBAC ✓ • Fully customisable ✓
22. jetstack.io KUBE-OIDC-PROXY Reverse proxy in Go 1.12 • Websockets necessary for kubectl exec/logs • Go 1.12 release notes: # pkg/proxy/proxy.go // set up proxy handler using proxy proxyHandler := httputil.NewSingleHostReverseProxy(url) proxyHandler.Transport = p proxyHandler.ErrorHandler = p.Error if err := p.serve(proxyHandler, stopCh); err != nil { return err }
23. jetstack.io KUBE-OIDC-PROXY Testing • go test can be used for E2E tests • Using Kind we spin up real control planes • kubeadm config changes quite a bit: # pkg/e2e/e2e_test.go#L67 kubeadmConfig := `metadata: name: config networking: serviceSubnet: 10.0.0.0/16` if v.Compare(v13) >= 0 { kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1beta1 kind: ClusterConfiguration `, kubeadmConfig) } else if v.Compare(v12) < 0 { kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1alpha2 kind: MasterConfiguration `, kubeadmConfig) } else { kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1alpha3 kind: ClusterConfiguration `, kubeadmConfig) }
24. jetstack.io DEMO Components • Cert-Manager: Manages self-signed and Let’s Encrypt certificates • Contour: Ingress controller, allows E2E encryption using TLS-SNI Passthrough • Dex: Implements OIDC with connecting to GitHub • Gangway: UI for getting a valid kubeconfig • kube-oidc-proxy: For auth and reverse proxying request
25. jetstack.io DEMO Architecture • Dex runs only on GKE • Digitalocean Kubernetes and AWS EKS identical • External-DNS setups records in GCP Cloud DNS dex.k8s.at gangway-gke.k8s.at kube-oidc-proxy-gke.k8s.at gangway-eks.k8s.at kube-oidc-proxy-eks.k8s.at gangway-dok.k8s.at kube-oidc-proxy-dok.k8s.at
26. jetstack.io DEMO Terraform • Spin up infrastructure • Generate secrets (CA, Cookie, …) • Write a JSON file which is used for the manifests • Poor support for Oauth2 Providers
27. jetstack.io DEMO Jsonnet for manifests • Reuse components from Bitnami’s Prod Runtime • Provide components as reusable modules and arrange as required • Use a config.json to configure the stack individually # demo/manifests/main.jsonnet kube_oidc_proxy: kube_oidc_proxy { local this = self, base_domain:: $.base_domain, p:: $.p, metadata:: { metadata+: { namespace: $.namespace, }, }, config+: { oidc+: { issuerURL: 'https://' + $.dex_domain, clientID: $.config.gangway.client_id, }, },
28. jetstack.io DEMO Jsonnet for manifests deployment+: { spec+: { replicas: $.default_replicas, }, }, certificate: cert_manager.Certificate( $.namespace, this.name, $.cert_manager.letsencryptProd, [this.domain] ), ingressRoute: IngressRouteTLSPassthrough( $.namespace, this.name, this.domain, this.name, 443 ), }
29. Demo time jetstack.io
30. jetstack.io DEMO • Browse to https://k8s.at • Don’t forget to start the repo # Get pods $ kubectl get pods -n default NAME READY STATUS RESTARTS AGE News-84f68cc647-zzzzz 1/1 Running 0 1m # Watch the logs $ kubectl logs -n default news-84f68cc647-zzzzz # Edit config map $ kubectl edit configmap -n default hello-world
31. jetstack.io LESSONS LEARNED • Dex and CRD backend »beta« • Jsonnet feels like the right tool to write your manifests in ◦ Base32 is easy to write • Go 1.12 came out at the right time
32. jetstack.io FUTURE Ideas • We have 0.1 tagged, try it for yourself! • Try and break it! • Auditing? ◦ Dynamic audit configuration (alpha in v1.13) • Other auth methods than OIDC • Maybe forward non OIDC auth to the real control plane
33. Source code/Issue Tracker/PRs • github.com/jetstack/kube-oidc-proxy Talk at KubeCon: • Portable, Universal Single Sign-On for Your Clusters Miguel Martinez, Bitnami Tuesday, May 21 • 14:50 - 15:25 jetstack.io
34. jetstack.io KUBE-OIDC-PROXY Sequence diagram
35. jetstack.io