»Consistent user authentication in multi-cloud hosted Kubernetes clusters«

As hosted Kubernetes solutions mature, it becomes ever more compelling to operate clusters across multiple cloud providers. A general point of friction can often be the differences in how you are able to authenticate to those clusters. Cloud providers tend to integrate their own proprietary solutions and hosted control planes lack the flexibility to use authentication providers and audit sinks.

During this talk I will show how a reverse proxy in front of the Kubernetes API can implement uniform OIDC authentication across hosted Kubernetes solutions.

Transcript

1. jetstack.io
   Kube-OIDC-Proxy
   Consistent user
   authentication across
   clusters hosted in
   multiple clouds
   Presented by
   Christian Simon
   

   View Slide

2. jetstack.io
   WHO AM I?
   Christian Simon
   ● Using Kubernetes for 4+ years
   ● Started kube-lego
   ● Working on Tarmak
   simonswine
   simonswine
   

   View Slide

3. jetstack.io
   WHO IS JOSH?
   Josh Van Leeuwen
   ● Joined as intern during his final year of university
   ● Now working full time as Solutions Engineer
   ● Developer of kube-oidc-proxy
   joshvanl
   

   View Slide

4. jetstack.io
   WHO ARE JETSTACK?
   Jetstack
   ● Cert-Manager, Tarmak, …
   ● Professional services: Consulting, Training, ...
   ● UK-based, remote friendly company
   ● Nice place to work: We are hiring!
   jetstackhq
   jetstack
   jetstackhq
   

   View Slide

5. jetstack.io
   AGENDA
   ● Introduction
   ● Foundations
   ○ OAuth2
   ○ JWT
   ○ OpenID Connect (OIDC)
   ○ Authn/z in
   ● Kube-OIDC-Proxy
   ● Demo
   

   View Slide

6. jetstack.io
   INTRODUCTION
   Authentication on Managed Kubernetes
   ● Bespoke IAM
   ● Lock-in
   ● No integration with existing systems
   (on-premises etc.)
   ● Inconsistent RBAC across multiple
   providers/accounts
   ● Lack of customisation
   

   View Slide

7. Authentication Foundations
   jetstack.io
   

   View Slide

8. jetstack.io
   FOUNDATIONS
   JSON Web Tokens (JWTs)
   ● Standardised signed token, to assert certain metadata (RFC7519)
   ● Defined on top of JSON Web Signature (RFC7515) and JSON
   Web Encryption (RFC7516)
   ● JWS: Consists of three base64 encoded parts separated by a dot:
   ..
   ● JWT defines fields on the payload (iss, aud, exp, iat, sub)
   

   View Slide

9. jetstack.io
   FOUNDATIONS
   JSON Web Tokens (JWTs) - Example
   eyJhbGciOiJSUzI1NiIsImtpZCI6IjkyNDU0ZDAzZD
   M1NTg0NmE3ODYxYjBlOWVlYTFlZDY4MTJjYTZmMmUi
   fQ
   .
   eyJpc3MiOiJodHRwczovL2RleC5jaHJpc3RpYW4tZ2
   NwLmpldHN0YWNrLm5ldCIsInN1YiI6IkNnWXlNak13
   TkRnU0JtZHBkR2gxWWciLCJhdWQiOiJXWlpHUUpjRU
   N2VVE2anByMVI5Z05CZDAiLCJleHAiOjE1NTQ1NjA1
   MjUsImlhdCI6MTU1NDQ3NDEyNSwiYXRfaGFzaCI6Ik
   13OFhtLWJqaFduN0NIVG1LT0x4RGciLCJlbWFpbCI6
   InNpbW9uQHN3aW5lLmRlIiwiZW1haWxfdmVyaWZpZW
   QiOnRydWUsImdyb3VwcyI6WyJqZXRzdGFjazpqZXRz
   dGFjay1kZXZzIl0sIm5hbWUiOiJDaHJpc3RpYW4gU2
   ltb24ifQ
   .
   
   

   View Slide

10. jetstack.io
    FOUNDATIONS
    JSON Web Tokens (JWTs) - Example
    { “alg":"RS256",
    "Kid":"92454d03d355846a7861b0e9eea1ed6812ca6f2e"
    }
    .
    { "iss":"https://dex.christian-gcp.jetstack.net",
    "sub":"CgYyMjMwNDgSBmdpdGh1Yg",
    "aud":"WZZGQJcECvUQ6jpr1R9gNBd0",
    "exp":1554560525,
    "iat":1554474125,
    "at_hash":"Mw8Xm-bjhWn7CHTmKOLxDg",
    "email":"simon@swine.de",
    "email_verified":true,
    "groups":["jetstack:jetstack-devs"],
    "name":"Christian Simon"}
    .
    
    

    View Slide

11. jetstack.io
    FOUNDATIONS
    OAuth2
    ● Industry standard for authentication
    through a third party
    ● Defines different grant types, which are
    different flows of handling token
    exchange with provider
    ● During the token exchange a long lived
    refresh_token can be issued, to
    renew expired access_tokens
    (called offline access)
    

    View Slide

12. jetstack.io
    FOUNDATIONS
    OAuth2 - Naming
    ● Resource Owner: End user that logs into an application
    ● Client: Application that you log into
    ● Auth provider: (e.g. Google, GitHub,...). Trusted provider that can
    check the identity of the resource owners
    ● Authorization Server: Server issuing access_tokens on the resource
    owners’ behalf (typically run by auth provider)
    ● Resource Server: Hosts profile information about resource owners
    (typically run by auth provider)
    

    View Slide

13. jetstack.io
    FOUNDATIONS
    OAuth2 - Authorization Code Grant
    Resource Owner Client
    Authorization
    Server
    Resource Server
    Authorization code request
    Consent and login
    Request resource
    with access token
    Respond with resource
    Token exchange
    Auth Provider
    

    View Slide

14. jetstack.io
    FOUNDATIONS
    OpenID Connect
    ● Identity standard on top of OAuth2 (Spec 1.0)
    ● Specifies a format for a JWT which is called id_token
    ● Defines standard claims, which define fields that are gonna be returned in
    the JWT payload
    ● Some providers implement OIDC: Google, Azure, Auth0, Okta
    ● Dex is an open source implementation of OIDC, which can connect to
    other providers otherwise not supporting it
    ● Provides a discovery mechanism for API endpoints
    

    View Slide

15. jetstack.io
    FOUNDATIONS
    OpenID Connect (OIDC)
    Resource Owner
    Kubernetes API
    server
    Authorization
    Server
    Resource Server
    Consent and login
    Request resource with ID
    token
    Respond JSON Web Token
    (Resource + Signature)
    Do discovery for metadata and
    public keys
    Response
    Auth Provider
    

    View Slide

16. jetstack.io
    FOUNDATIONS
    Kubernetes - Authentication & Authorization
    Authorization
    Mutating/Validating
    Webhooks
    Can they?
    (RBAC)
    Authentication
    ...
    kube-apiserver
    Authorization
    Admission Control
    User / Service »Who is the client?«
    Service accounts/Mutual
    TLS/Basic Auth/Token/OIDC
    »Is the client allowed to do
    this?»
    RBAC/Impersonation
    »Validation of payload»
    Mutation/Validation hooks,
    policies, quota ....
    

    View Slide

17. jetstack.io
    FOUNDATIONS
    Kubernetes - Impersonation
    ● Impersonation allows to mimic another user in the Kubernetes API
    ● Permission is controllable via RBAC statements
    ● Headers for users
    Impersonate-User: jane.doe@example.com
    ● Header for groups, can be repeated:
    Impersonate-Group: developers
    Impersonate-Group: admins
    

    View Slide

18. Kube-OIDC-Proxy
    jetstack.io
    

    View Slide

19. jetstack.io
    KUBE-OIDC-PROXY
    Authentication
    Authorization
    Mutating/Validating
    Webhooks
    Can they?
    (RBAC)
    Authentication
    ...
    kube-apiserver
    Authorization
    Admission Control
    kube-oidc-proxy
    »Who is the client?«
    Service account of
    kube-oidc-proxy
    »Is the client allowed to do this?»
    Impersonation allowed
    Action allowed for chosen user
    »Validation of payload»
    Mutation/Validation hooks,
    policies, quota ....
    User / Service
    »Who is the
    client?«
    OIDC auth
    

    View Slide

20. jetstack.io
    KUBE-OIDC-PROXY
    Authentication failed
    ● OIDC validation failed:
    401 Unauthorized
    ● No claim username existing in the JWT:
    403 Forbidden
    ● Request to kube-oidc-proxy contains impersonation headers:
    403 Forbidden
    

    View Slide

21. jetstack.io
    KUBE-OIDC-PROXY
    Reverse Proxy on Managed Kubernetes
    ● Consistent IAM for all the providers ✓
    ● No Lock in ✓
    ● Integration with existing systems ✓
    ● Consistent RBAC ✓
    ● Fully customisable ✓
    

    View Slide

22. jetstack.io
    KUBE-OIDC-PROXY
    Reverse proxy in Go 1.12
    ● Websockets necessary for kubectl exec/logs
    ● Go 1.12 release notes:
    # pkg/proxy/proxy.go
    // set up proxy handler using proxy
    proxyHandler := httputil.NewSingleHostReverseProxy(url)
    proxyHandler.Transport = p
    proxyHandler.ErrorHandler = p.Error
    if err := p.serve(proxyHandler, stopCh); err != nil {
    return err
    }
    

    View Slide

23. jetstack.io
    KUBE-OIDC-PROXY
    Testing
    ● go test can be
    used for E2E tests
    ● Using Kind we spin
    up real control
    planes
    ● kubeadm config
    changes quite a
    bit:
    # pkg/e2e/e2e_test.go#L67
    kubeadmConfig := `metadata:
    name: config
    networking:
    serviceSubnet: 10.0.0.0/16`
    if v.Compare(v13) >= 0 {
    kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1beta1
    kind: ClusterConfiguration
    `, kubeadmConfig)
    } else if v.Compare(v12) < 0 {
    kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1alpha2
    kind: MasterConfiguration
    `, kubeadmConfig)
    } else {
    kubeadmConfig = fmt.Sprint(`apiVersion: kubeadm.k8s.io/v1alpha3
    kind: ClusterConfiguration
    `, kubeadmConfig)
    }
    

    View Slide

24. jetstack.io
    DEMO
    Components
    ● Cert-Manager: Manages self-signed and Let’s Encrypt certificates
    ● Contour: Ingress controller, allows E2E encryption using TLS-SNI
    Passthrough
    ● Dex: Implements OIDC with connecting to GitHub
    ● Gangway: UI for getting a valid kubeconfig
    ● kube-oidc-proxy: For auth and reverse proxying request
    

    View Slide

25. jetstack.io
    DEMO
    Architecture
    ● Dex runs only on GKE
    ● Digitalocean Kubernetes and
    AWS EKS identical
    ● External-DNS setups records in
    GCP Cloud DNS
    dex.k8s.at
    gangway-gke.k8s.at
    kube-oidc-proxy-gke.k8s.at
    gangway-eks.k8s.at
    kube-oidc-proxy-eks.k8s.at
    gangway-dok.k8s.at
    kube-oidc-proxy-dok.k8s.at
    

    View Slide

26. jetstack.io
    DEMO
    Terraform
    ● Spin up infrastructure
    ● Generate secrets (CA, Cookie, …)
    ● Write a JSON file which is used for the manifests
    ● Poor support for Oauth2 Providers
    

    View Slide

27. jetstack.io
    DEMO
    Jsonnet for manifests
    ● Reuse components from
    Bitnami’s Prod Runtime
    ● Provide components as
    reusable modules and
    arrange as required
    ● Use a config.json to configure
    the stack individually
    # demo/manifests/main.jsonnet
    kube_oidc_proxy: kube_oidc_proxy {
    local this = self,
    base_domain:: $.base_domain,
    p:: $.p,
    metadata:: {
    metadata+: {
    namespace: $.namespace,
    },
    },
    config+: {
    oidc+: {
    issuerURL: 'https://' + $.dex_domain,
    clientID: $.config.gangway.client_id,
    },
    },
    

    View Slide

28. jetstack.io
    DEMO
    Jsonnet for manifests
    deployment+: {
    spec+: {
    replicas: $.default_replicas,
    },
    },
    certificate: cert_manager.Certificate(
    $.namespace,
    this.name,
    $.cert_manager.letsencryptProd,
    [this.domain]
    ),
    ingressRoute: IngressRouteTLSPassthrough(
    $.namespace,
    this.name,
    this.domain,
    this.name,
    443
    ),
    }
    

    View Slide

29. Demo time
    jetstack.io
    

    View Slide

30. jetstack.io
    DEMO
    ● Browse to https://k8s.at
    ● Don’t forget to start the repo
    # Get pods
    $ kubectl get pods -n default
    NAME READY STATUS RESTARTS AGE
    News-84f68cc647-zzzzz 1/1 Running 0 1m
    # Watch the logs
    $ kubectl logs -n default news-84f68cc647-zzzzz
    # Edit config map
    $ kubectl edit configmap -n default hello-world
    

    View Slide

31. jetstack.io
    LESSONS LEARNED
    ● Dex and CRD backend »beta«
    ● Jsonnet feels like the right tool to write your manifests in
    ○ Base32 is easy to write
    ● Go 1.12 came out at the right time
    

    View Slide

32. jetstack.io
    FUTURE
    Ideas
    ● We have 0.1 tagged, try it for yourself!
    ● Try and break it!
    ● Auditing?
    ○ Dynamic audit configuration (alpha in v1.13)
    ● Other auth methods than OIDC
    ● Maybe forward non OIDC auth to the real control plane
    

    View Slide

33. Source code/Issue Tracker/PRs
    ● github.com/jetstack/kube-oidc-proxy
    Talk at KubeCon:
    ● Portable, Universal Single Sign-On
    for Your Clusters
    Miguel Martinez, Bitnami
    Tuesday, May 21 • 14:50 - 15:25
    jetstack.io
    

    View Slide

34. jetstack.io
    KUBE-OIDC-PROXY
    Sequence diagram
    

    View Slide

35. jetstack.io
    

    View Slide