Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Kubernetes on bare metal: SSL

Kubernetes on bare metal: SSL

Maxim Filatov

January 24, 2020
Tweet

More Decks by Maxim Filatov

Other Decks in Technology

Transcript

  1. Server components Server components Kubelet node agent Pod runner healthchecks

    CSR Scheduler availability performance capacity Kube Proxy manage iptables TCP/UDP streams IP forward Apiserver REST operations frontend to the cluster's shared state ETCD Primary DB Cluster state CNI Container Network Interf Manage container netwo Controller Manager control loops Kubectl API client 3 . 1
  2. digitalSignature digitalSignature Purpose: asserted when the subject public key is

    used to verify digital signatures, other than signatures on certi cates (bit 5) and CRLs (bit 6), used to provide a content commitment service that protects against the signing entity falsely denying some action. In the case of later con ict, a reliable third party may determine the authenticity of the signed data. 4 . 3
  3. keyEncipherment keyEncipherment Purpose: asserted when the subject public key is

    used for enciphering private or secret keys, i.e., for key transport. For example, this bit shall be set when an RSA public key is to be used for encrypting a symmetric content-decryption key or an asymmetric private key. 4 . 4
  4. keyAgreement keyAgreement Purpose: asserted when the subject public key is

    used for . For example, when a Di e-Hellman key is to be used for key management, then this bit is set. key agreement 4 . 5
  5. serverAuth serverAuth Purpose: TLS WWW server authentication Key usage bits

    that may be consistent: , or digitalSignature keyEncipherment keyAgreement 5 . 3
  6. clientAuth clientAuth Purpose: TLS WWW client authentication Key usage bits

    that may be consistent: and/or digitalSignature keyAgreement 5 . 4
  7. Etcd Etcd ETCD cluster ETCD1 Primary DB Cluster state ETCD2

    Primary DB Cluster state ETCD3 Primary DB Cluster state 6 . 1
  8. Etcd: SSL Etcd: SSL ETCD: server to server ETCD1 Peer

    Certificate ETCD2 Peer Certificate ETCD3 Peer Certificate 6 . 2
  9. Etcd <-> Apiserver Etcd <-> Apiserver ETCD: client to server

    Apiserver ETCD Client Certificate ETCD1 Server Certificate ETCD2 Server Certificate ETCD3 Server Certificate 6 . 3
  10. Etcd <-> Apiserver Etcd <-> Apiserver ETCD: client to server

    Apiserver ETCD Client Certificate :2379 Connects to each member of etcd cluster ETCD1 Server Certificate :2379 Peer Certificate :2380 ETCD2 Server Certificate :2379 Peer Certificate :2380 ETCD3 Server Certificate :2379 Peer Certificate :2380 6 . 4
  11. Keypairs Keypairs Etcd peer CA => etcd peer Usages: server

    auth client auth Etcd server CA => etcd server Usages: server auth Etcd server CA => etcd client Usages: client auth 6 . 6
  12. So far so good So far so good CA: 2

    Etcd peer Etcd server Keypairs: 3+ etcd_peer etcd_server etcd_client 7 . 1
  13. Keypairs Keypairs Cluster signing CA => apiserver Usages: server auth

    Cluster signing CA => controller_manager Usages: server auth Cluster signing CA => scheduler Usages: server auth 8 . 3
  14. So far so good So far so good CA: 3

    Etcd peer Etcd server Kubernetes cluster signing Keypairs: 6+ etcd_peer etcd_server etcd_client apiserver controller_manager scheduler 9 . 1
  15. CSR CSR $ kubectl get csr NAME AGE REQUESTOR CONDITION

    csr-22jvq 1m system:node:kube01 Pending 10 . 4
  16. Guts Guts $ kubectl get csr csr-22jvq -o yaml apiVersion:

    certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-22jvq spec: groups: - system:nodes - system:authenticated request: LS0tL... usages: - digital signature - key encipherment - server auth username: system:node:kube01 10 . 5
  17. Guts Guts $ kubectl get csr csr-22jvq -o jsonpath='{.spec.request}' |\

    base64 -D | openssl req -noout -text Certificate Request: Data: Version: 0 (0x0) Subject: O=system:nodes, CN=system:node:kube01 ... Attributes: Requested Extensions: X509v3 Subject Alternative Name: DNS:kube01, IP Address:192.168.0.1 10 . 6
  18. Result Result $ kubectl get csr csr-22jvq -o \ jsonpath='{.status.certificate}'

    | \ base64 -D|openssl x509 -noout -text Certificate: Issuer: C=RU, ST=Moscow, L=Moscow, O=Kubernetes, OU=CA, CN=Kubern Subject: O=system:nodes, CN=system:node:kube01 ... X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Alternative Name: DNS:kube01, IP Address:192.168.0.1 10 . 8
  19. So far so good So far so good CA: 3

    Etcd peer Etcd server Kubernetes cluster signing Keypairs: 9+ … kubelet_client kubelet-current * num of kubelets * 2 11 . 1
  20. Service Accounts Service Accounts $ kubectl get sa NAME SECRETS

    AGE default 1 1h $ kubectl get secret NAME default-token-78jt9 TYPE DATA AGE kubernetes.io/service-account-token 3 1h 12 . 1
  21. Service Accounts: secret Service Accounts: secret anatomy anatomy $ kubectl

    get secrets default-token-78jt9 -o yaml | \ yq r - 'data' ca.crt: LS0tLS1CRUdJTiB... namespace: ZGVmYXVsdA== token: ZXlKaGJHY2lPaUpT... 12 . 2
  22. Guts Guts $ kubectl get secrets default-token-78jt9 \ -o jsonpath='{.data.token}'

    | base64 -D eyJhbGciOiJSUzI1NiIsImtpZCI6ImRIQm51UGV0TkNaZV9KV0k1SjRtSmlfeS1BU 12 . 3
  23. Apiserver & Controller Apiserver & Controller Manager Manager apiserver controller-manager

    --service-account-key-file=/path/to/service_account_ca.pem --service-account-private-key-file=/path/to/service_account_ca-k 12 . 6
  24. So far so good So far so good CA: 4

    Etcd peer Etcd server Kubernetes cluster signing Service accounts Keypairs: 9+ … 13 . 1
  25. Aggregation Layer Aggregation Layer https://kubernetes.io/docs/tasks/access- kubernetes-api/con gure-aggregation-layer/ $ kubectl get

    apiservices.apiregistration.k8s.io NAME SERVICE v1. Local v1.apps Local v1.authentication.k8s.io Local ... v1beta1.external.metrics.k8s.io monitoring/prometheus-adap v1beta1.metrics.k8s.io kube-system/metrics-server v1beta1.webhook.cert-manager.io kube-system/cert-manager-w ... 14 . 1
  26. Service Service $ kubectl get svc -n kube-system cert-manager-webhook NAME

    TYPE CLUSTER-IP EXTERNAL-IP PO cert-manager-webhook ClusterIP 10.222.206.2 none 4 14 . 2
  27. extension-apiserver- extension-apiserver- authentication configmap authentication configmap $ kubectl get cm

    -n kube-system extension-apiserver-authenticatio data: client-ca-file: |- -----BEGIN CERTIFICATE----- ... requestheader-allowed-names: '["front-proxy-client"]' requestheader-client-ca-file: |- -----BEGIN CERTIFICATE----- ... requestheader-extra-headers-prefix: '["X-Remote-Extra-"]' requestheader-group-headers: '["X-Remote-Group"]' requestheader-username-headers: '["X-Remote-User"]' 14 . 6
  28. Flow Flow 1. Authentication | Authorization User Apiserver 2. Send

    data to aggregated apiserver Apiserver proxy_client_keypair --requestheader-username-headers --requestheader-group-headers --requestheader-extra-headers-prefix Aggregated apiserver kube-system/extension-apiserver-authentication cm clus 14 . 7
  29. So far so good So far so good CA: 5

    Etcd peer Etcd server Kubernetes cluster signing Service accounts Requestheader client Keypairs: 10+ … Requestheader client CA => proxy_client 15 . 1
  30. Rotation: Etcd peer Rotation: Etcd peer Reissue CA Issue new

    keypairs Deploy to etcd nodes Restart instances 16 . 2
  31. Rotation: Etcd server Rotation: Etcd server Reissue CA Issue new

    etcd server(s) keypair(s) Deploy to etcd nodes Restart instances Issue new etcd client keypair Deploy to apiserver nodes Restart instances 16 . 3
  32. Rotation: Kubernetes cluster Rotation: Kubernetes cluster signing (apiserver) signing (apiserver)

    Reissue CA Issue new apiserver(s) keypair(s) Deploy to apiserver nodes Restart instances 16 . 4
  33. Rotation: Kubernetes cluster Rotation: Kubernetes cluster signing (pods and nodes)

    signing (pods and nodes) Delete all serviceaccount token containing secrets Restart all corresponding pods Delete kubelet requested keypairs (kubelet-*- current.pem) Restart all kubelets Twice approve all new CSRs kubectl certificate approve \ $(kubectl get csr|grep Pending|awk '{print $1}') 16 . 5
  34. Rotation: Service accounts Rotation: Service accounts Reissue CA Deploy to

    apiserver nodes Restart instances Delete all serviceaccount token containing secrets Restart all corresponding pods 16 . 6
  35. Rotation: Requestheader client Rotation: Requestheader client Reissue CA Issue new

    proxy_client keypair Deploy to apiserver nodes Restart instances Restart all corresponding pods 16 . 7