Kubernetes on bare metal: SSL

Video: https://youtu.be/5YF2IEq68sc?t=189

Transcript

1. K8S on bare metal: SSL K8S on bare metal: SSL 1
2. 2 . 1
3. Server components Server components Kubelet node agent Pod runner healthchecks CSR Scheduler availability performance capacity Kube Proxy manage iptables TCP/UDP streams IP forward Apiserver REST operations frontend to the cluster's shared state ETCD Primary DB Cluster state CNI Container Network Interf Manage container netwo Controller Manager control loops Kubectl API client 3 . 1
4. Key usages Key usages 4 . 1
5. digitalSignature contentCommitment keyEncipherment dataEncipherment keyAgreement keyCertSign cRLSign encipherOnly decipherOnly RFC5280, section 4.2.1.3 RFC5280, section 4.2.1.3 4 . 2
6. digitalSignature digitalSignature Purpose: asserted when the subject public key is used to verify digital signatures, other than signatures on certi cates (bit 5) and CRLs (bit 6), used to provide a content commitment service that protects against the signing entity falsely denying some action. In the case of later con ict, a reliable third party may determine the authenticity of the signed data. 4 . 3
7. keyEncipherment keyEncipherment Purpose: asserted when the subject public key is used for enciphering private or secret keys, i.e., for key transport. For example, this bit shall be set when an RSA public key is to be used for encrypting a symmetric content-decryption key or an asymmetric private key. 4 . 4
8. keyAgreement keyAgreement Purpose: asserted when the subject public key is used for . For example, when a Di e-Hellman key is to be used for key management, then this bit is set. key agreement 4 . 5
9. Extended key usages Extended key usages 5 . 1
10. serverAuth clientAuth codeSigning emailProtection timeStamping OCSPSigning RFC5280, section 4.2.1.12 RFC5280, section 4.2.1.12 5 . 2
11. serverAuth serverAuth Purpose: TLS WWW server authentication Key usage bits that may be consistent: , or digitalSignature keyEncipherment keyAgreement 5 . 3
12. clientAuth clientAuth Purpose: TLS WWW client authentication Key usage bits that may be consistent: and/or digitalSignature keyAgreement 5 . 4
13. Etcd Etcd ETCD cluster ETCD1 Primary DB Cluster state ETCD2 Primary DB Cluster state ETCD3 Primary DB Cluster state 6 . 1
14. Etcd: SSL Etcd: SSL ETCD: server to server ETCD1 Peer Certificate ETCD2 Peer Certificate ETCD3 Peer Certificate 6 . 2
15. Etcd <-> Apiserver Etcd <-> Apiserver ETCD: client to server Apiserver ETCD Client Certificate ETCD1 Server Certificate ETCD2 Server Certificate ETCD3 Server Certificate 6 . 3
16. Etcd <-> Apiserver Etcd <-> Apiserver ETCD: client to server Apiserver ETCD Client Certificate :2379 Connects to each member of etcd cluster ETCD1 Server Certificate :2379 Peer Certificate :2380 ETCD2 Server Certificate :2379 Peer Certificate :2380 ETCD3 Server Certificate :2379 Peer Certificate :2380 6 . 4
17. CA CA Etcd peer CA Etcd server CA 6 . 5
18. Keypairs Keypairs Etcd peer CA => etcd peer Usages: server auth client auth Etcd server CA => etcd server Usages: server auth Etcd server CA => etcd client Usages: client auth 6 . 6
19. Apiserver Apiserver --etcd-cafile=/path/to/ca-etcd_server.pem --etcd-certfile=/path/to/etcd_client.pem --etcd-keyfile=/path/to/etcd_client-key.pem 6 . 7
20. Etcd servers Etcd servers -client-cert-auth=true -peer-client-cert-auth=true -cert-file=/path/to/etcd_server.pem -key-file=/path/to/etcd_server-key.pem -peer-cert-file=/path/to/etcd_peer.pem -peer-key-file=/path/to/etcd_peer-key.pem -peer-trusted-ca-file=/path/to/ca-etcd_peer.pem -trusted-ca-file=/path/to/ca-etcd_server.pem 6 . 8
21. So far so good So far so good CA: 2 Etcd peer Etcd server Keypairs: 3+ etcd_peer etcd_server etcd_client 7 . 1
22. Master node Master node Master Node Scheduler Apiserver Controller Manager 8 . 1
23. CA CA Cluster Signing CA 8 . 2
24. Keypairs Keypairs Cluster signing CA => apiserver Usages: server auth Cluster signing CA => controller_manager Usages: server auth Cluster signing CA => scheduler Usages: server auth 8 . 3
25. Apiserver Apiserver --client-ca-file=/path/to/ca-cluster_signing.pem --tls-cert-file=/path/to/apiserver.pem --tls-private-key-file=/path/to/apiserver-key.pem --secure-port=8443 8 . 4
26. Controller Manager Controller Manager --cluster-signing-cert-file=/path/to/ca-cluster_signing.pem --cluster-signing-key-file=/path/to/ca-cluster_signing-key.pem --tls-cert-file=/path/to/controller_manager.pem --tls-private-key-file=/path/to/controller_manager-key.pem 8 . 5
27. Scheduler Scheduler --tls-cert-file=/path/to/scheduler.pem --tls-private-key-file=/path/to/scheduler-key.pem 8 . 6
28. So far so good So far so good CA: 3 Etcd peer Etcd server Kubernetes cluster signing Keypairs: 6+ etcd_peer etcd_server etcd_client apiserver controller_manager scheduler 9 . 1
29. Kubelets Kubelets 10 . 1
30. Keypairs Keypairs Cluster signing CA => kubelet_client Usages: client auth 10 . 2
31. Apiserver Apiserver --kubelet-client-certificate=/path/to/kubelet_client.pem --kubelet-client-key=/path/to/kubelet_client-key.pem --kubelet-https=true 10 . 3
32. CSR CSR $ kubectl get csr NAME AGE REQUESTOR CONDITION csr-22jvq 1m system:node:kube01 Pending 10 . 4
33. Guts Guts $ kubectl get csr csr-22jvq -o yaml apiVersion: certificates.k8s.io/v1beta1 kind: CertificateSigningRequest metadata: name: csr-22jvq spec: groups: - system:nodes - system:authenticated request: LS0tL... usages: - digital signature - key encipherment - server auth username: system:node:kube01 10 . 5
34. Guts Guts $ kubectl get csr csr-22jvq -o jsonpath='{.spec.request}' |\ base64 -D | openssl req -noout -text Certificate Request: Data: Version: 0 (0x0) Subject: O=system:nodes, CN=system:node:kube01 ... Attributes: Requested Extensions: X509v3 Subject Alternative Name: DNS:kube01, IP Address:192.168.0.1 10 . 6
35. Approve Approve $ kubectl certificate approve csr-22jvq certificatesigningrequest.certificates.k8s.io/csr-22jvq approved 10 . 7
36. Result Result $ kubectl get csr csr-22jvq -o \ jsonpath='{.status.certificate}' | \ base64 -D|openssl x509 -noout -text Certificate: Issuer: C=RU, ST=Moscow, L=Moscow, O=Kubernetes, OU=CA, CN=Kubern Subject: O=system:nodes, CN=system:node:kube01 ... X509v3 extensions: X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Alternative Name: DNS:kube01, IP Address:192.168.0.1 10 . 8
37. So far so good So far so good CA: 3 Etcd peer Etcd server Kubernetes cluster signing Keypairs: 9+ … kubelet_client kubelet-current * num of kubelets * 2 11 . 1
38. Service Accounts Service Accounts $ kubectl get sa NAME SECRETS AGE default 1 1h $ kubectl get secret NAME default-token-78jt9 TYPE DATA AGE kubernetes.io/service-account-token 3 1h 12 . 1
39. Service Accounts: secret Service Accounts: secret anatomy anatomy $ kubectl get secrets default-token-78jt9 -o yaml | \ yq r - 'data' ca.crt: LS0tLS1CRUdJTiB... namespace: ZGVmYXVsdA== token: ZXlKaGJHY2lPaUpT... 12 . 2
40. Guts Guts $ kubectl get secrets default-token-78jt9 \ -o jsonpath='{.data.token}' | base64 -D eyJhbGciOiJSUzI1NiIsImtpZCI6ImRIQm51UGV0TkNaZV9KV0k1SjRtSmlfeS1BU 12 . 3
41. Guts Guts Image courtesy of jwt.io 12 . 4
42. Service Accounts: CA Service Accounts: CA Service Accounts CA 12 . 5
43. Apiserver & Controller Apiserver & Controller Manager Manager apiserver controller-manager --service-account-key-file=/path/to/service_account_ca.pem --service-account-private-key-file=/path/to/service_account_ca-k 12 . 6
44. So far so good So far so good CA: 4 Etcd peer Etcd server Kubernetes cluster signing Service accounts Keypairs: 9+ … 13 . 1
45. Aggregation Layer Aggregation Layer https://kubernetes.io/docs/tasks/access- kubernetes-api/con gure-aggregation-layer/ $ kubectl get apiservices.apiregistration.k8s.io NAME SERVICE v1. Local v1.apps Local v1.authentication.k8s.io Local ... v1beta1.external.metrics.k8s.io monitoring/prometheus-adap v1beta1.metrics.k8s.io kube-system/metrics-server v1beta1.webhook.cert-manager.io kube-system/cert-manager-w ... 14 . 1
46. Service Service $ kubectl get svc -n kube-system cert-manager-webhook NAME TYPE CLUSTER-IP EXTERNAL-IP PO cert-manager-webhook ClusterIP 10.222.206.2 none 4 14 . 2
47. CA CA Requestheader 14 . 3
48. Keypairs Keypairs proxy_client Usages: client auth 14 . 4
49. Apiserver Apiserver --proxy-client-cert-file=/path/to/front_proxy_client.pem --proxy-client-key-file=/path/to/front_proxy_client-key.pem --requestheader-allowed-names=front-proxy-client --requestheader-client-ca-file=/path/to/ca-requestheader.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User 14 . 5
50. extension-apiserver- extension-apiserver- authentication configmap authentication configmap $ kubectl get cm -n kube-system extension-apiserver-authenticatio data: client-ca-file: |- -----BEGIN CERTIFICATE----- ... requestheader-allowed-names: '["front-proxy-client"]' requestheader-client-ca-file: |- -----BEGIN CERTIFICATE----- ... requestheader-extra-headers-prefix: '["X-Remote-Extra-"]' requestheader-group-headers: '["X-Remote-Group"]' requestheader-username-headers: '["X-Remote-User"]' 14 . 6
51. Flow Flow 1. Authentication | Authorization User Apiserver 2. Send data to aggregated apiserver Apiserver proxy_client_keypair --requestheader-username-headers --requestheader-group-headers --requestheader-extra-headers-prefix Aggregated apiserver kube-system/extension-apiserver-authentication cm clus 14 . 7
52. So far so good So far so good CA: 5 Etcd peer Etcd server Kubernetes cluster signing Service accounts Requestheader client Keypairs: 10+ … Requestheader client CA => proxy_client 15 . 1
53. Rotation Rotation Etcd peer Etcd server Kubernetes cluster signing Service accounts Requestheader client 16 . 1
54. Rotation: Etcd peer Rotation: Etcd peer Reissue CA Issue new keypairs Deploy to etcd nodes Restart instances 16 . 2
55. Rotation: Etcd server Rotation: Etcd server Reissue CA Issue new etcd server(s) keypair(s) Deploy to etcd nodes Restart instances Issue new etcd client keypair Deploy to apiserver nodes Restart instances 16 . 3
56. Rotation: Kubernetes cluster Rotation: Kubernetes cluster signing (apiserver) signing (apiserver) Reissue CA Issue new apiserver(s) keypair(s) Deploy to apiserver nodes Restart instances 16 . 4
57. Rotation: Kubernetes cluster Rotation: Kubernetes cluster signing (pods and nodes) signing (pods and nodes) Delete all serviceaccount token containing secrets Restart all corresponding pods Delete kubelet requested keypairs (kubelet-*- current.pem) Restart all kubelets Twice approve all new CSRs kubectl certificate approve \ $(kubectl get csr|grep Pending|awk '{print $1}') 16 . 5
58. Rotation: Service accounts Rotation: Service accounts Reissue CA Deploy to apiserver nodes Restart instances Delete all serviceaccount token containing secrets Restart all corresponding pods 16 . 6
59. Rotation: Requestheader client Rotation: Requestheader client Reissue CA Issue new proxy_client keypair Deploy to apiserver nodes Restart instances Restart all corresponding pods 16 . 7
60. Questions? Questions? 17 . 1