Slide 1
Slide 1 text
K8S on bare metal: SSL
K8S on bare metal: SSL
1
Slide 2
Slide 2 text
2 . 1
Slide 3
Slide 3 text
Server components
Server components
Kubelet
node agent
Pod runner
healthchecks
CSR
Scheduler
availability
performance
capacity
Kube Proxy
manage iptables
TCP/UDP streams
IP forward
Apiserver
REST operations
frontend to the
cluster's shared state
ETCD
Primary DB
Cluster state
CNI
Container Network Interf
Manage container netwo
Controller Manager
control loops
Kubectl
API client
3 . 1
Slide 4
Slide 4 text
Key usages
Key usages
4 . 1
Slide 5
Slide 5 text
digitalSignature
contentCommitment
keyEncipherment
dataEncipherment
keyAgreement
keyCertSign
cRLSign
encipherOnly
decipherOnly
RFC5280, section 4.2.1.3
RFC5280, section 4.2.1.3
4 . 2
Slide 6
Slide 6 text
digitalSignature
digitalSignature
Purpose: asserted when the subject public key is
used to verify digital signatures, other than
signatures on certi cates (bit 5) and CRLs (bit 6),
used to provide a content commitment service that
protects against the signing entity falsely denying
some action. In the case of later con ict, a reliable
third party may determine the authenticity of the
signed data.
4 . 3
Slide 7
Slide 7 text
keyEncipherment
keyEncipherment
Purpose: asserted when the subject public key is
used for enciphering private or secret keys, i.e., for
key transport. For example, this bit shall be set
when an RSA public key is to be used for encrypting
a symmetric content-decryption key or an
asymmetric private key.
4 . 4
Slide 8
Slide 8 text
keyAgreement
keyAgreement
Purpose: asserted when the subject public key is
used for . For example, when a
Di e-Hellman key is to be used for key
management, then this bit is set.
key agreement
4 . 5
Slide 9
Slide 9 text
Extended key usages
Extended key usages
5 . 1
Slide 10
Slide 10 text
serverAuth
clientAuth
codeSigning
emailProtection
timeStamping
OCSPSigning
RFC5280, section 4.2.1.12
RFC5280, section 4.2.1.12
5 . 2
Slide 11
Slide 11 text
serverAuth
serverAuth
Purpose: TLS WWW server authentication
Key usage bits that may be consistent:
, or
digitalSignature keyEncipherment
keyAgreement
5 . 3
Slide 12
Slide 12 text
clientAuth
clientAuth
Purpose: TLS WWW client authentication
Key usage bits that may be consistent:
and/or
digitalSignature keyAgreement
5 . 4
Slide 13
Slide 13 text
Etcd
Etcd
ETCD cluster
ETCD1
Primary DB
Cluster state
ETCD2
Primary DB
Cluster state
ETCD3
Primary DB
Cluster state
6 . 1
Slide 14
Slide 14 text
Etcd: SSL
Etcd: SSL
ETCD: server to server
ETCD1
Peer Certificate
ETCD2
Peer Certificate ETCD3
Peer Certificate
6 . 2
Slide 15
Slide 15 text
Etcd <-> Apiserver
Etcd <-> Apiserver
ETCD: client to server
Apiserver
ETCD Client Certificate
ETCD1
Server Certificate
ETCD2
Server Certificate
ETCD3
Server Certificate
6 . 3
Slide 16
Slide 16 text
Etcd <-> Apiserver
Etcd <-> Apiserver
ETCD: client to server
Apiserver
ETCD Client Certificate :2379
Connects to each member of etcd cluster
ETCD1
Server Certificate :2379
Peer Certificate :2380
ETCD2
Server Certificate :2379
Peer Certificate :2380
ETCD3
Server Certificate :2379
Peer Certificate :2380
6 . 4
Slide 17
Slide 17 text
CA
CA
Etcd peer CA
Etcd server CA
6 . 5
Slide 18
Slide 18 text
Keypairs
Keypairs
Etcd peer CA => etcd peer
Usages:
server auth
client auth
Etcd server CA => etcd server
Usages:
server auth
Etcd server CA => etcd client
Usages:
client auth
6 . 6
Slide 19
Slide 19 text
Apiserver
Apiserver
--etcd-cafile=/path/to/ca-etcd_server.pem
--etcd-certfile=/path/to/etcd_client.pem
--etcd-keyfile=/path/to/etcd_client-key.pem
6 . 7
Slide 20
Slide 20 text
Etcd servers
Etcd servers
-client-cert-auth=true
-peer-client-cert-auth=true
-cert-file=/path/to/etcd_server.pem
-key-file=/path/to/etcd_server-key.pem
-peer-cert-file=/path/to/etcd_peer.pem
-peer-key-file=/path/to/etcd_peer-key.pem
-peer-trusted-ca-file=/path/to/ca-etcd_peer.pem
-trusted-ca-file=/path/to/ca-etcd_server.pem
6 . 8
Slide 21
Slide 21 text
So far so good
So far so good
CA: 2
Etcd peer
Etcd server
Keypairs: 3+
etcd_peer
etcd_server
etcd_client
7 . 1
Slide 22
Slide 22 text
Master node
Master node
Master Node
Scheduler
Apiserver
Controller Manager
8 . 1
Slide 23
Slide 23 text
CA
CA
Cluster Signing CA
8 . 2
Slide 24
Slide 24 text
Keypairs
Keypairs
Cluster signing CA => apiserver
Usages:
server auth
Cluster signing CA => controller_manager
Usages:
server auth
Cluster signing CA => scheduler
Usages:
server auth
8 . 3
Slide 25
Slide 25 text
Apiserver
Apiserver
--client-ca-file=/path/to/ca-cluster_signing.pem
--tls-cert-file=/path/to/apiserver.pem
--tls-private-key-file=/path/to/apiserver-key.pem
--secure-port=8443
8 . 4
Slide 26
Slide 26 text
Controller Manager
Controller Manager
--cluster-signing-cert-file=/path/to/ca-cluster_signing.pem
--cluster-signing-key-file=/path/to/ca-cluster_signing-key.pem
--tls-cert-file=/path/to/controller_manager.pem
--tls-private-key-file=/path/to/controller_manager-key.pem
8 . 5
Slide 27
Slide 27 text
Scheduler
Scheduler
--tls-cert-file=/path/to/scheduler.pem
--tls-private-key-file=/path/to/scheduler-key.pem
8 . 6
Slide 28
Slide 28 text
So far so good
So far so good
CA: 3
Etcd peer
Etcd server
Kubernetes cluster signing
Keypairs: 6+
etcd_peer
etcd_server
etcd_client
apiserver
controller_manager
scheduler
9 . 1
Slide 29
Slide 29 text
Kubelets
Kubelets
10 . 1
Slide 30
Slide 30 text
Keypairs
Keypairs
Cluster signing CA => kubelet_client
Usages:
client auth
10 . 2
Slide 31
Slide 31 text
Apiserver
Apiserver
--kubelet-client-certificate=/path/to/kubelet_client.pem
--kubelet-client-key=/path/to/kubelet_client-key.pem
--kubelet-https=true
10 . 3
Slide 32
Slide 32 text
CSR
CSR
$ kubectl get csr
NAME AGE REQUESTOR CONDITION
csr-22jvq 1m system:node:kube01 Pending
10 . 4
Slide 33
Slide 33 text
Guts
Guts
$ kubectl get csr csr-22jvq -o yaml
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: csr-22jvq
spec:
groups:
- system:nodes
- system:authenticated
request: LS0tL...
usages:
- digital signature
- key encipherment
- server auth
username: system:node:kube01
10 . 5
Slide 34
Slide 34 text
Guts
Guts
$ kubectl get csr csr-22jvq -o jsonpath='{.spec.request}' |\
base64 -D | openssl req -noout -text
Certificate Request:
Data:
Version: 0 (0x0)
Subject: O=system:nodes, CN=system:node:kube01
...
Attributes:
Requested Extensions:
X509v3 Subject Alternative Name:
DNS:kube01, IP Address:192.168.0.1
10 . 6
Slide 35
Slide 35 text
Approve
Approve
$ kubectl certificate approve csr-22jvq
certificatesigningrequest.certificates.k8s.io/csr-22jvq
approved
10 . 7
Slide 36
Slide 36 text
Result
Result
$ kubectl get csr csr-22jvq -o \
jsonpath='{.status.certificate}' | \
base64 -D|openssl x509 -noout -text
Certificate:
Issuer: C=RU, ST=Moscow, L=Moscow, O=Kubernetes, OU=CA, CN=Kubern
Subject: O=system:nodes, CN=system:node:kube01
...
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication
X509v3 Subject Alternative Name:
DNS:kube01, IP Address:192.168.0.1
10 . 8
Slide 37
Slide 37 text
So far so good
So far so good
CA: 3
Etcd peer
Etcd server
Kubernetes cluster signing
Keypairs: 9+
…
kubelet_client
kubelet-current * num of kubelets * 2
11 . 1
Slide 38
Slide 38 text
Service Accounts
Service Accounts
$ kubectl get sa
NAME SECRETS AGE
default 1 1h
$ kubectl get secret
NAME
default-token-78jt9
TYPE DATA AGE
kubernetes.io/service-account-token 3 1h
12 . 1
Slide 39
Slide 39 text
Service Accounts: secret
Service Accounts: secret
anatomy
anatomy
$ kubectl get secrets default-token-78jt9 -o yaml | \
yq r - 'data'
ca.crt: LS0tLS1CRUdJTiB...
namespace: ZGVmYXVsdA==
token: ZXlKaGJHY2lPaUpT...
12 . 2
Slide 40
Slide 40 text
Guts
Guts
$ kubectl get secrets default-token-78jt9 \
-o jsonpath='{.data.token}' | base64 -D
eyJhbGciOiJSUzI1NiIsImtpZCI6ImRIQm51UGV0TkNaZV9KV0k1SjRtSmlfeS1BU
12 . 3
Slide 41
Slide 41 text
Guts
Guts
Image courtesy of jwt.io
12 . 4
Slide 42
Slide 42 text
Service Accounts: CA
Service Accounts: CA
Service Accounts CA
12 . 5
Slide 43
Slide 43 text
Apiserver & Controller
Apiserver & Controller
Manager
Manager
apiserver
controller-manager
--service-account-key-file=/path/to/service_account_ca.pem
--service-account-private-key-file=/path/to/service_account_ca-k
12 . 6
Slide 44
Slide 44 text
So far so good
So far so good
CA: 4
Etcd peer
Etcd server
Kubernetes cluster signing
Service accounts
Keypairs: 9+
…
13 . 1
Slide 45
Slide 45 text
Aggregation Layer
Aggregation Layer
https://kubernetes.io/docs/tasks/access-
kubernetes-api/con gure-aggregation-layer/
$ kubectl get apiservices.apiregistration.k8s.io
NAME SERVICE
v1. Local
v1.apps Local
v1.authentication.k8s.io Local
...
v1beta1.external.metrics.k8s.io monitoring/prometheus-adap
v1beta1.metrics.k8s.io kube-system/metrics-server
v1beta1.webhook.cert-manager.io kube-system/cert-manager-w
...
14 . 1
Slide 46
Slide 46 text
Service
Service
$ kubectl get svc -n kube-system cert-manager-webhook
NAME TYPE CLUSTER-IP EXTERNAL-IP PO
cert-manager-webhook ClusterIP 10.222.206.2 none 4
14 . 2
Slide 47
Slide 47 text
CA
CA
Requestheader
14 . 3
Slide 48
Slide 48 text
Keypairs
Keypairs
proxy_client
Usages:
client auth
14 . 4
Slide 49
Slide 49 text
Apiserver
Apiserver
--proxy-client-cert-file=/path/to/front_proxy_client.pem
--proxy-client-key-file=/path/to/front_proxy_client-key.pem
--requestheader-allowed-names=front-proxy-client
--requestheader-client-ca-file=/path/to/ca-requestheader.pem
--requestheader-extra-headers-prefix=X-Remote-Extra-
--requestheader-group-headers=X-Remote-Group
--requestheader-username-headers=X-Remote-User
14 . 5
Slide 50
Slide 50 text
extension-apiserver-
extension-apiserver-
authentication configmap
authentication configmap
$ kubectl get cm -n kube-system extension-apiserver-authenticatio
data:
client-ca-file: |-
-----BEGIN CERTIFICATE-----
...
requestheader-allowed-names: '["front-proxy-client"]'
requestheader-client-ca-file: |-
-----BEGIN CERTIFICATE-----
...
requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
requestheader-group-headers: '["X-Remote-Group"]'
requestheader-username-headers: '["X-Remote-User"]'
14 . 6
Slide 51
Slide 51 text
Flow
Flow
1. Authentication | Authorization
User
Apiserver
2. Send data to aggregated apiserver
Apiserver
proxy_client_keypair
--requestheader-username-headers
--requestheader-group-headers
--requestheader-extra-headers-prefix
Aggregated apiserver
kube-system/extension-apiserver-authentication cm
clus
14 . 7
Slide 52
Slide 52 text
So far so good
So far so good
CA: 5
Etcd peer
Etcd server
Kubernetes cluster signing
Service accounts
Requestheader client
Keypairs: 10+
…
Requestheader client CA => proxy_client
15 . 1
Slide 53
Slide 53 text
Rotation
Rotation
Etcd peer
Etcd server
Kubernetes cluster signing
Service accounts
Requestheader client
16 . 1
Slide 54
Slide 54 text
Rotation: Etcd peer
Rotation: Etcd peer
Reissue CA
Issue new keypairs
Deploy to etcd nodes
Restart instances
16 . 2
Slide 55
Slide 55 text
Rotation: Etcd server
Rotation: Etcd server
Reissue CA
Issue new etcd server(s) keypair(s)
Deploy to etcd nodes
Restart instances
Issue new etcd client keypair
Deploy to apiserver nodes
Restart instances
16 . 3
Slide 56
Slide 56 text
Rotation: Kubernetes cluster
Rotation: Kubernetes cluster
signing (apiserver)
signing (apiserver)
Reissue CA
Issue new apiserver(s) keypair(s)
Deploy to apiserver nodes
Restart instances
16 . 4
Slide 57
Slide 57 text
Rotation: Kubernetes cluster
Rotation: Kubernetes cluster
signing (pods and nodes)
signing (pods and nodes)
Delete all serviceaccount token containing
secrets
Restart all corresponding pods
Delete kubelet requested keypairs (kubelet-*-
current.pem)
Restart all kubelets
Twice approve all new CSRs
kubectl certificate approve \
$(kubectl get csr|grep Pending|awk '{print $1}')
16 . 5
Slide 58
Slide 58 text
Rotation: Service accounts
Rotation: Service accounts
Reissue CA
Deploy to apiserver nodes
Restart instances
Delete all serviceaccount token containing
secrets
Restart all corresponding pods
16 . 6
Slide 59
Slide 59 text
Rotation: Requestheader client
Rotation: Requestheader client
Reissue CA
Issue new proxy_client keypair
Deploy to apiserver nodes
Restart instances
Restart all corresponding pods
16 . 7
Slide 60
Slide 60 text
Questions?
Questions?
17 . 1