Kubernetes on prem - Cluster rollout with kubespray

Slides zum Vortrag
(aktualisiert für die IT-Tage 2025)

Transcript

1. Dirk Weil, GEDOPLAN GmbH Kubenetes on prem Mit Kubespray den eigenen Kubernetes-Cluster aufsetzen
2. Dirk Weil Studied Informatik at RWTH Aachen Living in Bielefeld CEO of GEDOPLAN GmbH (www.gedoplan.de) JEE since 1999 Speaker and author 2 gedoplan.de Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen
3. Objective Host a Kubernetes cluster on-prem Why? Test environments, playgrounds Learning Small, cheap (?) prod environments How? Bare Metal on Linux Hosts Load Balancer (MetalLB) Ingress Controller (Ingress-Nginx, cert-manager) Cluster Storage (Ceph) Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 3 gedoplan.de
4. Options kubeadm k3s … kubespray Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 4 gedoplan.de
5. Prearrangement Hosts with basic Linux installation e.g. Debian, Ubuntu, … ssh Server User with SSH key and sudo right Additional unused partition Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 5 gedoplan.de k8s-141 k8s-142 k8s-143
6. Prearrangement Workstation with Container environment (e. g. Docker Desktop) Alternative: Local Ansible installation https://www.ansible.com/ open source IT automation engine utilizes so-called playbooks (~Scripts) Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 6 gedoplan.de k8s-141 k8s-142 k8s-143
7. kubespray Ansible playbook for K8s rollout https://github.com/kubernetes-sigs/kubespray quay.io/kubespray/kubespray:v2.25.0 Topology configuration in so-called inventory Template in kubespray/inventory/sample Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 7 gedoplan.de Hosts and their duties Properties for various functionalities
8. kubespray Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 8 gedoplan.de all: hosts: k8s-141: ansible_host: k8s-141.gedoplan.intra k8s-142: ansible_host: k8s-142.gedoplan.intra k8s-143: ansible_host: k8s-143.gedoplan.intra children: kube_control_plane: hosts: k8s-141: kube_node: hosts: k8s-142: k8s-143: etcd: hosts: k8s-141: k8s-142: k8s-143: k8s_cluster: children: kube_control_plane: kube_node: calico_rr: hosts: {} hosts.yaml (.yml/.ini) Hosts Control Plane Worker Storage
9. kubespray Rollout K8s cluster Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 9 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray > docker run --rm -it \ -v "$(pwd)"/inventory:/kubespray/inventory \ -v ~/.ssh/id_rsa:/root/.ssh/id_rsa \ quay.io/kubespray/kubespray:v2.25.0 \ ansible-playbook –u your_user_id -i inventory/hosts.yaml -b -K cluster.yml BECOME password: PLAY [Check Ansible version] ************************************************************* Friday 05 December 2025 07:25:27 +0000 (0:00:00.051) 0:00:00.051 ********** TASK [Check 2.16.4 <= Ansible version <= 2.17.0] ***************************************** Ok: [localhost] => { “changed”: false, “msg”: “All assertions passed” } Friday 05 December 2025 07:25:27 +0000 (0:00:00.049) 0:00:00.101 ***** … PLAY RECAP ******************************************************************************* k8s-141 : ok=736 changed=140 unreachable=0 failed=0 … k8s-142 : ok=578 changed=110 unreachable=0 failed=0 … k8s-143 : ok=578 changed=110 unreachable=0 failed=0 … localhost : ok=3 changed=0 unreachable=0 failed=0 … /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray > docker run --rm -it \ -v "$(pwd)"/inventory:/kubespray/inventory \ -v ~/.ssh/id_rsa:/root/.ssh/id_rsa \ quay.io/kubespray/kubespray:v2.25.0 \ ansible-playbook –u your_user_id -i inventory/hosts.yaml -b -K cluster.yml BECOME password: PLAY [Check Ansible version] ************************************************************* Friday 05 December 2025 07:25:27 +0000 (0:00:00.051) 0:00:00.051 ********** TASK [Check 2.16.4 <= Ansible version <= 2.17.0] ***************************************** Ok: [localhost] => { “changed”: false, “msg”: “All assertions passed” } Friday 05 December 2025 07:25:27 +0000 (0:00:00.049) 0:00:00.101 ***** … PLAY RECAP ******************************************************************************* k8s-141 : ok=736 changed=140 unreachable=0 failed=0 … k8s-142 : ok=578 changed=110 unreachable=0 failed=0 … k8s-143 : ok=578 changed=110 unreachable=0 failed=0 … localhost : ok=3 changed=0 unreachable=0 failed=0 …
10. Load Balancer: MetalLB Open Source Load Balancer Implementation https://metallb.universe.tf/ Layer 2 Mode (ARP / NDP) Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 10 gedoplan.de k8s- 141 k8s- 142 k8s- 143 ARP Request „who has 192.168.10.140?“ ARP Response „me! I am 6c:c2:17:6d:32:b5“
11. Load Balancer: MetalLB kubespray group_vars: Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 11 gedoplan.de … metallb_enabled: true metallb_speaker_enabled: "{{ metallb_enabled }}" metallb_namespace: "metallb-system" metallb_protocol: "layer2" metallb_config: address_pools: primary: ip_range: - 192.168.10.140/32 auto_assign: true layer2: - primary … group_vars/k8s_cluster/addons.yml … kube_proxy_strict_arp: true … group_vars/k8s_cluster/k8s-cluster.yml Use strict ARP in kube-proxy (answer ARP requests for active service endpoints only) Activate MetalLB in layer 2 mode
12. Load Balancer: MetalLB Components include: Controller (Deployment, Service) IP (de)allocation to LoadBalancer services Speaker (DaemonSet) Respond to ARP requests (incl. leader election and failover) Address pool Level 2 (ARP) advertisement Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 12 gedoplan.de apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: primary namespace: metallb-system spec: addresses: - 192.168.10.140/32 autoAssign: true apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: primary namespace: metallb-system spec: ipAddressPools: - primary
13. Ingress Controller: Ingress-Nginx Multiplex (and loadbalance) web requests Hostname, path → background service Configuration by K8s ingresses Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 13 gedoplan.de Ingress-Nginx Service foo Service bar same IP
14. Ingress Controller: Ingress-Nginx kubespray group_vars: DaemonSet and Service bound to MetalLB allocated IP Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 14 gedoplan.de … ingress_nginx_enabled: true ingress_nginx_service_type: LoadBalancer ingress_nginx_namespace: "ingress-nginx" ingress_nginx_insecure_port: 80 ingress_nginx_secure_port: 443 ingress_nginx_class: nginx ingress_nginx_default: true … group_vars/k8s_cluster/addons.yml Activate Ingress-Nginx and let MetalLB assign IP
15. Certificate Management: cert-manager Auto-create TLS certificates Includes usage in ingresses Certificate issuers for various CAs Lets Encrypt Private CA … https://cert-manager.io Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 15 gedoplan.de
16. Certificate Management: cert-manager kubespray group_vars: Issuer for private CA Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 16 gedoplan.de … cert_manager_enabled: true cert_manager_namespace: "cert-manager" … group_vars/k8s_cluster/addons.yml Activate cert-manager apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: demo-internal spec: ca: secretName: demo-internal-ca apiVersion: v1 kind: Secret metadata: name: demo-internal-ca type: Opaque stringData: tls.crt: | -----BEGIN CERTIFICATE----- MIIDxTCCAq2gAwIBAgIBADANBgkqhkiG9w0 … -----END CERTIFICATE----- tls.key: | -----BEGIN PRIVATE KEY----- MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKc … -----END PRIVATE KEY----- Self-signed cert for CA
17. Certificate Management: cert-manager Ingress using cert-manager certificates: Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 17 gedoplan.de apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami annotations: cert-manager.io/cluster-issuer: demo-internal spec: rules: - host: whoami.demo.gedoplan.demo http: paths: - pathType: Prefix path: "/" backend: service: name: whoami port: number: 80 tls: - hosts: - whoami.demo.gedoplan.demo secretName: whoami-demo-gedoplan-demo-tls cert-manager issuer Secret will be created by cert-manager
18. Certificate Management: cert-manager Let‘s Encrypt issuer Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 18 gedoplan.de apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@gedoplan.de privateKeySecretRef: name: letsencrypt solvers: - http01: ingress: ingressClassName: nginx apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: whoami annotations: cert-manager.io/cluster-issuer: letsencrypt … ACME http01 challenge (alternative: dns01) Select issuer in ingress
19. deploy └── examples ├── cluster.yaml ├── common.yaml ├── crds.yaml ├── csi │ └── rbd │ └── storageclass.yaml ├── csi-operator.yaml ├── dashboard-ingress-https.yaml └── operator.yaml Cluster Storage: Ceph distributed storage system file, block and object storage Simple installation: Rook (https://rook.io) automates deployment and management of Ceph https://github.com/rook/rook.git Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 19 gedoplan.de Files used for demo
20. Cluster Storage: Ceph Deploy the Rook Operator Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 20 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 1-operator namespace/rook-ceph created customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created … clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE rook-ceph-operator-757cdc49bb-pswkm 1/1 Running 0 6m26s /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 1-operator namespace/rook-ceph created customresourcedefinition.apiextensions.k8s.io/cephblockpoolradosnamespaces.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephblockpools.ceph.rook.io created customresourcedefinition.apiextensions.k8s.io/cephbucketnotifications.ceph.rook.io created … clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-osd created clusterrolebinding.rbac.authorization.k8s.io/rook-ceph-system created configmap/rook-ceph-operator-config created deployment.apps/rook-ceph-operator created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE rook-ceph-operator-757cdc49bb-pswkm 1/1 Running 0 6m26s wait for it rook-ceph/1-operator ├── common.yaml ├── crds.yaml ├── csi-operator.yaml ├── kustomization.yaml └── operator.yaml
21. rook-ceph/2-cluster ├── cluster.yaml └── kustomization.yaml Cluster Storage: Ceph Create Ceph Cluster Devices must be empty (wipefs -a …) Ceph config directory must not exist (/var/lib/rook) Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 21 gedoplan.de /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 2-cluster cephcluster.ceph.rook.io/rook-ceph created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE csi-cephfsplugin-6s4rx 2/2 Running 0 11m csi-cephfsplugin-provisioner-7f4d578c49-9rzv9 5/5 Running 0 11m … rook-ceph-osd-0-c544b7d7d-gh2wb 2/2 Running 0 9m58s rook-ceph-osd-1-649fdbf45c-87m5x 2/2 Running 0 9m57s rook-ceph-osd-prepare-k8s-142-qxgl4 0/1 Completed 0 9m30s rook-ceph-osd-prepare-k8s-143-5bglq 0/1 Completed 0 9m27s /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 2-cluster cephcluster.ceph.rook.io/rook-ceph created ❯ watch kubectl -n rook-ceph get pod NAME READY STATUS RESTARTS AGE csi-cephfsplugin-6s4rx 2/2 Running 0 11m csi-cephfsplugin-provisioner-7f4d578c49-9rzv9 5/5 Running 0 11m … rook-ceph-osd-0-c544b7d7d-gh2wb 2/2 Running 0 9m58s rook-ceph-osd-1-649fdbf45c-87m5x 2/2 Running 0 9m57s rook-ceph-osd-prepare-k8s-142-qxgl4 0/1 Completed 0 9m30s rook-ceph-osd-prepare-k8s-143-5bglq 0/1 Completed 0 9m27s wait for it … storage: useAllNodes: false useAllDevices: false nodes: - name: k8s-142 devices: - name: sda2 - name: k8s-143 devices: - name: sda3
22. rook-ceph/3-storageclass ├── kustomization.yaml └── storageclass.yaml Cluster Storage: Ceph Create Storage Class Block: block storage to be consumed by a pod (RWO) Shared Filesystem: filesystem to be shared by pods (RWX) Object: S3 compatible object store Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 22 gedoplan.de … annotations: "storageclass.kubernetes.io/is-default-class": "true" Demo: block storage facultative /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 3-storageclass storageclass.storage.k8s.io/rook-ceph-block created cephblockpool.ceph.rook.io/replicapool created /opt/prj/gedoplan/showcase/k8s-on-prem-kubespray/rook-ceph ❯ kubectl apply -k 3-storageclass storageclass.storage.k8s.io/rook-ceph-block created cephblockpool.ceph.rook.io/replicapool created
23. Cluster Storage: Ceph Publish Ceph Dashboard Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 23 gedoplan.de apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rook-ceph-mgr-dashboard namespace: rook-ceph annotations: cert-manager.io/cluster-issuer: demo-internal nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" nginx.ingress.kubernetes.io/server-snippet: | proxy_ssl_verify off; spec: rules: - host: ceph.gedoplan.demo http: paths: - path: / pathType: Prefix backend: service: name: rook-ceph-mgr-dashboard port: name: https-dashboard tls: - hosts: - ceph.gedoplan.demo secretName: ceph-gedoplan-demo-tls tune to your needs
24. Cluster Storage: Ceph Consume storage Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen 24 gedoplan.de apiVersion: apps/v1 kind: StatefulSet metadata: name: pg spec: serviceName: pg selector: matchLabels: name: pg volumeClaimTemplates: - metadata: name: pg-data spec: storageClassName: rook-ceph-block accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 1Gi template: … PVC using block storage (SC can be ommitted, if default)
25. More Demo project github.com/GEDOPLAN/k8s-on-prem-kubespray Slides speakerdeck.com/dirkweil gedoplan.de Trainings in Berlin, Bielefeld, Köln, inhouse Reviews, Coaching Development Teams Contact dirk.weil@gedoplan.de linkedin.com/in/dirk-weil-49940683 25 gedoplan.de Kubernetes on prem - mit Kubespray den eigenen Kubernetes-Cluster aufsetzen