Kubernetes the Hardware Way

Kubernetes is a powerful system for operating application containers across a cluster of machines. In this talk, we'll explore CoreOS cluster provisioning and Kubernetes setup on hardware. To start, we'll cover PXE network setup and Ignition, CoreOS's built-in early-boot provisioning tool. Then we'll discuss bootcfg, a service which matches machines to profiles to provision complete clusters. We'll walk through PXE booting machines, installation to disk, and automated provisioning of a multi-node etcd key-value store and multi-node Kubernetes cluster. We’ll show how the approach extends across machines and to provisioning many different kinds of clusters, including "self-hosted" Kubernetes and rktnetes.

https://github.com/coreos/coreos-baremetal

Transcript

1. Kubernetes on Hardware CoreOS Provisioning Dalton Hubble @dghubble | dalton.hubble@coreos.com
3. Kubernetes the Hardware Way
4. Setup 0. Physical Setup
5. ibm0 nuc0 nuc1 nuc2 super micro0, micro1 (not shown)
7. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
8. Preboot eXecution Environment (PXE) • Clients boot correct image and configuration • Discover, download, and run an NBP • Environment must provide: ◦ Initiation mechanism ◦ Network Services for client NICs firmware ◦ Boot firmware APIs for the NBP
9. NBP • Load kernel & initrd • PXELINUX • iPXE ◦ Flash or Chain ◦ Config script file ◦ HTTP
10. iPXE Config #!ipxe kernel /coreos/1153.0.0/coreos_production_pxe.vmlinuz initrd /coreos/1153.0.0/coreos_production_pxe_image.cpio.gz boot
11. iPXE Config #!ipxe kernel /coreos/1153.0.0/coreos_production_pxe.vmlinuz initrd /coreos/1153.0.0/coreos_production_pxe_image.cpio.gz boot
12. DHCP, TFTP, DNS Option 60 Vendor Class Data Option 77 User Class Data Option 54 Boot Server IP Option 67 boot filename (NBP)
13. PXE Client DHCP Server TFTP Server Config Server boot server, filename PXEClient 67/UDP
14. PXE Client DHCP Server TFTP Server Config Server iPXE Client undionly.kpxe boot server, filename PXEClient 67/UDP 69/UDP
15. PXE Client DHCP Server TFTP Server Config Server iPXE Client undionly.kpxe boot server, filename boot server, filename PXEClient PXEClient, ipxe 67/UDP 69/UDP
16. PXE Client DHCP Server TFTP Server Config Server iPXE Client undionly.kpxe boot server, filename boot.ipxe (stub) boot server, filename PXEClient PXEClient, ipxe 67/UDP 69/UDP HTTP
17. PXE Client DHCP Server TFTP Server Config Server iPXE Client undionly.kpxe boot server, filename boot.ipxe (stub) iPXE Config boot server, filename PXEClient PXEClient, ipxe 67/UDP 69/UDP HTTP
18. Client Machines • BIOS or UEFI set to network boot • Boot order • BMC ◦ IPMI - remote power, boot order, console mgtm
19. rkt run coreos.com/dnsmasq:v0.3.0
20. PXE Demo
21. PXE-enabled DHCP
24. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
25. CoreOS Ignition
26. CoreOS Ignition • Early-boot disk provisioning (initramfs) ◦ Partition and format disks ◦ Write systemd units, networkd units, files ◦ Create users and groups • Runs once • Atomic
27. {"ignition":{"version":"2.0.0","config":{}},"storage":{"disks":[{"device":"/dev/sda","wipeTable":true,"partitions":[{"label":"ROOT","number":0,"size":0," start":0}]}],"filesystems":[{"name":"root","mount":{"device":"/dev/sda1","format":"ext4","create":{"force":true,"options":["-LROOT"]}}}],"files":[{"files ystem":"root","path":"/etc/kubernetes/manifests/kube-proxy.yaml","contents":{"source":"data:,apiVersion%3A%20v1%0Akind%3A%20Pod%0Ameta data%3A%0A%20%20name%3A%20kube-proxy%0A%20%20namespace%3A%20kube-system%0Aspec%3A%0A%20%20hostNetwork%3A%20true%0 A%20%20containers%3A%0A%20%20-%20name%3A%20kube-proxy%0A%20%20%20%20image%3A%20registry.example.com%3A5000%2Fhyperkub e%3Av1.3.6_coreos.0%0A%20%20%20%20command%3A%0A%20%20%20%20-%20%2Fhyperkube%0A%20%20%20%20-%20proxy%0A%20%20%20 %20-%20--master%3Dhttp%3A%2F%2F127.0.0.1%3A8080%0A%20%20%20%20securityContext%3A%0A%20%20%20%20%20%20privileged%3A%20t rue%0A%20%20%20%20volumeMounts%3A%0A%20%20%20%20-%20mountPath%3A%20%2Fetc%2Fssl%2Fcerts%0A%20%20%20%20%20%20nam e%3A%20ssl-certs-host%0A%20%20%20%20%20%20readOnly%3A%20true%0A%20%20volumes%3A%0A%20%20-%20hostPath%3A%0A%20%20%2 0%20%20%20path%3A%20%2Fusr%2Fshare%2Fca-certificates%0A%20%20%20%20name%3A%20ssl-certs-host%0A","verification":{}},"user":{},"grou p":{}},{"filesystem":"root","path":"/etc/kubernetes/manifests/kube-apiserver.yaml","contents":{"source":"data:,apiVersion%3A%20v1%0Akind%3A%2 0Pod%0Ametadata%3A%0A%20%20name%3A%20kube-apiserver%0A%20%20namespace%3A%20kube-system%0Aspec%3A%0A%20%20hostNetwo rk%3A%20true%0A%20%20containers%3A%0A%20%20-%20name%3A%20kube-apiserver%0A%20%20%20%20image%3A%20registry.example.com %3A5000%2Fhyperkube%3Av1.3.6_coreos.0%0A%20%20%20%20command%3A%0A%20%20%20%20-%20%2Fhyperkube%0A%20%20%20%20-%20a piserver%0A%20%20%20%20-%20--bind-address%3D0.0.0.0%0A%20%20%20%20-%20--etcd-servers%3Dhttp%3A%2F%2Fnode1.example.com%3A23 79%0A%20%20%20%20-%20--allow-privileged%3Dtrue%0A%20%20%20%20-%20--service-cluster-ip-range%3D10.3.0.0%2F24%0A%20%20%20%20- %20--secure-port%3D443%0A%20%20%20%20-%20--admission-control%3DNamespaceLifecycle%2CLimitRanger%2CServiceAccount%2CResourceQu ota%0A%20%20%20%20-%20--tls-cert-file%3D%2Fetc%2Fkubernetes%2Fssl%2Fapiserver.pem%0A%20%20%20%20-%20--tls-private-key-file%3D%2 Fetc%2Fkubernetes%2Fssl%2Fapiserver-key.pem%0A%20%20%20%20-%20--client-ca-file%3D%2Fetc%2Fkubernetes%2Fssl%2Fca.pem%0A%20%20 %20%20-%20--service-account-key-file%3D%2Fetc%2Fkubernetes%2Fssl%2Fapiserver-key.pem%0A%20%20%20%20-%20--runtime-config%3Dexten sions%2Fv1beta1%2Fnetworkpolicies%3Dtrue%0A%20%20%20%20livenessProbe%3A%0A%20%20%20%20%20%20httpGet%3A%0A%20%20%20%2 0%20%20%20%20host%3A%20127.0.0.1%0A%20%20%20%20%20%20%20%20port%3A%208080%0A%20%20%20%20%20%20%20%20path%3A%20 %2Fhealthz%0A%20%20%20%20%20%20initialDelaySeconds%3A%2015%0A%20%20%20%20%20%20timeoutSeconds%3A%2015%0A%20%20%20% 20ports%3A%0A%20%20%20%20-%20containerPort%3A%20443%0A%20%20%20%20%20%20hostPort%3A%20443%0A%20%...
28. CoreOS bootcfg
29. bootcfg Matcher Group “selector”: { “mac”: "52:54:00:a1:9c:ae” } Profile iPXE Ignition
30. CoreOS bootcfg • Matches machines to Profiles • Serves templated config files ◦ iPXE Config ◦ Ignition Config ◦ metadata, cloud-config, kickstart • gRPC API, generate client-bindings
31. Write a Unit File systemd: units: - name: etcd2.service enable: true dropins: - name: 40-etcd-cluster.conf contents: | [Service] Environment="ETCD_NAME={{.etcd_name}}" Environment="ETCD_ADVERTISE_CLIENT_URLS=http://{{.domain_name}}:2379" Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{.domain_name}}:2380" Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379" Environment="ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380" Environment="ETCD_INITIAL_CLUSTER={{.etcd_initial_cluster}}"
32. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
33. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
34. iPXE Config #!ipxe kernel /coreos/1153.0.0/coreos_production_pxe.vmlinuz coreos.first_boot coreos.config.url=http://bootcfg.foo/ignition?uuid=${uuid}&mac=$ {net0/mac:hexhyp} initrd /coreos/1153.0.0/coreos_production_pxe_image.cpio.gz boot
35. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
36. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
37. Userspace Bootload Early Userspace Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
38. Early Userspace Userspace Bootload Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
39. Userspace Bootload Early Userspace Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
40. Early Userspace Userspace Bootload Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
41. Userspace Bootload Early Userspace Disk Boot Ignition coreos-m etadata m ulti-user system d netw orkd GRUB etcd netw orkd
42. Revisit Mini-Demo
43. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
44. Kubernetes
45. Kubernetes Controller etcd flannel kubelet rkt | Docker Kubernetes control plane pods Worker etcd (proxy) flannel kubelet rkt | Docker App App
46. kernel rkt | Docker etcd flannel systemd
47. kernel rkt | Docker etcd flannel kubelet systemd App Libs App Libs App Libs App Libs
48. Kubelet systemd: units: - name: kubelet.service enable: true contents: | ... ExecStart=/usr/lib/coreos/kubelet-wrapper \ --api-servers={{.k8s_controller_endpoint}} \ --config=/etc/kubernetes/manifests \ ...
49. kubelet Rkt | Docker kubelet Rkt | Docker Kubernetes (static) Controller Worker
50. /etc/kubernetes/manifests (controller) • kube-apiserver • controller-manager • scheduler • kube-proxy • kube-dns
51. /etc/kubernetes/manifests (worker) • kube-proxy
52. kube-proxy.yaml storage: files: - path: /etc/kubernetes/manifests/kube-proxy.yaml filesystem: root contents: inline: | apiVersion: v1 kind: Pod metadata: Name: kube-proxy
53. kubelet kube-proxy apiserver controller-manager scheduler kube-dns Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (static) Controller Worker
55. DEMO: Air-gapped PXE-> Kubernetes
56. What’s Next?
57. kubelet kube-proxy apiserver controller-manager scheduler kube-dns Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (self-hosted) Controller Worker H O S T kubelet kubelet
58. kubelet kube-proxy apiserver scheduler Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (self-hosted) Controller Worker H O S T kubelet kubelet controller-manager kube-dns
59. Tectonic Installer
62. CoreOS is running the world’s containers We’re hiring: careers@coreos.com sales@coreos.com 90+ Projects on GitHub, 1,000+ Contributors coreos.com Support plans, training and more OPEN SOURCE ENTERPRISE
63. dalton.hubble@coreos.com Twitter: @dghubble IRC: #coreos on Freenode QUESTIONS? Thanks! We’re hiring: coreos.com/careers
64. More Info? github.com/coreos/coreos-baremetal github.com/coreos/ignition