Kubernetes the Hardware Way

Kubernetes is a powerful system for operating application containers across a cluster of machines. In this talk, we'll explore CoreOS cluster provisioning and Kubernetes setup on hardware. To start, we'll cover PXE network setup and Ignition, CoreOS's built-in early-boot provisioning tool. Then we'll discuss matchbox, a service which matches machines to profiles to provision complete clusters. We'll walk through PXE booting machines, installation to disk, and automated provisioning of a multi-node Kubernetes cluster. We’ll show how the approach extends across machines and to provisioning many different kinds of clusters, including "self-hosted" Kubernetes and rktnetes.

Transcript

1. Kubernetes the Hardware Way CoreOS Provisioning Dalton Hubble @dghubble | dalton.hubble@coreos.com
2. Kubernetes the Hardware Way
3. Overview 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
4. github.com/coreos/matchbox (formerly coreos-baremetal)
5. Setup 0. Physical Setup
8. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
9. Preboot eXecution Environment (PXE) • Clients boot correct image and configuration • Discover, download, and run an NBP • Environment must provide: ◦ Initiation mechanism ◦ Network Services for client NICs ◦ Boot firmware APIs for the NBP
10. Initiation Mechanism • Boot order ◦ Disk first, then network ◦ Works well with auto-updates • BMC ◦ IPMI ◦ ipmitool -H <node> chassis bootdev pxe
11. Network Environment • DHCP (PXE-enabled) • TFTP • DNS
12. rkt run coreos.com/dnsmasq:v0.3.0
13. dnsmasq dhcp-userclass=set:ipxe,iPXE dhcp-boot=tag:#ipxe,undionly.kpxe dhcp-boot=tag:ipxe,http://matchbox.foo:8080/boot.ipxe
14. PXE Client DHCP Server TFTP Server matchbox boot server, filename PXEClient 67/UDP
15. PXE Client DHCP Server TFTP Server matchbox iPXE Client undionly.kpxe boot server, filename PXEClient 67/UDP 69/UDP
16. NBP • Load kernel & initrd • PXELINUX • iPXE (undionly.kpxe) ◦ Flash or Chain ◦ Config script file ◦ Knows HTTP
17. iPXE Config #!ipxe kernel /coreos/1153.0.0/coreos_production_pxe.vmlinuz initrd /coreos/1153.0.0/coreos_production_pxe_image.cpio.gz boot
18. iPXE Config #!ipxe kernel /coreos/1153.0.0/coreos_production_pxe.vmlinuz initrd /coreos/1153.0.0/coreos_production_pxe_image.cpio.gz boot
19. PXE Client DHCP Server TFTP Server matchbox iPXE Client undionly.kpxe boot server, filename boot server, filename PXEClient PXEClient, ipxe 67/UDP 69/UDP
20. dnsmasq dhcp-userclass=set:ipxe,iPXE dhcp-boot=tag:#ipxe,undionly.kpxe dhcp-boot=tag:ipxe,http://matchbox.foo:8080/boot.ipxe
21. PXE Client DHCP Server TFTP Server matchbox iPXE Client undionly.kpxe boot server, filename iPXE Config boot server, filename PXEClient PXEClient, ipxe 67/UDP 69/UDP HTTP
22. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
23. CoreOS Ignition
24. CoreOS Ignition • Early-boot disk provisioning (initramfs) ◦ Partition and format disks ◦ Write systemd units, networkd units, files ◦ Create users and groups • Runs once • Atomic
25. CoreOS matchbox
26. matchbox Matcher Group “selector”: { “mac”: "52:54:00:a1:9c:ae” } Profile iPXE Ignition
27. CoreOS matchbox • Matches machines to Profiles • Serves templated Ignition configs ◦ iPXE Config ◦ Ignition Config ◦ cloud-config, kickstart, etc. • Read-write gRPC API
28. Automated Steps 1. Network Booting 2. Provisioning 3. Kubernetes Bring-up
29. Kubernetes (static) k8s-controller.ign etcd flannel kubelet rkt | Docker Kubernetes control plane pods k8s-worker.ign etcd (proxy) flannel kubelet rkt | Docker App App
30. Write a systemd Unit systemd: units: - name: etcd2.service enable: true dropins: - name: 40-etcd-cluster.conf contents: | [Service] Environment="ETCD_NAME={{.etcd_name}}" Environment="ETCD_ADVERTISE_CLIENT_URLS=http://{{.domain_name}}:2379" Environment="ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{.domain_name}}:2380" Environment="ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379" Environment="ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380" Environment="ETCD_INITIAL_CLUSTER={{.etcd_initial_cluster}}"
31. kubelet Rkt | Docker kubelet Rkt | Docker Kubernetes (static) Controller Worker
32. Kubelet systemd: units: - name: kubelet.service enable: true contents: | ... ExecStart=/usr/lib/coreos/kubelet-wrapper \ --api-servers={{.k8s_controller_endpoint}} \ --pod-manifest-path=/etc/kubernetes/manifests \ ...
33. kernel rkt | Docker etcd flannel kubelet systemd App Libs App Libs App Libs App Libs
34. /etc/kubernetes/manifests (controller) • kube-apiserver • controller-manager • scheduler • kube-proxy • kube-dns
35. /etc/kubernetes/manifests (worker) • kube-proxy
36. kube-apiserver.yaml storage: files: - path: /etc/kubernetes/manifests/kube-apiserver.yaml filesystem: root contents: inline: | apiVersion: v1 ... - name: kube-apiserver image: quay.io/coreos/hyperkube:v1.5.2_coreos.0
37. kubelet kube-proxy apiserver controller-manager scheduler kube-dns Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (static) Controller Worker
39. DEMO: PXE to Kubernetes
40. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
41. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
42. iPXE Config #!ipxe kernel /coreos/1235.6.0/coreos_production_pxe.vmlinuz coreos.first_boot coreos.config.url=http://matchbox.foo/ignition?uuid=${uuid}&mac= ${mac:hexhyp} initrd /coreos/1235.6.0/coreos_production_pxe_image.cpio.gz boot
43. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
44. Userspace NBP Early Userspace Network Boot Ignition m ulti-user system d netw orkd iPXE coreos-install netw orkd reboot
45. Userspace Bootload Early Userspace Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
46. Early Userspace Userspace Bootload Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
47. Userspace Bootload Early Userspace Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd
48. Early Userspace Userspace Bootload Disk Boot Ignition m ulti-user system d netw orkd GRUB etcd netw orkd kubelet
49. Themes • Lightweight, auto-upgrading base OS • Leverage DHCP and PXE • Simple, powerful provisioning system • Minimize on-host configs • The cluster is your configuration manager
50. Where to go from here?
51. Tectonic Installer
54. kubelet kube-proxy apiserver controller-manager scheduler kube-dns Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (self-hosted) Controller Worker H O S T kubelet kubelet
55. kubelet kube-proxy apiserver scheduler Rkt | Docker kubelet kube-proxy Rkt | Docker Kubernetes (self-hosted) Controller Worker H O S T kubelet kubelet controller-manager kube-dns
56. rktnetes systemd: units: - name: kubelet.service enable: true contents: | ... ExecStart=/usr/lib/coreos/kubelet-wrapper \ --container-runtime=rkt \ ...
57. More Info? github.com/coreos/matchbox
58. CoreOS is Hiring! sales@coreos.com 90+ Projects on GitHub, 1,000+ Contributors coreos.com Support plans, training and more OPEN SOURCE ENTERPRISE