Bare Metal k8s Cluster with CoreOS Matchbox

Slide 1

Slide 1 text

Bare Metal k8s Cluster with CoreOS Matchbox André Veelken - DevOps Engineer 22.03.2018

Slide 2

Slide 2 text

Cloud K8s vs. on-prem - majority of k8s clusters on AWS or other clouds - easy to set up, see e.g. kops tool - expensive - slow / noisy neighbors - easy scaling - ecosystem makes things easier

Slide 3

Slide 3 text

Cloud K8s vs. on-prem - fast & powerful - cheap in comparison - scaling? - DHCP? - OS images & config - Loadbalancing?

Slide 4

Slide 4 text

K8s on bare metal: choose your weapon kubeadm kubespray Ansible Puppet Matchbox (network booting)

Slide 5

Slide 5 text

Why our choice? - @chaosaffe, iPXE and  packet.net - similar to datacenter - wanted: immutable   infrastructure on-prem - CoreOS Container Linux - without additional  config management tools

Slide 6

Slide 6 text

What is Matchbox? github.com/coreos/matchbox - HTTP server - iPXE server - config templating engine (ignition) - static asset server - matches servers by labels (e.g. MAC address)   to Profiles with iPXE configs,   Container Linux configs

Slide 7

Slide 7 text

Advantages of Matchbox - YAML rendering for ignition and variable expansion - TLS auf gRPC - Validation - Hash signatures - tested, part of Tectonic

Slide 8

Slide 8 text

Cluster setup: Matchbox server - Matchbox server installation via package, Docker, rkt, … - secured via iptables

Slide 9

Slide 9 text

Cluster setup: Terraform (IAC tool) % terraform apply -auto-approve

Slide 10

Slide 10 text

Cluster setup: DNS - Terraform creates DNS records from   Terraform state at AWS Route53 for:    - each control plane node - each worker node - api (kubectl endpoint) - all worker nodes

Slide 11

Slide 11 text

Cluster setup: OS image preparation - get-coreos script places images in /var/lib/matchbox/assets/coreos - structure: /var/lib/matchbox assets/ - free form, files, hashes groups/ - control plane, master groups ignition/ - config mgmt on Container Linux profiles/

Slide 12

Slide 12 text

Cluster setup: Matchbox profiles - defines OS image, boot params, ignition -

Slide 13

Slide 13 text

Cluster setup: bootkube - tool for launching self-hosted Kubernetes clusters - we create a k8s config on Container Linux with it - write IPs and MACs into  bootkube-render.sh and execute it

Slide 14

Slide 14 text

Cluster setup: bootkube, rollout - bootkube renders assets into  /var/lib/matchbox/assets - restart of control plane and worker nodes, they pull their assets

Slide 15

Slide 15 text

Cluster setup: bootkube, bootstrapping - on control plane node 1 - % systemctl start bootkube - % journalctl -f -u bootkube - certificates from bootkube in   /etc/kubernetes/secrets are needed

Slide 16

Slide 16 text

Cluster setup: done, woohoo! - use kubectl on a master to inspect cluster

Slide 17

Slide 17 text

Lessons learned - Matchbox server should be   at same location as cluster - use OEM version of Container Linux if needed, e.g. for packet - learned many lessons on k8s internals especially on control plane: etcd, manifests

Slide 18

Slide 18 text

- Typhoon k8s distro - uses Terraform for everything - unified workflow - write scripts for scale up & down

Slide 19

Slide 19 text

Sources - GitHub.com/coreos/matchbox - github.com/kubernetes- incubator/bootkube - typhoon.psdn.io/bare-metal/ - Artwork: github.com/ ashleymcnamara/gophers

Slide 20

Slide 20 text

- Kubernetes Community Slack #wg-onprem (former #sig-onprem) - packet.net Slack #k8s - in planning: k8s on baremetal informal gathering   at containerdays.io in June

Slide 21

Slide 21 text

Thank you! % terraform destroy -force