Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bare Metal k8s Cluster with CoreOS Matchbox

dreamIT
March 22, 2018

Bare Metal k8s Cluster with CoreOS Matchbox

from our last k8s meetup on 22. March 2018

https://meetu.ps/e/DZP9q/mjfLb/a

dreamIT

March 22, 2018
Tweet

More Decks by dreamIT

Other Decks in Technology

Transcript

  1. Cloud K8s vs. on-prem - majority of k8s clusters on

    AWS or other clouds - easy to set up, see e.g. kops tool - expensive - slow / noisy neighbors - easy scaling - ecosystem makes things easier
  2. Cloud K8s vs. on-prem - fast & powerful - cheap

    in comparison - scaling? - DHCP? - OS images & config - Loadbalancing?
  3. Why our choice? - @chaosaffe, iPXE and
 packet.net - similar

    to datacenter - wanted: immutable 
 infrastructure on-prem - CoreOS Container Linux - without additional
 config management tools
  4. What is Matchbox? github.com/coreos/matchbox - HTTP server - iPXE server

    - config templating engine (ignition) - static asset server - matches servers by labels (e.g. MAC address) 
 to Profiles with iPXE configs, 
 Container Linux configs
  5. Advantages of Matchbox - YAML rendering for ignition and variable

    expansion - TLS auf gRPC - Validation - Hash signatures - tested, part of Tectonic
  6. Cluster setup: DNS - Terraform creates DNS records from 


    Terraform state at AWS Route53 for:
 
 - each control plane node - each worker node - api (kubectl endpoint) - all worker nodes
  7. Cluster setup: OS image preparation - get-coreos script places images

    in /var/lib/matchbox/assets/coreos - structure: /var/lib/matchbox assets/ - free form, files, hashes groups/ - control plane, master groups ignition/ - config mgmt on Container Linux profiles/
  8. Cluster setup: bootkube - tool for launching self-hosted Kubernetes clusters

    - we create a k8s config on Container Linux with it - write IPs and MACs into
 bootkube-render.sh and execute it
  9. Cluster setup: bootkube, rollout - bootkube renders assets into
 /var/lib/matchbox/assets

    - restart of control plane and worker nodes, they pull their assets
  10. Cluster setup: bootkube, bootstrapping - on control plane node 1

    - % systemctl start bootkube - % journalctl -f -u bootkube - certificates from bootkube in 
 /etc/kubernetes/secrets are needed
  11. Lessons learned - Matchbox server should be 
 at same

    location as cluster - use OEM version of Container Linux if needed, e.g. for packet - learned many lessons on k8s internals especially on control plane: etcd, manifests
  12. - Typhoon k8s distro - uses Terraform for everything -

    unified workflow - write scripts for scale up & down
  13. - Kubernetes Community Slack #wg-onprem (former #sig-onprem) - packet.net Slack

    #k8s - in planning: k8s on baremetal informal gathering 
 at containerdays.io in June