Singapore Kubernetes Meetup - cluster bootstrap

Singapore Kubernetes Meetup - cluster bootstrap

Overview on how we bootstrap clusters with kops and Terraform at Honestbee

Highlight is using kops/channels to bootstrap a secured tiller

2f572006f3fd56861637353dc2bc523e?s=128

vincentdesmet

January 17, 2018
Tweet

Transcript

  1. 3.

    Kubernetes Bootstrap - Local - Minikube - ... - Hosted

    - Turnkey Cloud Solutions - Turnkey On-prem Solutions
  2. 4.

    Kubernetes Bootstrap - Local - Hosted - Google Container Engine

    (GKE) - Azure Container Service (AKS) - Amazon Elastic Container Service (EKS) - ... - Turnkey Cloud Solutions - Turnkey On-prem Solutions
  3. 5.

    Kubernetes Bootstrap - Local - Hosted - Turnkey Cloud Solutions

    - Open source AWS - Commercial - ... - Turnkey On-prem Solutions
  4. 8.

    Honestbee 2016 - k8s Bootstrap kz8s/tack ( 1.4 -> 1.5

    ) - CoreOS + Terraform! Issues: - Terraform 0.7 - Poor remote state support - Providers compiled within Terraform release - No or poor support for tls / templating / local providers ... - Hard to manage Etcd (lost cluster state a few times) - Hard to upgrade Kubernetes version (manual / slow)
  5. 9.

    2016 Exciting projects - kubeadm - ref docker swarm init

    … - Create cluster with join tokens - Manage TLS cert rotation - Secure (RBAC …) - But … status (q2 2017): - lacks support for HA - still alpha - kops: Kubernetes operations (upup) - Manage clusters… the k8s way - Self Hosting proposal - temp control-plane for bootstrap - core component of CoreOS/Tectonic
  6. 10.

    CoreOS Tectonic > Enterprise-ready Kubernetes clusters - CoreOS Container Linux

    (orchestrated, self-updating clusters) - Enterprise Security (integrate with LDAP via dex) - Management Console for cluster maintenance and operations (etcd / control-plane) - Built-in Monitoring (prometheus for autoscaling vs heapster) - Open Cloud Services announced kubecon 2017 (i.e.: vault)
  7. 11.

    CoreOS Tectonic - fundamentals - CoreOS Tectonic: terraform - Initially

    bundled Terraform patched by CoreOS - CoreOS added a lot of features to tf providers for Azure / AWS while building Tectonic (open source) - Tectonic vanilla mode (no Licenses, fully open source) - CoreOS + Ticketmaster engineers develop open source ALB Ingress controller - Terraform has grown a lot since 0.7 - Better state backend support - Better modules support - Better state management - Added support for Workspaces (environments) - ...
  8. 12.

    CoreOS Tectonic - fundamentals - Self Hosted (bootkube) - Simpler

    installation flow - Sustainable cluster lifecycle - Goals? - Reduce components required on host - Reduce files written to host (config management) - Improve introspection - Upgrade Kubernetes, using Kubernetes API - Easier HA
  9. 13.

    Kubernetes Layers As defined in the Self-Hosted control plane proposal

    - Layer 0: Kubelet - Layer 1: Etcd - Layer 2: API server - Layer 3: Scheduler / Controller Manager / Proxy - Layer 4: DNS and Addons
  10. 15.

    Bootkube - Self-hosting layers 1 to 4 since Q1 2017

    - System hosted kubelet Updated through node annotations - Periodic snapshotting for recovery - Depends on external DNS / LB - Note test coverage for version skew between control-plane components
  11. 16.

    Other projects using bootkube (too late for us, but which

    you may consider) - Typhoon: Minimal and free Kubernetes distribution (Terraform) - Archon: Operator to manage Kubernetes clusters - bootkube-terraform: (part of typhoon) tf module to render bootkube assets with terraform - ...
  12. 17.

    Tectonic conclusion (at the time of testing) - Could not

    create multiple clusters from same Tectonic install dir (lacked support for Terraform workspaces) - Takes time to study, understand and make changes (+ how to keep changes in line with upstream) - Some Licensed components were still bootstrapped in vanilla mode (required some clean up) - ...
  13. 18.

    Kops - Define clusters in Code (manifests) - clusterSpec -

    instanceGroups - Manages secrets and TLS assets - Manages node boot sequence (which uses cluster manifests) - Manages HA Etcd cluster with better recovery models than tack - Defines how to bootstrap Addon: channels - DNS / CNI / ...
  14. 19.

    Kubernetes vs Kops State store (etcd) State store (s3 /

    gcs / … ) API Server CLI Client (kubectl) CLI Client (kops) Controllers Cloud Resources Manifests Cloud provider (AWS / GCP / …) Manifests *ignoring scheduler ... Cloud provider (AWS / GCP / TF …) upup/cloudup
  15. 20.

    Kops Boot Sequence - nodeup component (config from state-store: clusterSpec):

    - Installs packages / copies in assets - Sets up protokube (to manage etcd ~EBS volumes) - Sets up the OS for kubelet (renders /etc/kubernetes/manifests ) - kubelet requests PodCIDR from kube-controllermanager for CNI - kubelet configures and starts CRI with PodCIDR config - kubelet handles and reports workloads assigned by scheduler
  16. 21.

    Kops Addon Channels - Kubernetes Addons: Bundles of resources that

    provide specific functionality - dns - dashboard - autoscaler - Addon Channels Multiple addons versioned together: kind: Addons metadata: name: beekeeper spec: addons: - name: tiller.addons.k8s.io manifest: tiller.addons.k8s.io/k8s-1.7.yaml kubernetesVersion: '>=1.7.0' selector: k8s-addon: tiller.addons.k8s.io version: 2.7.2 #helm version - name: namespaces.honestbee.io manifest: namespaces.honestbee.io/k8s-1.7.yaml kubernetesVersion: '>=1.7.0' selector: k8s-addon: namespaces.honestbee.io version: 1.1.2 - name: ...
  17. 22.

    Kops channels - kops logic to bootstrap and manage core

    kubernetes addons - Compile as separate binary: /go/src/k8s.io/kops# make channels - Apply addons from local or upstream channels ~/# channels apply channel -f beekeeper/addons.yaml NAME CURRENT UPDATE namespaces.honestbee.io 1.1.1 1.1.2 tiller.addons.k8s.io 2.7.0 2.7.2 Must specify --yes to update
  18. 23.

    Kops channels List addons currently installed: ~/# channels get addons

    NAMESPACE NAME VERSION CHANNEL kube-system core.addons.k8s.io 1.4.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system dns-controller.addons.k8s.io 1.7.1 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system kube-dns.addons.k8s.io 1.14.5 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system limit-range.addons.k8s.io 1.5.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system storage-aws.addons.k8s.io 1.6.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system kubernetes-dashboard 1.7.1 kubernetes-dashboard/addon.yaml kube-system monitoring-standalone 1.6.0 monitoring-standalone/addon.yaml kube-system tiller.addons.k8s.io 2.7.2 beekeeper/addons.yaml kube-system namespaces.honestbee.io 1.1.2 beekeeper/addons.yaml kube-system kube-state-metrics.addons.k8s.io v1.1.0-rc.0 beekeeper/addons.yaml
  19. 25.

    Kops caveats - Don’t use the default CIDRs (overlap with

    Docker bip) - Use Declarative Manifests instead of Imperative kops cli - Make sure to reserve resources for kubelet / docker / system - ...
  20. 26.

    Honestbee Kops-infra - Agenda - Terminology - Kops manifests -

    Cluster definitions and maintenance - Terraform modules - kops - bootstrap (templated channels & bootstrap.sh) - VPC peers, endpoints and routing
  21. 27.

    Kops-Infra: Terminology - cluster: a single AZ Kubernetes cluster -

    cluster_group: a group of clusters. For example “staging clusters”, “svc clusters”, … A cluster_group represents shared infra (1 VPC, hosted stateful service endpoints (RDS / ElastiCache), … ) - utilities subnet - subnet used for Hosted Services such as RDS / ElastiCache / ...
  22. 29.

    Kops Manifests Source control committed copy of Manifests. State store

    (s3 / gcs / … ) CLI Client (kops) Cloud Resources Manifests Cloud provider (AWS / GCP / TF …) upup/cloudup
  23. 31.

    Kops Manifests - Sample use cases For example: Add tags

    to cluster Auto Scaling groups and EC2 Instances
  24. 33.
  25. 37.

    HB/Kops Infra - TF Modules - TF kops module -

    TF bootstrap module - VPC peering / endpoints / routing / … modules - vault / vault-controller
  26. 38.

    TF kops module Set up kops pre-requisites: - ssh keys

    - S3 bucket for Kops state-store - VPC to host cluster_groups - Route53 Hosted Zone for DNS resolution - Bastion Hosts - ...
  27. 39.

    TF Bootstrap Module Bootstrap cluster after creation - Templated Namespaces

    (RBAC, Registry Secrets, …) - channels - templated beekeeper channel - Autoscaler - Rendered Honestbee Namespace templates - Tiller - Render config for chart bootstrap - Bootstrap.sh (bootstrap charts) - Continue bootstrap using Helm (external / internal ingress, datadog, …)