Kubernetes Cluster Bootstrap (at Honestbee, on AWS) Vincent De Smet DevOps @ Honestbee @vincentdesmet

Kubernetes Bootstrap - Local - Hosted - Turnkey Cloud Solutions - Turnkey On-prem Solutions

Kubernetes Bootstrap - Local - Minikube - ... - Hosted - Turnkey Cloud Solutions - Turnkey On-prem Solutions

Kubernetes Bootstrap - Local - Hosted - Google Container Engine (GKE) - Azure Container Service (AKS) - Amazon Elastic Container Service (EKS) - ... - Turnkey Cloud Solutions - Turnkey On-prem Solutions

Kubernetes Bootstrap - Local - Hosted - Turnkey Cloud Solutions - Open source AWS - Commercial - ... - Turnkey On-prem Solutions

Kubernetes Bootstrap - We like:

CoreOS 2016 - k8s bootstrap - kube-aws custom yaml + golang no terraform ...

Honestbee 2016 - k8s Bootstrap kz8s/tack ( 1.4 -> 1.5 ) - CoreOS + Terraform! Issues: - Terraform 0.7 - Poor remote state support - Providers compiled within Terraform release - No or poor support for tls / templating / local providers ... - Hard to manage Etcd (lost cluster state a few times) - Hard to upgrade Kubernetes version (manual / slow)

2016 Exciting projects - kubeadm - ref docker swarm init … - Create cluster with join tokens - Manage TLS cert rotation - Secure (RBAC …) - But … status (q2 2017): - lacks support for HA - still alpha - kops: Kubernetes operations (upup) - Manage clusters… the k8s way - Self Hosting proposal - temp control-plane for bootstrap - core component of CoreOS/Tectonic

CoreOS Tectonic > Enterprise-ready Kubernetes clusters - CoreOS Container Linux (orchestrated, self-updating clusters) - Enterprise Security (integrate with LDAP via dex) - Management Console for cluster maintenance and operations (etcd / control-plane) - Built-in Monitoring (prometheus for autoscaling vs heapster) - Open Cloud Services announced kubecon 2017 (i.e.: vault)

CoreOS Tectonic - fundamentals - CoreOS Tectonic: terraform - Initially bundled Terraform patched by CoreOS - CoreOS added a lot of features to tf providers for Azure / AWS while building Tectonic (open source) - Tectonic vanilla mode (no Licenses, fully open source) - CoreOS + Ticketmaster engineers develop open source ALB Ingress controller - Terraform has grown a lot since 0.7 - Better state backend support - Better modules support - Better state management - Added support for Workspaces (environments) - ...

CoreOS Tectonic - fundamentals - Self Hosted (bootkube) - Simpler installation flow - Sustainable cluster lifecycle - Goals? - Reduce components required on host - Reduce files written to host (config management) - Improve introspection - Upgrade Kubernetes, using Kubernetes API - Easier HA

Kubernetes Layers As defined in the Self-Hosted control plane proposal - Layer 0: Kubelet - Layer 1: Etcd - Layer 2: API server - Layer 3: Scheduler / Controller Manager / Proxy - Layer 4: DNS and Addons

Kubernetes full control plane Source: @kubernetesonarm

Bootkube - Self-hosting layers 1 to 4 since Q1 2017 - System hosted kubelet Updated through node annotations - Periodic snapshotting for recovery - Depends on external DNS / LB - Note test coverage for version skew between control-plane components

Other projects using bootkube (too late for us, but which you may consider) - Typhoon: Minimal and free Kubernetes distribution (Terraform) - Archon: Operator to manage Kubernetes clusters - bootkube-terraform: (part of typhoon) tf module to render bootkube assets with terraform - ...

Tectonic conclusion (at the time of testing) - Could not create multiple clusters from same Tectonic install dir (lacked support for Terraform workspaces) - Takes time to study, understand and make changes (+ how to keep changes in line with upstream) - Some Licensed components were still bootstrapped in vanilla mode (required some clean up) - ...

Kops - Define clusters in Code (manifests) - clusterSpec - instanceGroups - Manages secrets and TLS assets - Manages node boot sequence (which uses cluster manifests) - Manages HA Etcd cluster with better recovery models than tack - Defines how to bootstrap Addon: channels - DNS / CNI / ...

Kubernetes vs Kops State store (etcd) State store (s3 / gcs / … ) API Server CLI Client (kubectl) CLI Client (kops) Controllers Cloud Resources Manifests Cloud provider (AWS / GCP / …) Manifests *ignoring scheduler ... Cloud provider (AWS / GCP / TF …) upup/cloudup

Kops Boot Sequence - nodeup component (config from state-store: clusterSpec): - Installs packages / copies in assets - Sets up protokube (to manage etcd ~EBS volumes) - Sets up the OS for kubelet (renders /etc/kubernetes/manifests ) - kubelet requests PodCIDR from kube-controllermanager for CNI - kubelet configures and starts CRI with PodCIDR config - kubelet handles and reports workloads assigned by scheduler

Kops Addon Channels - Kubernetes Addons: Bundles of resources that provide specific functionality - dns - dashboard - autoscaler - Addon Channels Multiple addons versioned together: kind: Addons metadata: name: beekeeper spec: addons: - name: tiller.addons.k8s.io manifest: tiller.addons.k8s.io/k8s-1.7.yaml kubernetesVersion: '>=1.7.0' selector: k8s-addon: tiller.addons.k8s.io version: 2.7.2 #helm version - name: namespaces.honestbee.io manifest: namespaces.honestbee.io/k8s-1.7.yaml kubernetesVersion: '>=1.7.0' selector: k8s-addon: namespaces.honestbee.io version: 1.1.2 - name: ...

Kops channels - kops logic to bootstrap and manage core kubernetes addons - Compile as separate binary: /go/src/k8s.io/kops# make channels - Apply addons from local or upstream channels ~/# channels apply channel -f beekeeper/addons.yaml NAME CURRENT UPDATE namespaces.honestbee.io 1.1.1 1.1.2 tiller.addons.k8s.io 2.7.0 2.7.2 Must specify --yes to update

Kops channels List addons currently installed: ~/# channels get addons NAMESPACE NAME VERSION CHANNEL kube-system core.addons.k8s.io 1.4.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system dns-controller.addons.k8s.io 1.7.1 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system kube-dns.addons.k8s.io 1.14.5 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system limit-range.addons.k8s.io 1.5.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system storage-aws.addons.k8s.io 1.6.0 s3://state-store/cluster.hb.sg/addons/ bootstrap-channel.yaml kube-system kubernetes-dashboard 1.7.1 kubernetes-dashboard/addon.yaml kube-system monitoring-standalone 1.6.0 monitoring-standalone/addon.yaml kube-system tiller.addons.k8s.io 2.7.2 beekeeper/addons.yaml kube-system namespaces.honestbee.io 1.1.2 beekeeper/addons.yaml kube-system kube-state-metrics.addons.k8s.io v1.1.0-rc.0 beekeeper/addons.yaml

Kops Workshop See https://github.com/honestbee/terraform-workshop/tree/master/kops

Kops caveats - Don’t use the default CIDRs (overlap with Docker bip) - Use Declarative Manifests instead of Imperative kops cli - Make sure to reserve resources for kubelet / docker / system - ...

Honestbee Kops-infra - Agenda - Terminology - Kops manifests - Cluster definitions and maintenance - Terraform modules - kops - bootstrap (templated channels & bootstrap.sh) - VPC peers, endpoints and routing

Kops-Infra: Terminology - cluster: a single AZ Kubernetes cluster - cluster_group: a group of clusters. For example “staging clusters”, “svc clusters”, … A cluster_group represents shared infra (1 VPC, hosted stateful service endpoints (RDS / ElastiCache), … ) - utilities subnet - subnet used for Hosted Services such as RDS / ElastiCache / ...

Cluster Group with 2 Clusters

Kops Manifests Source control committed copy of Manifests. State store (s3 / gcs / … ) CLI Client (kops) Cloud Resources Manifests Cloud provider (AWS / GCP / TF …) upup/cloudup

Kops Manifests - Sample use cases For example: Change worker instance types

Kops Manifests - Sample use cases For example: Add tags to cluster Auto Scaling groups and EC2 Instances

Kops Manifests - Sample use cases For example: Enable alpha api flags

Kops Manifests - Sample use cases For example: Add permissions to the node or master instance Roles

Kops Manifests - Sample use cases For example: securing namespaced Tiller to listen localhost only

Kops Manifests - Sample use cases For example: configuring Tiller max history (templated)

Kops Manifests - Sample use cases For example: set kube-reserved and eviction

HB/Kops Infra - TF Modules - TF kops module - TF bootstrap module - VPC peering / endpoints / routing / … modules - vault / vault-controller

TF kops module Set up kops pre-requisites: - ssh keys - S3 bucket for Kops state-store - VPC to host cluster_groups - Route53 Hosted Zone for DNS resolution - Bastion Hosts - ...

TF Bootstrap Module Bootstrap cluster after creation - Templated Namespaces (RBAC, Registry Secrets, …) - channels - templated beekeeper channel - Autoscaler - Rendered Honestbee Namespace templates - Tiller - Render config for chart bootstrap - Bootstrap.sh (bootstrap charts) - Continue bootstrap using Helm (external / internal ingress, datadog, …)

Kops logo TBD

Thank you!