Two years of Kubernetes on AWS

This talk describes how things looked like trying to run Kubernetes on AWS two years ago (October 2016), how they are today (October 2018) and where the community is going.

Transcript

1. Two years of Kubernetes on AWS Raffaele Di Fazio - @x0rg Photo by José Alejandro Cuffi
2. whoami @x0rg
3. Agenda Photo by Estée Janssens • Two years ago • Today • Where are we going (and what is needed)
5. October 2016 • Kubernetes ~1 year old • No “standard” deployment architecture • Provisioners’ lack of features
6. 2016’s Architecture Photo by Anthony DELANOIX on Unsplash
8. No deployment architecture • Multi-AZ? • Multi-region? • Etcd on the master or not? • Multi master or not?
9. To multi-AZ or not to multi-AZ? • Multi AZ setups the majority • EBS volume are per AZ • Cluster autoscaler not zone aware
10. Multi region? • Not a thing with Kubernetes in 2016 • Big promises: Federation
11. From: Scale into Multi-Cloud with containers
12. Multi-master • Opinionated • Kops supporting single master by default • Kube-aws supporting multi-master • Availability vs Cost&Simplicity
13. Multi or single master • Multi master meant increased availability... • … and increased costs • In a true HA setup: 3 master, 5 etcd => 8 instances
14. etcd • The essential part of Kubernetes • Etcd version 2 • Bad performance • Needs special care (backup, compaction, …)
15. And of course… docker! • Several cases of docker “hanging” • On GKE:
16. Provisioning tools Photo by Thomas Kvistholt on Unsplash
17. Main alternatives • Kops (v1.4) • Kube-aws (v0.8) • Plenty of people starting their own provisioner • Kubeadm just started (launched with Kubernetes 1.4)
18. Kops v1.4 • Works pretty well • Already lots of code (node agent, etc.) • Tries to work across different clouds
19. Kops, getting there
20. Kube-AWS • Code relatively simple • Supported only CoreOS • Little community compared to Kops
21. More questions • Monitoring • Logging • Autoscaling (nodes vs pods) • Security best practices • Authn, Authz • Overlay network configuration • Load balancing / Ingress traffic (ELB, ELBv2) • Automated cluster updates
22. Photo by Franck V. on Unsplash
23. October 2018 • Core stable • New features • Even more provisioning tools… and a managed solution • Architecture (partially) stabilized
24. Core (kind of) stable • Deployments, configmaps, etc. are not updated so much • We still find some quirks in the basic part of the system
26. New features • One release every 3 months, no LTS • Lots of features! • Stability can be a challenge
28. New features - stay up to date! • Best approach: continuous updates • Use a managed solution: GKE, AKS, EKS • Build automation around OSS tools
29. Federation “Note: Federation V1, the current Kubernetes federation API which reuses the Kubernetes API resources ‘as is’, is currently considered alpha for many of its features. There is no clear path to evolve the API to GA; however, there is a Federation V2 effort in progress to implement a dedicated federation API apart from the Kubernetes API.”
30. Federation “Note: Federation V1, the current Kubernetes federation API which reuses the Kubernetes API resources ‘as is’, is currently considered alpha for many of its features. There is no clear path to evolve the API to GA; however, there is a Federation V2 effort in progress to implement a dedicated federation API apart from the Kubernetes API."
31. Provisioning tools Photo by 贝莉儿 NG on Unsplash
33. EKS • Managed HA control pane is a big deal • Relatively cheap: $0.20 per hour + cost of nodes • Vanilla Kubernetes
34. EKS (cont’d) • Still at version 1.10 with no announcements • Control plane updated without notice • Needs tooling and automation to upgrade worker nodes • https://github.com/weaveworks/eksctl
35. Kops • Matured and adopted • Has somehow internally an opinionated view ◦ Etcd on the masters + EBS volumes ◦ Docker version installed by node agent
36. Kops • Contains a lot of hidden experiment ◦ ClusterBundle ◦ EtcdManager • Stateful aware cluster upgrades... never merged
37. More provisioners • Kubeadm based ◦ Kubicorn ◦ Heptio quickstart • Effort on getting a community version around an API first approach => cluster API
38. https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/
40. Cluster API • Community effort • Rewriting all code from scratch • Still in very early stage
41. Where are we going Photo by Franck V. on Unsplash
42. Service meshes “A service mesh is a configurable infrastructure layer for a microservices application. It makes communication between service instances flexible, reliable, and fast.”
45. Application management • Kubernetes is a platform to build PaaSes • https://github.com/knative • https://github.com/zalando-incubator/stackset-controller • ...
46. What is needed Photo by James Baldwin on Unsplash
47. Contribute! • Fix things upstream, even if it is painful!
48. Share your horror stories
49. Share your horror stories • 101 ways to crash your cluster (youtube) • A million ways to crash your cluster • Fallacies of distributed computing with Kubernetes on AWS
50. That was all! @x0rg Photo by rawpixel on Unsplash