Two years of Kubernetes on AWS

Slide 1

Slide 1 text

Two years of Kubernetes on AWS Raffaele Di Fazio - @x0rg Photo by José Alejandro Cuffi

Slide 2

Slide 2 text

whoami @x0rg

Slide 3

Slide 3 text

Agenda Photo by Estée Janssens ● Two years ago ● Today ● Where are we going (and what is needed)

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

October 2016 ● Kubernetes ~1 year old ● No “standard” deployment architecture ● Provisioners’ lack of features

Slide 6

Slide 6 text

2016’s Architecture Photo by Anthony DELANOIX on Unsplash

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No deployment architecture ● Multi-AZ? ● Multi-region? ● Etcd on the master or not? ● Multi master or not?

Slide 9

Slide 9 text

To multi-AZ or not to multi-AZ? ● Multi AZ setups the majority ● EBS volume are per AZ ● Cluster autoscaler not zone aware

Slide 10

Slide 10 text

Multi region? ● Not a thing with Kubernetes in 2016 ● Big promises: Federation

Slide 11

Slide 11 text

From: Scale into Multi-Cloud with containers

Slide 12

Slide 12 text

Multi-master ● Opinionated ● Kops supporting single master by default ● Kube-aws supporting multi-master ● Availability vs Cost&Simplicity

Slide 13

Slide 13 text

Multi or single master ● Multi master meant increased availability... ● … and increased costs ● In a true HA setup: 3 master, 5 etcd => 8 instances

Slide 14

Slide 14 text

etcd ● The essential part of Kubernetes ● Etcd version 2 ● Bad performance ● Needs special care (backup, compaction, …)

Slide 15

Slide 15 text

And of course… docker! ● Several cases of docker “hanging” ● On GKE:

Slide 16

Slide 16 text

Provisioning tools Photo by Thomas Kvistholt on Unsplash

Slide 17

Slide 17 text

Main alternatives ● Kops (v1.4) ● Kube-aws (v0.8) ● Plenty of people starting their own provisioner ● Kubeadm just started (launched with Kubernetes 1.4)

Slide 18

Slide 18 text

Kops v1.4 ● Works pretty well ● Already lots of code (node agent, etc.) ● Tries to work across different clouds

Slide 19

Slide 19 text

Kops, getting there

Slide 20

Slide 20 text

Kube-AWS ● Code relatively simple ● Supported only CoreOS ● Little community compared to Kops

Slide 21

Slide 21 text

More questions ● Monitoring ● Logging ● Autoscaling (nodes vs pods) ● Security best practices ● Authn, Authz ● Overlay network configuration ● Load balancing / Ingress traffic (ELB, ELBv2) ● Automated cluster updates

Slide 22

Slide 22 text

Photo by Franck V. on Unsplash

Slide 23

Slide 23 text

October 2018 ● Core stable ● New features ● Even more provisioning tools… and a managed solution ● Architecture (partially) stabilized

Slide 24

Slide 24 text

Core (kind of) stable ● Deployments, configmaps, etc. are not updated so much ● We still find some quirks in the basic part of the system

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

New features ● One release every 3 months, no LTS ● Lots of features! ● Stability can be a challenge

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

New features - stay up to date! ● Best approach: continuous updates ● Use a managed solution: GKE, AKS, EKS ● Build automation around OSS tools

Slide 29

Slide 29 text

Federation “Note: Federation V1, the current Kubernetes federation API which reuses the Kubernetes API resources ‘as is’, is currently considered alpha for many of its features. There is no clear path to evolve the API to GA; however, there is a Federation V2 effort in progress to implement a dedicated federation API apart from the Kubernetes API.”

Slide 30

Slide 30 text

Slide 31

Slide 31 text

Provisioning tools Photo by 贝莉儿 NG on Unsplash

Slide 32

Slide 32 text

No content

Slide 33

Slide 33 text

EKS ● Managed HA control pane is a big deal ● Relatively cheap: $0.20 per hour + cost of nodes ● Vanilla Kubernetes

Slide 34

Slide 34 text

EKS (cont’d) ● Still at version 1.10 with no announcements ● Control plane updated without notice ● Needs tooling and automation to upgrade worker nodes ● https://github.com/weaveworks/eksctl

Slide 35

Slide 35 text

Kops ● Matured and adopted ● Has somehow internally an opinionated view ○ Etcd on the masters + EBS volumes ○ Docker version installed by node agent

Slide 36

Slide 36 text

Kops ● Contains a lot of hidden experiment ○ ClusterBundle ○ EtcdManager ● Stateful aware cluster upgrades... never merged

Slide 37

Slide 37 text

More provisioners ● Kubeadm based ○ Kubicorn ○ Heptio quickstart ● Effort on getting a community version around an API first approach => cluster API

Slide 38

Slide 38 text

https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Cluster API ● Community effort ● Rewriting all code from scratch ● Still in very early stage

Slide 41

Slide 41 text

Where are we going Photo by Franck V. on Unsplash

Slide 42

Slide 42 text

Service meshes “A service mesh is a configurable infrastructure layer for a microservices application. It makes communication between service instances flexible, reliable, and fast.”

Slide 43

Slide 43 text

No content

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Application management ● Kubernetes is a platform to build PaaSes ● https://github.com/knative ● https://github.com/zalando-incubator/stackset-controller ● ...

Slide 46

Slide 46 text

What is needed Photo by James Baldwin on Unsplash

Slide 47

Slide 47 text

Contribute! ● Fix things upstream, even if it is painful!

Slide 48

Slide 48 text

Share your horror stories

Slide 49

Slide 49 text

Share your horror stories ● 101 ways to crash your cluster (youtube) ● A million ways to crash your cluster ● Fallacies of distributed computing with Kubernetes on AWS

Slide 50

Slide 50 text

That was all! @x0rg Photo by rawpixel on Unsplash