Two years of Kubernetes on AWS

Two years of Kubernetes on AWS

This talk describes how things looked like trying to run Kubernetes on AWS two years ago (October 2016), how they are today (October 2018) and where the community is going.

F020dfb4a8617b7acd70245afdc1fa02?s=128

Raffaele Di Fazio

October 16, 2018
Tweet

Transcript

  1. Two years of Kubernetes on AWS Raffaele Di Fazio -

    @x0rg Photo by José Alejandro Cuffi
  2. whoami @x0rg

  3. Agenda Photo by Estée Janssens • Two years ago •

    Today • Where are we going (and what is needed)
  4. None
  5. October 2016 • Kubernetes ~1 year old • No “standard”

    deployment architecture • Provisioners’ lack of features
  6. 2016’s Architecture Photo by Anthony DELANOIX on Unsplash

  7. None
  8. No deployment architecture • Multi-AZ? • Multi-region? • Etcd on

    the master or not? • Multi master or not?
  9. To multi-AZ or not to multi-AZ? • Multi AZ setups

    the majority • EBS volume are per AZ • Cluster autoscaler not zone aware
  10. Multi region? • Not a thing with Kubernetes in 2016

    • Big promises: Federation
  11. From: Scale into Multi-Cloud with containers

  12. Multi-master • Opinionated • Kops supporting single master by default

    • Kube-aws supporting multi-master • Availability vs Cost&Simplicity
  13. Multi or single master • Multi master meant increased availability...

    • … and increased costs • In a true HA setup: 3 master, 5 etcd => 8 instances
  14. etcd • The essential part of Kubernetes • Etcd version

    2 • Bad performance • Needs special care (backup, compaction, …)
  15. And of course… docker! • Several cases of docker “hanging”

    • On GKE:
  16. Provisioning tools Photo by Thomas Kvistholt on Unsplash

  17. Main alternatives • Kops (v1.4) • Kube-aws (v0.8) • Plenty

    of people starting their own provisioner • Kubeadm just started (launched with Kubernetes 1.4)
  18. Kops v1.4 • Works pretty well • Already lots of

    code (node agent, etc.) • Tries to work across different clouds
  19. Kops, getting there

  20. Kube-AWS • Code relatively simple • Supported only CoreOS •

    Little community compared to Kops
  21. More questions • Monitoring • Logging • Autoscaling (nodes vs

    pods) • Security best practices • Authn, Authz • Overlay network configuration • Load balancing / Ingress traffic (ELB, ELBv2) • Automated cluster updates
  22. Photo by Franck V. on Unsplash

  23. October 2018 • Core stable • New features • Even

    more provisioning tools… and a managed solution • Architecture (partially) stabilized
  24. Core (kind of) stable • Deployments, configmaps, etc. are not

    updated so much • We still find some quirks in the basic part of the system
  25. None
  26. New features • One release every 3 months, no LTS

    • Lots of features! • Stability can be a challenge
  27. None
  28. New features - stay up to date! • Best approach:

    continuous updates • Use a managed solution: GKE, AKS, EKS • Build automation around OSS tools
  29. Federation “Note: Federation V1, the current Kubernetes federation API which

    reuses the Kubernetes API resources ‘as is’, is currently considered alpha for many of its features. There is no clear path to evolve the API to GA; however, there is a Federation V2 effort in progress to implement a dedicated federation API apart from the Kubernetes API.”
  30. Federation “Note: Federation V1, the current Kubernetes federation API which

    reuses the Kubernetes API resources ‘as is’, is currently considered alpha for many of its features. There is no clear path to evolve the API to GA; however, there is a Federation V2 effort in progress to implement a dedicated federation API apart from the Kubernetes API."
  31. Provisioning tools Photo by 贝莉儿 NG on Unsplash

  32. None
  33. EKS • Managed HA control pane is a big deal

    • Relatively cheap: $0.20 per hour + cost of nodes • Vanilla Kubernetes
  34. EKS (cont’d) • Still at version 1.10 with no announcements

    • Control plane updated without notice • Needs tooling and automation to upgrade worker nodes • https://github.com/weaveworks/eksctl
  35. Kops • Matured and adopted • Has somehow internally an

    opinionated view ◦ Etcd on the masters + EBS volumes ◦ Docker version installed by node agent
  36. Kops • Contains a lot of hidden experiment ◦ ClusterBundle

    ◦ EtcdManager • Stateful aware cluster upgrades... never merged
  37. More provisioners • Kubeadm based ◦ Kubicorn ◦ Heptio quickstart

    • Effort on getting a community version around an API first approach => cluster API
  38. https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/

  39. None
  40. Cluster API • Community effort • Rewriting all code from

    scratch • Still in very early stage
  41. Where are we going Photo by Franck V. on Unsplash

  42. Service meshes “A service mesh is a configurable infrastructure layer

    for a microservices application. It makes communication between service instances flexible, reliable, and fast.”
  43. None
  44. None
  45. Application management • Kubernetes is a platform to build PaaSes

    • https://github.com/knative • https://github.com/zalando-incubator/stackset-controller • ...
  46. What is needed Photo by James Baldwin on Unsplash

  47. Contribute! • Fix things upstream, even if it is painful!

  48. Share your horror stories

  49. Share your horror stories • 101 ways to crash your

    cluster (youtube) • A million ways to crash your cluster • Fallacies of distributed computing with Kubernetes on AWS
  50. That was all! @x0rg Photo by rawpixel on Unsplash