$30 off During Our Annual Pro Sale. View Details »

KubeCon 2016: Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond

KubeCon 2016: Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond

Brandon Philips

November 08, 2016
Tweet

More Decks by Brandon Philips

Other Decks in Programming

Transcript

  1. Brandon Philips @brandonphilips | brandon@coreos.com | coreos.com Kubernetes v1.4 and

    Beyond Self-hosted, Scale, and Federation Stickers at the Front
  2. 90+ Projects on GitHub, 1,000+ Contributors OPEN SOURCE Secure solutions,

    support plans, training + more ENTERPRISE CoreOS is Running the World’s Containers Linux
  3. Scaling With etcd v3

  4. etcd Overview • Introduced in 2013 by CoreOS • Primary

    datastore of Kubernetes • Replicated consistent key/value • Revisions and event stream • Auto-leader election for availability
  5. http://play.etcd.io Quick Demo of Why etcd

  6. - etcd v2.0 could support ~1000 node - Snapshot: ~200MB

    - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API
  7. Performance - 1k Keys

  8. Performance - etcd v2 - 600K keys Snapshot caused performance

    degradation
  9. Performance - etcd v2 - 600K keys Snapshot triggered elections

  10. Performance - Zookeeper Defaults

  11. Performance - Zookeeper Defaults Snapshot triggered election

  12. Performance - etcd3 /zk snapshot disabled

  13. None
  14. Memory - 512MB data - 2M 256B keys 10GB 2.4GB

    0.8GB
  15. Tons of Testing for Reliability

  16. - etcd v2.0 could support ~1000 node - Snapshot: ~200MB

    - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API
  17. etcd Operator • Operates etcd clusters on k8s • Handles

    common tasks: ◦ Resize ◦ Upgrade ◦ Backup ◦ Healing • Configured by resource
  18. Self-Hosted The Big Ideas

  19. gcc // gcc source code #include <stdio.h> int main() {

    compile_c(argv[1]); } gcc
  20. go // golang source code package main import "os" func

    main() { compile_go(os.Args[1:]) } go
  21. None
  22. $ uname -s minix $ gcc linux.c

  23. $ uname -s minix $ gcc linux.c

  24. None
  25. $ uname -s linux $ gcc linux.c

  26. $ uname -s linux $ gcc linux.c

  27. Self-Hosted Applied to Kubernetes

  28. AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC

    v1.4.3 EC2 EC2 EC2 EBS EBS EBS
  29. AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC

    v1.4.4 EC2 EC2 EC2 EBS EBS EBS
  30. Azure APIs k8s API kubectl VirtualNet v1.4.3 VM VM VM

    PD PD PD
  31. Azure APIs k8s API kubectl VirtualNet v1.4.4 VM VM VM

    PD PD PD
  32. k8s API kubectl Ethernet v1.4.3 Bare Metal Bare Metal Bare

    Metal SAN SAN SAN
  33. k8s API kubectl Ethernet v1.4.4 Bare Metal Bare Metal Bare

    Metal SAN SAN SAN
  34. - Use Kubernetes compute, net, storage abstractions and APIs -

    Let infrastructure tools focus on infrastructure - Foundation to automate cluster upgrades Self-hosting Goals
  35. Kubernetes Running Kubernetes $ kubectl -n kube-system get daemonsets NAME

    DESIRED CURRENT NODE-SELECTOR AGE kube-apiserver 2 2 master=true 4d kube-proxy 3 3 <none> 4d $ kubectl -n kube-system get deployments NAME DESIRED CURRENT UP-TO-DATE kube-controller-manager 1 1 1 kube-scheduler 2 2 2
  36. $ kubectl edit deployments kube-scheduler Kubernetes Upgrading Kubernetes

  37. Self-Hosted Trying it Out

  38. Bootkube Quickstart Vagrant, AWS, GCE Quickstart github.com/kubernetes-incubator/bootkube

  39. Self-Hosted Next Steps

  40. KubeCeption: A Story of Self-Hosted Kubernetes Aaron Levy, CoreOS Tomorrow

    9am Willow B For more information
  41. Federation The Big Ideas

  42. API Server • API server is state of Kubernetes •

    Designed for single failure domain • Can handle 1000s of machines
  43. Federation API Server

  44. Federation API Server • Holds state for multiple clusters •

    Spans multiple failure domains • Coordinates between clusters not for clusters
  45. Federation Scheduling

  46. Federation API kubelet apiserver scheduler controller

  47. Federation API kubelet apiserver scheduler controller

  48. Federation API kubelet apiserver scheduler controller kubelet apiserver scheduler controller

  49. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler

    controller nginx
  50. Federation Services

  51. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler

    controller nginx
  52. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler

    controller nginx nginx service
  53. Federation High-Availability

  54. LoadBalancer federation-apiserver New York San Francisco federation-controller etcd federation-apiserver federation-controller

    etcd LoadBalancer
  55. LoadBalancer federation-apiserver New York San Francisco federation-controller Federation Control Plane

    A: A: etcd federation-apiserver federation-controller etcd LoadBalancer kubernetes-federation.example.com A: A:
  56. LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12

    A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer
  57. LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12

    A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer
  58. Federation Support Today • Ingress/Services • Namespaces • ReplicaSets •

    Secrets • DNS (Cloud provider)
  59. Federation Next Steps

  60. Federation Next Steps • Try out the federation setup guides

    ◦ Kelsey's works easier than the official guides • Get involved with SIG Federation
  61. A Shared Vision Self-hosted, etcd, Federation

  62. Kubernetes Controlled Kubernetes

  63. Leveraging Tested Designs

  64. • Make self-hosted default for Kubernetes clusters • Easy to

    operate etcd for single and federated • Bring federation to market on these technologies My Goals from Here
  65. tectonic.com/summit - @TectonicSummit December 12 & 13 2016 - New

    York City
  66. Thank you! Brandon Philips @brandonphilips | brandon@coreos.com | tectonic.com We’re

    hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers