KubeCon 2016: Self-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond

Transcript

1. Brandon Philips @brandonphilips | brandon@coreos.com | coreos.com Kubernetes v1.4 and Beyond Self-hosted, Scale, and Federation Stickers at the Front
2. 90+ Projects on GitHub, 1,000+ Contributors OPEN SOURCE Secure solutions, support plans, training + more ENTERPRISE CoreOS is Running the World’s Containers Linux
3. Scaling With etcd v3
4. etcd Overview • Introduced in 2013 by CoreOS • Primary datastore of Kubernetes • Replicated consistent key/value • Revisions and event stream • Auto-leader election for availability
5. http://play.etcd.io Quick Demo of Why etcd
6. - etcd v2.0 could support ~1000 node - Snapshot: ~200MB - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API
7. Performance - 1k Keys
8. Performance - etcd v2 - 600K keys Snapshot caused performance degradation
9. Performance - etcd v2 - 600K keys Snapshot triggered elections
10. Performance - Zookeeper Defaults
11. Performance - Zookeeper Defaults Snapshot triggered election
12. Performance - etcd3 /zk snapshot disabled
14. Memory - 512MB data - 2M 256B keys 10GB 2.4GB 0.8GB
15. Tons of Testing for Reliability
16. - etcd v2.0 could support ~1000 node - Snapshot: ~200MB - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API
17. etcd Operator • Operates etcd clusters on k8s • Handles common tasks: ◦ Resize ◦ Upgrade ◦ Backup ◦ Healing • Configured by resource
18. Self-Hosted The Big Ideas
19. gcc // gcc source code #include <stdio.h> int main() { compile_c(argv[1]); } gcc
20. go // golang source code package main import "os" func main() { compile_go(os.Args[1:]) } go
22. $ uname -s minix $ gcc linux.c
23. $ uname -s minix $ gcc linux.c
25. $ uname -s linux $ gcc linux.c
26. $ uname -s linux $ gcc linux.c
27. Self-Hosted Applied to Kubernetes
28. AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC v1.4.3 EC2 EC2 EC2 EBS EBS EBS
29. AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC v1.4.4 EC2 EC2 EC2 EBS EBS EBS
30. Azure APIs k8s API kubectl VirtualNet v1.4.3 VM VM VM PD PD PD
31. Azure APIs k8s API kubectl VirtualNet v1.4.4 VM VM VM PD PD PD
32. k8s API kubectl Ethernet v1.4.3 Bare Metal Bare Metal Bare Metal SAN SAN SAN
33. k8s API kubectl Ethernet v1.4.4 Bare Metal Bare Metal Bare Metal SAN SAN SAN
34. - Use Kubernetes compute, net, storage abstractions and APIs - Let infrastructure tools focus on infrastructure - Foundation to automate cluster upgrades Self-hosting Goals
35. Kubernetes Running Kubernetes $ kubectl -n kube-system get daemonsets NAME DESIRED CURRENT NODE-SELECTOR AGE kube-apiserver 2 2 master=true 4d kube-proxy 3 3 <none> 4d $ kubectl -n kube-system get deployments NAME DESIRED CURRENT UP-TO-DATE kube-controller-manager 1 1 1 kube-scheduler 2 2 2
36. $ kubectl edit deployments kube-scheduler Kubernetes Upgrading Kubernetes
37. Self-Hosted Trying it Out
38. Bootkube Quickstart Vagrant, AWS, GCE Quickstart github.com/kubernetes-incubator/bootkube
39. Self-Hosted Next Steps
40. KubeCeption: A Story of Self-Hosted Kubernetes Aaron Levy, CoreOS Tomorrow 9am Willow B For more information
41. Federation The Big Ideas
42. API Server • API server is state of Kubernetes • Designed for single failure domain • Can handle 1000s of machines
43. Federation API Server
44. Federation API Server • Holds state for multiple clusters • Spans multiple failure domains • Coordinates between clusters not for clusters
45. Federation Scheduling
46. Federation API kubelet apiserver scheduler controller
47. Federation API kubelet apiserver scheduler controller
48. Federation API kubelet apiserver scheduler controller kubelet apiserver scheduler controller
49. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx
50. Federation Services
51. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx
52. kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx nginx service
53. Federation High-Availability
54. LoadBalancer federation-apiserver New York San Francisco federation-controller etcd federation-apiserver federation-controller etcd LoadBalancer
55. LoadBalancer federation-apiserver New York San Francisco federation-controller Federation Control Plane A: A: etcd federation-apiserver federation-controller etcd LoadBalancer kubernetes-federation.example.com A: A:
56. LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12 A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer
57. LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12 A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer
58. Federation Support Today • Ingress/Services • Namespaces • ReplicaSets • Secrets • DNS (Cloud provider)
59. Federation Next Steps
60. Federation Next Steps • Try out the federation setup guides ◦ Kelsey's works easier than the official guides • Get involved with SIG Federation
61. A Shared Vision Self-hosted, etcd, Federation
62. Kubernetes Controlled Kubernetes
63. Leveraging Tested Designs
64. • Make self-hosted default for Kubernetes clusters • Easy to operate etcd for single and federated • Bring federation to market on these technologies My Goals from Here
65. tectonic.com/summit - @TectonicSummit December 12 & 13 2016 - New York City
66. Thank you! Brandon Philips @brandonphilips | brandon@coreos.com | tectonic.com We’re hiring in all departments! Email: careers@coreos.com Positions: coreos.com/ careers