Tweet
Tweet

Slide 1

Slide 1 text

Brandon Philips @brandonphilips | [email protected] | coreos.com Kubernetes v1.4 and Beyond Self-hosted, Scale, and Federation Stickers at the Front

Slide 2

Slide 2 text

90+ Projects on GitHub, 1,000+ Contributors OPEN SOURCE Secure solutions, support plans, training + more ENTERPRISE CoreOS is Running the World’s Containers Linux

Slide 3

Slide 3 text

Scaling With etcd v3

Slide 4

Slide 4 text

etcd Overview ● Introduced in 2013 by CoreOS ● Primary datastore of Kubernetes ● Replicated consistent key/value ● Revisions and event stream ● Auto-leader election for availability

Slide 5

Slide 5 text

http://play.etcd.io Quick Demo of Why etcd

Slide 6

Slide 6 text

- etcd v2.0 could support ~1000 node - Snapshot: ~200MB - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API

Slide 7

Slide 7 text

Performance - 1k Keys

Slide 8

Slide 8 text

Performance - etcd v2 - 600K keys Snapshot caused performance degradation

Slide 9

Slide 9 text

Performance - etcd v2 - 600K keys Snapshot triggered elections

Slide 10

Slide 10 text

Performance - Zookeeper Defaults

Slide 11

Slide 11 text

Performance - Zookeeper Defaults Snapshot triggered election

Slide 12

Slide 12 text

Performance - etcd3 /zk snapshot disabled

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

Memory - 512MB data - 2M 256B keys 10GB 2.4GB 0.8GB

Slide 15

Slide 15 text

Tons of Testing for Reliability

Slide 16

Slide 16 text

- etcd v2.0 could support ~1000 node - Snapshot: ~200MB - etcd v3.0 tested on ~3000 node + history - Snapshot: 1GB+ - Testing underway for 5000+ nodes Other parts of Kubernetes are scaling limit now etcd v3 + Kubernetes API

Slide 17

Slide 17 text

etcd Operator ● Operates etcd clusters on k8s ● Handles common tasks: ○ Resize ○ Upgrade ○ Backup ○ Healing ● Configured by resource

Slide 18

Slide 18 text

Self-Hosted The Big Ideas

Slide 19

Slide 19 text

gcc // gcc source code #include int main() { compile_c(argv[1]); } gcc

Slide 20

Slide 20 text

go // golang source code package main import "os" func main() { compile_go(os.Args[1:]) } go

Slide 21

Slide 21 text

No content

Slide 22

Slide 22 text

$ uname -s minix $ gcc linux.c

Slide 23

Slide 23 text

$ uname -s minix $ gcc linux.c

Slide 24

Slide 24 text

No content

Slide 25

Slide 25 text

$ uname -s linux $ gcc linux.c

Slide 26

Slide 26 text

$ uname -s linux $ gcc linux.c

Slide 27

Slide 27 text

Self-Hosted Applied to Kubernetes

Slide 28

Slide 28 text

AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC v1.4.3 EC2 EC2 EC2 EBS EBS EBS

Slide 29

Slide 29 text

AWS APIs kube-aws kargo kops k8s API kubectl AWS VPC v1.4.4 EC2 EC2 EC2 EBS EBS EBS

Slide 30

Slide 30 text

Azure APIs k8s API kubectl VirtualNet v1.4.3 VM VM VM PD PD PD

Slide 31

Slide 31 text

Azure APIs k8s API kubectl VirtualNet v1.4.4 VM VM VM PD PD PD

Slide 32

Slide 32 text

k8s API kubectl Ethernet v1.4.3 Bare Metal Bare Metal Bare Metal SAN SAN SAN

Slide 33

Slide 33 text

k8s API kubectl Ethernet v1.4.4 Bare Metal Bare Metal Bare Metal SAN SAN SAN

Slide 34

Slide 34 text

- Use Kubernetes compute, net, storage abstractions and APIs - Let infrastructure tools focus on infrastructure - Foundation to automate cluster upgrades Self-hosting Goals

Slide 35

Slide 35 text

Kubernetes Running Kubernetes $ kubectl -n kube-system get daemonsets NAME DESIRED CURRENT NODE-SELECTOR AGE kube-apiserver 2 2 master=true 4d kube-proxy 3 3 4d $ kubectl -n kube-system get deployments NAME DESIRED CURRENT UP-TO-DATE kube-controller-manager 1 1 1 kube-scheduler 2 2 2

Slide 36

Slide 36 text

$ kubectl edit deployments kube-scheduler Kubernetes Upgrading Kubernetes

Slide 37

Slide 37 text

Self-Hosted Trying it Out

Slide 38

Slide 38 text

Bootkube Quickstart Vagrant, AWS, GCE Quickstart github.com/kubernetes-incubator/bootkube

Slide 39

Slide 39 text

Self-Hosted Next Steps

Slide 40

Slide 40 text

KubeCeption: A Story of Self-Hosted Kubernetes Aaron Levy, CoreOS Tomorrow 9am Willow B For more information

Slide 41

Slide 41 text

Federation The Big Ideas

Slide 42

Slide 42 text

API Server ● API server is state of Kubernetes ● Designed for single failure domain ● Can handle 1000s of machines

Slide 43

Slide 43 text

Federation API Server

Slide 44

Slide 44 text

Federation API Server ● Holds state for multiple clusters ● Spans multiple failure domains ● Coordinates between clusters not for clusters

Slide 45

Slide 45 text

Federation Scheduling

Slide 46

Slide 46 text

Federation API kubelet apiserver scheduler controller

Slide 47

Slide 47 text

Federation API kubelet apiserver scheduler controller

Slide 48

Slide 48 text

Federation API kubelet apiserver scheduler controller kubelet apiserver scheduler controller

Slide 49

Slide 49 text

kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx

Slide 50

Slide 50 text

Federation Services

Slide 51

Slide 51 text

kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx

Slide 52

Slide 52 text

kubelet apiserver scheduler controller nginx Federation API kubelet apiserver scheduler controller nginx nginx service

Slide 53

Slide 53 text

Federation High-Availability

Slide 54

Slide 54 text

LoadBalancer federation-apiserver New York San Francisco federation-controller etcd federation-apiserver federation-controller etcd LoadBalancer

Slide 55

Slide 55 text

LoadBalancer federation-apiserver New York San Francisco federation-controller Federation Control Plane A: A: etcd federation-apiserver federation-controller etcd LoadBalancer kubernetes-federation.example.com A: A:

Slide 56

Slide 56 text

LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12 A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer

Slide 57

Slide 57 text

LoadBalancer federation-apiserver New York San Francisco federation-controller kubernetes-federation.example.com A: 50.31.17.12 A: 72.45.32.48 etcd federation-apiserver federation-controller etcd LoadBalancer

Slide 58

Slide 58 text

Federation Support Today ● Ingress/Services ● Namespaces ● ReplicaSets ● Secrets ● DNS (Cloud provider)

Slide 59

Slide 59 text

Federation Next Steps

Slide 60

Slide 60 text

Federation Next Steps ● Try out the federation setup guides ○ Kelsey's works easier than the official guides ● Get involved with SIG Federation

Slide 61

Slide 61 text

A Shared Vision Self-hosted, etcd, Federation

Slide 62

Slide 62 text

Kubernetes Controlled Kubernetes

Slide 63

Slide 63 text

Leveraging Tested Designs

Slide 64

Slide 64 text

● Make self-hosted default for Kubernetes clusters ● Easy to operate etcd for single and federated ● Bring federation to market on these technologies My Goals from Here

Slide 65

Slide 65 text

tectonic.com/summit - @TectonicSummit December 12 & 13 2016 - New York City

Slide 66

Slide 66 text

Thank you! Brandon Philips @brandonphilips | [email protected] | tectonic.com We’re hiring in all departments! Email: [email protected] Positions: coreos.com/ careers