Bootstrapping Kubernetes on CoreOS

Support for my talk at the CoreOS Meetup in Munich (May 2015)

Transcript

1. COREOS MEETUP MUNICH MAY 2015 Bootstrapping Kubernetes on CoreOS A helmsman for our ship
2. COREOS MEETUP MUNICH MAY 2015 Who I am Antoine Cotten Technical operations engineer @ Freeletics GmbH What I do • Virtualisation, private clouds • Automation, conf management • Docker orchestration • Continuous integration
3. COREOS MEETUP MUNICH MAY 2015 http://jwalczyk.blogspot.de
4. COREOS MEETUP MUNICH MAY 2015 Big picture Docker ecosystem http://comp.photo777.org
5. COREOS MEETUP MUNICH MAY 2015 Docker ecosystem Networking Scheduling Storage Scaling Fault tolerance Load-balancing Composition High availability etc… Challenges
6. COREOS MEETUP MUNICH MAY 2015 Docker ecosystem
7. COREOS MEETUP MUNICH MAY 2015 Contents Introduction to Kubernetes Integration with CoreOS How-Tos
8. COREOS MEETUP MUNICH MAY 2015 Introduction to Kubernetes
9. COREOS MEETUP MUNICH MAY 2015 “How would you design your infrastructure if you couldn't login? Ever.” –Kelsey Hightower Product Manager @ CoreOS Introduction to Kubernetes
10. COREOS MEETUP MUNICH MAY 2015 Components Introduction to Kubernetes master minion container runtime Control plane Node
11. COREOS MEETUP MUNICH MAY 2015 Components Introduction to Kubernetes master API key/value store scheduler controller manager minion kubelet container runtime proxy Control plane Node
12. COREOS MEETUP MUNICH MAY 2015 Work units Introduction to Kubernetes Node pod cont cont cont Pod Collocated group of containers
13. COREOS MEETUP MUNICH MAY 2015 Work units Introduction to Kubernetes Node Node Node Node Replication controller pod pod pod pod pod Replication controller Ensures n-replicas running at one time
14. COREOS MEETUP MUNICH MAY 2015 Node Node Node Node Work units Introduction to Kubernetes Service Service pod pod pod pod pod Network abstraction, maps endpoint to set of pods
15. COREOS MEETUP MUNICH MAY 2015 Kube up! ~/D/kubernetes-0.17.1 ❯❯❯ cluster/kube-up.sh Starting cluster using provider: aws Creating vpc. Creating subnet. Creating Internet Gateway. Associating route table. Configuring route table. Adding route to route table. Creating security group. Starting Master Waiting for master to be ready Attempt to check for master node [master running @54.76.244.21] Attempt to check for salt-master [salt-master running] Starting Minion (kube17-minion-1) Starting Minion (kube17-minion-2) Sleeping for 3 seconds... Minion kube17-minion-1 running Minion kube17-minion-2 running Waiting 3 minutes for cluster to settle Kubernetes cluster created. Wrote config for aws_kube17 to /Users/acotten/.kube/config Sanity checking cluster... Attempt 1 to check Docker on node kube17-minion-1 @ 54.77.104.208 ... [working] Attempt 1 to check Docker on node kube17-minion-2 @ 52.16.246.103 ... [working] Kubernetes cluster is running. The master is running at: https://54.76.244.21 The user name and password to use is located in /Users/acotten/.kube/config. Cluster validation succeeded Introduction to Kubernetes
16. COREOS MEETUP MUNICH MAY 2015 Integration with CoreOS
17. COREOS MEETUP MUNICH MAY 2015 Project backers Integration with CoreOS $12M
18. COREOS MEETUP MUNICH MAY 2015
19. COREOS MEETUP MUNICH MAY 2015
20. COREOS MEETUP MUNICH MAY 2015 Project backers Integration with CoreOS Nick Weaver - CoreOS Fest - 4th May
21. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage
22. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage /registry/namespaces /registry/services /registry/serviceaccounts /registry/ranges /registry/minions /registry/pods /registry/secrets /registry/controllers /registry/events key/value store etcdctl ls /registry ~ ❯❯❯
23. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage etcdctl ls /registry/minions /registry/minions/node1 /registry/minions/node2 { "kind": "Node", "apiVersion": "v1beta3", "metadata": { "name": "node1", "selfLink": "/api/v1beta3/nodes/node1", "uid": "319de440-fe0f-11e4-9c86-0a4b42026aad", "resourceVersion": "264159", "creationTimestamp": "2015-05-19T10:09:53Z" }, "spec": { "externalID": "i-709f9196" }, "status": { "capacity": { "cpu": "2", "memory": "4050736Ki" }, ... key/value store ~ ❯❯❯ ~ ❯❯❯ etcdctl get /registry/minions/node1
24. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage kubectl get nodes -o json key/value store ~ ❯❯❯ { "kind": "Node", "apiVersion": "v1beta3", "metadata": { "name": "node1", "selfLink": "/api/v1beta3/nodes/node1", "uid": "319de440-fe0f-11e4-9c86-0a4b42026aad", "resourceVersion": "264159", "creationTimestamp": "2015-05-19T10:09:53Z" }, "spec": { "externalID": "i-709f9196" }, "status": { "capacity": { "cpu": "2", "memory": "4050736Ki" }, ...
25. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage ce2a822cea30bfca: name=default peerURLs=http://localhost:2380 clientURLs=http://localhost:2379 master API key/value store scheduler controller manager minion kubelet container runtime proxy Control plane minion kubelet container runtime proxy Nodes etcdctl member list ~ ❯❯❯
26. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Storage etcdctl member list ce2a822cea30bfca: name=master peerURLs=http://10.0.0.1:2380 clientURLs=http://10.0.0.1:2379 dczfhvo9fga8fmu8: name=node1 peerURLs=http://10.0.0.2:2380 clientURLs=http://10.0.0.2:2379 lao9h7jolp5gn19e: name=node2 peerURLs=http://10.0.0.3:2380 clientURLs=http://10.0.0.3:2379 master API key/value store scheduler controller manager minion kubelet container runtime proxy Control plane Nodes minion kubelet container runtime proxy key/value store key/value store ~ ❯❯❯
27. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime
28. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime
29. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime container runtime Release 0.17.0 […] • kubelet: Add container runtime option for rkt. #7952 (yifan-gu) • kubelet/rkt: Remove build label. #7916 (yifan-gu) • kubelet/rkt: Add GetPods() for rkt. #7599 (yifan-gu) • kubelet/rkt: Add KillPod() and GetPodStatus() for rkt. #7605 (yifan-gu) • kubelet/rkt: Add basic rkt runtime routines. #7465 (yifan-gu) • kubelet/rkt: Add podInfo. #7555 (yifan-gu) • kubelet/rkt: Add routines for converting kubelet pod to rkt pod. #7543 (yifan-gu) • kubelet/rkt: Add RunPod() for rkt. #7589 (yifan-gu) • kubelet/rkt: Add RunInContainer()/ExecInContainer()/PortForward(). #7553 (yifan-gu) • kubelet/rkt: Add SyncPod() to rkt. #7611 (yifan-gu) • kubelet: Add container runtime option for rkt. #7900 (yifan-gu) • kubelet/rkt: Add docker prefix to image string. #7803 (yifan-gu) • kubelet/rkt: Inject dependencies to rkt. #7849 (yifan-gu) • kubelet/rkt: Remove dependencies on rkt.store #7859 (yifan-gu)
30. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime Usage of kubelet: --address=0.0.0.0: The IP address for the info server to serve on (set to 0.0.0.0 for all interfaces) --allow_dynamic_housekeeping=true: Whether to allow the housekeeping interval to be dynamic --allow-privileged=false: If true, allow containers to request privileged mode. [default=false] --alsologtostderr=false: log to standard error as well as files --api-servers=[]: List of Kubernetes API servers for publishing events, and reading pods and services […] --container_runtime="docker": The container runtime to use. Possible values: 'docker', ‘rkt'. […] kubelet --help ~ ❯❯❯
31. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE api-1 10.144.71.27 10.0.21.55 app=api,branch=master Running 24 minutes api freeletics/rails Running 24 m api-2 10.144.71.28 10.0.21.55 app=api,branch=master Running 24 minutes api freeletics/rails Running 24 m api-3 10.144.51.24 10.0.21.56 app=api,branch=master Running 25 minutes api freeletics/rails Running 25 m api-4 10.144.51.25 10.0.21.56 app=api,branch=master Running 24 minutes api freeletics/rails Running 24 m web-1 10.144.71.23 10.0.21.55 app=web,branch=master Running 50 minutes web freeletics/web Running 50 m web-2 10.144.51.23 10.0.21.56 app=web,branch=master Running 50 minutes web freeletics/web Running 50 m kubectl get pods ~ ❯❯❯
32. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Container runtime kubectl get pods ~ ❯❯❯ No ACI* support (yet), Docker images only Need recent built (rkt pull-request merged recently) *App Container Images Current release (v0.17.1): POD IP CONTAINER(S) IMAGE(S) HOST LABELS STATUS CREATED MESSAGE api-1 10.144.71.27 10.0.21.55 app=api,branch=master Pending 24 minutes api freeletics/rails Pending 24 m api-2 10.144.71.28 10.0.21.55 app=api,branch=master Pending 24 minutes api freeletics/rails Pending 24 m api-3 10.144.51.24 10.0.21.56 app=api,branch=master Pending 25 minutes api freeletics/rails Pending 25 m api-4 10.144.51.25 10.0.21.56 app=api,branch=master Pending 24 minutes api freeletics/rails Pending 24 m web-1 10.144.71.23 10.0.21.55 app=web,branch=master Pending 50 minutes web freeletics/web Pending 50 m web-2 10.144.51.23 10.0.21.56 app=web,branch=master Pending 50 minutes web freeletics/web Pending 50 m
33. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Networking all containers can communicate with all other containers without NAT all nodes can communicate with all containers (and vice-versa) without NAT the IP that a container sees itself as is the same IP that others see it as Kubernetes fundamental requirements:
34. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Networking minion @ 172.20.0.10 container runtime cont cont cont minion @ 172.20.0.11 container runtime cont cont cont BRIDGE @10.244.1.0/24 @10.244.2.0/24
35. COREOS MEETUP MUNICH MAY 2015 flannel
36. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Networking flannel start early-docker config key in etcd mk /coreos.com/network/config { "Network": “10.1.0.0/16", "Backend": {"Type": “vxlan"} } start docker 1. Read etcd net/conf key 2. Create TUN/TAP device 3. Create bridge (/24) 4. Export config (subnet.env) 1. Source subnet.env 2. Start with --bip=$SUBNET
37. COREOS MEETUP MUNICH MAY 2015 flannel 10.1.0.0/16 dev flannel0 proto kernel scope link src 10.1.15.0 10.1.0.0/16 dev flannel0 proto kernel scope link src 10.1.20.0
38. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Networking minion @ 172.20.0.10 container runtime cont cont cont minion @ 172.20.0.11 container runtime cont cont cont BRIDGE @10.244.1.0/24 @10.244.2.0/24 flannel automatic!
39. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Networking minion @ 172.20.0.10 container runtime cont cont cont minion @ 172.20.0.11 container runtime cont cont cont BRIDGE @10.244.1.0/24 @10.244.2.0/24 flannel automatic!
40. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Node registration kube-register https://github.com/kelseyhightower/kube-register
41. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Node registration fleetctl list-machines MACHINE IP METADATA 0cf3e4b702 10.0.0.2 k8srole=node d643df618d 10.0.0.3 k8srole=node d9bef61cea 10.0.0.1 k8srole=master controller manager kube-register \ --api-endpoint=localhost:8080 \ --fleet-endpoint=unix:///var/run/fleet.sock \ --metadata=k8srole=node registered machine: 10.0.0.2 registered machine: 10.0.0.3 ~ ❯❯❯ ~ ❯❯❯
42. COREOS MEETUP MUNICH MAY 2015 Core components Integration with CoreOS Node registration controller manager NAME LABELS STATUS 10.0.0.2 <none> Ready 10.0.0.3 <none> Ready kubectl get nodes ~ ❯❯❯
43. COREOS MEETUP MUNICH MAY 2015 How-Tos
44. COREOS MEETUP MUNICH MAY 2015 References https://coreos.com/blog/running-kubernetes-example-on-CoreOS-part-1/ Running Kubernetes Example on CoreOS https://github.com/GoogleCloudPlatform/kubernetes/ Kubernetes GitHub repo part-2/ https://github.com/kelseyhightower/kubestack/ Kubestack, manage Kubernetes with Packer and Terraform