Kubernetes Anywhere! Ilya Dmitrichenko Kubernetes Anywhere: Cluster portability with Weave Net @errordeveloper 20/01/2016 

Problem Outline • Overview of Kubernetes cluster architecture • Decisions to be made when deploying a cluster • Variety of existing examples on the Internet 

Problem Outline • Overview of Kubernetes cluster architecture • Decisions to be made when deploying a cluster • Variety of existing examples on the Internet 

Problem Outline • Overview of Kubernetes cluster architecture • Decisions to be made when deploying a cluster • Variety of existing examples on the Internet 

Problem Outline • Let’s make cluster deployment • simpler to implement in any environment • more robust and easier to manage 

Architecture Overview 

Main Kubernetes Cluster Components 

etcd1 etcd2 etcd3 

API Server [kube-apiserver] etcd1 etcd2 etcd3 

Scheduler [kube-scheduler] API Server [kube-apiserver] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes [kube-proxy] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes [kube-proxy] [kubelet] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes [kube-proxy] [kubelet] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes [kube-proxy] [kubelet] etcd1 etcd2 etcd3 

Controller Manager [kube-controller-manager] Scheduler [kube-scheduler] API Server [kube-apiserver] Cluster Nodes [kube-proxy] [kubelet] etcd1 etcd2 etcd3 

kube-apiserver [...] \ --etcd-servers=http://etcd1:2379,http://etcd2:2379,http://etcd3:2379 kube-controller-manager [...] \ --master=http://localhost:8080 kube-scheduler [...] \ --master=http://localhost:8080 kube-proxy [...] \ --master=http://kube0:8080 kubelet [...] \ --api-servers=http://kube0:8080 etcd1:2379 etcd2:2379 etcd3:2379 Cluster Component Discovery Options 

kube-apiserver [...] \ --etcd-servers=http://etcd1:2379,http://etcd2:2379,http://etcd3:2379 kube-controller-manager [...] \ --master=http://localhost:8080 kube-scheduler [...] \ --master=http://localhost:8080 kube-proxy [...] \ --master=http://kube0:8080 kubelet [...] \ --api-servers=http://kube0:8080 etcd1:2379 etcd2:2379 etcd3:2379 etcdX.cluster.internal Master kube0.cluster.internal [172.17.28.0.14] Nodes kubeX.cluster.internal [172.17.28.0.0/16] 

Deployment Choices • Which cloud? May be PaaS? What instance size? • Overlay network? What’s the L3 vs L2 again? • Virtualised or may be actually bare-metal? 

Deployment Choices • Can I use my favourite Linux distribution? • Is CoreOS so much better or may be Atomic is? • Deployment automation? I might like to try X… 

More Questions!? • How to CI/CD? …I still barely understand those! • May be Mesos would be easier? • How to docker my DB? How do I backup etcd? 

Existing Examples github.com/kubernetes/kubernetes 

repo: github.com/kubernetes/kubernetes code: cluster/rackspace/cloud-config/node-cloud-config.yaml - path: /opt/bin/regen-apiserver-list.sh permissions: 0755 content: | #!/bin/sh m=$(echo $(etcdctl ls --recursive /corekube/apiservers \ | cut -d/ -f4 | sort) | tr ' ' ,) mkdir -p /run/kubelet echo "APISERVER_IPS=$m" > /run/kubelet/apiservers.env echo “FIRST_APISERVER_URL=https://${m%%\,*}:6443” \ >> /run/kubelet/apiservers.env 

repo: github.com/kubernetes/kubernetes code: cluster/rackspace/cloud-config/node-cloud-config.yaml - name: kubelet-sighup.path command: start content: | [Path] PathChanged=/run/kubelet/apiservers.env - name: kubelet-sighup.service command: start content: | [Service] ExecStart=/usr/bin/pkill -SIGHUP -f kubelet 

repo: github.com/kubernetes/kubernetes code: cluster/rackspace/cloud-config/node-cloud-config.yaml - name: kube-proxy-sighup.path command: start content: | [Path] PathChanged=/run/kubelet/apiservers.env - name: kube-proxy-sighup.service command: start content: | [Service] ExecStart=/usr/bin/pkill -SIGHUP -f kube-proxy 

repo: github.com/kubernetes/kubernetes code: cluster/rackspace/cloud-config/kubelet.service ... [Service] EnvironmentFile=/run/kubelet/apiservers.env ExecStartPre=/run/config-kubelet.sh ExecStart=/opt/bin/kubelet \ --address=$private_ipv4 \ --api-servers=${FIRST_APISERVER_URL} \ ... 

repo: github.com/kubernetes/kubernetes code: cluster/aws/util.sh function get_elbs_in_vpc { # ELB doesn't seem to be on the same platform as the rest of AWS; doesn't support filtering aws elb --output json describe-load-balancers | \ python -c "import json,sys; lst = [str(lb['LoadBalancerName']) for lb in json.load(sys.stdin) ['LoadBalancerDescriptions'] if lb['VPCId'] == '$1']; print('\n'.join(lst))" } 

repo: github.com/kubernetes/kubernetes code: cluster/aws/util.sh function wait-master() { detect-master > $LOG # TODO(justinsb): This is really not necessary any more # Wait 3 minutes for cluster to come up. We hit it with a "highstate" after that to # make sure that everything is well configured. # TODO: Can we poll here? local i for (( i=0; i < 6*3; i++)); do printf "." sleep 10 done echo "Re-running salt highstate" ssh -oStrictHostKeyChecking=no -i "${AWS_SSH_KEY}" ${SSH_USER}@${KUBE_MASTER_IP} \ sudo salt '*' state.highstate > $LOG # This might loop forever if there was some uncaught error during start up until $(curl --insecure --user ${KUBE_USER}:${KUBE_PASSWORD} --max-time 5 \ --fail --output $LOG --silent https://${KUBE_MASTER_IP}/healthz); do printf "." sleep 2 done } 

> wc -l cluster/{gce,aws}/util.sh 1493 cluster/gce/util.sh 1449 cluster/aws/util.sh repo: github.com/kubernetes/kubernetes Many lines of fun! 

> find cluster/saltbase -name “*.sls”\ | wc -l 42 ## Salt YAML files > find cluster/saltbase -name “*.sls”\ | xargs cat | wc -l 1529 ## Lines of YAML repo: github.com/kubernetes/kubernetes Have you done any Salt? 

How about “non-official” ones? github.com/ansibl8s/setup-kubernetes github.com/Samsung-AG/kraken github.com/coreos/coreos-kubernetes Plenty of great examples if you’d like some of: Ansible and/or CoreOS and maybe Terrform… (Those are just the ones I had a brief look at) 

Oh… and you still need to chose and figure out all the other things! 

Kubernetes Anywhere github.com/weaveworks/weave-kubernetes-anywhere 

Project Goals • Dramatically simplify Kubernetes deployment • Easiest way to get started • Scale-out to any infrastructure seamlessly 

Project Goals • Enable complete portability, zero config • Allow user to move or clone the entire cluster • Make TLS setup fully transparent 

Approach • 100% containerised deployment • Use Weave Net as a cluster management network • Works with any provisioning/config tools 

Kubernetes Anywhere in a nutshell 

Step 1: Infrastructure Setup Let's say you'd like to have a cluster of 5 servers with Docker installed • 3 dedicated etcd hosts ($KUBE_ETCD_1, $KUBE_ETCD_2, $KUBE_ETCD_3) • 1 host running all master components ($KUBE_MASTER_0) • 2 worker nodes ($KUBE_WORKER_1, $KUBE_WORKER_2) 

Step 2: Install Weave Net On all of the machines run: sudo curl --location --silent git.io/weave \ --output /usr/local/bin/weave sudo chmod +x /usr/local/bin/weave 

Step 3: Launch Weave Net On all of the machines run: weave launch-router \ $KUBE_ETCD_1 $KUBE_ETCD_2 $KUBE_ETCD_3 \ $KUBE_MASTER_0 \ $KUBE_WORKER_1 $KUBE_WORKER_2 weave launch-proxy --rewrite-inspect weave expose -h "$(hostname).weave.local" eval $(weave env) 

Step 3: Launch etcd cluster On each of the 3 etcd hosts run these commands in turns: docker run -d -e ETCD_CLUSTER_SIZE=3 \ --name=etcd1 weaveworks/kubernetes-anywhere:etcd docker run -d -e ETCD_CLUSTER_SIZE=3 \ --name=etcd2 weaveworks/kubernetes-anywhere:etcd docker run -d -e ETCD_CLUSTER_SIZE=3 \ --name=etcd3 weaveworks/kubernetes-anywhere:etcd 

Step 4: Master components On $KUBE_MASTER_0 run these Docker commands: docker run -d --name=kube-apiserver \ -e ETCD_CLUSTER_SIZE=3 \ weaveworks/kubernetes-anywhere:apiserver docker run -d --name=kube-controller-manager \ weaveworks/kubernetes-anywhere:controller-manager docker run -d --name=kube-scheduler \ weaveworks/kubernetes-anywhere:scheduler 

Step 5.1: Worker components On $KUBE_WORKER_1 and $KUBE_WORKER_2 run: docker run \ --volume="/:/rootfs" \ --volume="/var/run/weave/weave.sock:/weave.sock" \ weaveworks/kubernetes-anywhere:tools \ setup-kubelet-volumes 

Step 5.2: Worker components On $KUBE_WORKER_1 and $KUBE_WORKER_2 run kubelet: docker run -d \ --name=kubelet \ --privileged=true --net=host --pid=host \ --volumes-from=kubelet-volumes \ weaveworks/kubernetes-anywhere:kubelet 

Step 5.3: Worker components On $KUBE_WORKER_1 and $KUBE_WORKER_2 run kube-proxy: docker run -d \ --name=kube-proxy \ --privileged=true --net=host --pid=host \ weaveworks/kubernetes-anywhere:proxy 

Step 6: Live demo! 

Try it yourself! github.com/weaveworks/weave-kubernetes-anywhere @errordeveloper @weaveworks