Kubernetes Beyond the basics @pbakker

Paul Bakker @pbakker

Automated, production ready Kubernetes cluster in steps 9

Step Understanding Kubernetes 0 Terminology, and concepts to build upon

Nodes, Pods, Controllers Docker container Docker container Docker container Docker container Pods Node Docker container Docker container Docker container Docker container Pods Node Docker container Docker container Docker container Replication Controller Master schedules schedules

Deployment 101 Push your Docker container Create a new replication controller JSON file kubectl  create  -­‐f  mycontroller.json   Replication Controller creates Pods

mycontroller.json  "spec":{              "replicas":3,              "selector":{                    "name":"frontend"              },              "template":{                    "metadata":{                          "labels":{                                "name":"frontend"                          }                    },                    "spec":{                          "containers":[                                {                                      "name":"php-­‐redis",                                      "image":"kubernetes/example-­‐guestbook-­‐php-­‐redis:v2",                                      "ports":[                                            {                                                  "containerPort":80                                            }                                      ]                                }                          ]                    }              }

Scaling kubectl  scale                                                                   —replicas=10  myreplication-­‐controller

Updating my app Create a new Replication Controller JSON file kubectl create -f my-new-rc.json Scale down and delete old RC

Step Automated deployment (simplistic) 1 This kubectl stuff seems a lot of typing!

The simplest Automated deployment Don’t use kubectl, use the API! Build server creates Replication Controller using REST Build server destroys old cluster using REST

Docker container Docker container Docker container Docker container Node Docker registry Build Server Docker container Docker container Docker container Docker container Node push Create RC Docker container Docker container Docker container Replication Controller Master schedules schedules API

Curl example curl  -­‐X  POST       http://k8-­‐master:8080/api/v1beta3/namespaces/default/replicationcontrollers  -­‐d  '{   #Pod  definition   }’

What about downtime? Not quite there yet

Step Load balancing 2 Our containers are running, but how do we access them!?

Pods come and go Pods have dynamic IP addresses First try - Kubernetes Services A service is a proxy to your Pods Fixed IP P O D S E
 R
 V
 I
 C E

Docker container Docker container Docker container Docker container Pods Node Docker container Docker container Docker container Docker container Pods Node MyService HTTP Virtual IP Virtual IP Fixed IP

What about SSL offloading? … better load balancing? … redirects, rewrites, etc? … and that “fixed” IP can’t be reached!? Services - Not quite right

Services are for communication within the k8 network (inter Pod communication) Services - A Hammer and screws…

Docker container Docker container Docker container Docker container Pods Node Docker container Docker container Docker container Docker container Pods Node Vulcan Proxy HTTP Virtual IP Virtual IP Fixed IP Custom load balancer etcd

Choosing a load balancer Vulcan uses etcd for all its config Can use Nginx / HA-proxy with templating

So you’re telling me… —link doesn’t work!? And now you’re telling me… —I can’t see my Pods!?

Step Weave 3 Each Pod gets its own IP Access Pods from outside k8

Docker container Docker container Docker container Docker container Node Docker container Docker container Docker container Docker container Pods Node Vulcan Proxy HTTP Virtual IP Virtual IP Fixed IP Weave network Pods

Step Load balancer registration 4 How does the load balancer keep track of pods?

Watch the Kubernetes API Kubernetes API Vulcan etcd Registrator Watch New Vulcan Config Uses new backends

Amdatu Vulcanized Open source tool connecting Kubernetes and Vulcan Written in Go Available as Docker container

Amdatu Vulcanized Kubernetes API Vulcan etcd Amdatu Vulcanized Watch New Vulcan Config Uses new backends

Step Blue / Green deployment 5 Auto deploy is great, but downtime not so much

Step 5 - Blue / Green Scale up new cluster Wait until healthy Switch backend in Load Balancer Dispose old cluster

How do we know a Pod is healthy? Its RUNNING status is not sufficient… Is the app fully started?

Introduce App level health checks Docker container Docker container Docker container Docker container Node Docker container Docker container Docker container Docker container Pods Node Deployer GET /health GET /health Pods Deploy Server

Running a Deployer This whole things starts be to complex! Our build server can’t access the Pods … how do we health check?

Kubernets API etcd Deployer Build Server Start deployment Kubernets API Kubernets API Kubernets API Pods GET /health Create RC Switch Vulcan Backend

Kubernets API Vulcan etcd Deployer Build Server Start deployment Kubernets API Kubernets API Kubernets API Pods GET /health Create RC Switch Vulcan Backend Amdatu Vulcanized Watch Create backends Read config

Deployment descriptor { "useHealthCheck": true, "newVersion": "${bamboo.deploy.version}", "appName": "todo", "replicas": 2, "vulcanFrontend": "rti-todo.amdatu.com", “podspec": { …. } }

"podspec": { "containers": [{ "image": “amdatu/mycontainer", "name": "todo", "ports": [{ "containerPort": 8080 }], "env": [ { "name": "version", "value": "${bamboo.deploy.version}" } ]}] }

One more thing… We need to tell Kubernetes replicas need to run on different machines! Docker container Docker container Docker container Docker container Node Docker container Docker container Docker container Docker container Node Create RC Docker container Docker container Docker container Replication Controller Master S E R V I C E

Deployment demo Demo

No content

Step Canary deployment 6

Canary deployments Different strategy for the Deployer Add Replication Controller But don’t change the running cluster

K8 Node K8 Node K8 Node K8 Node Prod pod Canary Main Replication Controller K8 Node K8 Node K8 Node K8 Node Canary pod Canary Replication Controller Vulcan

Step Persistent data 7 How to deploy Mongo/MySQL/ElasticSearch in Kubernetes?

You don’t

Kubernetes is great for… Stateless containers Running lots of containers together Moving containers around

Datastores scaling mechanics Reactive scaling makes less sense Cluster should be tuned Scaling is expensive

Infra server(s) K8 Master K8 Node K8 Node K8 Node K8 Node K8 Node Vulcan Vulcanized Deployer Mongo Cluster ElasticSearch Cluster … Cluster Cluster topology

Step Logging 8 kubectl logs mypod?

Logging Centralised logging is key in a dynamic environment Assume you can’t access a pod ELK is very useful for this

Logging Docker container Docker container Docker container Docker container Docker container Docker container Docker container Docker container LogStash ElasticSearch Kibana

Logging example OSGi app OSGi LogService SLF4J Kafka

Logging example Kafka LogStash ElasticSearch Infra components Kibana Dashboard Developer

No content

Step Configuration 9 Passing config to containers

Use environment variables dbName=todo-­‐app   host=${mongo} myconfig.cfg "podspec":  {                "env":  [                    {                      "name":  "mongo",                      "value":  "10.100.2.4"                      }, Deployment descriptor Approach 1

Use etcd etcd=localhost:2375 myconfig.cfg /apps/config/demo-­‐app etcd Approach 2 [    {      "name":  "mongo",      "value":  "10.100.2.4"    }   ]

And if you don’t want to do all this yourself…. RTI Fully managed Kubernetes based clusters Logging and Monitoring Automated deployments Not your standard PAAS

Thank you! Blog: http://paulbakker.io Twitter: @pbakker Mail: paul.bakker@luminis.eu