@_hilbert_ atosatto Andrea Tosatto @Open-Xchange Mixed VMs and Containers workloads in KUBERNETES 4DevOps.CH 

WHOAMI 

OPEN-XCHANGE Open-Source company based in Germany. Its main product is AppSuite, a webmail product with advanced real-time collaboration and sharing features. In 2015, OX merged with Dovecot and PowerDNS becoming a group of “kickass internet hackers”. Our software stack is mainly composed by C/C++ and Java backend services. The AppSuite frontend is mainly written in BackboneJS. Sometime OXers also write Perl. We’re polyglot. @openexchange www.open-xchange.com 

POWERDNS Started in 1999. Open-Source since 2007. Part of Open-Xchange since 2015. Powering more than: 30% of the internet hosted domains in the world, 75% of the DNSSEC domains in the world, 150mln of internet users. A piece of blazing fast C++ code powering the Internet. Sometimes we also use Python. ;-) @PowerDNS powerdns/pdns 

WHOAMI (2) (me) Borg Kubernetes Docker Swarm OpenStack Linux Containers Kernel-based VM Omega 

THIS TALK WILL BE A ABOUT EXTENDING KUBERNETES 

“THE CLOUD ERA” 

“GIFEE” 

“CONTAINERS (R)EVOLUTION” 

MULTI-LAYER ORCHESTRATION CONTAINER ENGINE […] CONTAINER ENGINE CONTAINER ORCHESTRATOR APP “A” APP “B” APP “C” APP “D” APP “B” APP “E” VMs ORCHESTRATOR VM 01 VM 0M […] VM N1 VM NM […] PHYSICAL HOST N PHYSICAL HOST 1 [...] 

APPLICATIONS ORCHESTRATION PHYSICAL HOST 1 PHYSICAL HOST N [...] CONTAINER ENGINE APPLICATION ORCHESTRATOR APP “A” APP “B” APP “C” VM […] CONTAINER ENGINE APP “B” APP “D” APP “E” VM […] 

Kubernetes Started by Google in June 2014. Inspired by Google’s Borg and Omega. Donated in 2015 to the Cloud Native Computing Foundation. “Kubernetes is more than just a “container orchestrator”. It aims to eliminate the burden of orchestrating physical/virtual compute, network, and storage infrastructure, and enable application operators and developers to focus entirely on container-centric primitives for self-service operation. Kubernetes also provides a stable, portable foundation (a platform) for building customized workflows and higher-level automation.” > https://github.com/kubernetes/community/blob/master/contributors/design-proposals/architecture.md 

KubeLang Container CronJob DaemonSet Deployment Job Pod ReplicaSet ReplicationController StatefulSet Endpoints Ingress Service ConfigMap Secret PersistentVolumeClaim Volume HorizontalPod Autoscaler PersistentVolume ThirdPartyResource … > https://kubernetes.io/docs/resources-reference/v1.5/ 

K8S NODE Kubernetes Control Plane STATE EtcD SCHEDULING Scheduler API & LOGICS API Server K8S MASTER K8S NODE K8S NODE RUNTIME Docker ACCESS Kube Proxy CONTROL Kubelet 

Kubelet Watches for pods that have been assigned to its node Periodically executes any requested container liveness probes Reports the status of the pod back to the rest of the system Reports the status of the node back to the rest of the system Mounts the pod’s required volumes Downloads the pod’s secrets Runs the pod’s containers CONTAINER EXECUTION STATUS REPORTING 

WITH THE PROPER “THING-LET” WE CAN SCHEDULE ALMOST ANYTHING IN KUBERNETES 

KubeLang (again) Container CronJob DaemonSet Deployment Job Pod ReplicaSet ReplicationController StatefulSet Endpoints Ingress Service ConfigMap Secret PersistentVolumeClaim Volume HorizontalPod Autoscaler PersistentVolume ThirdPartyResource VM 

Custom Kubernetes Resources Kubernetes allows to define custom resources through the special “ThirdPartyResource” kind When a user creates a new ThirdPartyResource, the Kubernetes API Server reacts by creating a new, namespaced RESTful resource path. After the ThirdPartyResource object has been created you can create custom objects. Custom objects can contain custom fields. These fields can contain arbitrary JSON. > https://kubernetes.io/docs/concepts/ecosystem/thirdpartyresource/ 

Custom Kubernetes Resources 

Container Runtime Interface Kubelet communicates with the container runtime (or a CRI shim for the runtime) over Unix sockets using the gRPC framework, where kubelet acts as a client and the CRI shim as the server. The protocol buffers API includes two gRPC services, ImageService, and RuntimeService. The ImageService provides RPCs to manage images operations. The RuntimeService contains RPCs to manage the lifecycle of the pods and containers, as well as calls to interact with containers (exec/attach/port-forward). > http://blog.kubernetes.io/2016/12/container-runtime-interface-cri-in-kubernetes.html 

BEYOND CONTAINERS 

Virtlet Virtlet is a Kubernetes runtime server developed by Mirantis which allows you to run VM workloads, based on QCOW2 images. Virtlet consists of the following components: - Virtlet manager, implementing CRI interface for virtualization and image handling, - vmwrapper, which is responsible for preparing environment for emulator, currently qemu, - CRI Proxy, which provides the possibility to mix docker-shim and VM based workloads on the same k8s node. Mirantis/virtlet 

Virtlet Mirantis/virtlet VIRTLET CRI BOOTSTRAP CRI PROXY 

Virtlet Mirantis/virtlet The nodeAffinity spec that makes the pod run only on the nodes that have extraRuntime=virtlet label The virtlet/image-service/cirros image name is handled by virtlet runtime and actual image name passed to the runtime is image-service/cirros. In case of virtlet this means downloading QCOW2 image from http://image-service/cirros. In order to distinguish between runtimes CRI proxy adds prefixes to pod and container ids returned by the runtimes. 

KubeVirt KubeVirt extends Kubernetes by adding additional virtualization resource types through Kubernetes's third party resource concept. KubeVirt consists in: virt-api, the HTTP RESTfull entrypoint to manage the virtual machines within the cluster, virt-controller, the component managing the state of each VM within the Kubernetes cluster, virt-handler, the daemon running on each Kubernetes node providing an interface to libvirtd. kubevirt/kubevirt 

KubeVirt kubevirt/kubevirt 1. A client posts a new VM definition to the K8s API Server The K8s API Server validates the input and creates a VM TPR object. 2. The virt-controller observes the creation of the new VM object and creates a corresponding pod. 3. Kubernetes schedules the pod on a host. 4. The virt-controller observes that a pod for the VM got started and updates the nodeName field in the VM object. 5. The virt-handler (DaemonSet) observes that a VM got assigned to the host where it is running on and creates a corresponding domain using the local libvirtd instance using the VM Specification. 

WHAT’S NEXT 

WHAT’S NEXT The stabilization of the Container Runtime and Container Network Interfaces will enable different runtimes support to Kubernetes. VMs will hopefully become first-class citizens in K8S. The community is pushing K8S to become a single orchestration tool for any kind of compute workload. I hope K8S will learn from OpenStack and resists to the OpenStack syndrome. If you think K8S is too much and you’re looking for just an orchestrator, give Nomad (www.nomadproject.io) a chance! 

READ MORE > Google Borg https://research.google.com/pubs/pub43438.html http://queue.acm.org/detail.cfm?id=2898444 http://www.umbrant.com/blog/2015/mesos_omega_borg_survey.html > Google Omega https://research.google.com/pubs/pub41684.html https://www.wired.com/2013/04/google-john-wilkes-new-hackers/ https://www.nextplatform.com/2015/05/05/google-omega-to- become-part-of-borg-collective/ 

THANK YOU PS: WE’RE HIRING - https://www.open-xchange.com/jobs/ @_hilbert_ atosatto Andrea Tosatto @Open-Xchange [email protected]