Containers orchestration CNCF

This is a revamp talk I gave at meetup CNCF in Paris

Transcript

1. Containers orchestration An overview of Swarm, Kubernetes, Nomad and Mesos
2. I am Laurent Grangeau I love to automate things and run apps at scale. You can find me at @laurentgrangeau Hello!
3. Containerization 1
4. Containerization ◉Based on kernel capabilities cgroups namespaces ◉Isolation of processus, network, io, cpus, memory ◉Like a VM but not a VM
5. Inspired by Google
6. Containerization ◉LXC (Linux) ◉Docker (Docker Inc.) ◉Rkt (CoreOS) ◉LXD (Ubuntu) ◉They all use the same kernel features
7. What is orchestration ? 2
8. Orchestration ◉Before, deployment was mainly bash scripts ◉Then came Ansible, Chef, Puppet Standard way to test and deploy Allow developpers and DevOps to forget about what needs to happen ◉Then containers came We need to forget about what server will host a particular container, or how containers will be restarted, monitored and killed
9. Orchestration ◉Composing of architecture, tools, and processes used by humans to deliver a defined Service. ◉Stitching of software and hardware components together to deliver a defined Service. ◉Connecting and Automating of workflows when applicable to deliver a defined Service.
10. Why do I need orchestration ? 3
11. Orchestration ◉You might not need it for small apps ◉No orchestration == manual orchestration Place containers Scale containers Monitor containers Update containers ◉Operate containers at scale ? You need orchestration !
12. Orchestration ◉Cloud services are intended to scale-up arbitrarily and dynamically, without requiring direct human intervention to do so. ◉Cloud service delivery includes fulfillment assurance and billing. ◉Cloud services delivery entails workflows in various technical and business domains.
13. The new developer paradigm 3
14. Microservices Fine-grained, highly decoupled and atomic purpose centric services Multi-versioned Scalable Stateless Share-nothing Immutable Continuously delivered Distributed
15. Microservices ESB Service Service Service Service Service > Service Service Service Service Service From Enterprise Services buses to full-mesh topologies
16. Microservices Service consumer Service provider Registry 2. Find 1. Publish 3. Bind Leverage on a Service registry to discover where services are located
17. Resiliency & scalability Vertical > Horizontal ◉Apps designed for failure & scalability ◉Data to be externalized ◉Dumber infrastructure  Structured: MongoDB, Hadoop, Cassandra, Elastic Search...  Binaries: object storage with Ceph, OpenStack Swift...  Helpful patterns: stateless, multi-versioning, loose coupling...  Infrastructure rationalization  Low-cost, poor-SLA commodity
19. Matrix Docker Swarm Kubernetes Nomad Mesos Genesis & Purpose Easy to setup Declarative system Go, lightweight, modular, extensible Opinionated framework Go, lightweight, modular and extensible 3rd generation orchestrator Declarative Batch processing Declarative jobs spec. Support all major OS Go Distributed system kernel (stiches many machines into a logical cluster) Launched in 2009 C++ (Java, Python & C++ APIs) Marathon as a framework (Scala)
20. Matrix Docker Swarm Kubernetes Nomad Mesos Support & Momentum ~2000 commits, 12 core maintainer, Prod ready ~38000 commits, ~100 commiters Prod ready Under governance CNCF 113 contributors Nomad enterprise Q2 2017 Mesos : 78 contributors Marathon : 217 contributors Apache foundation Host & Service discovery Pull model, rate control, embedded dns, round robin, services Node register itself Addon DNS (SkyDNS) or env. variable Gossip (Serf), federated cluster, integrate w/ Consul (service discovery & monitoring) Mesos-DNS (SRV record) Marathon ensure that each ports are unique
21. Matrix Docker Swarm Kubernetes Nomad Mesos Scheduling Pluggable, filter, constraint, spread Pluggable Criteria (predicates and priorities) 2 phases (checking & ranking) Optimistically concurrent 2 level scheduler Provide reservations, over- subscription and preemption Modularity & Extensibility Pluggable network, authorization Pluggable, database or network for discovery Multiple runtime Addon (DNS, logging, …) Task drivers, provide resource isolation, extensible No pluggable, need to recompile Nomad Multiple framework available Shared libraries loaded on demand
22. Matrix Docker Swarm Kubernetes Nomad Mesos Update & Maintenance Node maybe active, drained and paused Rolling update Automated deployment, rolling update, support rollingback Consistently backward compatible Host maintenance Node can be drained Integrate with Packer, Consul, Terraform Log rotation, no log forward Rolling update Maintenance mode Backward compatible Marathon can do blue/green deployment
23. Matrix Docker Swarm Kubernetes Nomad Mesos Health Monitoring Monitoring nodes Healthcheck apps Monitoring nodes failure, resources 3 types healthcheck (http, exec, tcp) Cluster-level logging Monitoring via heartbeat Healthcheck via http, tcp, script More to come via Consul Resource via alloc- status Master tracks statistic and metrics (counters & gauges) Healthchecks (http/tcp) Event-stream integrated with LB
24. Matrix Docker Swarm Kubernetes Nomad Mesos Network & Load- balancing SDN overlay, Gossip protocol, DNS server, BYONetwork, IPVS, Mesh routing Pod (atomic, flat network, no nat, intrapod via localhost) Services load- balancing Dynamic port (20000 to 60000) Shared IP with node LB via Consul An IP per container (no shared IP) 3rd party network driver CNI isolator LB via tcp/http proxies Pluggable Secret Management Included in Docker 1.13 Data volumes or env variable Limited to 1mb Accessible within a namespace, no cross-namespace Integrate with Vault Secure access through a workflow Minimize secret exposure during bootstrapping Only supported by enterprise DC/OS No secret larger than 1mb
25. Matrix Docker Swarm Kubernetes Nomad Mesos High availibility & Scale Active/Standby managers, rescheduling on failure, no multi region HA supported Federated clusters, multi regions deployment Support up to 2000 nodes Horizontal pod autoscaling Distributed, HA using both leader election & state replication Shared state optimistic scheduler 1M container accross 5000 nodes in 5mn Multiple cluster, multi region, federated cluster Required Zookeeper for the quorum One leader at-a- time Greate asynchronous job, HA built-in « Golden standard » by Docker Used by AirBNB, Twitter, Apple, …
26. Docker Swarm Mode 4
27. Architecture
28. Installation ◉Manager docker swarm init ◉Nodes docker swarm join --token SWMTKN-1- 10bkye0rl1yuvp7plbymlldwb3pt401k6324dt9wsc5own848v- 51dyy2ovv8votedjj0ib7w4n4 192.168.99.100:2377
29. Swarm mode pros ◉Secure by default ◉Bundled with Docker ◉Have advanced features with enterprise edition UCP (RBAC, enforced security, support, Trusted registry…) ◉Full Docker experience ◉Swarm works. Swarm is simple and easy to deploy. ​1.12 eliminated the need for much third-party software Facilitates earlier stages of adoption by organizations viewing containers as faster VMs Now with built-in functionality for applications ◉Swarm is easy to extend, if can already know Docker APIs, you can customize Swarm ◉Still modular, but has stepped back here. ◉Moving very fast; eliminating gaps quickly.
30. Swarm mode cons ◉Distributed Application Bundle still work in progress ◉Suitable for orchestrating a combination of infrastructure containers Has only recently added capabilities falling into the application bucket ◉Swarm is a young project Advanced features forthcoming Natural expectation of caveats in functionality ◉No rebalancing, autoscaling or monitoring, yet ◉Only schedules Docker containers, not containers using other specifications. Does not schedule VMs or non-containerized processes Does not provide support for batch jobs ◉Need separate load-balancer for overlapping ingress ports ◉While dependency and affinity filters are available, Swarm does not provide the ability to enforce scheduling of two containers onto the same host or not at all. Filters facilitate sidecar pattern. No “pod” concept.
31. Kubernetes 5
32. Architecture
33. Installation – the hard way ◉Master Install kubectl Prepare images (etcd, hyperkube) Prepare certificates (CA_CERT, MASTER_CERT, MASTER_KEY, KUBELET_CERT, KUBELET_KEY) Prepare credentials (TOKEN=$(dd if=/dev/urandom bs=128 count=1 2>/dev/null | base64 | tr -d "=+/" | dd bs=32 count=1 2>/dev/null)) Install base softwares (docker, kubelet, kube-proxy) Prepare networking (iptables, overlay network) Bootstrap node (etcd, apiserver, controller manager, scheduler) Start cluster services (cluster DNS, cluster logging, container resource monitoring, gui)
34. Installation – the hard way ◉Minions Same as master…
35. Installation – the easiest way ◉Master kubeadm init ◉Minions kubeadm join --token=d1f88f.5754d925c7b618ed 172.31.34.218
36. Kubernetes pros ◉Easy to install (via kubeadm, but limited to single master) ◉Kubernetes can schedule docker or rkt containers ◉Inherently opinionated w/functionality built-in. Relatively easy to change its opinion Little to no third-party software needed Builds in many application-level concepts and services (petsets, jobsets, daemonsets, application packages / charts, etc.) Advanced storage/volume management ◉Project has most momentum ◉Project is arguably most extensible ◉Thorough project documentation ◉Supports multi-tenancy ◉Multi-master, cross-cluster federation, robust logging & metrics aggregation
37. Kubernetes cons ◉Extra work to have HA on multiple masters ◉Have to setup etcd, network plugins, DNS servers and certificates authorities ◉Only runs containerized applications ◉​ For those familiar with Docker-only, Kubernetes requires understanding of new concepts Powerful frameworks with more moving pieces beget complicated cluster deployment and management. ◉Lightweight graphical user interface ◉Does not provide as sophisticated techniques for resource utilization as Mesos
38. Hashicorp Nomad 6
39. Architecture
40. Installation ◉Server nomad agent –config=server.conf ◉server.conf bind_addr = "0.0.0.0" data_dir = "/var/lib/nomad" server { enabled = true bootstrap_expect = 3 }
41. Installation ◉Client nomad agent –config=client.conf ◉client.conf bind_addr = "0.0.0.0" data_dir = "/var/lib/nomad" client { enabled = true servers = [“172.16.1.1:4647”] }
42. Nomad pros ◉Integrated with other Hashicorp tools ◉Doesn’t depend on external systems or storage ◉Highly available ◉Support multi-datacenter and multi-region configurations ◉Easier to use ◉A single binary for both clients and servers ◉Supports different non-containerized tasks ◉Arguably the most advanced scheduler design ◉Upfront consideration of federation / hybrid cloud ◉Broad OS support
43. Nomad cons ◉Have to learn new concepts ◉No native service discovery (you have to rely on external systems) ◉No load balancing ◉No overlay network ◉No DNS server ◉Outside of scheduler, comparatively less sophisticated ◉Young project ◉Less relative momentum ◉Less relative adoption ◉Less extensible / pluggable
44. Mesos & Marathon 7
45. Architecture
46. Mesos pros ◉Universal Containerizer Abstract away from docker, rkt, runc, appc Can orchestrate native mesos containers ◉Can run multiple frameworks, including Kubernetes and Swarm. ◉Supports multi-tenancy. ◉Good for Big Data shops and job / task-oriented workloads.​ Good for mixed workloads and with data-locality policies ◉Mesos is powerful and scalable, battle-tested ​Good for multiple large things you need to do 10,000+ node cluster system ◉​Marathon UI is young, but promising.
47. Mesos cons ◉Steeper learning curve ◉Still needs 3rd party tools (Zookeeper) ◉Marathon interface could be more Docker friendly (hard to get at volumes and registry) ◉May need a dedicated infrastructure IT team An overly complex solution for small deployments
48. Conclusion 8
49.  Hard to predict a winner  Use the framework that fits most your environment  Others schedulers out there (Blox, Diego, etc.)  For full Docker experience, Swarm mode is the best choice
50. Any questions ? You can find me at ◉ @laurentgrangeau ◉ laurent.grangeau@gmail.com Thanks!