docker_on_mesos.pdf

Presentation from Docker meetup, Athens, GR, The Cube Athens
http://www.meetup.com/Docker-Athens/events/230640009/
Introduction on:
1) how to run docker containers on Mesos cluster
2) networking options for containers
3) routing containers with HAProxy
Keywords: Mesos, Marathon, Zookeeper, Docker, Python

Transcript

1. Docker containers on Mesos cluster Ioannis Petrousov May 13, 2016
2. Who am I sysadmin @betiator gaming technologies
3. Table of Contents 1 Intro 2 Mesos Definitions Components Topologies Masters coordination 3 Zookeeper Solve race conditions 4 Marathon Orchestrator 5 Installation 6 Launch containers 7 Routing 8 Outro
4. Intro What systems are necessary? What do I need to run docker containers on Mesos cluster? Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 4 / 30
5. Intro Main technologies Docker (obviously) Mesos (obviously) Zookeeper Marathon Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 5 / 30
6. Mesos Apache Mesos Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 6 / 30
7. Mesos Definitions Computer cluster https://www.techopedia.com/definition/6581/computer-cluster A computer cluster is a single logical unit consisting of multiple computers that are linked through a LAN. The networked computers essentially act as a single, much more powerful machine. Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 7 / 30
8. Mesos Definitions Define Mesos A distributed systems kernel Mesos is built using the same principles as the Linux kernel, only at a different level of abstraction. The Mesos kernel runs on every machine and provides applications (e.g., Hadoop, Spark, Kafka, Elastic Search) with API’s for resource management and scheduling across entire datacenter and cloud environments. Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 8 / 30
9. Mesos Definitions Distributed kernel Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 9 / 30
10. Mesos Components Structural components Handles process scheduling (which node will run a task) Resource management (offers) Health checks (node 3 down) Runs the tasks Reports resources to master Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 10 / 30
11. Mesos Topologies Single server - AIO Simple approach Good starting point Not exactly a cluster Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 11 / 30
12. Mesos Topologies Multiple slaves Single master multiple nodes Single point of failure (master) Process will continue to run Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 12 / 30
13. Mesos Topologies Multiple masters and nodes No single point failure High availability Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 13 / 30
14. Mesos Masters coordination Which master controls the cluster? Which master should the nodes listen to? What if a master falls (power failure)? Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 14 / 30
15. Mesos Masters coordination Master chaos Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 15 / 30
16. Zookeeper Solve race conditions Zookeeper Distributed systems coordinator Maintains data of all the items Solves race conditions, deadlocks and inconsistencies Enables HA for Mesos masters Hadoop, Mesos, Pinterest Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 16 / 30
17. Marathon We need an orchestrator Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 17 / 30
18. Marathon Orchestrator Marathon A cluster-wide init and control system Starts long running tasks (Web server, Kafka,...) Supports Docker containers Start, stop, scale and configure containers WebUI and REST API Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 18 / 30
19. Installation Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 19 / 30
20. Installation Install AIO https://github.com/gpetrousov/mesos_AIO Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 20 / 30
21. Launch containers Launch a container Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 21 / 30
22. Launch containers Demo time! Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 22 / 30
23. Routing Routing containers Multiple containers running on multiple nodes require proper networking. What if one container needs to speak to another? What if we are running a container scaled to 30 instances spread across 10 nodes? How to support dynamic scaling of containers based on resource utilization? How we expose service (HTTP, HTTPS, APIs,...) containers to the outside world? Passing –link will not work on multiple hosts and dynamic ports assignments. Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 23 / 30
24. Routing HAProxy Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 24 / 30
25. Routing Solutions servicerouter.py haproxy-marathon-bridge marathon-lb hadog.py (https://github.com/gpetrousov/hadog) Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 25 / 30
26. Outro The result http://searchvirtualstorage.techtarget.com/definition/ hyper-convergence Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 26 / 30
27. Outro Pros and Cons Cons Not easy for beginners (too many systems) Somewhat difficult documentation A lot of black magic: Mesos, cgroups, Marathon CPU translation Resource fragmentation ... Pros Scalability Fault-tolerant replicated master and slaves using ZooKeeper Multi-resource scheduling (memory, CPU, disk, and ports) Java, Python and C++ APIs for developing new parallel applications ... Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 27 / 30
28. Outro Do your research Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 28 / 30
29. Outro References http://mesos.berkeley.edu/mesos_tech_report.pdf http://mesos.apache.org/ http://theza.ch/2014/09/17/ cpu-resources-in-docker-mesos-and-marathon/ https://www.digitalocean.com/community/tutorials/ how-to-configure-a-production-ready-mesosphere-cluster-on https://mesosphere.github.io/marathon/ https://zookeeper.apache.org/ Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 29 / 30
30. Outro http://petrousov.net/about/ Ioannis Petrousov Docker containers on Mesos cluster May 13, 2016 30 / 30