Introduction to Mesos

Abstracted Infrastructures - Apache Mesos w/ Micah Hausler - 5/19/16
Servers are great, but sometimes you need more than one. Or ten. Or a hundred. Are your servers really fully utilized? We’ll walk through through setting up a Mesos cluster and how bin-packing applications can optimize server use and simplify your (server) configuration.

http://www.meetup.com/chadevs/events/230792434/
https://www.youtube.com/watch?v=PdSmMT0IGx4
https://github.com/micahhausler/mesos-docker

Transcript

1. Apache Mesos by Micah Hausler an introduction to - abstracted infrastructures -
2. a short story.
3. you work for a startup.
5. that sells t-shirts.
8. you
9. selling t-shirts is great.
10. Tomorrow 8:00 am Marketing <-> You
11. “We need a portal that we can be updated constantly and provide customers and partners an interactive experience to promote our brand!” - marketing
13. you already have your process,
14. but it has to change here.
15. the last thing you want is to perform duplicate work.
16. happy web servers running your site happy database serving your site CPU: 80% CPU: 80%
17. happy web servers running your site happy database serving your site web server running that portal CPU: 80% CPU: 80% CPU: 20%
18. the present situation
20. we’d like to go from this, App1 40% App1 40% App1 40% App1 40% App2 20%
21. App1 40% App1 40% App1 40% App1 40% App2 20% to this.
22. • Introduced in 2010 - UC Berkley • “Sixty-four cores or 128 cores on a single chip looks a lot like 64 machines or 128 machines in a data center” -Benjamin Hindman • A Data Center Operating System
23. how does this actually work?
24. how does this actually work? Master
25. how does this actually work? Master Slave nodes
26. what about slave failure? Master Slave nodes
27. what about slave failure? Master Slave nodes
28. what about slave failure? Master Slave nodes
29. what about master failure? Master Slave nodes
30. what about master failure? Master Slave nodes Standby Masters
31. what about master failure? Master Slave nodes Standby Masters
32. Q: How do jobs get sent to Mesos?
33. Q: How do jobs get sent to Mesos? A: The scheduler API!
34. Mesos Master Mesos Salve 1 Mesos Salve n +
35. Mesos Master Mesos Salve 1 Mesos Salve n Framework A +
36. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A +
37. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A +
38. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A Job 1 Job 2 +
39. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A <task 1, slave1, 2cpu, 1gb, …> <task 2, slave1, 1cpu, 2gb, …> Job 1 Job 2 +
40. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A <task 1, slave1, 2cpu, 1gb, …> <task 2, slave1, 1cpu, 2gb, …> <framework A, task 1, slave1, 2cpu, 1gb, …> <framework A, task 2, slave1, 1cpu, 2gb, …> Job 1 Job 2 +
41. Mesos Master <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> <slave1, 4cpu, 4gb mem, 40 gb disk, ports[80,443,…]…> Mesos Salve 1 Mesos Salve n Framework A <task 1, slave1, 2cpu, 1gb, …> <task 2, slave1, 1cpu, 2gb, …> <framework A, task 1, slave1, 2cpu, 1gb, …> <framework A, task 2, slave1, 1cpu, 2gb, …> Task 1 Task2 Job 1 Job 2 +
42. so what are the frameworks?
43. so what are the frameworks? Singularity - by Hubspot Marathon - by Mesosphere Chronos and lots more
44. demo time.
45. who is using Mesos?
46. who is using Mesos?
47. other features. • slave roles • weighted resource assignments • framework resource reservations • maintenance API • multi-disk/partition assignment • slave recovery • startup file fetcher • oversubscription • quota (reserve resources for not-yet-created framework tasks)
48. questions?
49. links. • https://github.com/micahhausler/mesos-docker • https://mesos.apache.org/ • https://mesosphere.github.io/marathon/ • https://mesos.github.io/chronos/
50. Micah Hausler @micahhausler micah@skuid.com thanks! skuid is hiring • Site Reliability Engineers • NodeJS Backend Engineers • Frontend Engineers • Quality Engineers • System Engineers