Mesos: Ignite Talk at #DevopsDaysAustin 2015

#DevopsDaysAustin 2015

Transcript

1. @andypipe Scaling Twitter Infrastructure with Mesos Chris Aniszczyk (@cra)
2. Fail Whale and Growth
3. 300M+ 500M+ 80% Active users Tweets per day of users are outside the US 2006 2015 100TB compressed data per Twitter Scale
4. 1 2 3 4 5 6 7 8 9 0% 11% 22% 33% 0% 11% 22% 33% 0% 11% 22% 33% Mixed Utilization Devs requesting CPUs slows you down Static Partitioned Datacenters
5. 0% 11% 22% 33% 0% 11% 22% 33% 0% 11% 22% 33% 0% 33.333% 66.667% 100% Didn’t we learn as children that we should share :)? Resource Sharing!?
6. • Google was ahead of the game of managing warehouse scale cluster computing: http://youtube.com/watch? v=0ZFMlO98Jkc • NEW (2015): Google Borg paper (https:// research.google.com/pubs/ pub43438.html) • 2009: Universities (Berkeley) was doing research in this area, we decided to partner and hire researchers: https:// amplab.cs.berkeley.edu/tag/mesos/ Evaluated industry + academia
7. • May 2012: Took university research and spun into a independent open source project at the Apache Foundation: https://blog.twitter.com/2012/ incubating-apache-mesos • June 2013: https://twitter.com/ ApacheMesos/statuses/ 360039441500340224 (graduated) Enter Apache Mesos
8. Where does Mesos Fit?
9. build and run a PaaS on top of Mesos: Apache Aurora / Marathon use OpenStack or EC2 or physical machines to run Mesos Where does Mesos Fit?
10. Mesos treats your datacenter nodes as one big computer! Not as individual machines (or VMs), but as computational resources like CPU cores, memory, disks, network ports, etc… Mesos: Datacenters as 1 CPU
11. http://mesos.apache.org/documentation/latest/mesos-frameworks/ Frameworks are responsible for using resource offers from Mesos Mesos consists of Frameworks
12. http://mesos.apache.org/documentation/latest/mesos-frameworks/ Ecosystem of Frameworks
13. offer hostname 4 CPUs offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM offer hostname 4 CPUs 4 GB RAM Schedulers get resource offers Executors run tasks Mesos Framework = Scheduler + Executor Mesos at a High Level
14. Mesos Slave Hadoop task-tracker Mesos Executor Task #1 Task #2 ./ruby XYZ Mesos Slave Docker Executor Docker Executor java -jar XYZ.jar ./xyz Mesos Master Mesos Master Mesos Master Hadoop scheduler Marathon scheduler Zookeeper quorum Master schedules tasks to run on slaves’ available resources; slaves use executors to coordinate execution of tasks Tasks are the unit of execution
15. Mesos provides fine-grained resource Compute Node Mesos Slave Process Hadoop task-tracker Mesos Executor Task #1 Task #2 ruby XYZ Container (Cgroups) Executor Slaves isolate executors and tasks via containers (dotted line) Compute Node Mesos Slave Process Hadoop task-tracker Task #1 Task #2 Container (Cgroups) Task #3 Containers can GROW AND SHRINK as tasks
16. http://mesos.apache.org/documentation/latest/powered-by-mesos/ https://mesosphere.com/blog/2015/04/23/apple-details-j-a-r-v-i-s-the-mesos-framework-that-runs-siri/ #PoweredByMesos
17. http://mesos.apache.org http://www.meetup.com/Austin-Mesos-Users-Group/ Open Source Community
18. http://mesosphere.io/ Commercial Support
19. http://mesoscon.org MesosCon 2015!
20. Thank you for listening! Chris Aniszczyk (@cra) zx@twitter.com http://mesos.apache.org email: {user,dev}@mesos.apache.org https://mesosphere.com/learn/