Mesos+YARN = Myriad

Talk at Mesos User Group NYC, see http://www.meetup.com/Apache-Mesos-NYC-Meetup/events/224248923/

Transcript

1. © 2015 Mesosphere, Inc. All Rights Reserved. MESOS + YARN =
 1 Michael Hausenblas, Developer & Cloud Advocate | 2015-09-30 | Mesos NYC Meetup
2. © 2015 Mesosphere, Inc. All Rights Reserved. WHY? 2
3. © 2015 Mesosphere, Inc. All Rights Reserved. BENEFIT FROM TWO GREAT ECO SYSTEMS 3 Apache Mesos ecosystem Marathon, Kubernetes, Chronos, Elasticsearch, C*, Jenkins, Spark, MySQL, ArangoDB, Quobyte, Ceph, etc. Apache Hadoop & YARN ecosystem MapReduce, Hive, Spark, HBase, Impala, etc. Apache Myriad enables you to benefit from both ecosystems without code changes to neither Mesos or YARN
4. © 2015 Mesosphere, Inc. All Rights Reserved. STATIC
 PARTIONING 4
5. © 2015 Mesosphere, Inc. All Rights Reserved. DYNAMIC
 PARTIONING 5
6. © 2015 Mesosphere, Inc. All Rights Reserved. HOW DOES IT WORK? 6
7. © 2015 Mesosphere, Inc. All Rights Reserved. OVERVIEW 7 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
8. © 2015 Mesosphere, Inc. All Rights Reserved. CLIENT
 RESOURCE
 REQUEST 8 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
9. © 2015 Mesosphere, Inc. All Rights Reserved. APPLICATION MASTER – NODE MANAGER
 INTERACTION 9 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
10. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 10 • Resource Manager plugin/Myriad scheduler • REST end-point to flex up/down NodeManagers • Myriad Executor • Launches NodeManager • Mounts YARN’s cgroups under Mesos
11. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 11 https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md • Mesos Slave advertises resources to Mesos Master • YARN Node Manager started as a Mesos Task allotted 4 CPUs &
 8 GB RAM, Node Manager only advertise 3 CPUs & 7 GB RAM • Node Manager mounts YARN containers under cgroup hierarchy of Mesos task, e.g.:
 /sys/fs/cgroup/cpu/mesos/node- manager-task-id/container-1
12. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 12 https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md 1. Myriad launches a NodeManager 2. NodeManager advertises resources to ResourceManager 3. NodeManager launches YARN containers
13. © 2015 Mesosphere, Inc. All Rights Reserved. NEXT
 STEPS 13 • Resource Manager failover/discovery using Marathon/Mesos-DNS • Distribution of Hadoop binaries • Web Interface improvements • HA Myriad scheduler, task reconciliation • Launch Job History Server
14. © 2015 Mesosphere, Inc. All Rights Reserved. WHEN SHOULD
 I BE USING IT? 14
15. © 2015 Mesosphere, Inc. All Rights Reserved. DEMO 15
16. © 2015 Mesosphere, Inc. All Rights Reserved. 16
17. © 2015 Mesosphere, Inc. All Rights Reserved. WHERE CAN I GET IT? 17
18. © 2015 Mesosphere, Inc. All Rights Reserved. SET UP 18 • Roll your own (build from source) • MapR distribution • Mesosphere DCOS
19. © 2015 Mesosphere, Inc. All Rights Reserved. 19
20. © 2015 Mesosphere, Inc. All Rights Reserved.
21. © 2015 Mesosphere, Inc. All Rights Reserved. SEE ALSO … 21
22. © 2015 Mesosphere, Inc. All Rights Reserved. USER
 RESOURCES 22 http://mesos.apache.org https://github.com/apache/incubator-myriad http://apache-myriad.org http://mesosphere.com/product
23. © 2015 Mesosphere, Inc. All Rights Reserved. DEV
 RESOURCES 23 dev@myriad.incubator.apache.org https://issues.apache.org/jira/browse/MYRIAD http://myriad.incubator.apache.org
24. © 2015 Mesosphere, Inc. All Rights Reserved. WE ARE HIRING … San Francisco New York Hamburg https://mesosphere.com/careers/
25. © 2015 Mesosphere, Inc. All Rights Reserved. Q & A 25 • @mhausenblas • mhausenblas.info • @mesosphere • apache-myriad.org