Mesos+YARN = Myriad

Mesos+YARN = Myriad

5c3807aaaf0ffefe6c75e3dbbb8588b5?s=128

Michael Hausenblas

September 30, 2015
Tweet

Transcript

  1. © 2015 Mesosphere, Inc. All Rights Reserved. MESOS + YARN

    =
 1 Michael Hausenblas, Developer & Cloud Advocate | 2015-09-30 | Mesos NYC Meetup
  2. © 2015 Mesosphere, Inc. All Rights Reserved. WHY? 2

  3. © 2015 Mesosphere, Inc. All Rights Reserved. BENEFIT FROM TWO

    GREAT ECO SYSTEMS 3 Apache Mesos ecosystem Marathon, Kubernetes, Chronos, Elasticsearch, C*, Jenkins, Spark, MySQL, ArangoDB, Quobyte, Ceph, etc. Apache Hadoop & YARN ecosystem MapReduce, Hive, Spark, HBase, Impala, etc. Apache Myriad enables you to benefit from both ecosystems without code changes to neither Mesos or YARN
  4. © 2015 Mesosphere, Inc. All Rights Reserved. STATIC
 PARTIONING 4

  5. © 2015 Mesosphere, Inc. All Rights Reserved. DYNAMIC
 PARTIONING 5

  6. © 2015 Mesosphere, Inc. All Rights Reserved. HOW DOES IT

    WORK? 6
  7. © 2015 Mesosphere, Inc. All Rights Reserved. OVERVIEW 7 YARN

    Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
  8. © 2015 Mesosphere, Inc. All Rights Reserved. CLIENT
 RESOURCE
 REQUEST

    8 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
  9. © 2015 Mesosphere, Inc. All Rights Reserved. APPLICATION MASTER –

    NODE MANAGER
 INTERACTION 9 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.
  10. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 10

    • Resource Manager plugin/Myriad scheduler • REST end-point to flex up/down NodeManagers • Myriad Executor • Launches NodeManager • Mounts YARN’s cgroups under Mesos
  11. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 11

    https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md • Mesos Slave advertises resources to Mesos Master • YARN Node Manager started as a Mesos Task allotted 4 CPUs &
 8 GB RAM, Node Manager only advertise 3 CPUs & 7 GB RAM • Node Manager mounts YARN containers under cgroup hierarchy of Mesos task, e.g.:
 /sys/fs/cgroup/cpu/mesos/node- manager-task-id/container-1
  12. © 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 12

    https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md 1. Myriad launches a NodeManager 2. NodeManager advertises resources to ResourceManager 3. NodeManager launches YARN containers
  13. © 2015 Mesosphere, Inc. All Rights Reserved. NEXT
 STEPS 13

    • Resource Manager failover/discovery using Marathon/Mesos-DNS • Distribution of Hadoop binaries • Web Interface improvements • HA Myriad scheduler, task reconciliation • Launch Job History Server
  14. © 2015 Mesosphere, Inc. All Rights Reserved. WHEN SHOULD
 I

    BE USING IT? 14
  15. © 2015 Mesosphere, Inc. All Rights Reserved. DEMO 15

  16. © 2015 Mesosphere, Inc. All Rights Reserved. 16

  17. © 2015 Mesosphere, Inc. All Rights Reserved. WHERE CAN I

    GET IT? 17
  18. © 2015 Mesosphere, Inc. All Rights Reserved. SET UP 18

    • Roll your own (build from source) • MapR distribution • Mesosphere DCOS
  19. © 2015 Mesosphere, Inc. All Rights Reserved. 19

  20. © 2015 Mesosphere, Inc. All Rights Reserved.

  21. © 2015 Mesosphere, Inc. All Rights Reserved. SEE ALSO …

    21
  22. © 2015 Mesosphere, Inc. All Rights Reserved. USER
 RESOURCES 22

    http://mesos.apache.org https://github.com/apache/incubator-myriad http://apache-myriad.org http://mesosphere.com/product
  23. © 2015 Mesosphere, Inc. All Rights Reserved. DEV
 RESOURCES 23

    dev@myriad.incubator.apache.org https://issues.apache.org/jira/browse/MYRIAD http://myriad.incubator.apache.org
  24. © 2015 Mesosphere, Inc. All Rights Reserved. WE ARE HIRING

    … San Francisco New York Hamburg https://mesosphere.com/careers/
  25. © 2015 Mesosphere, Inc. All Rights Reserved. Q & A

    25 • @mhausenblas • mhausenblas.info • @mesosphere • apache-myriad.org