Tweet
Tweet

Slide 1

Slide 1 text

© 2015 Mesosphere, Inc. All Rights Reserved. MESOS + YARN =
 1 Michael Hausenblas, Developer & Cloud Advocate | 2015-09-30 | Mesos NYC Meetup

Slide 2

Slide 2 text

© 2015 Mesosphere, Inc. All Rights Reserved. WHY? 2

Slide 3

Slide 3 text

© 2015 Mesosphere, Inc. All Rights Reserved. BENEFIT FROM TWO GREAT ECO SYSTEMS 3 Apache Mesos ecosystem Marathon, Kubernetes, Chronos, Elasticsearch, C*, Jenkins, Spark, MySQL, ArangoDB, Quobyte, Ceph, etc. Apache Hadoop & YARN ecosystem MapReduce, Hive, Spark, HBase, Impala, etc. Apache Myriad enables you to benefit from both ecosystems without code changes to neither Mesos or YARN

Slide 4

Slide 4 text

© 2015 Mesosphere, Inc. All Rights Reserved. STATIC
 PARTIONING 4

Slide 5

Slide 5 text

© 2015 Mesosphere, Inc. All Rights Reserved. DYNAMIC
 PARTIONING 5

Slide 6

Slide 6 text

© 2015 Mesosphere, Inc. All Rights Reserved. HOW DOES IT WORK? 6

Slide 7

Slide 7 text

© 2015 Mesosphere, Inc. All Rights Reserved. OVERVIEW 7 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.

Slide 8

Slide 8 text

© 2015 Mesosphere, Inc. All Rights Reserved. CLIENT
 RESOURCE
 REQUEST 8 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.

Slide 9

Slide 9 text

© 2015 Mesosphere, Inc. All Rights Reserved. APPLICATION MASTER – NODE MANAGER
 INTERACTION 9 YARN Courtesy of: Apache Hadoop YARN—Moving beyond MapReduce and Batch Processing with Apache Hadoop 2, Arun C. Murth, Vinod Kumar Vavilapalli, Doug Eadline, Joseph Niemiec, Jeff Markham. Addison-Wesley ISBN 978-0-321-93450-5.

Slide 10

Slide 10 text

© 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 10 • Resource Manager plugin/Myriad scheduler • REST end-point to flex up/down NodeManagers • Myriad Executor • Launches NodeManager • Mounts YARN’s cgroups under Mesos

Slide 11

Slide 11 text

© 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 11 https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md • Mesos Slave advertises resources to Mesos Master • YARN Node Manager started as a Mesos Task allotted 4 CPUs &
 8 GB RAM, Node Manager only advertise 3 CPUs & 7 GB RAM • Node Manager mounts YARN containers under cgroup hierarchy of Mesos task, e.g.:
 /sys/fs/cgroup/cpu/mesos/node- manager-task-id/container-1

Slide 12

Slide 12 text

© 2015 Mesosphere, Inc. All Rights Reserved. MYRIAD
 INTERNALS 12 https://github.com/apache/incubator-myriad/blob/master/docs/how-it-works.md 1. Myriad launches a NodeManager 2. NodeManager advertises resources to ResourceManager 3. NodeManager launches YARN containers

Slide 13

Slide 13 text

© 2015 Mesosphere, Inc. All Rights Reserved. NEXT
 STEPS 13 • Resource Manager failover/discovery using Marathon/Mesos-DNS • Distribution of Hadoop binaries • Web Interface improvements • HA Myriad scheduler, task reconciliation • Launch Job History Server

Slide 14

Slide 14 text

© 2015 Mesosphere, Inc. All Rights Reserved. WHEN SHOULD
 I BE USING IT? 14

Slide 15

Slide 15 text

© 2015 Mesosphere, Inc. All Rights Reserved. DEMO 15

Slide 16

Slide 16 text

© 2015 Mesosphere, Inc. All Rights Reserved. 16

Slide 17

Slide 17 text

© 2015 Mesosphere, Inc. All Rights Reserved. WHERE CAN I GET IT? 17

Slide 18

Slide 18 text

© 2015 Mesosphere, Inc. All Rights Reserved. SET UP 18 • Roll your own (build from source) • MapR distribution • Mesosphere DCOS

Slide 19

Slide 19 text

© 2015 Mesosphere, Inc. All Rights Reserved. 19

Slide 20

Slide 20 text

© 2015 Mesosphere, Inc. All Rights Reserved.

Slide 21

Slide 21 text

© 2015 Mesosphere, Inc. All Rights Reserved. SEE ALSO … 21

Slide 22

Slide 22 text

© 2015 Mesosphere, Inc. All Rights Reserved. USER
 RESOURCES 22 http://mesos.apache.org https://github.com/apache/incubator-myriad http://apache-myriad.org http://mesosphere.com/product

Slide 23

Slide 23 text

© 2015 Mesosphere, Inc. All Rights Reserved. DEV
 RESOURCES 23 dev@myriad.incubator.apache.org https://issues.apache.org/jira/browse/MYRIAD http://myriad.incubator.apache.org

Slide 24

Slide 24 text

© 2015 Mesosphere, Inc. All Rights Reserved. WE ARE HIRING … San Francisco New York Hamburg https://mesosphere.com/careers/

Slide 25

Slide 25 text

© 2015 Mesosphere, Inc. All Rights Reserved. Q & A 25 • @mhausenblas • mhausenblas.info • @mesosphere • apache-myriad.org