Elastic Hadoop Clusters on Mesos with Apache Myriad (incubating)

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1 Apache Myriad (incubating) ApacheCon NA, May 2016 Mohit Soni Adam Bordelon mohit@ me@
2. © 2016 Mesosphere, Inc. All Rights Reserved. 2 AGENDA • Why Myriad ? • What is Myriad ? • Scheduling Modes • Demo ! Apache Mesos, Hadoop, and Myriad logos are trademarks of the Apache Software Foundation
3. © 2016 Mesosphere, Inc. All Rights Reserved. 3 WHY MYRIAD ? Mesos Agent Agent Agent Agent Agent Agent YARN NM NM NM NM NM NM Isolated clusters
4. © 2016 Mesosphere, Inc. All Rights Reserved. 4 UTILIZATION FLUCTUATES Server sprawl; Excess capacity
5. © 2016 Mesosphere, Inc. All Rights Reserved. 5 WHY MYRIAD ? Mesos Agent Agent Agent Agent Agent Agent YARN NM NM NM NM NM NM Isolated clusters
6. © 2016 Mesosphere, Inc. All Rights Reserved. 6 BREAKING SILOS Mesos Agent YARN Agent Agent Agent Agent Agent Agent Agent Agent Agent Agent Agent
7. © 2016 Mesosphere, Inc. All Rights Reserved. 7 THE PROBLEM WITH RESOURCE SILOS • Overprovision to handle maximum capacity • No elasticity to handle spikes/drops in load • Hadoop and non-Hadoop services cannot co-locate ◦ If they do, they need strong isolation • Even test/prod Hadoop clusters don’t co-locate ◦ Makes it difficult to share the data layer
8. © 2016 Mesosphere, Inc. All Rights Reserved. 8 AGENDA • Why Myriad ? • What is Myriad ? • Coarse vs Fine grained scheduling • Demo !
9. © 2016 Mesosphere, Inc. All Rights Reserved. 9 WHAT IS MYRIAD ? • A framework that delegates resources between Apache Mesos and Apache YARN ◦ Implements both Mesos Scheduler interface and YARN scheduler interface ◦ Takes Mesos resource offers and launches/grows NMs ◦ Can kill/shrink NMs to give resources back to Mesos • With Myriad, Mesos is your datacenter’s kernel, and YARN is a managed service running on top of it
10. © 2016 Mesosphere, Inc. All Rights Reserved. 10 MYRIAD ARCHITECTURE YARN RM Mesos Scheduler Web Server
11. © 2016 Mesosphere, Inc. All Rights Reserved. 11 MYRIAD ARCHITECTURE Myriad/RM ZK Agent NM HDFS A B C Agent NM HDFS B C1 C2 Agent NM HDFS A B C3 Mesos
12. © 2016 Mesosphere, Inc. All Rights Reserved. 12 AGENDA • Why Myriad ? • What is Myriad ? • Scheduling Modes • Demo !
13. © 2016 Mesosphere, Inc. All Rights Reserved. 13 SCHEDULING MODES • Coarse-grained • Fine-grained
14. © 2016 Mesosphere, Inc. All Rights Reserved. 14 COARSE GRAINED Myriad/RM Agent Agent Mesos Myriad API /api/flexup {“profile”: “medium”, “instances”: 1} HDFS HDFS
15. © 2016 Mesosphere, Inc. All Rights Reserved. 15 COARSE GRAINED Myriad/RM Agent Agent Mesos Myriad API /api/flexup {“profile”: “medium”, “instances”: 1} Offer HDFS HDFS
16. © 2016 Mesosphere, Inc. All Rights Reserved. 16 COARSE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM
17. © 2016 Mesosphere, Inc. All Rights Reserved. 17 COARSE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Resources allocated for future YARN containers
18. © 2016 Mesosphere, Inc. All Rights Reserved. 18 FINE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Resources allocated for future YARN containers Myriad API /api/flexup {“profile”: “zero”, “instances”: 1}
19. © 2016 Mesosphere, Inc. All Rights Reserved. 19 FINE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Resources allocated for future YARN containers Myriad API /api/flexup {“profile”: “zero”, “instances”: 1} Executor NM No upfront resource allocation
20. © 2016 Mesosphere, Inc. All Rights Reserved. 20 SCHEDULING YARN TASKS Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM YARN App Executor NM
21. © 2016 Mesosphere, Inc. All Rights Reserved. 21 COARSE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM YARN App Executor NM C3 C1 C2 Containers launched using previously allocated resources
22. © 2016 Mesosphere, Inc. All Rights Reserved. 22 FINE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM YARN App Executor NM C3 C1 C2 Offer
23. © 2016 Mesosphere, Inc. All Rights Reserved. 23 FINE GRAINED Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM YARN App Executor NM C3 C1 C2 C4 C5 Containers launched when resources from this agent are offered to Myriad
24. © 2016 Mesosphere, Inc. All Rights Reserved. 24 TASKS COMPLETE Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Executor NM C3 C4
25. © 2016 Mesosphere, Inc. All Rights Reserved. 25 TASKS COMPLETE Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Executor NM C3 C4 Freed resources remain allocated for future use Freed resources reclaimed by Mesos
26. © 2016 Mesosphere, Inc. All Rights Reserved. 26 ALL TASKS COMPLETE Myriad/RM Agent Agent Mesos HDFS HDFS Executor NM Executor NM Freed resources remain allocated for future use Freed resources reclaimed by Mesos
27. © 2016 Mesosphere, Inc. All Rights Reserved. 27 AGENDA • Why Myriad ? • What is Myriad ? • Scheduling Modes • Demo !
28. © 2016 Mesosphere, Inc. All Rights Reserved. 28 APACHE MESOS LOVES MYRIAD • Support Hadoop2: Run any YARN app, e.g. Hive, Pig • Sharing: Remove static partitioning, resource silos ◦ Borrow YARN resources when Tier-1 services spike ◦ Backfill unused capacity with best-effort Hadoop jobs • Portable: Works with unmodified Mesos distro
29. © 2016 Mesosphere, Inc. All Rights Reserved. 29 APACHE HADOOP LOVES MYRIAD • Elastic scaling: NM count and capacity • Fault-tolerant: Maintain RM/NM capacity • Cluster utilization: colocated services, shared resources • Isolation: Cpu, memory, disk, network, gpus, etc. • Multitenancy: many Hadoop clusters on one Mesos cluster • Portable: Works with any modern Hadoop2 distro
30. © 2016 Mesosphere, Inc. All Rights Reserved. 30 FEATURES IN MYRIAD 0.1 (DEC 2015) • Scale up/down NM capacity via REST API • Remote Distribution of RM/NM binaries • RM failure/discovery using Marathon/Mesos-DNS • Myriad HA, Task reconciliation • Job history server, Timeline server • UX: REST API and WebUI
31. © 2016 Mesosphere, Inc. All Rights Reserved. 31 FEATURES IN MYRIAD 0.2 (~MAY 2016) • Improved Fine-Grained Scaling • CGroups fixes • Dockerized NM • Support for Mesosphere DC/OS
32. © 2016 Mesosphere, Inc. All Rights Reserved. 32 UPCOMING FEATURES • Improved multitenancy • Revocable resources, reservations • Security integration • Autoscaling policies (data locality, node drain) • Docker networking, IP-per-container, etc. • Queuing theory scheduling
33. © 2016 Mesosphere, Inc. All Rights Reserved. 33 RESOURCES • http://myriad.incubator.apache.org • dev@myriad.incubator.apache.org • https://github.com/apache/incubator-myriad • https://issues.apache.org/jira/browse/MYRIAD