© 2016 Mesosphere, Inc. All Rights Reserved. 1Apache Myriad(incubating)ApacheCon NA, May 2016Mohit Soni Adam Bordelonmohit@ me@
View Slide
© 2016 Mesosphere, Inc. All Rights Reserved. 2AGENDA• Why Myriad ?• What is Myriad ?• Scheduling Modes• Demo !Apache Mesos, Hadoop, and Myriad logos are trademarks of the Apache Software Foundation
© 2016 Mesosphere, Inc. All Rights Reserved. 3WHY MYRIAD ?MesosAgent Agent AgentAgent Agent AgentYARNNM NM NMNM NM NMIsolated clusters
© 2016 Mesosphere, Inc. All Rights Reserved. 4UTILIZATION FLUCTUATESServer sprawl; Excess capacity
© 2016 Mesosphere, Inc. All Rights Reserved. 5WHY MYRIAD ?MesosAgent Agent AgentAgent Agent AgentYARNNM NM NMNM NM NMIsolated clusters
© 2016 Mesosphere, Inc. All Rights Reserved. 6BREAKING SILOSMesosAgentYARNAgent AgentAgent Agent AgentAgent Agent AgentAgent Agent Agent
© 2016 Mesosphere, Inc. All Rights Reserved. 7THE PROBLEM WITH RESOURCE SILOS● Overprovision to handle maximum capacity● No elasticity to handle spikes/drops in load● Hadoop and non-Hadoop services cannot co-locate○ If they do, they need strong isolation● Even test/prod Hadoop clusters don’t co-locate○ Makes it difficult to share the data layer
© 2016 Mesosphere, Inc. All Rights Reserved. 8AGENDA• Why Myriad ?• What is Myriad ?• Coarse vs Fine grained scheduling• Demo !
© 2016 Mesosphere, Inc. All Rights Reserved. 9WHAT IS MYRIAD ?● A framework that delegates resources between ApacheMesos and Apache YARN○ Implements both Mesos Scheduler interfaceand YARN scheduler interface○ Takes Mesos resource offers and launches/grows NMs○ Can kill/shrink NMs to give resources back to Mesos● With Myriad, Mesos is your datacenter’s kernel,and YARN is a managed service running on top of it
© 2016 Mesosphere, Inc. All Rights Reserved. 10MYRIAD ARCHITECTUREYARN RMMesos SchedulerWeb Server
© 2016 Mesosphere, Inc. All Rights Reserved. 11MYRIAD ARCHITECTUREMyriad/RMZKAgentNMHDFSABCAgentNMHDFSBC1C2AgentNMHDFSABC3Mesos
© 2016 Mesosphere, Inc. All Rights Reserved. 12AGENDA• Why Myriad ?• What is Myriad ?• Scheduling Modes• Demo !
© 2016 Mesosphere, Inc. All Rights Reserved. 13SCHEDULING MODES● Coarse-grained● Fine-grained
© 2016 Mesosphere, Inc. All Rights Reserved. 14COARSE GRAINEDMyriad/RMAgent AgentMesosMyriad API/api/flexup{“profile”: “medium”,“instances”: 1}HDFS HDFS
© 2016 Mesosphere, Inc. All Rights Reserved. 15COARSE GRAINEDMyriad/RMAgent AgentMesosMyriad API/api/flexup{“profile”: “medium”,“instances”: 1}OfferHDFS HDFS
© 2016 Mesosphere, Inc. All Rights Reserved. 16COARSE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NM
© 2016 Mesosphere, Inc. All Rights Reserved. 17COARSE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMResources allocated forfuture YARN containers
© 2016 Mesosphere, Inc. All Rights Reserved. 18FINE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMResources allocated forfuture YARN containersMyriad API/api/flexup{“profile”: “zero”,“instances”: 1}
© 2016 Mesosphere, Inc. All Rights Reserved. 19FINE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMResources allocated forfuture YARN containersMyriad API/api/flexup{“profile”: “zero”,“instances”: 1}Executor NMNo upfront resourceallocation
© 2016 Mesosphere, Inc. All Rights Reserved. 20SCHEDULING YARN TASKSMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMYARN AppExecutor NM
© 2016 Mesosphere, Inc. All Rights Reserved. 21COARSE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMYARN AppExecutor NMC3C1 C2Containerslaunchedusingpreviouslyallocatedresources
© 2016 Mesosphere, Inc. All Rights Reserved. 22FINE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMYARN AppExecutor NMC3C1 C2 Offer
© 2016 Mesosphere, Inc. All Rights Reserved. 23FINE GRAINEDMyriad/RMAgent AgentMesosHDFS HDFSExecutor NMYARN AppExecutor NMC3C1 C2C4 C5Containerslaunched whenresources fromthis agent areoffered to Myriad
© 2016 Mesosphere, Inc. All Rights Reserved. 24TASKS COMPLETEMyriad/RMAgent AgentMesosHDFS HDFSExecutor NM Executor NMC3C4
© 2016 Mesosphere, Inc. All Rights Reserved. 25TASKS COMPLETEMyriad/RMAgent AgentMesosHDFS HDFSExecutor NM Executor NMC3C4Freed resources remainallocated for future useFreed resourcesreclaimed by Mesos
© 2016 Mesosphere, Inc. All Rights Reserved. 26ALL TASKS COMPLETEMyriad/RMAgent AgentMesosHDFS HDFSExecutor NM Executor NMFreed resources remainallocated for future useFreed resourcesreclaimed by Mesos
© 2016 Mesosphere, Inc. All Rights Reserved. 27AGENDA• Why Myriad ?• What is Myriad ?• Scheduling Modes• Demo !
© 2016 Mesosphere, Inc. All Rights Reserved. 28APACHE MESOS LOVES MYRIAD● Support Hadoop2: Run any YARN app, e.g. Hive, Pig● Sharing: Remove static partitioning, resource silos○ Borrow YARN resources when Tier-1 services spike○ Backfill unused capacity with best-effort Hadoop jobs● Portable: Works with unmodified Mesos distro
© 2016 Mesosphere, Inc. All Rights Reserved. 29APACHE HADOOP LOVES MYRIAD● Elastic scaling: NM count and capacity● Fault-tolerant: Maintain RM/NM capacity● Cluster utilization: colocated services, shared resources● Isolation: Cpu, memory, disk, network, gpus, etc.● Multitenancy: many Hadoop clusters on one Mesos cluster● Portable: Works with any modern Hadoop2 distro
© 2016 Mesosphere, Inc. All Rights Reserved. 30FEATURES IN MYRIAD 0.1 (DEC 2015)● Scale up/down NM capacity via REST API● Remote Distribution of RM/NM binaries● RM failure/discovery using Marathon/Mesos-DNS● Myriad HA, Task reconciliation● Job history server, Timeline server● UX: REST API and WebUI
© 2016 Mesosphere, Inc. All Rights Reserved. 31FEATURES IN MYRIAD 0.2 (~MAY 2016)● Improved Fine-Grained Scaling● CGroups fixes● Dockerized NM● Support for Mesosphere DC/OS
© 2016 Mesosphere, Inc. All Rights Reserved. 32UPCOMING FEATURES● Improved multitenancy● Revocable resources, reservations● Security integration● Autoscaling policies (data locality, node drain)● Docker networking, IP-per-container, etc.● Queuing theory scheduling
© 2016 Mesosphere, Inc. All Rights Reserved. 33RESOURCES● http://myriad.incubator.apache.org● [email protected]● https://github.com/apache/incubator-myriad● https://issues.apache.org/jira/browse/MYRIAD