WITH RESOURCE SILOS • Overprovision to handle maximum capacity • No elasticity to handle spikes/drops in load • Hadoop and non-Hadoop services cannot co-locate ◦ If they do, they need strong isolation • Even test/prod Hadoop clusters don’t co-locate ◦ Makes it difficult to share the data layer
MYRIAD ? • A framework that delegates resources between Apache Mesos and Apache YARN ◦ Implements both Mesos Scheduler interface and YARN scheduler interface ◦ Takes Mesos resource offers and launches/grows NMs ◦ Can kill/shrink NMs to give resources back to Mesos • With Myriad, Mesos is your datacenter’s kernel, and YARN is a managed service running on top of it
LOVES MYRIAD • Support Hadoop2: Run any YARN app, e.g. Hive, Pig • Sharing: Remove static partitioning, resource silos ◦ Borrow YARN resources when Tier-1 services spike ◦ Backfill unused capacity with best-effort Hadoop jobs • Portable: Works with unmodified Mesos distro
MYRIAD 0.1 (DEC 2015) • Scale up/down NM capacity via REST API • Remote Distribution of RM/NM binaries • RM failure/discovery using Marathon/Mesos-DNS • Myriad HA, Task reconciliation • Job history server, Timeline server • UX: REST API and WebUI
mohit
mohit
reverentgeek
rmw
skipperchong
carmenhchung
keathley
productmarketing
clairelew
thoeni
jcasabona
cherdarchuk
