Elastic Hadoop Clusters on Mesos with Apache Myriad (incubating)

Transcript

1. © 2016 Mesosphere, Inc. All Rights Reserved. 1
   Apache Myriad
   (incubating)
   ApacheCon NA, May 2016
   Mohit Soni Adam Bordelon
   mohit@ me@
   

   View Slide

2. © 2016 Mesosphere, Inc. All Rights Reserved. 2
   AGENDA
   • Why Myriad ?
   • What is Myriad ?
   • Scheduling Modes
   • Demo !
   Apache Mesos, Hadoop, and Myriad logos are trademarks of the Apache Software Foundation
   

   View Slide

3. © 2016 Mesosphere, Inc. All Rights Reserved. 3
   WHY MYRIAD ?
   Mesos
   Agent Agent Agent
   Agent Agent Agent
   YARN
   NM NM NM
   NM NM NM
   Isolated clusters
   

   View Slide

4. © 2016 Mesosphere, Inc. All Rights Reserved. 4
   UTILIZATION FLUCTUATES
   Server sprawl; Excess capacity
   

   View Slide

5. © 2016 Mesosphere, Inc. All Rights Reserved. 5
   WHY MYRIAD ?
   Mesos
   Agent Agent Agent
   Agent Agent Agent
   YARN
   NM NM NM
   NM NM NM
   Isolated clusters
   

   View Slide

6. © 2016 Mesosphere, Inc. All Rights Reserved. 6
   BREAKING SILOS
   Mesos
   Agent
   YARN
   Agent Agent
   Agent Agent Agent
   Agent Agent Agent
   Agent Agent Agent
   

   View Slide

7. © 2016 Mesosphere, Inc. All Rights Reserved. 7
   THE PROBLEM WITH RESOURCE SILOS
   ● Overprovision to handle maximum capacity
   ● No elasticity to handle spikes/drops in load
   ● Hadoop and non-Hadoop services cannot co-locate
   ○ If they do, they need strong isolation
   ● Even test/prod Hadoop clusters don’t co-locate
   ○ Makes it difficult to share the data layer
   

   View Slide

8. © 2016 Mesosphere, Inc. All Rights Reserved. 8
   AGENDA
   • Why Myriad ?
   • What is Myriad ?
   • Coarse vs Fine grained scheduling
   • Demo !
   

   View Slide

9. © 2016 Mesosphere, Inc. All Rights Reserved. 9
   WHAT IS MYRIAD ?
   ● A framework that delegates resources between Apache
   Mesos and Apache YARN
   ○ Implements both Mesos Scheduler interface
   and YARN scheduler interface
   ○ Takes Mesos resource offers and launches/grows NMs
   ○ Can kill/shrink NMs to give resources back to Mesos
   ● With Myriad, Mesos is your datacenter’s kernel,
   and YARN is a managed service running on top of it
   

   View Slide

10. © 2016 Mesosphere, Inc. All Rights Reserved. 10
    MYRIAD ARCHITECTURE
    YARN RM
    Mesos Scheduler
    Web Server
    

    View Slide

11. © 2016 Mesosphere, Inc. All Rights Reserved. 11
    MYRIAD ARCHITECTURE
    Myriad/RM
    ZK
    Agent
    NM
    HDFS
    A
    B
    C
    Agent
    NM
    HDFS
    B
    C1
    C2
    Agent
    NM
    HDFS
    A
    B
    C3
    Mesos
    

    View Slide

12. © 2016 Mesosphere, Inc. All Rights Reserved. 12
    AGENDA
    • Why Myriad ?
    • What is Myriad ?
    • Scheduling Modes
    • Demo !
    

    View Slide

13. © 2016 Mesosphere, Inc. All Rights Reserved. 13
    SCHEDULING MODES
    ● Coarse-grained
    ● Fine-grained
    

    View Slide

14. © 2016 Mesosphere, Inc. All Rights Reserved. 14
    COARSE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    Myriad API
    /api/flexup
    {“profile”: “medium”,
    “instances”: 1}
    HDFS HDFS
    

    View Slide

15. © 2016 Mesosphere, Inc. All Rights Reserved. 15
    COARSE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    Myriad API
    /api/flexup
    {“profile”: “medium”,
    “instances”: 1}
    Offer
    HDFS HDFS
    

    View Slide

16. © 2016 Mesosphere, Inc. All Rights Reserved. 16
    COARSE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    

    View Slide

17. © 2016 Mesosphere, Inc. All Rights Reserved. 17
    COARSE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    Resources allocated for
    future YARN containers
    

    View Slide

18. © 2016 Mesosphere, Inc. All Rights Reserved. 18
    FINE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    Resources allocated for
    future YARN containers
    Myriad API
    /api/flexup
    {“profile”: “zero”,
    “instances”: 1}
    

    View Slide

19. © 2016 Mesosphere, Inc. All Rights Reserved. 19
    FINE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    Resources allocated for
    future YARN containers
    Myriad API
    /api/flexup
    {“profile”: “zero”,
    “instances”: 1}
    Executor NM
    No upfront resource
    allocation
    

    View Slide

20. © 2016 Mesosphere, Inc. All Rights Reserved. 20
    SCHEDULING YARN TASKS
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    YARN App
    Executor NM
    

    View Slide

21. © 2016 Mesosphere, Inc. All Rights Reserved. 21
    COARSE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    YARN App
    Executor NM
    C3
    C1 C2
    Containers
    launched
    using
    previously
    allocated
    resources
    

    View Slide

22. © 2016 Mesosphere, Inc. All Rights Reserved. 22
    FINE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    YARN App
    Executor NM
    C3
    C1 C2 Offer
    

    View Slide

23. © 2016 Mesosphere, Inc. All Rights Reserved. 23
    FINE GRAINED
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM
    YARN App
    Executor NM
    C3
    C1 C2
    C4 C5
    Containers
    launched when
    resources from
    this agent are
    offered to Myriad
    

    View Slide

24. © 2016 Mesosphere, Inc. All Rights Reserved. 24
    TASKS COMPLETE
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM Executor NM
    C3
    C4
    

    View Slide

25. © 2016 Mesosphere, Inc. All Rights Reserved. 25
    TASKS COMPLETE
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM Executor NM
    C3
    C4
    Freed resources remain
    allocated for future use
    Freed resources
    reclaimed by Mesos
    

    View Slide

26. © 2016 Mesosphere, Inc. All Rights Reserved. 26
    ALL TASKS COMPLETE
    Myriad/RM
    Agent Agent
    Mesos
    HDFS HDFS
    Executor NM Executor NM
    Freed resources remain
    allocated for future use
    Freed resources
    reclaimed by Mesos
    

    View Slide

27. © 2016 Mesosphere, Inc. All Rights Reserved. 27
    AGENDA
    • Why Myriad ?
    • What is Myriad ?
    • Scheduling Modes
    • Demo !
    

    View Slide

28. © 2016 Mesosphere, Inc. All Rights Reserved. 28
    APACHE MESOS LOVES MYRIAD
    ● Support Hadoop2: Run any YARN app, e.g. Hive, Pig
    ● Sharing: Remove static partitioning, resource silos
    ○ Borrow YARN resources when Tier-1 services spike
    ○ Backfill unused capacity with best-effort Hadoop jobs
    ● Portable: Works with unmodified Mesos distro
    

    View Slide

29. © 2016 Mesosphere, Inc. All Rights Reserved. 29
    APACHE HADOOP LOVES MYRIAD
    ● Elastic scaling: NM count and capacity
    ● Fault-tolerant: Maintain RM/NM capacity
    ● Cluster utilization: colocated services, shared resources
    ● Isolation: Cpu, memory, disk, network, gpus, etc.
    ● Multitenancy: many Hadoop clusters on one Mesos cluster
    ● Portable: Works with any modern Hadoop2 distro
    

    View Slide

30. © 2016 Mesosphere, Inc. All Rights Reserved. 30
    FEATURES IN MYRIAD 0.1 (DEC 2015)
    ● Scale up/down NM capacity via REST API
    ● Remote Distribution of RM/NM binaries
    ● RM failure/discovery using Marathon/Mesos-DNS
    ● Myriad HA, Task reconciliation
    ● Job history server, Timeline server
    ● UX: REST API and WebUI
    

    View Slide

31. © 2016 Mesosphere, Inc. All Rights Reserved. 31
    FEATURES IN MYRIAD 0.2 (~MAY 2016)
    ● Improved Fine-Grained Scaling
    ● CGroups fixes
    ● Dockerized NM
    ● Support for Mesosphere DC/OS
    

    View Slide

32. © 2016 Mesosphere, Inc. All Rights Reserved. 32
    UPCOMING FEATURES
    ● Improved multitenancy
    ● Revocable resources, reservations
    ● Security integration
    ● Autoscaling policies (data locality, node drain)
    ● Docker networking, IP-per-container, etc.
    ● Queuing theory scheduling
    

    View Slide

33. © 2016 Mesosphere, Inc. All Rights Reserved. 33
    RESOURCES
    ● http://myriad.incubator.apache.org
    ● dev@myriad.incubator.apache.org
    ● https://github.com/apache/incubator-myriad
    ● https://issues.apache.org/jira/browse/MYRIAD
    

    View Slide