Upgrade to Pro — share decks privately, control downloads, hide ads and more …

A Common Substrate For Distributed Systems With Apache Aurora And Apache Mesos

37b0fbbdf3dc2d989e8082708d50a939?s=47 dlester
November 18, 2014

A Common Substrate For Distributed Systems With Apache Aurora And Apache Mesos

Presented at ApacheCon Europe 2014



November 18, 2014


  1. A Common Substrate For Distributed Systems With Apache Aurora And

    Apache Mesos @davelester, OSS Advocate @ Twitter Apache Mesos and Aurora PMC Member November 2014, ApacheCon Europe
  2. Overview 1. Project Motivation and Roots 2. Mesos and Aurora

    Essentials 3. Ecosystem, Direction of Growth
  3. Overview 1. What Mesos is and why it matters 2.

    How Mesos and Aurora work 3. What else is happening in the community
  4. Out of Scope • More in-depth technical talk tomorrow, Tim

    St Clair’s ApacheCon talk, Nov 19th at 2:00pm. Incl Docker! • In depth on project’s research background, see Ben Hindman’s original paper from UC Berkeley, or talks • For framework/adopter talks, see: • Brenden Matthew, “Hadoop on Mesos” • Bill Farner, “Past, Present, Future of the Aurora Scheduler”
  5. Project Motivations and Roots

  6. Berkeley Data Analytics Stack Academic Roots Mesos (cluster management) Hadoop

    Distributed File System (HDFS) Tachyon (Distributed Memory-Centric Storage System) Spark Hadoop MR Hive SparkSQL MLib GraphX Spark Streaming Storm MPI BlinkDB Related External Release BDAS Release https://amplab.cs.berkeley.edu/software/
  7. What is Mesos? • A layer of abstraction between machines

    in a datacenter and applications that run • cluster manager • resource manager • Today, also a top-level software project at the Apache Software Foundation Academic Roots
  8. Key Problems Addressed • Fault tolerance • Resource efficiency and

    utilization Academic Roots
  9. Mesos Design Challenges 1. Each framework may have different scheduling

    needs 2. Must scale to tens of thousands of nodes running hundreds of jobs with millions of tasks 3. Must be fault-tolerant and highly available http://static.usenix.org/events/nsdi11/tech/full_papers/Hindman_new.pdf Academic Roots
  10. Goal: program against your datacenter like it’s a single pool

    of resources Apache Mesos Node Node Node Node Node Node Node Node Aurora Storm Spark Frameworks Machines Academic Roots
  11. Avoid Static Partitioning Before After Avoid clusters dedicated to specific

    purposes by pooling resources Academic Roots
  12. Google Borg • Google’s cluster management solution • Borg has

    “probably saved Google the cost of building an extra data center” • See: Wired Magazine, “Return of the Borg: How Twitter Rebuilt Google’s Secret Weapon” • Google helped fund AMP Lab, and John Wilkes spoke at MesosCon 2014 Industry Background www.wired.com/2013/03/google-borg-twitter-mesos/
  13. Mesos at Twitter • “Twitter Stack”, including Mesos, Aurora, Finagle

    • Hundreds of separate services with different owners • Managed by Site Reliability Engineer (SRE) team Industry Background
  14. Industry Background

  15. How does your system handle spikes like this? (Castle in

    the Sky, 8/03/2013 TPS record) Industry Background https://blog.twitter.com/2013/new-tweets-per-second-record-and-how
  16. “throw machines at the problem" vs improving the scalability of

    your system Industry Background
  17. • During the World Cup, ran on a large Ruby

    on Rails codebase. At that time, approximately 200 engineers pushing code • Needed a solution to isolate failure and isolate feature development Previously at Twitter Industry Background
  18. Industry Background https://twitter.com/raffi/status/488437255346741249

  19. Common pattern among companies; see Groupon talk, “Breaking up the

    monolith” http://vimeo.com/105880150 Industry Background
  20. Mesos Adopters Industry Background mesos.apache.org/documentation/latest/powered-by-mesos/

  21. Project Timeline • 2010, began as a research project at

    UC Berkeley AMPLab • Late 2010, became part of the Apache Incubator • Shortly after, Twitter began using Mesos • Early 2013, Airbnb releases Chronos • Summer 2013, became top-level project • 2013, Marathon and Aurora both open sourced • Summer 2014, first-ever MesosCon Mesos at Apache
  22. Then and Now (17mo later) June 2013 • 4 public

    adopters • 4 Apache releases • 2 OSS frameworks November 2014 • 48 public adopters • 19 Apache releases • 19+ OSS frameworks Mesos at Apache
  23. Mesos User Groups (MUGs) • Amsterdam • Atlanta, GA •

    London • Los Angeles / Santa Monica, CA • New York, NY • Paris • San Francisco, CA • Seattle Mesos at Apache mesos.apache.org/community/user-groups/
  24. Mesos and Aurora Essentials

  25. An operating system? “a collection of software that manages computer

    hardware resources and provides common services for computer programs.” - Wikipedia, as referenced by Ben Hindman
  26. Hello Datacenter OS “a collection of software that manages datacenter

    hardware resources and provides common services for computer programs.”
  27. Basic Mesos Architecture Worker Worker Worker Mesos Master Distributing/Scheduling tasks

    across a set of worker machines ZooKeeper Cluster Architecture & High Availability
  28. Schedulers/Frameworks :) :) ̄\_(ツ)_/ ̄ Mesos Master Example: scheduling an

    Aurora job on two machines. One box idles. Aurora ZooKeeper Cluster Architecture & High Availability
  29. Leveraging Containers Mesos Master Uses Linux Containers for CPU and

    memory isolation; containers size can change (be elastic) over time. Also: native Docker support. Aurora ZooKeeper Cluster Container ̄\_(ツ)_/ ̄ Container Architecture & High Availability
  30. Failure of a worker :) Mesos Master Example: failure detected

    by scheduler and rescheduled onto another machine. Aurora :) ZooKeeper Cluster :( Architecture & High Availability
  31. Failure of a master :( Leader election of master nodes

    whose state is replicated across other nodes. Workers connect to new Mesos master. ZooKeeper Cluster Mesos Master Master Worker Worker Worker Architecture & High Availability
  32. Resource Utilization Why care about resource utilization? • Less hardware

    required to run the same jobs, driving down costs • Easier to manage fewer machines
  33. Resource Utilization Resource Offers Mesos provides a set of resource

    “offers” (descriptions of available hardware resources) to a scheduler, which makes decisions for Mesos. ZooKeeper Cluster Mesos Master Aurora Worker Worker Worker
  34. Holy Grail of Resource Utilization Resource Utilization http://people.csail.mit.edu/matei/papers/2011/nsdi_mesos.pdf

  35. Quasar • Users specify performance target for applications instead of

    typical resource reservations • Machine-learning used to predict resource usage and for cluster scheduling • Research by Christina Delimitrou and Christos Kozyrakis at Stanford Resource Utilization http://www.industry-academia.org/download/2014-asplos-quasar-Stanford-paper.pdf
  36. Aurora’s Approach • Can mark production and non-production jobs; production

    jobs can pre-empt non-prod jobs • Additional priority system within each role Resource Utilization
  37. Apache Aurora Apache Mesos Node Node Node Node Node Node

    Node Node Apache Aurora Framework(s) Machines Apache Aurora Scheduler aurora.incubator.apache.org
  38. Key Aurora Features • “run 200 of these, forever” •

    Key features Aurora provides: • Deployment and scheduling of jobs • Rich DSL for defining services • Health checking • Battle-tested in production at Twitter for several years Apache Aurora Scheduler
  39. • Scheduling Diversity Constraints • Host and rack diversity •

    Job abstraction to bundle tasks • Ability to run multiple applications that are replicas of one another, and manage through a single point • Rolling Deploys • SLA Monitoring Apache Aurora Scheduler Key Aurora Features
  40. Example .aurora File Apache Aurora Scheduler import os hello_world_process =

    Process(name = 'hello_world', cmdline = 'echo hello world') hello_world_task = Task( resources = Resources(cpu = 0.1, ram = 16 * MB, disk = 16 * MB), processes = [hello_world_process]) hello_world_job = Job( cluster = 'cluster1', role = os.getenv('USER'), task = hello_world_task) jobs = [hello_world_job] hello_world.aurora aurora.incubator.apache.org/documentation/latest/tutorial/ aurora.incubator.apache.org/documentation/latest/tutorial/
  41. Example .aurora File include(‘hello_world.aurora') production_resources = Resources(cpu = 1.0, ram

    = 512 * MB, disk = 2 * GB) staging_resources = Resources(cpu = 0.1, ram = 32 * MB, disk = 512 * MB) hello_world_template = hello_world( name = "hello_world-{{cluster}}" task = hello_world(resources=production_resources)) jobs = [ # production jobs hello_world_template(cluster = 'cluster1', instances = 25), hello_world_template(cluster = 'cluster2', instances = 15), # staging jobs hello_world_template( cluster = 'local', instances = 1, task = hello_world(resources=staging_resources)), ] Apache Aurora Scheduler hello_world_productionized.aurora
  42. Navigating the (growing) Mesos Ecosystem

  43. Frameworks Services Aurora Marathon Kubernetes Singularity Big Data Spark Storm

    Hadoop Batch Chronos Jenkins Software Ecosystem http://mesos.apache.org/documentation/latest/mesos-frameworks/
  44. Framework Bindings Software Ecosystem • Build your own framework for

    the scheduler API, in your favorite language • Currently bindings in: C++, Java, Clojure, Haskell, Go.. or write your own bindings!
  45. Docker in Mesos 0.20.0 • First-class Docker support • Frameworks

    can now launch Docker containers for tasks • Specify the Docker image for containers, and supports Docker primitives incl volumes and entrypoints Software Ecosystem http://mesos.apache.org/blog/mesos-0-20-0-released/
  46. Kubernetes on Mesos • Similarities between Google’s Borg and Omega

    systems, and Mesos as a resource manager • Recently open sourced Kubernetes, a new project which is inspired by ideas in their own system • Commercial support: Microsoft, RedHat, IBM, Docker, Mesosphere • Being ported to Mesos as a scheduler Software Ecosystem https://github.com/mesosphere/kubernetes-mesos
  47. YARN alongside Mesos • Hadoop 2.0 (aka YARN) can work

    with Mesos • Presented at MesosCon 2014 • Prototype developed by eBay/Paypal to share resources across multiple resource managers, using a control plane • Future work being explored in this space Software Ecosystem http://www.youtube.com/watch?v=d7vZWm_xS9c
  48. Mesos at HubSpot • HubSpot built a custom framework called

    Singularity to run their services (prior to Aurora being open sourced) • They run entirely on AWS • Reduced hardware resources; QA environment runs at 50% of its previous capacity Use Cases https://mesosphere.com/resources/mesos-case-study-hubspot/
  49. Mesos at Coursera • Built a custom Mesos scheduler called

    Iguazú • Uses Docker containers to bundle existing code for long-running jobs • Relying on Mesos to manage how jobs are run Use Cases https://tech.coursera.org/blog/2014/11/17/long-running-jobs-at-coursera/
  50. Upcoming Mesos Features • Future Mesos features: • optimistic offers

    • storage primitives The Future!
  51. MesosCon 2015 • MesosCon 2015 will be co-located with LinuxCon

    in Seattle next August • Looking for sponsors, volunteers, local support The Future! events.linuxfoundation.org/events/mesoscon
  52. Learning More • Apache Mesos website • http://mesos.apache.org • Mesosphere

    tutorials • http://mesosphere.io The Future!
  53. Thank You Dave Lester @davelester dlester@twitter.com