Apache Mesos (Twitter Open Source Open House)

Short talk on motivation behind Apache Mesos and the resulting architecture as well as a look into how it's being used at Twitter.

Transcript

1. Benjamin  Hindman    –  @benh  
   Apache  Mesos  
   incubator.apache.org/mesos  
   @ApacheMesos  
   

   View Slide

2. history  
   Berkeley  research  project  including  Benjamin  
   Hindman,  Andy  Konwinski,  Matei  Zaharia,  Ali  
   Ghodsi,  Anthony  D.  Joseph,  Randy  Katz,  Scott  
   Shenker,  Ion  Stoica  
   http://incubator.apache.org/mesos/research.html  
   

   View Slide

3. Mesos  aims  to  make  it  easier  to  build  
   distributed  applications/frameworks  and  
   share  datacenter  resources  
   

   View Slide

4. applications/frameworks  
   services   analytics  
   

   View Slide

5. analytics  
   services  
   applications/frameworks  
   

   View Slide

6. deploying  things  today:  
   static  partitioning  
   Node   Node  
   Hadoop  
   Node   Node  
   service  
   …  
   

   View Slide

7. static  partitioning  
   considered  harmful  
   Node   Node  
   Hadoop  
   Node   Node  
   service  
   …  
   

   View Slide

8. static  partitioning  considered  harmful  
   hard  to  fully  utilize  machines  (e.g.,  72  GB  RAM  
   and  24  CPUs)  
   Node   Node  
   Hadoop  
   Node   Node  
   service  
   …  
   

   View Slide

9. static  partitioning  considered  harmful  
   harder  to  deal  with  failures  
   Node   Node  
   Hadoop  
   Node   Node  
   service  
   …  
   X  
   

   View Slide

10. static  partitioning  considered  harmful  
    Node   Node  
    Hadoop  
    Node   Node  
    service  
    …  
    Node   Node   Node  
    harder  to  scale  elastically  
    

    View Slide

11. Mesos  
    Node   Node  
    Hadoop  
    Node   Node  
    service  
    …  
    

    View Slide

12. level  of  indirection  
    Mesos  
    Node   Node   Node   Node  
    Hadoop   service  
    …  
    Node   Node  
    Hadoop  
    Node   Node  
    service  
    …  
    

    View Slide

13. a  “kernel”  for  the  datacenter  
    Mesos  
    Node   Node   Node   Node  
    Hadoop   service  
    …  
    Node   Node  
    Hadoop  
    Node   Node  
    service  
    …  
    

    View Slide

14. Twitter’s  “kernel”  for  the  datacenter  
    Mesos  
    Node   Node   Node   Node  
    Hadoop   service  
    …  
    Node   Node  
    Hadoop  
    Node   Node  
    service  
    …  
    

    View Slide

15. architecture  
    Mesos  
    master  
    Mesos  slave  
    Mesos  slave  
    

    View Slide

16. Mesos  
    master  
    Mesos  
    master  
    architecture  
    Mesos  
    master  
    Mesos  slave  
    Mesos  slave  
    

    View Slide

17. applications/frameworks  
    1.  scheduler  
    

    View Slide

18. architecture  
    Mesos  
    master  
    Mesos  slave  
    Mesos  slave  
    service  Y  
    scheduler  
    requests  resources,  
    assign  tasks  
    

    View Slide

19. frameworks  
    1.  scheduler  
    2.  executor  (optional,  if  you  don’t  just  want  to  
    run  a  single  command)  
    

    View Slide

20. architecture  
    Mesos  
    master  
    Mesos  slave  
    Mesos  slave  
    service  Y  
    scheduler  
    service  Y  
    task  
    (Netty  
    server)  
    service  Y  
    executor  
    Netty  
    Server  
    runs  tasks,  reports  
    status  updates  
    

    View Slide

21. architecture  
    service  X  
    scheduler  
    allocation  
    module  
    Mesos  
    master  
    Mesos  slave  
    Mesos  slave  
    decides  how  to  allocate  
    resources  
    service  Y  
    scheduler  
    service  Y  
    task  
    (Netty  
    server)  
    service  Y  
    executor  
    Netty  
    Server  
    

    View Slide

22. “two-­‐level  scheduling”  
    Mesos:  controls  resource  allocations  to  
    applications/frameworks  
    applications/frameworks:  make  decisions  about  
    what  to  run  
    

    View Slide

23. dominant  resource-­‐fairness  
    default  allocation  policy  (see  
    incubator.apache.org/mesos/research.html  for  
    more  info)    
    help  us  write  new  allocators!  
    

    View Slide

24. architecture  
    service  X  
    scheduler  
    allocation  
    module  
    Mesos  
    master  
    Mesos  slave  
    service  X  
    executor  
    Mesos  slave  
    task  
    launches,  isolates,  
    and  monitors  tasks  
    and  executors  
    service  Y  
    scheduler  
    service  Y  
    task  
    (Netty  
    server)  
    service  Y  
    executor  
    Netty  
    Server  
    request  
    offer  
    

    View Slide

25. “kernel”  primitives  for  building  
    frameworks  
    messaging  (unreliable)  
    mechanisms  for  high-­‐availability  
    fault-­‐detection  
    resource  isolation  (cgroups)  
    resource  monitoring  
    

    View Slide

26. Mesos  
    Mesos  
    Node   Node   Node   Node  
    Hadoop  
    …  
    Node   Node   Node   Node  
    Spark  
    

    View Slide

27. Mesos  at  Twitter  
    Mesos  
    Node   Node   Node   Node  
    Hadoop  
    …  
    Node   Node   Node   Node  
    Spark   Storm  
    

    View Slide

28. Twitter  framework  
    a  framework  that  makes  deploying  and  
    managing  productions  servers  easy  
    jobs/servers  are  submitted  to  the  framework  via  
    a  configuration  file  
    provides  mechanisms:  
    » rolling  restarts/updates  
    » relaunching  processes  after  failures  (if  requested)  
    » and  more!  
    

    View Slide

29. demo  
    

    View Slide

30. details  
    50,000+  lines  of  C++  
    libprocess  for  asynchronous  actor  style  
    concurrency  (github.com/libprocess)  
    APIs  in  C++,  Java,  Python  
    protobuf  for  data  transport,  data  types  
    zookeeper  support  for  high-­‐availability  
    linux  control  groups  support  (LXC/cgroups)  
    

    View Slide

31. frameworks  
    •  Hadoop  (0.20.205  and  0.20.2-­‐cdh3u3)  
    •  MPICH2  (Open  Source  MPI  framework)  
    •  Spark  (github.com/mesos/spark)  
    •  DPark  (github.com/douban/dpark)  
    •  Storm  (github.com/nathanmarz/storm)  
    

    View Slide

32. genomics  researchers  using  Hadoop  
    and  Spark  
    Building  a  new  framework  for  job  
    workflows,  wants  to  use  Spark  and  
    Hadoop  too  
    Built  DPark  (a  Python  clone  of  
    Spark),  also  running  MPI  
    Hadoop  and  Spark  used  by  machine  
    learning  researchers  
    

    View Slide

33. future  
    smarter  allocator  support  (priority,  weighted  
    fair-­‐sharing,  etc)  
    better  resource  monitoring/collection  
    other  primitives  for  building  applications/
    frameworks  systems?  
    other  frameworks!?  
    

    View Slide

34. try  it  out!  
    run  on  bare-­‐metal  or  virtual  machines  –  develop  
    against  Mesos  API  and  run  in  private  datacenter,  
    or  the  cloud,  or  both!  
    

    View Slide

35. questions?  
    incubator.apache.org/mesos  
    @ApacheMesos
    

    View Slide