Apache Mesos (Twitter Open Source Open House)

Short talk on motivation behind Apache Mesos and the resulting architecture as well as a look into how it's being used at Twitter.

Transcript

1. Benjamin  Hindman    –  @benh   Apache  Mesos   incubator.apache.org/mesos   @ApacheMesos  
2. history   Berkeley  research  project  including  Benjamin   Hindman,  Andy  Konwinski,  Matei  Zaharia,  Ali   Ghodsi,  Anthony  D.  Joseph,  Randy  Katz,  Scott   Shenker,  Ion  Stoica   http://incubator.apache.org/mesos/research.html  
3. Mesos  aims  to  make  it  easier  to  build   distributed  applications/frameworks  and   share  datacenter  resources  
4. applications/frameworks   services   analytics  
5. analytics   services   applications/frameworks  
6. deploying  things  today:   static  partitioning   Node   Node   Hadoop   Node   Node   service   …  
7. static  partitioning   considered  harmful   Node   Node   Hadoop   Node   Node   service   …  
8. static  partitioning  considered  harmful   hard  to  fully  utilize  machines  (e.g.,  72  GB  RAM   and  24  CPUs)   Node   Node   Hadoop   Node   Node   service   …  
9. static  partitioning  considered  harmful   harder  to  deal  with  failures   Node   Node   Hadoop   Node   Node   service   …   X  
10. static  partitioning  considered  harmful   Node   Node   Hadoop   Node   Node   service   …   Node   Node   Node   harder  to  scale  elastically  
11. Mesos   Node   Node   Hadoop   Node   Node   service   …  
12. level  of  indirection   Mesos   Node   Node   Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  
13. a  “kernel”  for  the  datacenter   Mesos   Node   Node   Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  
14. Twitter’s  “kernel”  for  the  datacenter   Mesos   Node   Node   Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  
15. architecture   Mesos   master   Mesos  slave   Mesos  slave  
16. Mesos   master   Mesos   master   architecture   Mesos   master   Mesos  slave   Mesos  slave  
17. applications/frameworks   1.  scheduler  
18. architecture   Mesos   master   Mesos  slave   Mesos  slave   service  Y   scheduler   requests  resources,   assign  tasks  
19. frameworks   1.  scheduler   2.  executor  (optional,  if  you  don’t  just  want  to   run  a  single  command)  
20. architecture   Mesos   master   Mesos  slave   Mesos  slave   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server   runs  tasks,  reports   status  updates  
21. architecture   service  X   scheduler   allocation   module   Mesos   master   Mesos  slave   Mesos  slave   decides  how  to  allocate   resources   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server  
22. “two-­‐level  scheduling”   Mesos:  controls  resource  allocations  to   applications/frameworks   applications/frameworks:  make  decisions  about   what  to  run  
23. dominant  resource-­‐fairness   default  allocation  policy  (see   incubator.apache.org/mesos/research.html  for   more  info)     help  us  write  new  allocators!  
24. architecture   service  X   scheduler   allocation   module   Mesos   master   Mesos  slave   service  X   executor   Mesos  slave   task   launches,  isolates,   and  monitors  tasks   and  executors   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server   request   offer  
25. “kernel”  primitives  for  building   frameworks   messaging  (unreliable)   mechanisms  for  high-­‐availability   fault-­‐detection   resource  isolation  (cgroups)   resource  monitoring  
26. Mesos   Mesos   Node   Node   Node   Node   Hadoop   …   Node   Node   Node   Node   Spark  
27. Mesos  at  Twitter   Mesos   Node   Node   Node   Node   Hadoop   …   Node   Node   Node   Node   Spark   Storm  
28. Twitter  framework   a  framework  that  makes  deploying  and   managing  productions  servers  easy   jobs/servers  are  submitted  to  the  framework  via   a  configuration  file   provides  mechanisms:   » rolling  restarts/updates   » relaunching  processes  after  failures  (if  requested)   » and  more!  
29. demo  
30. details   50,000+  lines  of  C++   libprocess  for  asynchronous  actor  style   concurrency  (github.com/libprocess)   APIs  in  C++,  Java,  Python   protobuf  for  data  transport,  data  types   zookeeper  support  for  high-­‐availability   linux  control  groups  support  (LXC/cgroups)  
31. frameworks   •  Hadoop  (0.20.205  and  0.20.2-­‐cdh3u3)   •  MPICH2  (Open  Source  MPI  framework)   •  Spark  (github.com/mesos/spark)   •  DPark  (github.com/douban/dpark)   •  Storm  (github.com/nathanmarz/storm)  
32. genomics  researchers  using  Hadoop   and  Spark   Building  a  new  framework  for  job   workflows,  wants  to  use  Spark  and   Hadoop  too   Built  DPark  (a  Python  clone  of   Spark),  also  running  MPI   Hadoop  and  Spark  used  by  machine   learning  researchers  
33. future   smarter  allocator  support  (priority,  weighted   fair-­‐sharing,  etc)   better  resource  monitoring/collection   other  primitives  for  building  applications/ frameworks  systems?   other  frameworks!?  
34. try  it  out!   run  on  bare-­‐metal  or  virtual  machines  –  develop   against  Mesos  API  and  run  in  private  datacenter,   or  the  cloud,  or  both!  
35. questions?   incubator.apache.org/mesos   @ApacheMesos