Apache Mesos (Twitter Open Source Open House)

Transcript

1. Benjamin  Hindman    –  @benh   Apache  Mesos   incubator.apache.org/mesos

     @ApacheMesos  

2. history   Berkeley  research  project  including  Benjamin   Hindman,  Andy

    Konwinski,  Matei  Zaharia,  Ali   Ghodsi,  Anthony  D.  Joseph,  Randy  Katz,  Scott   Shenker,  Ion  Stoica   http://incubator.apache.org/mesos/research.html  

3. Mesos  aims  to  make  it  easier  to  build   distributed

    applications/frameworks  and   share  datacenter  resources  

4. applications/frameworks   services   analytics  

5. analytics   services   applications/frameworks  

6. deploying  things  today:   static  partitioning   Node   Node

     Hadoop   Node   Node   service   …  

7. static  partitioning   considered  harmful   Node   Node  

   Hadoop   Node   Node   service   …  

8. static  partitioning  considered  harmful   hard  to  fully  utilize  machines

    (e.g.,  72  GB  RAM   and  24  CPUs)   Node   Node   Hadoop   Node   Node   service   …  

9. static  partitioning  considered  harmful   harder  to  deal  with  failures

     Node   Node   Hadoop   Node   Node   service   …   X  

10. static  partitioning  considered  harmful   Node   Node   Hadoop

      Node   Node   service   …   Node   Node   Node   harder  to  scale  elastically  

11. Mesos   Node   Node   Hadoop   Node  

    Node   service   …  

12. level  of  indirection   Mesos   Node   Node  

    Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  

13. a  “kernel”  for  the  datacenter   Mesos   Node  

    Node   Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  

14. Twitter’s  “kernel”  for  the  datacenter   Mesos   Node  

    Node   Node   Node   Hadoop   service   …   Node   Node   Hadoop   Node   Node   service   …  

15. architecture   Mesos   master   Mesos  slave   Mesos

     slave  

16. Mesos   master   Mesos   master   architecture  

    Mesos   master   Mesos  slave   Mesos  slave  

17. applications/frameworks   1.  scheduler  

18. architecture   Mesos   master   Mesos  slave   Mesos

     slave   service  Y   scheduler   requests  resources,   assign  tasks  

19. frameworks   1.  scheduler   2.  executor  (optional,  if  you

     don’t  just  want  to   run  a  single  command)  

20. architecture   Mesos   master   Mesos  slave   Mesos

     slave   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server   runs  tasks,  reports   status  updates  

21. architecture   service  X   scheduler   allocation   module

      Mesos   master   Mesos  slave   Mesos  slave   decides  how  to  allocate   resources   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server  

22. “two-­‐level  scheduling”   Mesos:  controls  resource  allocations  to   applications/frameworks

      applications/frameworks:  make  decisions  about   what  to  run  

23. dominant  resource-­‐fairness   default  allocation  policy  (see   incubator.apache.org/mesos/research.html  for

      more  info)     help  us  write  new  allocators!  

24. architecture   service  X   scheduler   allocation   module

      Mesos   master   Mesos  slave   service  X   executor   Mesos  slave   task   launches,  isolates,   and  monitors  tasks   and  executors   service  Y   scheduler   service  Y   task   (Netty   server)   service  Y   executor   Netty   Server   request   offer  

25. “kernel”  primitives  for  building   frameworks   messaging  (unreliable)  

    mechanisms  for  high-­‐availability   fault-­‐detection   resource  isolation  (cgroups)   resource  monitoring  

26. Mesos   Mesos   Node   Node   Node  

    Node   Hadoop   …   Node   Node   Node   Node   Spark  

27. Mesos  at  Twitter   Mesos   Node   Node  

    Node   Node   Hadoop   …   Node   Node   Node   Node   Spark   Storm  

28. Twitter  framework   a  framework  that  makes  deploying  and  

    managing  productions  servers  easy   jobs/servers  are  submitted  to  the  framework  via   a  configuration  file   provides  mechanisms:   » rolling  restarts/updates   » relaunching  processes  after  failures  (if  requested)   » and  more!  

29. demo  

30. details   50,000+  lines  of  C++   libprocess  for  asynchronous

     actor  style   concurrency  (github.com/libprocess)   APIs  in  C++,  Java,  Python   protobuf  for  data  transport,  data  types   zookeeper  support  for  high-­‐availability   linux  control  groups  support  (LXC/cgroups)  

31. frameworks   •  Hadoop  (0.20.205  and  0.20.2-­‐cdh3u3)   •  MPICH2

     (Open  Source  MPI  framework)   •  Spark  (github.com/mesos/spark)   •  DPark  (github.com/douban/dpark)   •  Storm  (github.com/nathanmarz/storm)  

32. genomics  researchers  using  Hadoop   and  Spark   Building  a

     new  framework  for  job   workflows,  wants  to  use  Spark  and   Hadoop  too   Built  DPark  (a  Python  clone  of   Spark),  also  running  MPI   Hadoop  and  Spark  used  by  machine   learning  researchers  

33. future   smarter  allocator  support  (priority,  weighted   fair-­‐sharing,  etc)

      better  resource  monitoring/collection   other  primitives  for  building  applications/ frameworks  systems?   other  frameworks!?  

34. try  it  out!   run  on  bare-­‐metal  or  virtual  machines

     –  develop   against  Mesos  API  and  run  in  private  datacenter,   or  the  cloud,  or  both!  

35. questions?   incubator.apache.org/mesos   @ApacheMesos