Upgrade to Pro — share decks privately, control downloads, hide ads and more …

mesos.devoxx.2014

50c1b0fe4cdb0e8e7992d6872cf6cfd7?s=47 Sam Bessalah
November 12, 2014

 mesos.devoxx.2014

50c1b0fe4cdb0e8e7992d6872cf6cfd7?s=128

Sam Bessalah

November 12, 2014
Tweet

Transcript

  1. ROOM 9 #devoxx #DV14 #mesos #docker @samklr Kernel for Distributed

    Applications Sam BESSALAH @samklr
  2. #Devoxx #DV14 #mesos #docker @samklr

  3. #Devoxx #DV14 #mesos #docker @samklr Example of a statically partionned

    cluster
  4. #Devoxx #DV14 #mesos #docker @samklr Akka/Play Jenkins Spark/Hadoop Redis Storm

  5. #Devoxx #DV14 #mesos #docker @samklr Issues with static partitionning -

    Increased Complexity : Machines/vm sprawl Software failures gets harder to handle as machines number grow - Huge overheard, on development, - Planning for failure and maintenance gets harder - Not really developer friendly. Long cycles to deployments.
  6. #Devoxx #DV14 #mesos #docker @samklr Resources utilisation under static partitionning

    Play/Akka Redis Hadoop
  7. #Devoxx #DV14 #mesos #docker @samklr Resources utilisation under static partitionning

    Play/Akka Redis Hadoop Put together, we would get
  8. #Devoxx #DV14 #mesos #docker @samklr Enters Google stage left ...

    Google “Borg” .Unofficial name. Moved to a new system called Omega Bundles all the cluster’s machines into a single pooled view of resources. Evolved to Omega.
  9. #Devoxx #DV14 #mesos #docker @samklr

  10. #Devoxx #DV14 #mesos #docker @samklr Mesos : Level of indirection

    1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently. 2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes.
  11. #Devoxx #DV14 #mesos #docker @samklr

  12. #Devoxx #DV14 #mesos #docker @samklr Quick View.

  13. #Devoxx #DV14 #mesos #docker @samklr Mesos Architecture

  14. #Devoxx #DV14 #mesos #docker @samklr http://mesos.apache.org/documentation/latest/mesos-architecture/

  15. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : gets information from the framework like environmental variables,configuration information and run tasks inside the mesos slaves.
  16. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
  17. #Devoxx #DV14 #mesos #docker @samklr

  18. #Devoxx #DV14 #mesos #docker @samklr Resources allocation process - Mesos

    allocate resources via “resources offers” - Resources are a snapshot of available resources that a scheduler will use to run tasks on mesos slaves.
  19. #Devoxx #DV14 #mesos #docker @samklr

  20. #Devoxx #DV14 #mesos #docker @samklr

  21. #Devoxx #DV14 #mesos #docker @samklr

  22. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers()

  23. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks()

  24. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3.

    lauchTask()
  25. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3.

    lauchTask() 4. statusUpdate()
  26. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3.

    lauchTask() 4. statusUpdate() 5. statusUpdate()
  27. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves

  28. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves Linux Containers

  29. #Devoxx #DV14 #mesos #docker @samklr Mesos uses Linux Containers for

    resource isolation. Mesos containers are built around cgroups , to provide isolation of CPU, memory, disk … and namespaces for network isolation. Started leveraging containerirsation since 2010
  30. #Devoxx #DV14 #mesos #docker @samklr

  31. #Devoxx #DV14 #mesos #docker @samklr Linux Containers

  32. #Devoxx #DV14 #mesos #docker @samklr Linux Containers LXC

  33. #Devoxx #DV14 #mesos #docker @samklr Linux Containers LXC Cgroups

  34. #Devoxx #DV14 #mesos #docker @samklr

  35. #Devoxx #DV14 #mesos #docker @samklr Mesos provides a native support

    for Docker Containers since 0.20
  36. #Devoxx #DV14 #mesos #docker @samklr

  37. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
  38. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under.
  39. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA -

    Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart.
  40. #Devoxx #DV14 #mesos #docker @samklr

  41. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is

    an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
  42. #Devoxx #DV14 #mesos #docker @samklr Marathon

  43. #Devoxx #DV14 #mesos #docker @samklr - A generic mesos framework

    to run long running services (web apps, etc …) - A distributed Init.d for the cluster. - Runs any linux binary without modification (Tomcat, Play, …) - Cluster wide process supervisor -Private PaaS - Service discovery - Provide a self service Rest API for deployment - Authentification & SSL - Placement constrainst (nodes, racks, etc. (Like with Spark) ) - Service discovery and load balancing via HAProxy What is Marathon ?
  44. #Devoxx #DV14 #mesos #docker @samklr Marathon workflow

  45. #Devoxx #DV14 #mesos #docker @samklr Marathon workflow

  46. #Devoxx #DV14 #mesos #docker @samklr POST /v2/apps { “id” :

    “IPython Notebook” “container “ : { “image “ : “docker: //ipython/notebook” “options” : [ . . . ] } } Marathon Rest APi
  47. #Devoxx #DV14 #mesos #docker @samklr Wants to scale by addind

    more instances .. PATCH /v2/apps { “instances” : 4 } Marathon Rest APi
  48. #Devoxx #DV14 #mesos #docker @samklr Conclusion

  49. #Devoxx #DV14 #mesos #docker @samklr Conclusion

  50. #Devoxx #DV14 #mesos #docker @samklr Conclusion Aurora

  51. #Devoxx #DV14 #mesos #docker @samklr Conclusion Kubernetes

  52. #Devoxx #DV14 #mesos #docker @samklr

  53. #Devoxx #DV14 #mesos #docker @samklr

  54. #Devoxx #DV14 #mesos #docker @samklr

  55. #Devoxx #DV14 #mesos #docker @samklr speakerdeck.com/samklr