mesos.devoxx.2014

Transcript

1. ROOM 9 #devoxx #DV14 #mesos #docker @samklr Kernel for Distributed Applications Sam BESSALAH @samklr
2. #Devoxx #DV14 #mesos #docker @samklr
3. #Devoxx #DV14 #mesos #docker @samklr Example of a statically partionned cluster
4. #Devoxx #DV14 #mesos #docker @samklr Akka/Play Jenkins Spark/Hadoop Redis Storm
5. #Devoxx #DV14 #mesos #docker @samklr Issues with static partitionning - Increased Complexity : Machines/vm sprawl Software failures gets harder to handle as machines number grow - Huge overheard, on development, - Planning for failure and maintenance gets harder - Not really developer friendly. Long cycles to deployments.
6. #Devoxx #DV14 #mesos #docker @samklr Resources utilisation under static partitionning Play/Akka Redis Hadoop
7. #Devoxx #DV14 #mesos #docker @samklr Resources utilisation under static partitionning Play/Akka Redis Hadoop Put together, we would get
8. #Devoxx #DV14 #mesos #docker @samklr Enters Google stage left ... Google “Borg” .Unofficial name. Moved to a new system called Omega Bundles all the cluster’s machines into a single pooled view of resources. Evolved to Omega.
9. #Devoxx #DV14 #mesos #docker @samklr
10. #Devoxx #DV14 #mesos #docker @samklr Mesos : Level of indirection 1. Run multiple types of distributed systems on the same cluster of machines, and dynamically share resources more eficiently. 2. Provide failure detection, task distribution, task tracking, task monitoring, low level resources management, and fine grained resources sharing. And scale to the thousands of nodes.
11. #Devoxx #DV14 #mesos #docker @samklr
12. #Devoxx #DV14 #mesos #docker @samklr Quick View.
13. #Devoxx #DV14 #mesos #docker @samklr Mesos Architecture
14. #Devoxx #DV14 #mesos #docker @samklr http://mesos.apache.org/documentation/latest/mesos-architecture/
15. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : gets information from the framework like environmental variables,configuration information and run tasks inside the mesos slaves.
16. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
17. #Devoxx #DV14 #mesos #docker @samklr
18. #Devoxx #DV14 #mesos #docker @samklr Resources allocation process - Mesos allocate resources via “resources offers” - Resources are a snapshot of available resources that a scheduler will use to run tasks on mesos slaves.
19. #Devoxx #DV14 #mesos #docker @samklr
20. #Devoxx #DV14 #mesos #docker @samklr
21. #Devoxx #DV14 #mesos #docker @samklr
22. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers()
23. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks()
24. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3. lauchTask()
25. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3. lauchTask() 4. statusUpdate()
26. #Devoxx #DV14 #mesos #docker @samklr 1- resourceOffers() 2. launchTasks() 3. lauchTask() 4. statusUpdate() 5. statusUpdate()
27. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves
28. #Devoxx #DV14 #mesos #docker @samklr Mesos slaves Linux Containers
29. #Devoxx #DV14 #mesos #docker @samklr Mesos uses Linux Containers for resource isolation. Mesos containers are built around cgroups , to provide isolation of CPU, memory, disk … and namespaces for network isolation. Started leveraging containerirsation since 2010
30. #Devoxx #DV14 #mesos #docker @samklr
31. #Devoxx #DV14 #mesos #docker @samklr Linux Containers
32. #Devoxx #DV14 #mesos #docker @samklr Linux Containers LXC
33. #Devoxx #DV14 #mesos #docker @samklr Linux Containers LXC Cgroups
34. #Devoxx #DV14 #mesos #docker @samklr
35. #Devoxx #DV14 #mesos #docker @samklr Mesos provides a native support for Docker Containers since 0.20
36. #Devoxx #DV14 #mesos #docker @samklr
37. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus
38. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under.
39. #Devoxx #DV14 #mesos #docker @samklr Fault recovery & HA - Mesos Master uses Zookeeper for leader election and leader discovery. - Mesos Master have a Registrar to log all the tasks and slaves info, implemented as a replicated log using MultiPaxos for consensus - Mesos has a slave recovery mechanism. So user’s tasks keep running whenever a slave proces goes under. Slaves checkpoint information to their local disks (task info, status updates, etc ), they recover checkpointed data and reconnect with old tasks when they restart.
40. #Devoxx #DV14 #mesos #docker @samklr
41. #Devoxx #DV14 #mesos #docker @samklr Mesos Framework A framework is an application that runs distributed applications on Mesos. It has two components : The Scheduler : Interacts with Mesos, to book resources and launch tasks within the mesos slaves. Executors : run tasks in mesos slaves. Mesos provides an API to build its own frameworks. And there are SDKs in mulitple languages.
42. #Devoxx #DV14 #mesos #docker @samklr Marathon
43. #Devoxx #DV14 #mesos #docker @samklr - A generic mesos framework to run long running services (web apps, etc …) - A distributed Init.d for the cluster. - Runs any linux binary without modification (Tomcat, Play, …) - Cluster wide process supervisor -Private PaaS - Service discovery - Provide a self service Rest API for deployment - Authentification & SSL - Placement constrainst (nodes, racks, etc. (Like with Spark) ) - Service discovery and load balancing via HAProxy What is Marathon ?
44. #Devoxx #DV14 #mesos #docker @samklr Marathon workflow
45. #Devoxx #DV14 #mesos #docker @samklr Marathon workflow
46. #Devoxx #DV14 #mesos #docker @samklr POST /v2/apps { “id” : “IPython Notebook” “container “ : { “image “ : “docker: //ipython/notebook” “options” : [ . . . ] } } Marathon Rest APi
47. #Devoxx #DV14 #mesos #docker @samklr Wants to scale by addind more instances .. PATCH /v2/apps { “instances” : 4 } Marathon Rest APi
48. #Devoxx #DV14 #mesos #docker @samklr Conclusion
49. #Devoxx #DV14 #mesos #docker @samklr Conclusion
50. #Devoxx #DV14 #mesos #docker @samklr Conclusion Aurora
51. #Devoxx #DV14 #mesos #docker @samklr Conclusion Kubernetes
52. #Devoxx #DV14 #mesos #docker @samklr
53. #Devoxx #DV14 #mesos #docker @samklr
54. #Devoxx #DV14 #mesos #docker @samklr
55. #Devoxx #DV14 #mesos #docker @samklr speakerdeck.com/samklr