Building a framework on Mesos: A case study with Jenkins

Transcript

1. Building a framework on Mesos: A case study with Jenkins @vinodkone
2. Architecture Master (leader) Spark Scheduler Hadoop Scheduler Jenkins Scheduler Master (standby) Master (standby) ZooKeeper Slave Hadoop Executor Task1 Spark Executor Task1 Slave Hadoop Executor Task1 Task2 Slave Jenkins Executor Task1
3. • Scheduler • receives resource offers • launches tasks • Executor * • executes tasks • launched by the slave • * optional (Command Executor) Slave cpus: 10, mem: 2GB Anatomy of a Framework Scheduler Master Offer cpus:10 mem: 2GB Executor Task cpus: 2 mem: 1GB
4. Writing a Framework • APIs and drivers • C++, Java, Python • see example frameworks (src/examples) • Open Source frameworks • Scala: Chronos, Spark • Java: Hadoop, Jenkins, Storm • Python: dpark
5. Scheduler Callbacks registered(frameworkId, masterInfo) resourceOffers(offers) statusUpdate(taskStatus) ...more Actions launchTasks(offerId, taskInfos, filters) killTask(taskId) declineOffer(offerId, filters) ...more
6. Resource offers @override public void resourceOffers(SchedulerDriver driver, List<Offer> offers) { for (Offer offer : offers) { bool matched = false; for (Request request : requests) { if (matches(offer, request) { matched = true; createMesosTask(offer, request); requests.remove(request); break; } } } } driver.launchTasks(offer.getId(), tasks, filters)
7. Executor Callbacks registered(executorInfo, frameworkInfo, slaveInfo) launchTask(taskInfo) killTask(taskId) ...more Actions sendStatusUpdate(taskStatus) sendFrameworkMessage(data) ...more
8. Status Updates • Executor to inform task state to scheduler • non-terminal updates (e.g., TASK_RUNNING) • terminal updates (e.g., TASK_FINISHED) • TASK_LOST: (e.g., slave terminated) • Status updates are crucial! • e.g., terminal update frees up resources
9. Things to keep in mind • Callbacks are single threaded • i.e., one callback at a time • should not block in a callback! • Actions are asynchronous • e.g., sendStatusUpdate() enqueues the update
10. Jenkins on Mesos +
11. Motivation • Jenkins clusters are statically allocated • order of magnitude smaller than compute cluster • Over utilized • very high build times • Under utilized • wastage of resources source: builds.apache.org
12. Goals • Make Jenkins elastic • auto scale based on workload • Harness the compute cluster • already spent $$$ on it • Resource isolation • helps with capacity planning
13. Jenkins Framework • Jenkins Scheduler • implemented as a Jenkins plugin • launches Jenkins slaves based on workload • kills idle Jenkins slaves • Command Executor
14. Slave Architecture Jenkins Master Job Queue Mesos Plugin Jenkins Scheduler Mesos Master Cloud Command Executor Jenkins Slave JN LP Jenkins
15. Installing the plugin • Getting the plugin • source code in mesos repo (git.apache.org/mesos) • build yourself : $ make jenkins • download directly from Jenkins repo (soon) • Installing the plugin • from Jenkins Web UI •$ cp mesos.hpi ${JENKINS_HOME}/plugins
16. Configuring the plugin • Go to /configure on Jenkins • add a new Mesos Cloud • give the address of Mesos master • Label the jobs as “mesos” • Caveat: experimental release!
17. Demo
18. Questions? @ApacheMesos http://mesos.apache.org