Using Infrastructure Nodes Wisely with Jenkins and Apache Mesos, for Fun and Profit

So it is release crunch time and your developers are hacking away. Your Jenkins CI build queue is getting longer because of lack of free slaves. What do you do? Allocate more dedicated hardware? OK, but what happens when the crunch time is over? Those slave instances will then be sitting idle. Not good! Do you wish that nodes in your infrastructure could be better utilized? Is your infrastructure a victim of "static partitioning"? How awesome would it be if there was a way to scale to multiple Jenkins slaves automatically, whenever needed - and then also scale back down when the work is done? This talk is about solving these problems. Pradeepto will cover all the concepts necessary to understand Apache Mesos, its architecture, what it does and how it does it. He will then demo Jenkins running over Apache Mesos and demonstrate the ability to scale up and down as needed. Code will be shared for better understanding of the solution and the concepts behind it.

Transcript

1. #jenkinsconf Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos London, 23 & 24 June 2015 1
2. #jenkinsconf whoami Pradeepto Bhattacharya / @pradeepto 2
3. #jenkinsconf Agenda  Your Current Build / Test Cluster ( and its problems )  Static Partitioning  Scaling and Limitations  Your Future Build / Test Cluster Using Mesos  Dynamic Partitioning  Benefits  Concepts  Jenkins Mesos Plugin  Demo 3
4. #jenkinsconf #jenkinsconf The Story So Far … (or our problem statement) 4
5. #jenkinsconf Your Usual (Build/Test) Cluster 5
6. #jenkinsconf Scaling and Limitations  Machines  You have have only so much machines!  Money  You have only so much money! 6
7. #jenkinsconf Enter Apache Mesos! 7
8. #jenkinsconf Apache Mesos  Very brief history  Omega and Borg  Benjamin Hindman and merry men!  Production grade  Who uses it right now?  http://mesos.apache.org/documentation/latest/powered-by-mesos/ 8
9. #jenkinsconf #jenkinsconf Concepts 9
10. #jenkinsconf An Operating System  OS for your cluster! 10
11. #jenkinsconf Node Abstraction 11
12. #jenkinsconf Your Old Cluster (but with a difference) 12
13. #jenkinsconf Dynamic Partitioning 13
14. #jenkinsconf Benefits  ROI  Better Utilization  Isolation (using cgroups on the top of HDFS)  Efficiency 14
15. #jenkinsconf Components  Mesos masters  Mesos slaves  Frameworks  Zookeeper 15
16. #jenkinsconf Architechture  Image Credit : http://mesos.apache.org/documentation/latest/mesos-architecture/ 16
17. #jenkinsconf Frameworks  Mesos Applications are called Frameworks  Runs on the top of Mesos  Consists of two components  Schedulers  Executors 17
18. #jenkinsconf Resource Offers 18
19. #jenkinsconf August Club of Frameworks 19
20. #jenkinsconf Frameworks Periodic Table  Image Credit : http://open.mesosphere.com/frameworks/ 20
21. #jenkinsconf #jenkinsconf Jenkins Mesos Plugin 21
22. #jenkinsconf Configuration (General) 22
23. #jenkinsconf Configuration (General) 23
24. #jenkinsconf #jenkinsconf Demo! 24
25. #jenkinsconf #jenkinsconf Concepts Again! 25
26. #jenkinsconf Slave Resources and Attributes  Key-Value pairs (characteristics of slave)  Predefined resources  cpu, memory, disks, ports  Custom attributes  os, rack, etc.  Can be passed to slave while running them via –attributes/--resources commandline option (separated by a semi-colon)  e.g. --attribute='os:centos'  --resources='cpus:24;mem:24576;disk:409600;ports:[21000-24000] 26
27. #jenkinsconf Strategies  Jenkins Master running stand-alone  Jenkins Master running on Mesos Cluster via Marathon 27
28. #jenkinsconf Gotchas!  LIBPROCESS_IP  # LIBPROCESS_IP=10.10.20.100 <process_trigger_command>  Firewall and blocked ports  Dirty slaves 28
29. #jenkinsconf References  https://github.com/jenkinsci/mesos-plugin  http://mesos.apache.org  https://www.cs.berkeley.edu/~alig/papers/mesos.pdf  http://clogeny.com/2015/03/25/build-your-own-data-center/  http://clogeny.com/2015/06/09/hyperscaling-applications-using-mesos-marathon/  http://www.ebaytechblog.com/2014/04/04/delivering-ebays-ci-solution-with-apache-mesos-part-i/  http://www.ebaytechblog.com/2014/05/12/delivering-ebays-ci-solution-with-apache-mesos-part-ii/ 29
30. #jenkinsconf Thanks  CloudBees and organisers of JUC Europe 2015  Contributors of Jenkins, Mesos, Mesos plugin  My Family  Kalpak, Shikha, Lenin  Girija and Rupali (my partners in CI related crimes) 30
31. #jenkinsconf Manning Mesos In Action Coupon Code : mlignazio 50% off Mesos in Action, all formats, valid through June 30 31
32. #jenkinsconf Questions? pradeeptob@gmail.com 32
33. #jenkinsconf Please Share Your Feedback • Did you find this session valuable? • Please share your thoughts in the • Jenkins User Conference Mobile App. • Find the session in the app and click • on the feedback area. 33