Jenkins as a Code - leveraging Groovy for infrastructure management

Jobs in Jenkins (or any other CI/CD tool) can be created and updated manually using GUI. It is ok if you have a few/several of them. However, most of the companies grow and you can quickly wake up with dozens or hundreds of jobs to maintain.

A new email address/alias to get notification about failed builds? Global migration to the better™ SCM? No way to do it manually with GUI in a convenient way. Manual scripts using Jenkins API? Better, but hard to test and maintain. The same applies to plugin installation, credentials etc. Luckily, there is a better way.

During the talk we will present how Jenkins Job DSL together with Ansible can be used to automatically provision Jenkins instance and maintain any number of jobs. We will show how to define jobs and views in Groovy based DSL and test automatically that the generated structures are exactly the same as expected in Jenkins. Expect live demo - we will setup fully functional Jenkins instance with just one click!

Co-presented with Łukasz Szczęsny.

Transcript

1. Jenkins as a code Leveraging Groovy for infrastructure management Łukasz Szczęsny & Marcin Zajączkowski Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog GR8Day Warsaw, 19th March 2016
2. About Łukasz Software engineer @ Uber FOSS and Open Hardware lover Co-organizer of the Warsaw Linux User Group Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
3. About Marcin Areas of expertise Automatic Testing / TDD Software Craftsmanship / Code Quality Java8 / Groovy Concurrency / Parallel Computing / Reactive Systems Deployment Automation / Continuous Delivery FOSS projects author and contributor, blogger Leads a small software house - Codearte targeted at clients who care about the quality Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
4. Agenda Manual Jenkins maintenance Job configuration as code Jenkins Job DSL Infrastructure as code Case study - Continuous Delivery in Jenkins Live demo Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
5. Manual Jenkins maintenance Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
6. Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
7. Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring problematic with dozens of jobs and plugins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
8. Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring problematic with dozens of jobs and plugins mission impossible with several microservices deployed in several countries for multiple products using deployment pipeline with several steps each Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
9. Automation to the rescue! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
10. Jenkins Job DSL Job configuration in code Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
11. (Jenkins) Job DSL - 2 parts Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
12. (Jenkins) Job DSL - 2 parts Domain Specific Language to specify job configuration Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
13. (Jenkins) Job DSL - 2 parts Domain Specific Language to specify job configuration Jenkins plugin to transform configuration DSL into real jobs in Jenkins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
14. Job DSL - part 1 - configuration Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
15. Job DSL - part 1 - configuration Groovy based DSL (Domain Specific Language) job/view/dashboard configuration developed as "normal" code in IDE with auto-completion type check Groovy magic if needed outside Jenkins instance Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
16. Job DSL - simple example j o b ( ' g r 8 d a y - w e b s i t e - p u b l i s h ' ) { s c m { g i t h u b ( ' g r 8 d a y / w e b s i t e ' ) } t r i g g e r s { s c m ( ' * / 1 5 * * * * ' ) } s t e p s { g r a d l e ( ' p u b l i s h ' ) } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
17. Job DSL - dynamic example S t r i n g r e p o = ' g r 8 d a y / m o b i l e - a p p ' U R L b r a n c h U r l = " h t t p s : / / a p i . g i t h u b . c o m / r e p o s / $ r e p o / b r a n c h e s " . t o U R L ( ) L i s t b r a n c h e s = n e w J s o n S l u r p e r ( ) . p a r s e T e x t ( b r a n c h U r l . t e x t ) b r a n c h e s . e a c h { b r a n c h - > S t r i n g s a f e B r a n c h N a m e = b r a n c h . n a m e . r e p l a c e A l l ( ' / ' , ' - ' ) j o b ( " $ r e p o - $ s a f e B r a n c h N a m e - b u i l d " ) { s c m { g i t h u b r e p o , b r a n c h . n a m e } t r i g g e r s { s c m ' H / 1 0 * * * * ' } s t e p s { g r a d l e ' c h e c k ' } } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
18. Job DSL - features Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
19. Job DSL - features comprehensive support for Jenkins Core stuff Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
20. Job DSL - features comprehensive support for Jenkins Core stuff extensive support for additional plugins over 180 plugins as of version 1.44 active community - continuous flow of new pull requests Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
21. Job DSL - features comprehensive support for Jenkins Core stuff extensive support for additional plugins over 180 plugins as of version 1.44 active community - continuous flow of new pull requests powerful c o n f i g u r a t i o n block for not yet supported features custom stuff virtually everything possible in XML should be achievable Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
22. Job DSL - part 2 - Jenkins plugin Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
23. Job DSL - part 2 - Jenkins plugin installed on Jenkins instance used in seed jobs on Jenkins leverages DSL configuration updates jobs & views in Jenkins to bring them to desired state by XML configuration files modification Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
24. Job DSL - benefits Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
25. Job DSL - benefits source code instead of XML or GUI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
26. Job DSL - benefits source code instead of XML or GUI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
27. Job DSL - benefits source code instead of XML or GUI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment scalable hundreds of jobs created/modified in seconds* Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
28. Job DSL - drawbacks/limitations Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
29. Job DSL - drawbacks/limitations quite steep learning curve can become hard to understand for complex configurations Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
30. Job DSL - drawbacks/limitations quite steep learning curve can become hard to understand for complex configurations small error in DSL can remove some/all jobs can be easily recreated, but without execution history Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
31. Job DSL - drawbacks/limitations quite steep learning curve can become hard to understand for complex configurations small error in DSL can remove some/all jobs can be easily recreated, but without execution history not suitable for global Jenkins configuration management credentials, machine provisioning, Jenkins and plugin update, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
32. Infrastructure Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
33. Infrastructure challenges Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
34. Infrastructure challenges install and configure Jenkins master install and configure all required dependencies install and configure plugins create and connect slaves add JDK installation configure authentication create credentials ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
35. Infrastructure toolbelt configuration management tools Ansible Puppet Chef Salt etc. Groovy console Jenkins CLI Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
36. Infrastructure toolbelt Slave management Swarm plugin Docker plugin SSH slaves Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
37. Continuous Delivery in Jenkins Case study Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
38. Continuous Delivery Clearly defined way how to transform source code into project deployed to production a set of steps arranged into pipeline unified way for various projects/variants/realms Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
39. Continuous Delivery in Jenkins not a first class citizen in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
40. Continuous Delivery in Jenkins not a first class citizen in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... no easy (and unified) way to setup usually even harder to maintain Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
41. Continuous Delivery - case study custom Continuous Delivery framework on top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
42. Continuous Delivery - case study custom Continuous Delivery framework on top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Ansible for infrastructure management Rundeck for deployment Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
43. Continuous Delivery - case study custom Continuous Delivery framework on top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Ansible for infrastructure management Rundeck for deployment open sourced to make live easier to others jenkins-pipeline-dsl - core library sample-jenkins-microservice-pipeline - sample pipeline Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
44. Live demo The whole delivery pipeline with one click! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
45. Summary Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
46. Summary automation is good for you Jenkins Job DSL is great together with infrastructure tools it is a match made in heaven looking forward to Jenkins 2.0 release to simplify Continuous Delivery pipelines creation Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
47. Questions? Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
48. Marcin Zajączkowski @SolidSoftBlog http://blog.solidsoft.info m.zajaczkowski@gmail.com IRC: szpak@freenode Łukasz Szczęsny @wybczu https://wybcz.pl luk@{uber.com,wybcz.pl} IRC: wybczu@freenode Thank you Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog