Jenkins as a code

Jenkins as a code

Co-authored with Łukasz Szczęsny.

Jobs in Jenkins (or any other CI/CD tool) can be created and updated manually using GUI. It is ok if you have a few/several of them. However, most of the companies grow and you can quickly wake up with dozens or hundreds of jobs to maintain.

A new email address/alias to get notification about failed builds? Global migration to the better™ SCM? No way to do it manually with GUI in a convenient way. It just doesn’t scale. Manual scripts using Jenkins API? Better, but hard to test and maintain. The same applies to plugin installation, credentials managements etc. Luckily, there is a better way.

During the talk we will present how Jenkins Job DSL together with Ansible can be used to automatically provision Jenkins instance and maintain any number of jobs. We will show how to define jobs and views in Groovy based DSL and test automatically that the generated structures are exactly the same as expected in Jenkins. Expect live demo - we will setup fully functional Jenkins instance with just one click!

P.S. Jenkins 2.0 brought a breath of fresh air to the ecosystem. Find out how it impacted Jenkins management.

220d0825b07706221aeae4751057ede8?s=128

Marcin Zajączkowski

June 24, 2016
Tweet

Transcript

  1. 2.

    About Łukasz Software engineer @ Uber FOSS and Open Hardware

    lover Co-organizer of the Warsaw Linux User Group Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  2. 3.

    About Marcin Areas of expertise Automatic Testing / TDD Software

    Craftsmanship / Code Quality Java8 / Groovy Concurrency / Parallel Computing / Reactive Systems Deployment Automation / Continuous Delivery FOSS projects author and contributor, blogger and trainer Leads a small software house - Codearte targeted at clients who care about the quality Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  3. 4.

    Agenda Manual Jenkins maintenance Job configuration as code Jenkins Job

    DSL Infrastructure as code Case study - Continuous Delivery in Jenkins Jenkins 2.x Live demo Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  4. 6.

    Manual Jenkins maintenance configuration via GUI does not scale slow,

    error prone, and boring Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  5. 7.

    Manual Jenkins maintenance configuration via GUI does not scale slow,

    error prone, and boring problematic with dozens of jobs and plugins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  6. 8.

    Manual Jenkins maintenance configuration via GUI does not scale slow,

    error prone, and boring problematic with dozens of jobs and plugins mission impossible with several microservices deployed in several countries for multiple products using deployment pipeline with several steps each Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  7. 10.

    Jenkins Job DSL Job configuration in code Łukasz Szczęsny &

    Marcin Zajączkowski @wybczu & @SolidSoftBlog
  8. 11.

    (Jenkins) Job DSL - 2 parts Domain Specific Language to

    specify job configuration Jenkins plugin to transform configuration DSL into real jobs in Jenkins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  9. 12.

    Job DSL - part 1 - configuration Łukasz Szczęsny &

    Marcin Zajączkowski @wybczu & @SolidSoftBlog
  10. 13.

    Job DSL - part 1 - configuration Groovy based DSL

    (Domain Specific Language) job/view/dashboard configuration developed as "normal" code in IDE with auto-completion type check automatic testing Groovy magic if needed outside Jenkins instance Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  11. 14.

    Job DSL - simple example job('devoxx-website-publish') { scm { github('devoxx/website')

    } triggers { scm('*/15 * * * *') } steps { shell('mkdir -p /srv/devoxx.pl') gradle('publish', '-PdestDir=/srv/devoxx.pl') } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  12. 15.

    Job DSL - dynamic example String repo = 'devoxx/mobile-app' URL

    branchUrl = "https://api.github.com/repos/$repo/branches".toURL() List branches = new JsonSlurper().parseText(branchUrl.text) branches.each { branch -> String safeBranchName = branch.name.replaceAll('/', '-') job("$repo-$safeBranchName-build") { scm { github repo, branch.name } triggers { scm 'H/10 * * * *' } steps { gradle 'check' } } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  13. 17.

    Job DSL - features comprehensive support for Jenkins Core stuff

    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  14. 18.

    Job DSL - features comprehensive support for Jenkins Core stuff

    extensive support for additional plugins almost 200 plugins as of version 1.47 active community - continuous flow of new pull requests Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  15. 19.

    Job DSL - features comprehensive support for Jenkins Core stuff

    extensive support for additional plugins almost 200 plugins as of version 1.47 active community - continuous flow of new pull requests powerful configuration block for not yet supported features custom stuff virtually everything possible in XML should be achievable in DSL Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  16. 20.

    Job DSL - part 2 - Jenkins plugin Łukasz Szczęsny

    & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  17. 21.

    Job DSL - part 2 - Jenkins plugin installed on

    Jenkins instance used in seed jobs on Jenkins leverages DSL configuration updates jobs & views in Jenkins to bring them to desired state Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  18. 23.

    Job DSL - benefits source code instead of XML or

    UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  19. 24.

    Job DSL - benefits source code instead of XML or

    UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  20. 25.

    Job DSL - benefits source code instead of XML or

    UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment scalable hundreds of jobs created/modified in seconds* Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  21. 27.

    Job DSL - drawbacks/limitations quite steep learning curve can become

    hard to understand for complex configurations Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  22. 28.

    Job DSL - drawbacks/limitations quite steep learning curve can become

    hard to understand for complex configurations small error in DSL can remove some/all jobs can be easily recreated, but without execution history Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  23. 29.

    Job DSL - drawbacks/limitations quite steep learning curve can become

    hard to understand for complex configurations small error in DSL can remove some/all jobs can be easily recreated, but without execution history not suitable for global Jenkins configuration management credentials, machine provisioning, Jenkins and plugin update, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  24. 32.

    Infrastructure challenges install and configure Jenkins master install and configure

    all required dependencies install and configure plugins create and connect slaves add JDK installation configure authentication create credentials ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  25. 33.

    Infrastructure toolbelt configuration management tools Ansible Puppet Chef Salt etc.

    Groovy + ${JENKINS_HOME}/groovy.init[.d] Jenkins CLI system-config-dsl-plugin Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  26. 34.

    Infrastructure toolbelt Slave management Swarm plugin Amazon EC2 plugin Docker

    plugin SSH slaves Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  27. 36.

    Continuous Delivery Clearly defined way how to transform source code

    into project deployed to production a set of steps arranged into pipeline unified way for various projects/variants/realms Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  28. 37.

    Continuous Delivery in Jenkins 1.x not a first class citizen

    in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  29. 38.

    Continuous Delivery in Jenkins 1.x not a first class citizen

    in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... no easy (and unified) way to setup usually even harder to maintain Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  30. 39.

    Continuous Delivery - case study custom Continuous Delivery framework on

    top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  31. 40.

    Continuous Delivery - case study custom Continuous Delivery framework on

    top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Ansible for infrastructure management Rundeck for deployment Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  32. 41.

    Continuous Delivery - case study custom Continuous Delivery framework on

    top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Ansible for infrastructure management Rundeck for deployment open sourced to make live easier to others jenkins-pipeline-dsl - core library sample-jenkins-microservice-pipeline - sample pipeline Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  33. 43.

    Jenkins 2.x Built-in support for delivery pipelines Pipeline Plugin Pipeline

    Stage View Plugin Jenkinsfile in code for pipeline definition Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  34. 44.

    Jenkins 2.x Built-in support for delivery pipelines Pipeline Plugin Pipeline

    Stage View Plugin Jenkinsfile in code for pipeline definition node { stage 'Checkout' git url: 'https://github.com/devoxx/website.git' stage 'Publish' sh "./gradlew publish" } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  35. 45.

    Jenkins 2.x - continued Improved usability new "Getting Started" experience

    new configuration page redesigned "Create Item" page project Blue Ocean ...but we (mostly) don't care about the UI ;) Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  36. 46.

    Jenkins 2.x - continued Improved usability new "Getting Started" experience

    new configuration page redesigned "Create Item" page project Blue Ocean ...but we (mostly) don't care about the UI ;) Fully backwards compatible * Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  37. 47.

    Jenkins 2.x (*) requires Servlet 3.1 no AJP support with

    the embedded Winstone-Jetty container use HTTP with reverse proxy Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  38. 48.

    Job DSL in Jenkins 2.x world works out-of-the-box jobs generated

    as in Jenkins 1.x Groovy 2.4 available (finally!) traits, static compilation and more to use in production code will live as long as the community is active Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  39. 49.

    Job DSL vs Pipeline plugin Job DSL pros: much wider

    support for plugins as of June 2016 Pipeline plugin catches up better testability outside Jenkins instance not only for pipelines maturity Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  40. 50.

    Job DSL vs Pipeline plugin - continued Pipeline plugin pros:

    flagship component of Jenkins 2 supported directly by CloudBees created with Continuous Delivery in mind pipeline as Code eye catchy visualization durability increasing popularity Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  41. 51.

    Live demo The whole delivery pipeline with one click! Łukasz

    Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  42. 53.

    Summary automation is good for you! Jenkins Job DSL is

    great together with infrastructure tools it is a match made in heaven delivery pipelines as the first class citizens in Jenkins 2.x UI and visualization is getting better all the time! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
  43. 55.

    Marcin Zajączkowski @SolidSoftBlog http://blog.solidsoft.info m.zajaczkowski@gmail.com IRC: szpak@freenode Łukasz Szczęsny @wybczu

    https://wybcz.pl luk@{wybcz.pl,uber.com} IRC: wybczu@freenode Thank you (and remember about the feedback!) Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog