Jenkins as a code - Speaker Deck

Slide 1

Slide 1 text

Jenkins as a code Łukasz Szczęsny & Marcin Zajączkowski

Slide 2

Slide 2 text

About Łukasz Software engineer @ Uber FOSS and Open Hardware lover Co-organizer of the Warsaw Linux User Group Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 3

Slide 3 text

About Marcin Areas of expertise Automatic Testing / TDD Software Craftsmanship / Code Quality Java8 / Groovy Concurrency / Parallel Computing / Reactive Systems Deployment Automation / Continuous Delivery FOSS projects author and contributor, blogger and trainer Leads a small software house - Codearte targeted at clients who care about the quality Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 4

Slide 4 text

Agenda Manual Jenkins maintenance Job configuration as code Jenkins Job DSL Infrastructure as code Case study - Continuous Delivery in Jenkins Jenkins 2.x Live demo Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 5

Slide 5 text

Manual Jenkins maintenance Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 6

Slide 6 text

Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 7

Slide 7 text

Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring problematic with dozens of jobs and plugins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 8

Slide 8 text

Manual Jenkins maintenance configuration via GUI does not scale slow, error prone, and boring problematic with dozens of jobs and plugins mission impossible with several microservices deployed in several countries for multiple products using deployment pipeline with several steps each Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 9

Slide 9 text

Automation to the rescue! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 10

Slide 10 text

Jenkins Job DSL Job configuration in code Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 11

Slide 11 text

(Jenkins) Job DSL - 2 parts Domain Specific Language to specify job configuration Jenkins plugin to transform configuration DSL into real jobs in Jenkins Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 12

Slide 12 text

Job DSL - part 1 - configuration Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 13

Slide 13 text

Job DSL - part 1 - configuration Groovy based DSL (Domain Specific Language) job/view/dashboard configuration developed as "normal" code in IDE with auto-completion type check automatic testing Groovy magic if needed outside Jenkins instance Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 14

Slide 14 text

Job DSL - simple example job('devoxx-website-publish') { scm { github('devoxx/website') } triggers { scm('*/15 * * * *') } steps { shell('mkdir -p /srv/devoxx.pl') gradle('publish', '-PdestDir=/srv/devoxx.pl') } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 15

Slide 15 text

Job DSL - dynamic example String repo = 'devoxx/mobile-app' URL branchUrl = "https://api.github.com/repos/$repo/branches".toURL() List branches = new JsonSlurper().parseText(branchUrl.text) branches.each { branch -> String safeBranchName = branch.name.replaceAll('/', '-') job("$repo-$safeBranchName-build") { scm { github repo, branch.name } triggers { scm 'H/10 * * * *' } steps { gradle 'check' } } } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 16

Slide 16 text

Job DSL - features Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 17

Slide 17 text

Job DSL - features comprehensive support for Jenkins Core stuff Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 18

Slide 18 text

Slide 19

Slide 19 text

Job DSL - features comprehensive support for Jenkins Core stuff extensive support for additional plugins almost 200 plugins as of version 1.47 active community - continuous flow of new pull requests powerful configuration block for not yet supported features custom stuff virtually everything possible in XML should be achievable in DSL Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 20

Slide 20 text

Job DSL - part 2 - Jenkins plugin Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 21

Slide 21 text

Job DSL - part 2 - Jenkins plugin installed on Jenkins instance used in seed jobs on Jenkins leverages DSL configuration updates jobs & views in Jenkins to bring them to desired state Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 22

Slide 22 text

Job DSL - benefits Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 23

Slide 23 text

Job DSL - benefits source code instead of XML or UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 24

Slide 24 text

Job DSL - benefits source code instead of XML or UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 25

Slide 25 text

Job DSL - benefits source code instead of XML or UI single source of truth manageable jobs and views backed by SCM reviewable - possibly with pull requests testable automatic "unit" testing pre-production environment scalable hundreds of jobs created/modified in seconds* Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 26

Slide 26 text

Job DSL - drawbacks/limitations Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 27

Slide 27 text

Job DSL - drawbacks/limitations quite steep learning curve can become hard to understand for complex configurations Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 28

Slide 28 text

Slide 29

Slide 29 text

Job DSL - drawbacks/limitations quite steep learning curve can become hard to understand for complex configurations small error in DSL can remove some/all jobs can be easily recreated, but without execution history not suitable for global Jenkins configuration management credentials, machine provisioning, Jenkins and plugin update, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 30

Slide 30 text

Infrastructure Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 31

Slide 31 text

Infrastructure challenges Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 32

Slide 32 text

Infrastructure challenges install and configure Jenkins master install and configure all required dependencies install and configure plugins create and connect slaves add JDK installation configure authentication create credentials ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 33

Slide 33 text

Infrastructure toolbelt configuration management tools Ansible Puppet Chef Salt etc. Groovy + ${JENKINS_HOME}/groovy.init[.d] Jenkins CLI system-config-dsl-plugin Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 34

Slide 34 text

Infrastructure toolbelt Slave management Swarm plugin Amazon EC2 plugin Docker plugin SSH slaves Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 35

Slide 35 text

Continuous Delivery in Jenkins Case study Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 36

Slide 36 text

Continuous Delivery Clearly defined way how to transform source code into project deployed to production a set of steps arranged into pipeline unified way for various projects/variants/realms Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 37

Slide 37 text

Continuous Delivery in Jenkins 1.x not a first class citizen in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 38

Slide 38 text

Continuous Delivery in Jenkins 1.x not a first class citizen in Jenkins 1.x bunch of jobs triggering each other can be emulated with various plugins Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ... no easy (and unified) way to setup usually even harder to maintain Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 39

Slide 39 text

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Continuous Delivery - case study custom Continuous Delivery framework on top of Jenkins Job DSL one standardized way for Continuous Delivery reused in all projects in the company Ansible for infrastructure management Rundeck for deployment open sourced to make live easier to others jenkins-pipeline-dsl - core library sample-jenkins-microservice-pipeline - sample pipeline Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 42

Slide 42 text

Jenkins 2.x Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 43

Slide 43 text

Jenkins 2.x Built-in support for delivery pipelines Pipeline Plugin Pipeline Stage View Plugin Jenkinsfile in code for pipeline definition Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 44

Slide 44 text

Jenkins 2.x Built-in support for delivery pipelines Pipeline Plugin Pipeline Stage View Plugin Jenkinsfile in code for pipeline definition node { stage 'Checkout' git url: 'https://github.com/devoxx/website.git' stage 'Publish' sh "./gradlew publish" } Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 45

Slide 45 text

Jenkins 2.x - continued Improved usability new "Getting Started" experience new configuration page redesigned "Create Item" page project Blue Ocean ...but we (mostly) don't care about the UI ;) Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 46

Slide 46 text

Jenkins 2.x - continued Improved usability new "Getting Started" experience new configuration page redesigned "Create Item" page project Blue Ocean ...but we (mostly) don't care about the UI ;) Fully backwards compatible * Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 47

Slide 47 text

Jenkins 2.x (*) requires Servlet 3.1 no AJP support with the embedded Winstone-Jetty container use HTTP with reverse proxy Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 48

Slide 48 text

Job DSL in Jenkins 2.x world works out-of-the-box jobs generated as in Jenkins 1.x Groovy 2.4 available (finally!) traits, static compilation and more to use in production code will live as long as the community is active Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 49

Slide 49 text

Job DSL vs Pipeline plugin Job DSL pros: much wider support for plugins as of June 2016 Pipeline plugin catches up better testability outside Jenkins instance not only for pipelines maturity Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 50

Slide 50 text

Job DSL vs Pipeline plugin - continued Pipeline plugin pros: flagship component of Jenkins 2 supported directly by CloudBees created with Continuous Delivery in mind pipeline as Code eye catchy visualization durability increasing popularity Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 51

Slide 51 text

Live demo The whole delivery pipeline with one click! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 52

Slide 52 text

Summary Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 53

Slide 53 text

Summary automation is good for you! Jenkins Job DSL is great together with infrastructure tools it is a match made in heaven delivery pipelines as the first class citizens in Jenkins 2.x UI and visualization is getting better all the time! Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 54

Slide 54 text

Questions? Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

Slide 55

Slide 55 text

Marcin Zajączkowski @SolidSoftBlog http://blog.solidsoft.info [email protected] IRC: szpak@freenode Łukasz Szczęsny @wybczu https://wybcz.pl luk@{wybcz.pl,uber.com} IRC: wybczu@freenode Thank you (and remember about the feedback!) Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog