Jenkins as a code

Co-authored with Łukasz Szczęsny.

Jobs in Jenkins (or any other CI/CD tool) can be created and updated manually using GUI. It is ok if you have a few/several of them. However, most of the companies grow and you can quickly wake up with dozens or hundreds of jobs to maintain.

A new email address/alias to get notification about failed builds? Global migration to the better™ SCM? No way to do it manually with GUI in a convenient way. It just doesn’t scale. Manual scripts using Jenkins API? Better, but hard to test and maintain. The same applies to plugin installation, credentials managements etc. Luckily, there is a better way.

During the talk we will present how Jenkins Job DSL together with Ansible can be used to automatically provision Jenkins instance and maintain any number of jobs. We will show how to define jobs and views in Groovy based DSL and test automatically that the generated structures are exactly the same as expected in Jenkins. Expect live demo - we will setup fully functional Jenkins instance with just one click!

P.S. Jenkins 2.0 brought a breath of fresh air to the ecosystem. Find out how it impacted Jenkins management.

Transcript

1. Jenkins as a code
   Łukasz Szczęsny & Marcin Zajączkowski
   

   View Slide

2. About Łukasz
   Software engineer @ Uber
   FOSS and Open Hardware lover
   Co-organizer of the Warsaw Linux User Group
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

3. About Marcin
   Areas of expertise
   Automatic Testing / TDD
   Software Craftsmanship / Code Quality
   Java8 / Groovy
   Concurrency / Parallel Computing / Reactive Systems
   Deployment Automation / Continuous Delivery
   FOSS projects author and contributor, blogger and trainer
   Leads a small software house - Codearte
   targeted at clients who care about the quality
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

4. Agenda
   Manual Jenkins maintenance
   Job configuration as code
   Jenkins Job DSL
   Infrastructure as code
   Case study - Continuous Delivery in Jenkins
   Jenkins 2.x
   Live demo
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

5. Manual Jenkins maintenance
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

6. Manual Jenkins maintenance
   configuration via GUI does not scale
   slow, error prone, and boring
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

7. Manual Jenkins maintenance
   configuration via GUI does not scale
   slow, error prone, and boring
   problematic with dozens of jobs and plugins
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

8. Manual Jenkins maintenance
   configuration via GUI does not scale
   slow, error prone, and boring
   problematic with dozens of jobs and plugins
   mission impossible with
   several microservices
   deployed in several countries
   for multiple products
   using deployment pipeline with several steps each
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

9. Automation to the rescue!
   Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
   

   View Slide

10. Jenkins Job DSL
    Job configuration in code
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

11. (Jenkins) Job DSL - 2 parts
    Domain Specific Language
    to specify job configuration
    Jenkins plugin
    to transform configuration DSL into real jobs in Jenkins
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

12. Job DSL - part 1 - configuration
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

13. Job DSL - part 1 - configuration
    Groovy based DSL (Domain Specific Language)
    job/view/dashboard configuration
    developed as "normal" code in IDE with
    auto-completion
    type check
    automatic testing
    Groovy magic if needed
    outside Jenkins instance
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

14. Job DSL - simple example
    job('devoxx-website-publish') {
    scm {
    github('devoxx/website')
    }
    triggers {
    scm('*/15 * * * *')
    }
    steps {
    shell('mkdir -p /srv/devoxx.pl')
    gradle('publish', '-PdestDir=/srv/devoxx.pl')
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

15. Job DSL - dynamic example
    String repo = 'devoxx/mobile-app'
    URL branchUrl = "https://api.github.com/repos/$repo/branches".toURL()
    List branches = new JsonSlurper().parseText(branchUrl.text)
    branches.each { branch ->
    String safeBranchName = branch.name.replaceAll('/', '-')
    job("$repo-$safeBranchName-build") {
    scm {
    github repo, branch.name
    }
    triggers {
    scm 'H/10 * * * *'
    }
    steps {
    gradle 'check'
    }
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

16. Job DSL - features
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

17. Job DSL - features
    comprehensive support for Jenkins Core stuff
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

18. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    almost 200 plugins as of version 1.47
    active community - continuous flow of new pull requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

19. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    almost 200 plugins as of version 1.47
    active community - continuous flow of new pull requests
    powerful configuration block for
    not yet supported features
    custom stuff
    virtually everything possible in XML should be achievable in DSL
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

20. Job DSL - part 2 - Jenkins plugin
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

21. Job DSL - part 2 - Jenkins plugin
    installed on Jenkins instance
    used in seed jobs on Jenkins
    leverages DSL configuration
    updates jobs & views in Jenkins
    to bring them to desired state
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

22. Job DSL - benefits
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

23. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

24. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

25. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    scalable
    hundreds of jobs created/modified in seconds*
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

26. Job DSL - drawbacks/limitations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

27. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

28. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

29. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    not suitable for global Jenkins configuration management
    credentials, machine provisioning, Jenkins and plugin update, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

30. Infrastructure
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

31. Infrastructure challenges
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

32. Infrastructure challenges
    install and configure Jenkins master
    install and configure all required dependencies
    install and configure plugins
    create and connect slaves
    add JDK installation
    configure authentication
    create credentials
    ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

33. Infrastructure toolbelt
    configuration management tools
    Ansible
    Puppet
    Chef
    Salt
    etc.
    Groovy + ${JENKINS_HOME}/groovy.init[.d]
    Jenkins CLI
    system-config-dsl-plugin
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

34. Infrastructure toolbelt
    Slave management
    Swarm plugin
    Amazon EC2 plugin
    Docker plugin
    SSH slaves
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

35. Continuous Delivery in Jenkins
    Case study
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

36. Continuous Delivery
    Clearly defined way how to transform source code
    into project deployed to production
    a set of steps arranged into pipeline
    unified way for various projects/variants/realms
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

37. Continuous Delivery in Jenkins 1.x
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

38. Continuous Delivery in Jenkins 1.x
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ...
    no easy (and unified) way to setup
    usually even harder to maintain
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

39. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

40. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

41. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    open sourced to make live easier to others
    jenkins-pipeline-dsl - core library
    sample-jenkins-microservice-pipeline - sample pipeline
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

42. Jenkins 2.x
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

43. Jenkins 2.x
    Built-in support for delivery pipelines
    Pipeline Plugin
    Pipeline Stage View Plugin
    Jenkinsfile in code for pipeline definition
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

44. Jenkins 2.x
    Built-in support for delivery pipelines
    Pipeline Plugin
    Pipeline Stage View Plugin
    Jenkinsfile in code for pipeline definition
    node {
    stage 'Checkout'
    git url: 'https://github.com/devoxx/website.git'
    stage 'Publish'
    sh "./gradlew publish"
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

45. Jenkins 2.x - continued
    Improved usability
    new "Getting Started" experience
    new configuration page
    redesigned "Create Item" page
    project Blue Ocean
    ...but we (mostly) don't care about the UI ;)
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

46. Jenkins 2.x - continued
    Improved usability
    new "Getting Started" experience
    new configuration page
    redesigned "Create Item" page
    project Blue Ocean
    ...but we (mostly) don't care about the UI ;)
    Fully backwards compatible *
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

47. Jenkins 2.x
    (*)
    requires Servlet 3.1
    no AJP support with the embedded Winstone-Jetty container
    use HTTP with reverse proxy
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

48. Job DSL in Jenkins 2.x world
    works out-of-the-box
    jobs generated as in Jenkins 1.x
    Groovy 2.4 available (finally!)
    traits, static compilation and more to use in production code
    will live as long as the community is active
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

49. Job DSL vs Pipeline plugin
    Job DSL pros:
    much wider support for plugins
    as of June 2016
    Pipeline plugin catches up
    better testability outside Jenkins instance
    not only for pipelines
    maturity
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

50. Job DSL vs Pipeline plugin - continued
    Pipeline plugin pros:
    flagship component of Jenkins 2
    supported directly by CloudBees
    created with Continuous Delivery in mind
    pipeline as Code
    eye catchy visualization
    durability
    increasing popularity
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

51. Live demo
    The whole delivery pipeline with one click!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

52. Summary
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

53. Summary
    automation is good for you!
    Jenkins Job DSL is great
    together with infrastructure tools it is a match made in heaven
    delivery pipelines as the first class citizens in Jenkins 2.x
    UI and visualization is getting better all the time!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

54. Questions?
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide

55. Marcin Zajączkowski
    @SolidSoftBlog
    http://blog.solidsoft.info
    m.zajaczkowski@gmail.com
    IRC: szpak@freenode
    Łukasz Szczęsny
    @wybczu
    https://wybcz.pl
    luk@{wybcz.pl,uber.com}
    IRC: wybczu@freenode
    Thank you
    (and remember about the feedback!)
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    

    View Slide