Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jenkins as a code

Jenkins as a code

Co-authored with Łukasz Szczęsny.

Jobs in Jenkins (or any other CI/CD tool) can be created and updated manually using GUI. It is ok if you have a few/several of them. However, most of the companies grow and you can quickly wake up with dozens or hundreds of jobs to maintain.

A new email address/alias to get notification about failed builds? Global migration to the better™ SCM? No way to do it manually with GUI in a convenient way. It just doesn’t scale. Manual scripts using Jenkins API? Better, but hard to test and maintain. The same applies to plugin installation, credentials managements etc. Luckily, there is a better way.

During the talk we will present how Jenkins Job DSL together with Ansible can be used to automatically provision Jenkins instance and maintain any number of jobs. We will show how to define jobs and views in Groovy based DSL and test automatically that the generated structures are exactly the same as expected in Jenkins. Expect live demo - we will setup fully functional Jenkins instance with just one click!

P.S. Jenkins 2.0 brought a breath of fresh air to the ecosystem. Find out how it impacted Jenkins management.

Marcin Zajączkowski

June 24, 2016
Tweet

More Decks by Marcin Zajączkowski

Other Decks in Programming

Transcript

  1. Jenkins as a code
    Łukasz Szczęsny & Marcin Zajączkowski

    View Slide

  2. About Łukasz
    Software engineer @ Uber
    FOSS and Open Hardware lover
    Co-organizer of the Warsaw Linux User Group
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  3. About Marcin
    Areas of expertise
    Automatic Testing / TDD
    Software Craftsmanship / Code Quality
    Java8 / Groovy
    Concurrency / Parallel Computing / Reactive Systems
    Deployment Automation / Continuous Delivery
    FOSS projects author and contributor, blogger and trainer
    Leads a small software house - Codearte
    targeted at clients who care about the quality
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  4. Agenda
    Manual Jenkins maintenance
    Job configuration as code
    Jenkins Job DSL
    Infrastructure as code
    Case study - Continuous Delivery in Jenkins
    Jenkins 2.x
    Live demo
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  5. Manual Jenkins maintenance
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  6. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  7. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    problematic with dozens of jobs and plugins
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  8. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    problematic with dozens of jobs and plugins
    mission impossible with
    several microservices
    deployed in several countries
    for multiple products
    using deployment pipeline with several steps each
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  9. Automation to the rescue!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  10. Jenkins Job DSL
    Job configuration in code
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  11. (Jenkins) Job DSL - 2 parts
    Domain Specific Language
    to specify job configuration
    Jenkins plugin
    to transform configuration DSL into real jobs in Jenkins
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  12. Job DSL - part 1 - configuration
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  13. Job DSL - part 1 - configuration
    Groovy based DSL (Domain Specific Language)
    job/view/dashboard configuration
    developed as "normal" code in IDE with
    auto-completion
    type check
    automatic testing
    Groovy magic if needed
    outside Jenkins instance
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  14. Job DSL - simple example
    job('devoxx-website-publish') {
    scm {
    github('devoxx/website')
    }
    triggers {
    scm('*/15 * * * *')
    }
    steps {
    shell('mkdir -p /srv/devoxx.pl')
    gradle('publish', '-PdestDir=/srv/devoxx.pl')
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  15. Job DSL - dynamic example
    String repo = 'devoxx/mobile-app'
    URL branchUrl = "https://api.github.com/repos/$repo/branches".toURL()
    List branches = new JsonSlurper().parseText(branchUrl.text)
    branches.each { branch ->
    String safeBranchName = branch.name.replaceAll('/', '-')
    job("$repo-$safeBranchName-build") {
    scm {
    github repo, branch.name
    }
    triggers {
    scm 'H/10 * * * *'
    }
    steps {
    gradle 'check'
    }
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  16. Job DSL - features
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  17. Job DSL - features
    comprehensive support for Jenkins Core stuff
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  18. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    almost 200 plugins as of version 1.47
    active community - continuous flow of new pull requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  19. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    almost 200 plugins as of version 1.47
    active community - continuous flow of new pull requests
    powerful configuration block for
    not yet supported features
    custom stuff
    virtually everything possible in XML should be achievable in DSL
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  20. Job DSL - part 2 - Jenkins plugin
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  21. Job DSL - part 2 - Jenkins plugin
    installed on Jenkins instance
    used in seed jobs on Jenkins
    leverages DSL configuration
    updates jobs & views in Jenkins
    to bring them to desired state
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  22. Job DSL - benefits
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  23. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  24. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  25. Job DSL - benefits
    source code instead of XML or UI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    scalable
    hundreds of jobs created/modified in seconds*
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  26. Job DSL - drawbacks/limitations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  27. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  28. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  29. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    not suitable for global Jenkins configuration management
    credentials, machine provisioning, Jenkins and plugin update, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  30. Infrastructure
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  31. Infrastructure challenges
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  32. Infrastructure challenges
    install and configure Jenkins master
    install and configure all required dependencies
    install and configure plugins
    create and connect slaves
    add JDK installation
    configure authentication
    create credentials
    ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  33. Infrastructure toolbelt
    configuration management tools
    Ansible
    Puppet
    Chef
    Salt
    etc.
    Groovy + ${JENKINS_HOME}/groovy.init[.d]
    Jenkins CLI
    system-config-dsl-plugin
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  34. Infrastructure toolbelt
    Slave management
    Swarm plugin
    Amazon EC2 plugin
    Docker plugin
    SSH slaves
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  35. Continuous Delivery in Jenkins
    Case study
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  36. Continuous Delivery
    Clearly defined way how to transform source code
    into project deployed to production
    a set of steps arranged into pipeline
    unified way for various projects/variants/realms
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  37. Continuous Delivery in Jenkins 1.x
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  38. Continuous Delivery in Jenkins 1.x
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline Plugin, ...
    no easy (and unified) way to setup
    usually even harder to maintain
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  39. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  40. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  41. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    open sourced to make live easier to others
    jenkins-pipeline-dsl - core library
    sample-jenkins-microservice-pipeline - sample pipeline
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  42. Jenkins 2.x
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  43. Jenkins 2.x
    Built-in support for delivery pipelines
    Pipeline Plugin
    Pipeline Stage View Plugin
    Jenkinsfile in code for pipeline definition
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  44. Jenkins 2.x
    Built-in support for delivery pipelines
    Pipeline Plugin
    Pipeline Stage View Plugin
    Jenkinsfile in code for pipeline definition
    node {
    stage 'Checkout'
    git url: 'https://github.com/devoxx/website.git'
    stage 'Publish'
    sh "./gradlew publish"
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  45. Jenkins 2.x - continued
    Improved usability
    new "Getting Started" experience
    new configuration page
    redesigned "Create Item" page
    project Blue Ocean
    ...but we (mostly) don't care about the UI ;)
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  46. Jenkins 2.x - continued
    Improved usability
    new "Getting Started" experience
    new configuration page
    redesigned "Create Item" page
    project Blue Ocean
    ...but we (mostly) don't care about the UI ;)
    Fully backwards compatible *
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  47. Jenkins 2.x
    (*)
    requires Servlet 3.1
    no AJP support with the embedded Winstone-Jetty container
    use HTTP with reverse proxy
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  48. Job DSL in Jenkins 2.x world
    works out-of-the-box
    jobs generated as in Jenkins 1.x
    Groovy 2.4 available (finally!)
    traits, static compilation and more to use in production code
    will live as long as the community is active
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  49. Job DSL vs Pipeline plugin
    Job DSL pros:
    much wider support for plugins
    as of June 2016
    Pipeline plugin catches up
    better testability outside Jenkins instance
    not only for pipelines
    maturity
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  50. Job DSL vs Pipeline plugin - continued
    Pipeline plugin pros:
    flagship component of Jenkins 2
    supported directly by CloudBees
    created with Continuous Delivery in mind
    pipeline as Code
    eye catchy visualization
    durability
    increasing popularity
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  51. Live demo
    The whole delivery pipeline with one click!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  52. Summary
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  53. Summary
    automation is good for you!
    Jenkins Job DSL is great
    together with infrastructure tools it is a match made in heaven
    delivery pipelines as the first class citizens in Jenkins 2.x
    UI and visualization is getting better all the time!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  54. Questions?
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  55. Marcin Zajączkowski
    @SolidSoftBlog
    http://blog.solidsoft.info
    [email protected]
    IRC: [email protected]
    Łukasz Szczęsny
    @wybczu
    https://wybcz.pl
    [email protected]{wybcz.pl,uber.com}
    IRC: [email protected]
    Thank you
    (and remember about the feedback!)
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide