Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jenkins as a Code - leveraging Groovy for infrastructure management

Jenkins as a Code - leveraging Groovy for infrastructure management

Jobs in Jenkins (or any other CI/CD tool) can be created and updated manually using GUI. It is ok if you have a few/several of them. However, most of the companies grow and you can quickly wake up with dozens or hundreds of jobs to maintain.

A new email address/alias to get notification about failed builds? Global migration to the better™ SCM? No way to do it manually with GUI in a convenient way. Manual scripts using Jenkins API? Better, but hard to test and maintain. The same applies to plugin installation, credentials etc. Luckily, there is a better way.

During the talk we will present how Jenkins Job DSL together with Ansible can be used to automatically provision Jenkins instance and maintain any number of jobs. We will show how to define jobs and views in Groovy based DSL and test automatically that the generated structures are exactly the same as expected in Jenkins. Expect live demo - we will setup fully functional Jenkins instance with just one click!

Co-presented with Łukasz Szczęsny.

Marcin Zajączkowski

March 19, 2016
Tweet

More Decks by Marcin Zajączkowski

Other Decks in Programming

Transcript

  1. Jenkins as a code
    Leveraging Groovy for infrastructure management
    Łukasz Szczęsny & Marcin Zajączkowski
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog
    GR8Day Warsaw, 19th March 2016

    View Slide

  2. About Łukasz
    Software engineer @ Uber
    FOSS and Open Hardware lover
    Co-organizer of the Warsaw Linux User Group
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  3. About Marcin
    Areas of expertise
    Automatic Testing / TDD
    Software Craftsmanship / Code Quality
    Java8 / Groovy
    Concurrency / Parallel Computing / Reactive Systems
    Deployment Automation / Continuous Delivery
    FOSS projects author and contributor, blogger
    Leads a small software house - Codearte
    targeted at clients who care about the quality
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  4. Agenda
    Manual Jenkins maintenance
    Job configuration as code
    Jenkins Job DSL
    Infrastructure as code
    Case study - Continuous Delivery in Jenkins
    Live demo
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  5. Manual Jenkins maintenance
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  6. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  7. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    problematic with dozens of jobs and plugins
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  8. Manual Jenkins maintenance
    configuration via GUI does not scale
    slow, error prone, and boring
    problematic with dozens of jobs and plugins
    mission impossible with
    several microservices
    deployed in several countries
    for multiple products
    using deployment pipeline with several steps each
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  9. Automation to the rescue!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  10. Jenkins Job DSL
    Job configuration in code
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  11. (Jenkins) Job DSL - 2 parts
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  12. (Jenkins) Job DSL - 2 parts
    Domain Specific Language
    to specify job configuration
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  13. (Jenkins) Job DSL - 2 parts
    Domain Specific Language
    to specify job configuration
    Jenkins plugin
    to transform configuration DSL into real jobs in Jenkins
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  14. Job DSL - part 1 - configuration
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  15. Job DSL - part 1 - configuration
    Groovy based DSL (Domain Specific Language)
    job/view/dashboard configuration
    developed as "normal" code in IDE with
    auto-completion
    type check
    Groovy magic if needed
    outside Jenkins instance
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  16. Job DSL - simple example
    j
    o
    b
    (
    '
    g
    r
    8
    d
    a
    y
    -
    w
    e
    b
    s
    i
    t
    e
    -
    p
    u
    b
    l
    i
    s
    h
    '
    ) {
    s
    c
    m {
    g
    i
    t
    h
    u
    b
    (
    '
    g
    r
    8
    d
    a
    y
    /
    w
    e
    b
    s
    i
    t
    e
    '
    )
    }
    t
    r
    i
    g
    g
    e
    r
    s {
    s
    c
    m
    (
    '
    *
    /
    1
    5 * * * *
    '
    )
    }
    s
    t
    e
    p
    s {
    g
    r
    a
    d
    l
    e
    (
    '
    p
    u
    b
    l
    i
    s
    h
    '
    )
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  17. Job DSL - dynamic example
    S
    t
    r
    i
    n
    g r
    e
    p
    o = '
    g
    r
    8
    d
    a
    y
    /
    m
    o
    b
    i
    l
    e
    -
    a
    p
    p
    '
    U
    R
    L b
    r
    a
    n
    c
    h
    U
    r
    l = "
    h
    t
    t
    p
    s
    :
    /
    /
    a
    p
    i
    .
    g
    i
    t
    h
    u
    b
    .
    c
    o
    m
    /
    r
    e
    p
    o
    s
    /
    $
    r
    e
    p
    o
    /
    b
    r
    a
    n
    c
    h
    e
    s
    "
    .
    t
    o
    U
    R
    L
    (
    )
    L
    i
    s
    t b
    r
    a
    n
    c
    h
    e
    s = n
    e
    w J
    s
    o
    n
    S
    l
    u
    r
    p
    e
    r
    (
    )
    .
    p
    a
    r
    s
    e
    T
    e
    x
    t
    (
    b
    r
    a
    n
    c
    h
    U
    r
    l
    .
    t
    e
    x
    t
    )
    b
    r
    a
    n
    c
    h
    e
    s
    .
    e
    a
    c
    h { b
    r
    a
    n
    c
    h -
    >
    S
    t
    r
    i
    n
    g s
    a
    f
    e
    B
    r
    a
    n
    c
    h
    N
    a
    m
    e = b
    r
    a
    n
    c
    h
    .
    n
    a
    m
    e
    .
    r
    e
    p
    l
    a
    c
    e
    A
    l
    l
    (
    '
    /
    '
    , '
    -
    '
    )
    j
    o
    b
    (
    "
    $
    r
    e
    p
    o
    -
    $
    s
    a
    f
    e
    B
    r
    a
    n
    c
    h
    N
    a
    m
    e
    -
    b
    u
    i
    l
    d
    "
    ) {
    s
    c
    m {
    g
    i
    t
    h
    u
    b r
    e
    p
    o
    , b
    r
    a
    n
    c
    h
    .
    n
    a
    m
    e
    }
    t
    r
    i
    g
    g
    e
    r
    s {
    s
    c
    m '
    H
    /
    1
    0 * * * *
    '
    }
    s
    t
    e
    p
    s {
    g
    r
    a
    d
    l
    e '
    c
    h
    e
    c
    k
    '
    }
    }
    }
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  18. Job DSL - features
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  19. Job DSL - features
    comprehensive support for Jenkins Core stuff
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  20. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    over 180 plugins as of version 1.44
    active community - continuous flow of new pull
    requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  21. Job DSL - features
    comprehensive support for Jenkins Core stuff
    extensive support for additional plugins
    over 180 plugins as of version 1.44
    active community - continuous flow of new pull
    requests
    powerful c
    o
    n
    f
    i
    g
    u
    r
    a
    t
    i
    o
    n block for
    not yet supported features
    custom stuff
    virtually everything possible in XML should be achievable
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  22. Job DSL - part 2 - Jenkins plugin
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  23. Job DSL - part 2 - Jenkins plugin
    installed on Jenkins instance
    used in seed jobs on Jenkins
    leverages DSL configuration
    updates jobs & views in Jenkins
    to bring them to desired state
    by XML configuration files modification
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  24. Job DSL - benefits
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  25. Job DSL - benefits
    source code instead of XML or GUI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  26. Job DSL - benefits
    source code instead of XML or GUI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  27. Job DSL - benefits
    source code instead of XML or GUI
    single source of truth
    manageable jobs and views
    backed by SCM
    reviewable - possibly with pull requests
    testable
    automatic "unit" testing
    pre-production environment
    scalable
    hundreds of jobs created/modified in seconds*
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  28. Job DSL - drawbacks/limitations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  29. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex
    configurations
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  30. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex
    configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  31. Job DSL - drawbacks/limitations
    quite steep learning curve
    can become hard to understand for complex
    configurations
    small error in DSL can remove some/all jobs
    can be easily recreated, but without execution history
    not suitable for global Jenkins configuration management
    credentials, machine provisioning, Jenkins and plugin
    update, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  32. Infrastructure
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  33. Infrastructure challenges
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  34. Infrastructure challenges
    install and configure Jenkins master
    install and configure all required dependencies
    install and configure plugins
    create and connect slaves
    add JDK installation
    configure authentication
    create credentials
    ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  35. Infrastructure toolbelt
    configuration management tools
    Ansible
    Puppet
    Chef
    Salt
    etc.
    Groovy console
    Jenkins CLI
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  36. Infrastructure toolbelt
    Slave management
    Swarm plugin
    Docker plugin
    SSH slaves
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  37. Continuous Delivery in Jenkins
    Case study
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  38. Continuous Delivery
    Clearly defined way how to transform source code
    into project deployed to production
    a set of steps arranged into pipeline
    unified way for various projects/variants/realms
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  39. Continuous Delivery in Jenkins
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline
    Plugin, ...
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  40. Continuous Delivery in Jenkins
    not a first class citizen in Jenkins 1.x
    bunch of jobs triggering each other
    can be emulated with various plugins
    Delivery Pipeline Plugin, Build Flow Plugin, Pipeline
    Plugin, ...
    no easy (and unified) way to setup
    usually even harder to maintain
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  41. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  42. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  43. Continuous Delivery - case study
    custom Continuous Delivery framework
    on top of Jenkins Job DSL
    one standardized way for Continuous Delivery
    reused in all projects in the company
    Ansible for infrastructure management
    Rundeck for deployment
    open sourced to make live easier to others
    jenkins-pipeline-dsl - core library
    sample-jenkins-microservice-pipeline - sample pipeline
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  44. Live demo
    The whole delivery pipeline with one click!
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  45. Summary
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  46. Summary
    automation is good for you
    Jenkins Job DSL is great
    together with infrastructure tools it is a match made in
    heaven
    looking forward to Jenkins 2.0 release to simplify
    Continuous Delivery pipelines creation
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  47. Questions?
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide

  48. Marcin Zajączkowski
    @SolidSoftBlog
    http://blog.solidsoft.info
    [email protected]
    IRC: [email protected]
    Łukasz Szczęsny
    @wybczu
    https://wybcz.pl
    [email protected]{uber.com,wybcz.pl}
    IRC: [email protected]
    Thank you
    Łukasz Szczęsny & Marcin Zajączkowski @wybczu & @SolidSoftBlog

    View Slide