Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DW|JW 2019 - Docker and Jenkins [as code]

Oleg Nenashev
December 04, 2019

DW|JW 2019 - Docker and Jenkins [as code]

Docker is everywhere in the CI/CD pipelines and in development tools. Jenkins, one of the most popular automation servers, is actively evolving to improve its support of container technologies and static configurations in images. The Configuration as Code plugin is a new milestone which enables managing Jenkins configurations via YAML. Together with Docker, this plugin offers many ways to produce ready-to-fly Jenkins images for any environments. In our talk, we will describe official master and agent images offered by the Jenkins project. What's inside them? How do you configure images with JCasC and Groovy hooks? How do you use these approaches together? How do CloudBees Jenkins Distribution and CloudBees Assurance program help with plugin management? And what's next for Jenkins Docker packaging and JCasC?

Presentation by Mark Waite and me at DevOps World | Jenkins World 2019. https://sched.co/UauW

Oleg Nenashev

December 04, 2019
Tweet

More Decks by Oleg Nenashev

Other Decks in Technology

Transcript

  1. Docker and Jenkins [as code]

    View Slide

  2. © 2019 All Rights Reserved. 2
    About us
    @oleg_nenashev
    oleg-nenashev
    @MarkEWaite
    MarkEWaite
    Long-time contributors
    Platform SIG
    https://jenkins.io/sigs/platform
    Docker image maintainers
    CloudBees

    View Slide

  3. © 2019 All Rights Reserved. 3
    About you

    View Slide

  4. © 2019 All Rights Reserved. 4
    Today
    • Docker packaging for Jenkins
    • Configuration as Code for Jenkins and Docker
    • What about CloudBees?
    • What’s next for Jenkins and Docker?
    HINT: Join us at the
    community booth for
    deep dive!

    View Slide

  5. © 2019 All Rights Reserved. 5
    CI/CD. Why containers?
    Config Management
    Fast provisioning
    Clean environments
    Disposability
    Clustering 5

    View Slide

  6. © 2019 All Rights Reserved. 6

    View Slide

  7. © 2019 All Rights Reserved. 7
    Docker
    Docker Pipeline
    Docker Build Steps
    Yet Another Docker
    ...
    Kubernetes
    Kubernetes Pipeline
    Jenkins & Docker - Plugins

    View Slide

  8. © 2019 All Rights Reserved. 8
    Example: Docker Pipeline Plugin
    https://plugins.jenkins.io/docker-workflow
    docker.image('onenashev/gcc-riscv:6.4').inside {
    checkout scm
    sh "make clean test"
    }

    View Slide

  9. © 2019 All Rights Reserved. 9
    Jenkins & Docker integrations keep evolving
    https://plugins.jenkins.io/docker-workflow

    View Slide

  10. © 2019 All Rights Reserved. 10
    Jenkins runs in Docker

    View Slide

  11. © 2019 All Rights Reserved. 11
    Jenkins & Docker. Packaging
    https://hub.docker.com/r/jenkins/

    View Slide

  12. © 2019 All Rights Reserved. 12
    ● jenkins/slave – base image with Remoting
    ● jenkins/jnlp-slave – JNLP agents
    ● jenkins/ssh-slave – SSH agents
    ● jenkins/remoting-kafka-agent - Kafka Agents
    Official Agent Images
    https://hub.docker.com/r/jenkins/

    View Slide

  13. © 2019 All Rights Reserved. 13
    Remoting over Apache Kafka
    GSoC 2018 / 2019
    Jenkins networking
    over Apache Kafka
    Docker images
    Helm charts for agents
    and Masters
    https://github.com/jenkinsci/remoting-kafka-plugin

    View Slide

  14. © 2019 All Rights Reserved. 14
    Jenkins Masters images
    • Production - jenkins/jenkins
    • Weekly and LTS releases
    • OpenJDK 8 and 11
    • Debian and CentOS
    • Alpine and Debian Slim for OpenJDK 8

    View Slide

  15. © 2019 All Rights Reserved. 15
    Jenkins Masters images
    • Production - jenkins/jenkins
    • Weekly and LTS releases
    • OpenJDK 8 and 11
    • Debian and CentOS
    • Alpine and Debian Slim for OpenJDK 8
    • Experimental - jenkins4eval/jenkins
    • ARM, ARM 64, IBM s/390, PowerPC 64 LE
    • Continuous delivery for Jenkins master branches
    • More platforms are coming soon

    View Slide

  16. © 2019 All Rights Reserved. 16
    Official image: jenkins/jenkins
    docker run --rm \
    -p 8080:8080 \
    -p 50000:50000 \
    -v jenkins_home:/var/jenkins_home \
    jenkins/jenkins:lts
    https://github.com/jenkinsci/docker

    View Slide

  17. © 2019 All Rights Reserved. 17
    Jenkins Image is not enough
    ● No plugins
    ● No configuration
    ● No jobs

    View Slide

  18. © 2019 All Rights Reserved. 18

    View Slide

  19. © 2019 All Rights Reserved. 19
    Configuration as Code in Jenkins
    Jobs
    System
    configuration

    View Slide

  20. © 2019 All Rights Reserved. 20
    Docker images
    One FROM to rule the world

    View Slide

  21. © 2019 All Rights Reserved. 21
    Extending Docker images
    Image source:
    http://container.farm/special-extended-container/
    YOUR
    STUFF

    View Slide

  22. © 2019 All Rights Reserved. 22
    Extending Docker images
    Tools
    Environment
    Configuration
    Plugins
    Docs: https://github.com/jenkinsci/docker

    View Slide

  23. © 2019 All Rights Reserved. 23
    Dockerfile

    View Slide

  24. © 2019 All Rights Reserved. 24
    Plugin management

    View Slide

  25. © 2019 All Rights Reserved. 25
    Plugin bundling
    jenkins/jenkins:2.190.3

    View Slide

  26. © 2019 All Rights Reserved. 26
    Plugin bundling
    jenkins/jenkins:2.190.3
    # Pinned version
    blueocean:1.8.2
    # Latest
    antisamy-markup-formatter
    matrix-auth:latest
    # Latest from the Experimental update center
    git-client:experimental
    # Incrementals (JEP-305)
    workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.21-rc591.43d37d4d080a
    plugins.txt

    View Slide

  27. © 2019 All Rights Reserved. 27
    Limitations
    • Latest versions of non-specified
    dependencies are installed
    • No “Check for updates” feature
    • Plaintext Format
    • No YAML
    • No Dependabot integration, etc.

    View Slide

  28. © 2019 All Rights Reserved. 28
    Plugin installation manager tool
    https://github.com/jenkinsci/plugin-installation-manager-tool
    • CLI tool
    • Java, not bash script
    • YAML support
    • Updates listing
    • Transient dependency
    management
    plugins:
    artifactId: git
    source:
    version: latest
    artifactId: job-import-plugin
    source:
    version: 3.1
    artifactId: docker
    artifactId: docker-commons
    source:
    version: 2.4.4
    artifactId: matrix-project

    View Slide

  29. © 2019 All Rights Reserved. 29
    Configurations

    View Slide

  30. © 2019 All Rights Reserved. 30
    Groovy Hooks [1/2]
    • Standard Groovy engine
    • Triggered by Jenkins
    • Direct access to the runtime
    • Extensible hook engine
    • 2 hook types in the core: init, boot-failure
    https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

    View Slide

  31. © 2019 All Rights Reserved. 31
    Groovy Hooks [2/2]
    Script locations:
    • WEB-INF/HOOK.groovy in jenkins.war
    • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war
    • $JENKINS_HOME/HOOK.groovy
    • $JENKINS_HOME/HOOK.groovy.d/*.groovy
    Execution in alphabetical order
    https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

    View Slide

  32. © 2019 All Rights Reserved. 32
    Jenkins Startup Flow
    JOBS_LOADED
    EXTENSIONS_AUGMENTED
    COMPLETED
    Groovy Hooks: init()
    PLUGINS_STARTED
    . . .

    View Slide

  33. © 2019 All Rights Reserved. 33
    Example
    https://github.com/oleg-nenashev/demo-jenkins-config-as-code

    View Slide

  34. © 2019 All Rights Reserved. 34
    Adding Groovy hooks to the standard image
    Just put Groovy scripts in the right place!
    https://github.com/oleg-nenashev/demo-jenkins-config-as-code

    View Slide

  35. © 2019 All Rights Reserved. 35
    https://github.com/oleg-nenashev/demo-jenkins-config-as-code

    View Slide

  36. © 2019 All Rights Reserved. 36
    More examples
    oleg-nenashev/demo-jenkins-config-as-code
    Praqma/JenkinsAsCodeReference
    librecores/librecores-ci

    View Slide

  37. © 2019 All Rights Reserved. 37
    Add JCasC Plugin!
    Configuration-as-Code Plugin
    https://plugins.jenkins.io/configuration-as-code

    View Slide

  38. © 2019 All Rights Reserved. 38
    JCasC Plugin
    ● Sub-project in Jenkins
    ● Feb 2018 - first alpha
    ● Sep 2018 - 1.0
    ● Now - 1.35
    ● Configuration via YAML
    jenkinsci/configuration-as-code-plugin
    gitter.im/jenkinsci/configuration-as-code-plugin

    View Slide

  39. © 2019 All Rights Reserved. 39
    jenkins:
    systemMessage: "JCasC Demo"
    numExecutors: 1
    scmCheckoutRetryCount: 4
    mode: NORMAL
    securityRealm:
    local:
    allowsSignup: false
    users:
    - id: demoAdmin
    password: ${adminpw}
    jenkins.yaml
    YAML specification

    View Slide

  40. © 2019 All Rights Reserved. 40
    • Configuration export from Jenkins
    • CLI and REST API for configuration management
    • Pre-flight validations and dry-run
    • Support for many plugins out-of-the-box
    • External credentials
    • Job/Folder management via JobDSL
    Features

    View Slide

  41. © 2019 All Rights Reserved. 41
    Adding JCasC to the Docker image
    Configuration file
    Needs JCasC Plugin

    configuration-as-code:1.35

    View Slide

  42. © 2019 All Rights Reserved. 42
    Combining JCasC and Groovy hooks
    JOBS_LOADED
    EXTENSIONS_AUGMENTED
    COMPLETED
    Groovy Hooks: init()
    JCasC
    PLUGINS_STARTED
    . . .
    Groovy hooks run after JCasC
    Hooks can fine-tune the configuration:
    • Missing integrations
    • Dynamic scripting
    • Job configurations and migrations
    Use JCasC where possible!

    View Slide

  43. © 2019 All Rights Reserved. 43
    Comparison
    JCasC Plugin Groovy Hooks
    YAML – declarative definition
    Low barrier to entry
    Limited integrations
    Forward compatibility risks
    External rollback
    Apply changes w/o restart
    No debugging, YAGNI?
    Dry run mode + validation
    Export from UI
    CLI and REST API for CM
    Plugin management
    Groovy – general-purpose language
    High barrier to entry
    Can manage everything
    Compatible if API is stable
    External rollback
    Restart is required
    Debugging support
    Plugin Management

    View Slide

  44. © 2019 All Rights Reserved. 44
    Comparison
    JCasC Plugin Groovy Hooks
    YAML – declarative definition
    Low barrier to entry
    Limited integrations
    Forward compatibility risks
    External rollback
    Apply changes w/o restart
    No debugging, YAGNI?
    Dry run mode + validation
    Export from UI
    CLI and REST API for CM
    Plugin management
    Groovy – general-purpose language
    High barrier to entry
    Can manage everything
    Compatible if API is stable
    External rollback
    Restart is required
    Debugging support
    Plugin Management
    Use JCasC when possible, combine
    with Groovy hooks when needed

    View Slide

  45. Mark’s success story
    Fast setup, better interactive testing

    View Slide

  46. © 2019 All Rights Reserved. 46
    Git Plugin Development - Don’t Break the Users!
    • Automated tests are great, but…
    • Most git plugin bugs are not detected by automated tests
    • Git plugin users don’t like being broken by plugin upgrades
    • Choices
    • Automate everything, don’t release until it is all automated
    • Interactive tests of interesting and useful cases in addition to
    automation
    • Interactive test configuration is COMPLICATED
    • Platforms - Alpine, CentOS, Debian, Fedora, FreeBSD, OpenSUSE,
    Windows
    • Git versions - 1.8, 1.9, 2.11, 2.17, 2.23, 2.24, …
    • ANSWER: Simplify test setup by managing it in a repository (several repos)

    View Slide

  47. © 2019 All Rights Reserved. 47
    Faster Interactive Test Setup - Docker & Jenkins
    Public Repo
    • LTS
    • Debian
    • Debian Slim
    • Alpine
    • JDK 11
    • LTS, plugins
    • Debian
    • Debian Slim
    • Alpine
    • JDK 11
    Private Repo
    • LTS, plugins, & credentials
    • Debian
    • Debian Slim
    • Alpine
    • JDK 11
    • LTS, plugins, credentials & agents
    • Debian
    • Debian Slim
    • Alpine
    • JDK 11
    https://github.com/MarkEWaite/docker-lfs
    $ docker_build.py
    $ docker_run.py --clean

    View Slide

  48. © 2019 All Rights Reserved. 48
    Fully Configured Jenkins
    • 30+ agents
    • 10+ git versions
    • Linux - Debian, CentOS, Ubuntu
    • Two versions of each
    • FreeBSD & Windows agents
    • Raspberry Pi 4 agent
    • Cloud and local agents
    • 1000+ test jobs
    • Specific bug check jobs
    • Easy to start, easy to stop
    • Any Linux machine in network
    https://github.com/MarkEWaite/docker-lfs

    View Slide

  49. © 2019 All Rights Reserved. 49
    © 2019 CloudBees, Inc. All Rights Reserved.
    What about CloudBees?

    View Slide

  50. © 2019 All Rights Reserved. 50
    Products
    ● All products are available as Docker images
    ○ https://hub.docker.com/u/cloudbees
    ● Helm charts for CloudBees Jenkins Distribution
    ● CloudBees Jenkins Distribution and CloudBees Core
    run on-premise or in the cloud
    ○ Docker
    ○ Kubernetes - Helm Charts
    ○ Virtual machines
    ○ Bare metal

    View Slide

  51. © 2019 All Rights Reserved. 51
    Plugins
    ● All Jenkins Docker plugins are available to customers
    ● Extended SLA for several plugins
    https://docs.cloudbees.com/plugins

    View Slide

  52. © 2019 All Rights Reserved. 52
    JCasC in CloudBees products
    ● Tier 2 (Compatible) plugin
    ○ SLA: Bugfixes and Security fixes
    ○ CloudBees Jenkins Support
    ○ CloudBees Jenkins Distribution
    ● Technical preview for CloudBees Core
    ○ Standalone and Managed masters
    ○ Configuration Bundles feature
    http://bit.ly/jcasc-cloudbees

    View Slide

  53. © 2019 All Rights Reserved. 53
    Plugin Management
    ● Added value features for Docker
    ● CloudBees Assurance Program and Beekeper
    ○ https://docs.cloudbees.com/docs/admin-resources/latest/
    assurance-program/
    ● Plugin catalog for CloudBees Core
    ○ https://docs.cloudbees.com/docs/admin-resources/latest/
    plugin-management/configuring-plugin-catalogs

    View Slide

  54. © 2019 All Rights Reserved. 54
    CloudBees
    Assurance Program
    ● Verified plugin sets
    ● Upgrade assistant
    ● Administrative monitors
    ● Opt-in automatic upgrade

    View Slide

  55. © 2019 All Rights Reserved. 55
    Lists the set of plugins, plugin versions and
    plugin dependencies that CloudBees has
    tested for compatibility and security.
    Administrative dashboard provides a
    centralized view of the monitored
    Jenkins plugins, recommended actions,
    and configuration options available.

    View Slide

  56. © 2019 All Rights Reserved. 56
    JCasC in CloudBees Jenkins Distribution
    ● Combines with CloudBees Assurance Program
    ● Documentation and Quick Start
    ○ http://bit.ly/cjd-jcasc
    ● Demo
    ○ https://github.com/cloudbees-oss/cjd-jcasc-demo

    View Slide

  57. © 2019 All Rights Reserved. 57
    Back to Jenkins and Docker

    View Slide

  58. © 2019 All Rights Reserved. 58
    Upcoming changes in JCasC
    ● Hardening the implementation
    ● New features
    ● Plugin compatibility is a focus
    ○ Fixes in ~50 plugins
    ● Jenkins core architectural changes
    ○ Read-only configuration UI
    ○ Initialization milestones
    ● Developer tools (VS Code Plugin, etc.)

    View Slide

  59. © 2019 All Rights Reserved. 59
    Upcoming changes in Docker
    ● New master images / platforms
    ○ Windows
    ○ Ubuntu
    ○ UBI
    ○ AdoptOpenJDK with HotSpot
    ○ AdoptOpenJDK with OpenJ9
    ● Multi-platform Docker image
    ● New agent images

    View Slide

  60. © 2019 All Rights Reserved. 60
    Takeaways
    1. Dockerize Jenkins
    2. Use & extend official images
    3. Use static configurations, pin
    all versions
    4. Use Configuration-as-Code

    View Slide

  61. © 2019 All Rights Reserved. 61
    It is a great time to participate!
    Join Platform SIG
    • https://jenkins.io/sigs/platform/
    • https://gitter.im/jenkinsci/platform-sig
    Contribute
    • https://jenkins.io/participate/
    • https://github.com/jenkinsci/docker
    • https://github.com/jenkinsci/configuration-as-code-plugin

    View Slide

  62. © 2019 All Rights Reserved. 62
    Join us at the community booth!
    • Any questions
    • More demos
    • Custom image
    packager
    • JCasC experts
    • How to contribute?
    • ...

    View Slide

  63. View Slide

  64. © 2019 All Rights Reserved. 64
    DevOptics
    Software Delivery
    Visibility & Insights
    Core
    Unified Software
    Delivery &
    Governance
    CodeShip
    CI/CD as a
    Service
    Flow
    Adaptive Release
    Orchestration
    DevOps Excellence
    Jenkins
    CloudBees Jenkins
    Distribution
    CloudBees Jenkins X
    Distribution
    24x7 Technical Support
    Assisted Updates
    Support
    Accelerator
    Training
    Customer Success Managers
    DevOps Consultants
    Rollout
    Feature
    Flag
    Management
    Continuous Delivery Products and Services

    View Slide