DW|JW 2019 - Docker and Jenkins [as code]

568e3391c8b528f2b255443e4cca27ca?s=47 Oleg Nenashev
December 04, 2019

DW|JW 2019 - Docker and Jenkins [as code]

Docker is everywhere in the CI/CD pipelines and in development tools. Jenkins, one of the most popular automation servers, is actively evolving to improve its support of container technologies and static configurations in images. The Configuration as Code plugin is a new milestone which enables managing Jenkins configurations via YAML. Together with Docker, this plugin offers many ways to produce ready-to-fly Jenkins images for any environments. In our talk, we will describe official master and agent images offered by the Jenkins project. What's inside them? How do you configure images with JCasC and Groovy hooks? How do you use these approaches together? How do CloudBees Jenkins Distribution and CloudBees Assurance program help with plugin management? And what's next for Jenkins Docker packaging and JCasC?

Presentation by Mark Waite and me at DevOps World | Jenkins World 2019. https://sched.co/UauW

568e3391c8b528f2b255443e4cca27ca?s=128

Oleg Nenashev

December 04, 2019
Tweet

Transcript

  1. Docker and Jenkins [as code]

  2. © 2019 All Rights Reserved. 2 About us @oleg_nenashev oleg-nenashev

    @MarkEWaite MarkEWaite Long-time contributors Platform SIG https://jenkins.io/sigs/platform Docker image maintainers CloudBees
  3. © 2019 All Rights Reserved. 3 About you

  4. © 2019 All Rights Reserved. 4 Today • Docker packaging

    for Jenkins • Configuration as Code for Jenkins and Docker • What about CloudBees? • What’s next for Jenkins and Docker? HINT: Join us at the community booth for deep dive!
  5. © 2019 All Rights Reserved. 5 CI/CD. Why containers? Config

    Management Fast provisioning Clean environments Disposability Clustering 5
  6. © 2019 All Rights Reserved. 6

  7. © 2019 All Rights Reserved. 7 Docker Docker Pipeline Docker

    Build Steps Yet Another Docker ... Kubernetes Kubernetes Pipeline Jenkins & Docker - Plugins
  8. © 2019 All Rights Reserved. 8 Example: Docker Pipeline Plugin

    https://plugins.jenkins.io/docker-workflow docker.image('onenashev/gcc-riscv:6.4').inside { checkout scm sh "make clean test" }
  9. © 2019 All Rights Reserved. 9 Jenkins & Docker integrations

    keep evolving https://plugins.jenkins.io/docker-workflow
  10. © 2019 All Rights Reserved. 10 Jenkins runs in Docker

  11. © 2019 All Rights Reserved. 11 Jenkins & Docker. Packaging

    https://hub.docker.com/r/jenkins/
  12. © 2019 All Rights Reserved. 12 • jenkins/slave – base

    image with Remoting • jenkins/jnlp-slave – JNLP agents • jenkins/ssh-slave – SSH agents • jenkins/remoting-kafka-agent - Kafka Agents Official Agent Images https://hub.docker.com/r/jenkins/
  13. © 2019 All Rights Reserved. 13 Remoting over Apache Kafka

    GSoC 2018 / 2019 Jenkins networking over Apache Kafka Docker images Helm charts for agents and Masters https://github.com/jenkinsci/remoting-kafka-plugin
  14. © 2019 All Rights Reserved. 14 Jenkins Masters images •

    Production - jenkins/jenkins • Weekly and LTS releases • OpenJDK 8 and 11 • Debian and CentOS • Alpine and Debian Slim for OpenJDK 8
  15. © 2019 All Rights Reserved. 15 Jenkins Masters images •

    Production - jenkins/jenkins • Weekly and LTS releases • OpenJDK 8 and 11 • Debian and CentOS • Alpine and Debian Slim for OpenJDK 8 • Experimental - jenkins4eval/jenkins • ARM, ARM 64, IBM s/390, PowerPC 64 LE • Continuous delivery for Jenkins master branches • More platforms are coming soon
  16. © 2019 All Rights Reserved. 16 Official image: jenkins/jenkins docker

    run --rm \ -p 8080:8080 \ -p 50000:50000 \ -v jenkins_home:/var/jenkins_home \ jenkins/jenkins:lts https://github.com/jenkinsci/docker
  17. © 2019 All Rights Reserved. 17 Jenkins Image is not

    enough • No plugins • No configuration • No jobs
  18. © 2019 All Rights Reserved. 18

  19. © 2019 All Rights Reserved. 19 Configuration as Code in

    Jenkins Jobs System configuration
  20. © 2019 All Rights Reserved. 20 Docker images One FROM

    to rule the world
  21. © 2019 All Rights Reserved. 21 Extending Docker images Image

    source: http://container.farm/special-extended-container/ YOUR STUFF
  22. © 2019 All Rights Reserved. 22 Extending Docker images Tools

    Environment Configuration Plugins Docs: https://github.com/jenkinsci/docker
  23. © 2019 All Rights Reserved. 23 Dockerfile

  24. © 2019 All Rights Reserved. 24 Plugin management

  25. © 2019 All Rights Reserved. 25 Plugin bundling jenkins/jenkins:2.190.3

  26. © 2019 All Rights Reserved. 26 Plugin bundling jenkins/jenkins:2.190.3 #

    Pinned version blueocean:1.8.2 # Latest antisamy-markup-formatter matrix-auth:latest # Latest from the Experimental update center git-client:experimental # Incrementals (JEP-305) workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.21-rc591.43d37d4d080a plugins.txt
  27. © 2019 All Rights Reserved. 27 Limitations • Latest versions

    of non-specified dependencies are installed • No “Check for updates” feature • Plaintext Format • No YAML • No Dependabot integration, etc.
  28. © 2019 All Rights Reserved. 28 Plugin installation manager tool

    https://github.com/jenkinsci/plugin-installation-manager-tool • CLI tool • Java, not bash script • YAML support • Updates listing • Transient dependency management plugins: artifactId: git source: version: latest artifactId: job-import-plugin source: version: 3.1 artifactId: docker artifactId: docker-commons source: version: 2.4.4 artifactId: matrix-project
  29. © 2019 All Rights Reserved. 29 Configurations

  30. © 2019 All Rights Reserved. 30 Groovy Hooks [1/2] •

    Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • 2 hook types in the core: init, boot-failure https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script
  31. © 2019 All Rights Reserved. 31 Groovy Hooks [2/2] Script

    locations: • WEB-INF/HOOK.groovy in jenkins.war • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script
  32. © 2019 All Rights Reserved. 32 Jenkins Startup Flow JOBS_LOADED

    EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() PLUGINS_STARTED . . .
  33. © 2019 All Rights Reserved. 33 Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code

  34. © 2019 All Rights Reserved. 34 Adding Groovy hooks to

    the standard image Just put Groovy scripts in the right place! https://github.com/oleg-nenashev/demo-jenkins-config-as-code
  35. © 2019 All Rights Reserved. 35 https://github.com/oleg-nenashev/demo-jenkins-config-as-code

  36. © 2019 All Rights Reserved. 36 More examples oleg-nenashev/demo-jenkins-config-as-code Praqma/JenkinsAsCodeReference

    librecores/librecores-ci
  37. © 2019 All Rights Reserved. 37 Add JCasC Plugin! Configuration-as-Code

    Plugin https://plugins.jenkins.io/configuration-as-code
  38. © 2019 All Rights Reserved. 38 JCasC Plugin • Sub-project

    in Jenkins • Feb 2018 - first alpha • Sep 2018 - 1.0 • Now - 1.35 • Configuration via YAML jenkinsci/configuration-as-code-plugin gitter.im/jenkinsci/configuration-as-code-plugin
  39. © 2019 All Rights Reserved. 39 jenkins: systemMessage: "JCasC Demo"

    numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml YAML specification
  40. © 2019 All Rights Reserved. 40 • Configuration export from

    Jenkins • CLI and REST API for configuration management • Pre-flight validations and dry-run • Support for many plugins out-of-the-box • External credentials • Job/Folder management via JobDSL Features
  41. © 2019 All Rights Reserved. 41 Adding JCasC to the

    Docker image Configuration file Needs JCasC Plugin … configuration-as-code:1.35 …
  42. © 2019 All Rights Reserved. 42 Combining JCasC and Groovy

    hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . Groovy hooks run after JCasC Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations Use JCasC where possible!
  43. © 2019 All Rights Reserved. 43 Comparison JCasC Plugin Groovy

    Hooks YAML – declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for CM Plugin management Groovy – general-purpose language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support Plugin Management
  44. © 2019 All Rights Reserved. 44 Comparison JCasC Plugin Groovy

    Hooks YAML – declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for CM Plugin management Groovy – general-purpose language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support Plugin Management Use JCasC when possible, combine with Groovy hooks when needed
  45. Mark’s success story Fast setup, better interactive testing

  46. © 2019 All Rights Reserved. 46 Git Plugin Development -

    Don’t Break the Users! • Automated tests are great, but… • Most git plugin bugs are not detected by automated tests • Git plugin users don’t like being broken by plugin upgrades • Choices • Automate everything, don’t release until it is all automated • Interactive tests of interesting and useful cases in addition to automation • Interactive test configuration is COMPLICATED • Platforms - Alpine, CentOS, Debian, Fedora, FreeBSD, OpenSUSE, Windows • Git versions - 1.8, 1.9, 2.11, 2.17, 2.23, 2.24, … • ANSWER: Simplify test setup by managing it in a repository (several repos)
  47. © 2019 All Rights Reserved. 47 Faster Interactive Test Setup

    - Docker & Jenkins Public Repo • LTS • Debian • Debian Slim • Alpine • JDK 11 • LTS, plugins • Debian • Debian Slim • Alpine • JDK 11 Private Repo • LTS, plugins, & credentials • Debian • Debian Slim • Alpine • JDK 11 • LTS, plugins, credentials & agents • Debian • Debian Slim • Alpine • JDK 11 https://github.com/MarkEWaite/docker-lfs $ docker_build.py $ docker_run.py --clean
  48. © 2019 All Rights Reserved. 48 Fully Configured Jenkins •

    30+ agents • 10+ git versions • Linux - Debian, CentOS, Ubuntu • Two versions of each • FreeBSD & Windows agents • Raspberry Pi 4 agent • Cloud and local agents • 1000+ test jobs • Specific bug check jobs • Easy to start, easy to stop • Any Linux machine in network https://github.com/MarkEWaite/docker-lfs
  49. © 2019 All Rights Reserved. 49 © 2019 CloudBees, Inc.

    All Rights Reserved. What about CloudBees?
  50. © 2019 All Rights Reserved. 50 Products • All products

    are available as Docker images ◦ https://hub.docker.com/u/cloudbees • Helm charts for CloudBees Jenkins Distribution • CloudBees Jenkins Distribution and CloudBees Core run on-premise or in the cloud ◦ Docker ◦ Kubernetes - Helm Charts ◦ Virtual machines ◦ Bare metal
  51. © 2019 All Rights Reserved. 51 Plugins • All Jenkins

    Docker plugins are available to customers • Extended SLA for several plugins https://docs.cloudbees.com/plugins
  52. © 2019 All Rights Reserved. 52 JCasC in CloudBees products

    • Tier 2 (Compatible) plugin ◦ SLA: Bugfixes and Security fixes ◦ CloudBees Jenkins Support ◦ CloudBees Jenkins Distribution • Technical preview for CloudBees Core ◦ Standalone and Managed masters ◦ Configuration Bundles feature http://bit.ly/jcasc-cloudbees
  53. © 2019 All Rights Reserved. 53 Plugin Management • Added

    value features for Docker • CloudBees Assurance Program and Beekeper ◦ https://docs.cloudbees.com/docs/admin-resources/latest/ assurance-program/ • Plugin catalog for CloudBees Core ◦ https://docs.cloudbees.com/docs/admin-resources/latest/ plugin-management/configuring-plugin-catalogs
  54. © 2019 All Rights Reserved. 54 CloudBees Assurance Program •

    Verified plugin sets • Upgrade assistant • Administrative monitors • Opt-in automatic upgrade
  55. © 2019 All Rights Reserved. 55 Lists the set of

    plugins, plugin versions and plugin dependencies that CloudBees has tested for compatibility and security. Administrative dashboard provides a centralized view of the monitored Jenkins plugins, recommended actions, and configuration options available.
  56. © 2019 All Rights Reserved. 56 JCasC in CloudBees Jenkins

    Distribution • Combines with CloudBees Assurance Program • Documentation and Quick Start ◦ http://bit.ly/cjd-jcasc • Demo ◦ https://github.com/cloudbees-oss/cjd-jcasc-demo
  57. © 2019 All Rights Reserved. 57 Back to Jenkins and

    Docker
  58. © 2019 All Rights Reserved. 58 Upcoming changes in JCasC

    • Hardening the implementation • New features • Plugin compatibility is a focus ◦ Fixes in ~50 plugins • Jenkins core architectural changes ◦ Read-only configuration UI ◦ Initialization milestones • Developer tools (VS Code Plugin, etc.)
  59. © 2019 All Rights Reserved. 59 Upcoming changes in Docker

    • New master images / platforms ◦ Windows ◦ Ubuntu ◦ UBI ◦ AdoptOpenJDK with HotSpot ◦ AdoptOpenJDK with OpenJ9 • Multi-platform Docker image • New agent images
  60. © 2019 All Rights Reserved. 60 Takeaways 1. Dockerize Jenkins

    2. Use & extend official images 3. Use static configurations, pin all versions 4. Use Configuration-as-Code
  61. © 2019 All Rights Reserved. 61 It is a great

    time to participate! Join Platform SIG • https://jenkins.io/sigs/platform/ • https://gitter.im/jenkinsci/platform-sig Contribute • https://jenkins.io/participate/ • https://github.com/jenkinsci/docker • https://github.com/jenkinsci/configuration-as-code-plugin
  62. © 2019 All Rights Reserved. 62 Join us at the

    community booth! • Any questions • More demos • Custom image packager • JCasC experts • How to contribute? • ...
  63. None
  64. © 2019 All Rights Reserved. 64 DevOptics Software Delivery Visibility

    & Insights Core Unified Software Delivery & Governance CodeShip CI/CD as a Service Flow Adaptive Release Orchestration DevOps Excellence Jenkins CloudBees Jenkins Distribution CloudBees Jenkins X Distribution 24x7 Technical Support Assisted Updates Support Accelerator Training Customer Success Managers DevOps Consultants Rollout Feature Flag Management Continuous Delivery Products and Services