Tweet
Tweet

Slide 1

Slide 1 text

Docker and Jenkins [as code]

Slide 2

Slide 2 text

© 2019 All Rights Reserved. 2 About us @oleg_nenashev oleg-nenashev @MarkEWaite MarkEWaite Long-time contributors Platform SIG https://jenkins.io/sigs/platform Docker image maintainers CloudBees

Slide 3

Slide 3 text

© 2019 All Rights Reserved. 3 About you

Slide 4

Slide 4 text

© 2019 All Rights Reserved. 4 Today • Docker packaging for Jenkins • Configuration as Code for Jenkins and Docker • What about CloudBees? • What’s next for Jenkins and Docker? HINT: Join us at the community booth for deep dive!

Slide 5

Slide 5 text

© 2019 All Rights Reserved. 5 CI/CD. Why containers? Config Management Fast provisioning Clean environments Disposability Clustering 5

Slide 6

Slide 6 text

© 2019 All Rights Reserved. 6

Slide 7

Slide 7 text

© 2019 All Rights Reserved. 7 Docker Docker Pipeline Docker Build Steps Yet Another Docker ... Kubernetes Kubernetes Pipeline Jenkins & Docker - Plugins

Slide 8

Slide 8 text

© 2019 All Rights Reserved. 8 Example: Docker Pipeline Plugin https://plugins.jenkins.io/docker-workflow docker.image('onenashev/gcc-riscv:6.4').inside { checkout scm sh "make clean test" }

Slide 9

Slide 9 text

© 2019 All Rights Reserved. 9 Jenkins & Docker integrations keep evolving https://plugins.jenkins.io/docker-workflow

Slide 10

Slide 10 text

© 2019 All Rights Reserved. 10 Jenkins runs in Docker

Slide 11

Slide 11 text

© 2019 All Rights Reserved. 11 Jenkins & Docker. Packaging https://hub.docker.com/r/jenkins/

Slide 12

Slide 12 text

© 2019 All Rights Reserved. 12 ● jenkins/slave – base image with Remoting ● jenkins/jnlp-slave – JNLP agents ● jenkins/ssh-slave – SSH agents ● jenkins/remoting-kafka-agent - Kafka Agents Official Agent Images https://hub.docker.com/r/jenkins/

Slide 13

Slide 13 text

© 2019 All Rights Reserved. 13 Remoting over Apache Kafka GSoC 2018 / 2019 Jenkins networking over Apache Kafka Docker images Helm charts for agents and Masters https://github.com/jenkinsci/remoting-kafka-plugin

Slide 14

Slide 14 text

© 2019 All Rights Reserved. 14 Jenkins Masters images • Production - jenkins/jenkins • Weekly and LTS releases • OpenJDK 8 and 11 • Debian and CentOS • Alpine and Debian Slim for OpenJDK 8

Slide 15

Slide 15 text

© 2019 All Rights Reserved. 15 Jenkins Masters images • Production - jenkins/jenkins • Weekly and LTS releases • OpenJDK 8 and 11 • Debian and CentOS • Alpine and Debian Slim for OpenJDK 8 • Experimental - jenkins4eval/jenkins • ARM, ARM 64, IBM s/390, PowerPC 64 LE • Continuous delivery for Jenkins master branches • More platforms are coming soon

Slide 16

Slide 16 text

© 2019 All Rights Reserved. 16 Official image: jenkins/jenkins docker run --rm \ -p 8080:8080 \ -p 50000:50000 \ -v jenkins_home:/var/jenkins_home \ jenkins/jenkins:lts https://github.com/jenkinsci/docker

Slide 17

Slide 17 text

© 2019 All Rights Reserved. 17 Jenkins Image is not enough ● No plugins ● No configuration ● No jobs

Slide 18

Slide 18 text

© 2019 All Rights Reserved. 18

Slide 19

Slide 19 text

© 2019 All Rights Reserved. 19 Configuration as Code in Jenkins Jobs System configuration

Slide 20

Slide 20 text

© 2019 All Rights Reserved. 20 Docker images One FROM to rule the world

Slide 21

Slide 21 text

© 2019 All Rights Reserved. 21 Extending Docker images Image source: http://container.farm/special-extended-container/ YOUR STUFF

Slide 22

Slide 22 text

© 2019 All Rights Reserved. 22 Extending Docker images Tools Environment Configuration Plugins Docs: https://github.com/jenkinsci/docker

Slide 23

Slide 23 text

© 2019 All Rights Reserved. 23 Dockerfile

Slide 24

Slide 24 text

© 2019 All Rights Reserved. 24 Plugin management

Slide 25

Slide 25 text

© 2019 All Rights Reserved. 25 Plugin bundling jenkins/jenkins:2.190.3

Slide 26

Slide 26 text

© 2019 All Rights Reserved. 26 Plugin bundling jenkins/jenkins:2.190.3 # Pinned version blueocean:1.8.2 # Latest antisamy-markup-formatter matrix-auth:latest # Latest from the Experimental update center git-client:experimental # Incrementals (JEP-305) workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.21-rc591.43d37d4d080a plugins.txt

Slide 27

Slide 27 text

© 2019 All Rights Reserved. 27 Limitations • Latest versions of non-specified dependencies are installed • No “Check for updates” feature • Plaintext Format • No YAML • No Dependabot integration, etc.

Slide 28

Slide 28 text

© 2019 All Rights Reserved. 28 Plugin installation manager tool https://github.com/jenkinsci/plugin-installation-manager-tool • CLI tool • Java, not bash script • YAML support • Updates listing • Transient dependency management plugins: artifactId: git source: version: latest artifactId: job-import-plugin source: version: 3.1 artifactId: docker artifactId: docker-commons source: version: 2.4.4 artifactId: matrix-project

Slide 29

Slide 29 text

© 2019 All Rights Reserved. 29 Configurations

Slide 30

Slide 30 text

© 2019 All Rights Reserved. 30 Groovy Hooks [1/2] • Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • 2 hook types in the core: init, boot-failure https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

Slide 31

Slide 31 text

© 2019 All Rights Reserved. 31 Groovy Hooks [2/2] Script locations: • WEB-INF/HOOK.groovy in jenkins.war • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script

Slide 32

Slide 32 text

© 2019 All Rights Reserved. 32 Jenkins Startup Flow JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() PLUGINS_STARTED . . .

Slide 33

Slide 33 text

© 2019 All Rights Reserved. 33 Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Slide 34

Slide 34 text

© 2019 All Rights Reserved. 34 Adding Groovy hooks to the standard image Just put Groovy scripts in the right place! https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Slide 35

Slide 35 text

© 2019 All Rights Reserved. 35 https://github.com/oleg-nenashev/demo-jenkins-config-as-code

Slide 36

Slide 36 text

© 2019 All Rights Reserved. 36 More examples oleg-nenashev/demo-jenkins-config-as-code Praqma/JenkinsAsCodeReference librecores/librecores-ci

Slide 37

Slide 37 text

© 2019 All Rights Reserved. 37 Add JCasC Plugin! Configuration-as-Code Plugin https://plugins.jenkins.io/configuration-as-code

Slide 38

Slide 38 text

© 2019 All Rights Reserved. 38 JCasC Plugin ● Sub-project in Jenkins ● Feb 2018 - first alpha ● Sep 2018 - 1.0 ● Now - 1.35 ● Configuration via YAML jenkinsci/configuration-as-code-plugin gitter.im/jenkinsci/configuration-as-code-plugin

Slide 39

Slide 39 text

© 2019 All Rights Reserved. 39 jenkins: systemMessage: "JCasC Demo" numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml YAML specification

Slide 40

Slide 40 text

© 2019 All Rights Reserved. 40 • Configuration export from Jenkins • CLI and REST API for configuration management • Pre-flight validations and dry-run • Support for many plugins out-of-the-box • External credentials • Job/Folder management via JobDSL Features

Slide 41

Slide 41 text

© 2019 All Rights Reserved. 41 Adding JCasC to the Docker image Configuration file Needs JCasC Plugin … configuration-as-code:1.35 …

Slide 42

Slide 42 text

© 2019 All Rights Reserved. 42 Combining JCasC and Groovy hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . Groovy hooks run after JCasC Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations Use JCasC where possible!

Slide 43

Slide 43 text

© 2019 All Rights Reserved. 43 Comparison JCasC Plugin Groovy Hooks YAML – declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for CM Plugin management Groovy – general-purpose language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support Plugin Management

Slide 44

Slide 44 text

© 2019 All Rights Reserved. 44 Comparison JCasC Plugin Groovy Hooks YAML – declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for CM Plugin management Groovy – general-purpose language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support Plugin Management Use JCasC when possible, combine with Groovy hooks when needed

Slide 45

Slide 45 text

Mark’s success story Fast setup, better interactive testing

Slide 46

Slide 46 text

© 2019 All Rights Reserved. 46 Git Plugin Development - Don’t Break the Users! • Automated tests are great, but… • Most git plugin bugs are not detected by automated tests • Git plugin users don’t like being broken by plugin upgrades • Choices • Automate everything, don’t release until it is all automated • Interactive tests of interesting and useful cases in addition to automation • Interactive test configuration is COMPLICATED • Platforms - Alpine, CentOS, Debian, Fedora, FreeBSD, OpenSUSE, Windows • Git versions - 1.8, 1.9, 2.11, 2.17, 2.23, 2.24, … • ANSWER: Simplify test setup by managing it in a repository (several repos)

Slide 47

Slide 47 text

© 2019 All Rights Reserved. 47 Faster Interactive Test Setup - Docker & Jenkins Public Repo • LTS • Debian • Debian Slim • Alpine • JDK 11 • LTS, plugins • Debian • Debian Slim • Alpine • JDK 11 Private Repo • LTS, plugins, & credentials • Debian • Debian Slim • Alpine • JDK 11 • LTS, plugins, credentials & agents • Debian • Debian Slim • Alpine • JDK 11 https://github.com/MarkEWaite/docker-lfs $ docker_build.py $ docker_run.py --clean

Slide 48

Slide 48 text

© 2019 All Rights Reserved. 48 Fully Configured Jenkins • 30+ agents • 10+ git versions • Linux - Debian, CentOS, Ubuntu • Two versions of each • FreeBSD & Windows agents • Raspberry Pi 4 agent • Cloud and local agents • 1000+ test jobs • Specific bug check jobs • Easy to start, easy to stop • Any Linux machine in network https://github.com/MarkEWaite/docker-lfs

Slide 49

Slide 49 text

© 2019 All Rights Reserved. 49 © 2019 CloudBees, Inc. All Rights Reserved. What about CloudBees?

Slide 50

Slide 50 text

© 2019 All Rights Reserved. 50 Products ● All products are available as Docker images ○ https://hub.docker.com/u/cloudbees ● Helm charts for CloudBees Jenkins Distribution ● CloudBees Jenkins Distribution and CloudBees Core run on-premise or in the cloud ○ Docker ○ Kubernetes - Helm Charts ○ Virtual machines ○ Bare metal

Slide 51

Slide 51 text

© 2019 All Rights Reserved. 51 Plugins ● All Jenkins Docker plugins are available to customers ● Extended SLA for several plugins https://docs.cloudbees.com/plugins

Slide 52

Slide 52 text

© 2019 All Rights Reserved. 52 JCasC in CloudBees products ● Tier 2 (Compatible) plugin ○ SLA: Bugfixes and Security fixes ○ CloudBees Jenkins Support ○ CloudBees Jenkins Distribution ● Technical preview for CloudBees Core ○ Standalone and Managed masters ○ Configuration Bundles feature http://bit.ly/jcasc-cloudbees

Slide 53

Slide 53 text

© 2019 All Rights Reserved. 53 Plugin Management ● Added value features for Docker ● CloudBees Assurance Program and Beekeper ○ https://docs.cloudbees.com/docs/admin-resources/latest/ assurance-program/ ● Plugin catalog for CloudBees Core ○ https://docs.cloudbees.com/docs/admin-resources/latest/ plugin-management/configuring-plugin-catalogs

Slide 54

Slide 54 text

© 2019 All Rights Reserved. 54 CloudBees Assurance Program ● Verified plugin sets ● Upgrade assistant ● Administrative monitors ● Opt-in automatic upgrade

Slide 55

Slide 55 text

© 2019 All Rights Reserved. 55 Lists the set of plugins, plugin versions and plugin dependencies that CloudBees has tested for compatibility and security. Administrative dashboard provides a centralized view of the monitored Jenkins plugins, recommended actions, and configuration options available.

Slide 56

Slide 56 text

© 2019 All Rights Reserved. 56 JCasC in CloudBees Jenkins Distribution ● Combines with CloudBees Assurance Program ● Documentation and Quick Start ○ http://bit.ly/cjd-jcasc ● Demo ○ https://github.com/cloudbees-oss/cjd-jcasc-demo

Slide 57

Slide 57 text

© 2019 All Rights Reserved. 57 Back to Jenkins and Docker

Slide 58

Slide 58 text

© 2019 All Rights Reserved. 58 Upcoming changes in JCasC ● Hardening the implementation ● New features ● Plugin compatibility is a focus ○ Fixes in ~50 plugins ● Jenkins core architectural changes ○ Read-only configuration UI ○ Initialization milestones ● Developer tools (VS Code Plugin, etc.)

Slide 59

Slide 59 text

© 2019 All Rights Reserved. 59 Upcoming changes in Docker ● New master images / platforms ○ Windows ○ Ubuntu ○ UBI ○ AdoptOpenJDK with HotSpot ○ AdoptOpenJDK with OpenJ9 ● Multi-platform Docker image ● New agent images

Slide 60

Slide 60 text

© 2019 All Rights Reserved. 60 Takeaways 1. Dockerize Jenkins 2. Use & extend official images 3. Use static configurations, pin all versions 4. Use Configuration-as-Code

Slide 61

Slide 61 text

© 2019 All Rights Reserved. 61 It is a great time to participate! Join Platform SIG • https://jenkins.io/sigs/platform/ • https://gitter.im/jenkinsci/platform-sig Contribute • https://jenkins.io/participate/ • https://github.com/jenkinsci/docker • https://github.com/jenkinsci/configuration-as-code-plugin

Slide 62

Slide 62 text

© 2019 All Rights Reserved. 62 Join us at the community booth! • Any questions • More demos • Custom image packager • JCasC experts • How to contribute? • ...

Slide 63

Slide 63 text

No content

Slide 64

Slide 64 text

© 2019 All Rights Reserved. 64 DevOptics Software Delivery Visibility & Insights Core Unified Software Delivery & Governance CodeShip CI/CD as a Service Flow Adaptive Release Orchestration DevOps Excellence Jenkins CloudBees Jenkins Distribution CloudBees Jenkins X Distribution 24x7 Technical Support Assisted Updates Support Accelerator Training Customer Success Managers DevOps Consultants Rollout Feature Flag Management Continuous Delivery Products and Services