Docker and Jenkins [as Code]

The Jenkins Configuration as Code plugin is a new milestone for Jenkins configuration management. Together with Docker or Kubernetes the plugin allows managing the entire Jenkins configuration. Let's talk about official master and agent images offered by the Jenkins project. What’s inside them? How to configure images with JCasC and Groovy hooks, how to use these approaches together? And, finally, how to simplify packaging of custom Jenkins images and define the entire system [as code]?

This slidedeck will be periodically updated to newer versions of the talk

Transcript

1. © 2019 CloudBees, Inc. All Rights Reserved. Docker and Jenkins [as code] Oleg Nenashev, CloudBees Docker Geneva meetup, Mar 21, 2019 1
2. © 2019 CloudBees, Inc. All Rights Reserved. Today… 2
3. © 2019 CloudBees, Inc. All Rights Reserved. > whoami @oleg_nenashev oleg-nenashev • Based in Neuchatel, Switzerland • Principal SW Engineer @ CloudBees • Jenkins Core Maintainer & Ambassador • Swiss Jenkins Meetup organizer 3
4. © 2019 CloudBees, Inc. All Rights Reserved. • Building local community • Sharing experiences • Looking for speakers Swiss Jenkins Area Meetup 4 https://www.meetup.com/Swiss-Jenkins-Area-Meetup https://twitter.com/JenkinsMeetupCH
5. © 2019 CloudBees, Inc. All Rights Reserved. > whoami -docker -casc • Using Configuration-as-code since 2011 • Using Docker since 2015 (for development) • Jenkins Docker Packaging maintainer • JCasC plugin contributor • Custom WAR Packager maintainer 5
6. © 2019 CloudBees, Inc. All Rights Reserved. We’re Building the World’s First End-to-End Continuous Software Delivery System About CloudBees 6 https://www.cloudbees.com
7. © 2019 CloudBees, Inc. All Rights Reserved. > whoami -cloudbees 7 • CloudBees Jenkins Distribution • CloudBees Jenkins Support • Community (e.g. Java 11 support, Jenkinsfile Runner) https://www.cloudbees.com
8. © 2019 CloudBees, Inc. All Rights Reserved. About you 8
9. © 2019 CloudBees, Inc. All Rights Reserved. Agenda • Docker & Jenkins • Extending official images • Build your own Jenkins images! 9 These slides: http://bit.ly/docker-genev a-jenkins-as-code
10. © 2019 CloudBees, Inc. All Rights Reserved. CI/CD. Why containers? Config Management Fast provisioning Clean environments Disposability Clustering 10
11. © 2019 CloudBees, Inc. All Rights Reserved. 11
12. © 2019 CloudBees, Inc. All Rights Reserved. Configuration as Code in Jenkins Jobs System configuration 12 12
13. © 2019 CloudBees, Inc. All Rights Reserved. Configuration as Code in Jenkins Jobs System configuration 13 13
14. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins and Docker 14
15. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins & Docker. Packaging https://hub.docker.com/r/jenkins/ 15
16. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins and Docker • Build agents and steps 16
17. © 2019 CloudBees, Inc. All Rights Reserved. Official agent images • jenkins/slave – base image with Remoting • jenkins/jnlp-slave – JNLP agents • jenkins/ssh-slave – SSH agents • jenkins/remoting-kafka-agent – Remoting Kafka Plugin 17 https://hub.docker.com/r/jenkins/
18. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins & Docker. Plugins Docker Plugin Docker Pipeline Plugin Docker Custom Build Environment Plugin … Yet Another Docker Plugin Kubernetes Plugin Kubernetes Pipeline Plugin … 18
19. © 2019 CloudBees, Inc. All Rights Reserved. Example: Docker Pipeline Plugin https://plugins.jenkins.io/docker-workflow docker.image('onenashev/gcc-riscv:6.4-rc').inside { checkout scm sh "make clean test" } 19
20. © 2019 CloudBees, Inc. All Rights Reserved. Example: Kubernetes Plugin def label = "gcc-riscv-${UUID.randomUUID().toString()}" podTemplate(name: 'test', label: label, containers: [ containerTemplate(name: 'gcc-riscv', image: 'onenashev/gcc-riscv:6.4-rc’, resourceRequestCpu: ‘2’, resourceLimitMemory: ‘8Gi' ttyEnabled: true, command: 'cat')]) { node(label) { checkout scm sh "make clean test" step([$class: 'TapPublisher', testResults: 'output/test/report.tap', ...]) } } https://plugins.jenkins.io/kubernetes 20
21. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins and Docker • Build agents and steps • Master images 21
22. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins masters in Docker • jenkins/jenkins • jenkinsci/blueocean (being deprecated) • experimental images 22
23. © 2019 CloudBees, Inc. All Rights Reserved. Official image: jenkins/jenkins docker run --rm \ -p 8080:8080 \ -p 50000:50000 \ -v jenkins_home:/var/jenkins_home \ jenkins/jenkins:lts 23 See also: https://batmat.net/2018/09/07/how-to-run-and-upgrade- jenkins-using-the-official-docker-image/
24. © 2019 CloudBees, Inc. All Rights Reserved. Official image: jenkins/jenkins • Standard image • Weekly and LTS releases • No bundled plugins, only detached ones • Debian, Alpine, Slim • Java 8, Java 11 24 https://github.com/jenkinsci/docker
25. © 2019 CloudBees, Inc. All Rights Reserved. Extending images Official images are designed to be extended Managed as code 25
26. © 2019 CloudBees, Inc. All Rights Reserved. Extending images Image source: http://container.farm/special-extended-container/ YOUR STUFF 26
27. © 2019 CloudBees, Inc. All Rights Reserved. Extending Docker images Dockerfile Plugin bundling Groovy Hooks Environment and Java Options Docs: https://github.com/jenkinsci/docker 27
28. © 2019 CloudBees, Inc. All Rights Reserved. Dockerfile 28
29. © 2019 CloudBees, Inc. All Rights Reserved. Plugin bundling 29
30. © 2019 CloudBees, Inc. All Rights Reserved. Plugin bundling 30 # Pinned version blueocean:1.8.2 # Latest antisamy-markup-formatter matrix-auth:latest # Latest from the Experimental update center git-client:experimental # Incrementals (JEP-305) workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.21-rc591.43d37d4d080a plugins.txt
31. © 2019 CloudBees, Inc. All Rights Reserved. Groovy Hooks [1/2] • Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • 2 hook types in the core: init, boot-failure https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 31
32. © 2019 CloudBees, Inc. All Rights Reserved. Groovy Hooks [2/2] Script locations: • WEB-INF/HOOK.groovy in jenkins.war • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy Execution in alphabetical order 32 https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script
33. © 2019 CloudBees, Inc. All Rights Reserved. Jenkins Startup Flow JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() PLUGINS_STARTED . . . 33
34. © 2019 CloudBees, Inc. All Rights Reserved. Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code 34
35. © 2019 CloudBees, Inc. All Rights Reserved. Adding Groovy hooks to the standard image Just put Groovy scripts in the right place! https://github.com/oleg-nenashev/demo-jenkins-config-as-code 35
36. © 2019 CloudBees, Inc. All Rights Reserved. More examples oleg-nenashev/demo-jenkins-config-as-code Praqma/JenkinsAsCodeReference librecores/librecores-ci 36
37. © 2019 CloudBees, Inc. All Rights Reserved. Add JCasC Plugin! Configuration-as-Code Plugin https://plugins.jenkins.io/configuration-as-code 37
38. © 2019 CloudBees, Inc. All Rights Reserved. 38 JCasC Plugin • Sub-project in Jenkins • Feb 2018 - first alpha • Sep 2018 - 1.0 • Now - 1.8 • Configuration via YAML jenkinsci/configuration-as-code-plugin
39. © 2019 CloudBees, Inc. All Rights Reserved. jenkins: systemMessage: "JCasC Demo" numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml 39
40. © 2019 CloudBees, Inc. All Rights Reserved. • YAML specification • Configuration export from Jenkins • CLI and REST API for configuration management • Pre-flight validations and dry-run • Support of many plugins out-of-the-box Features 40
41. © 2019 CloudBees, Inc. All Rights Reserved. Adding JCasC to the Docker image YAML auto-discovery in 1.0+ Needs JCasC Plugin [and JCasC Support?] … configuration-as-code:1.1 configuration-as-code-support:1.1 … 41
42. © 2019 CloudBees, Inc. All Rights Reserved. Demos https://github.com/jenkinsci/configuration-as-code-plugi n/tree/master/demos 42
43. © 2019 CloudBees, Inc. All Rights Reserved. https://github.com/oleg-nenashev/demo- jenkins-config-as-code/tree/casc-plugin 43
44. © 2019 CloudBees, Inc. All Rights Reserved. Demo. Local Pipeline Development Env Intellij IDEA Filesystem SCM Plugin • Documentation, Syntax • Static analysis • Debug (only for hooks) Local Jenkins instance Source Code (local .git repos) • Configuration-as-Code • Same as production • Repos with libs • Jenkinsfile JENKINS_HOME (volume) 44
45. © 2019 CloudBees, Inc. All Rights Reserved. JCasC Limitations (temporary?) • Not all plugins are supported • YAML - No dynamic logic / scripting • Startup Chicken&Egg for plugins & jobs • Handling of removed sections • Export from UI – many open issues • No read-only admin access 45
46. © 2019 CloudBees, Inc. All Rights Reserved. Combining JCasC and Groovy hooks 46 JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . Groovy hooks run after JCasC Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations Use JCasC where possible!
47. © 2019 CloudBees, Inc. All Rights Reserved. Comparison JCasC Plugin Groovy Hooks YAML – standard declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for CM Plugin management Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support Plugin Management 47
48. © 2019 CloudBees, Inc. All Rights Reserved. Too many options? 48 https://www.flickr.com/photos/ silgeland/2645819709
49. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager https://jenkins.io/blog/2018/10/16/custom-war-packager/ • Managed by a config YAML • Produces ready-to fly Jenkinsfile bundles 49 https://github.com/jenkinsci/custom-war-packager
50. © 2019 CloudBees, Inc. All Rights Reserved. Single-shot masters. Build Flow Custom WAR Packager Core and Plugins Configuration-as-code Packaging Jenkins WARs, Docker images, or Jenkinsfile Runner 50
51. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager Inputs • Base Docker image • Jenkins WAR • Plugins and Custom libraries • System properties • Self-configuration: JCasC & System Groovy Hooks 51
52. © 2019 CloudBees, Inc. All Rights Reserved. Quick Demo (packaging) https://github.com/jenkinsci/custom-war-packager/ tree/master/demo/casc 52
53. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager YAML [1/2] Bundle settings: 53
54. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager YAML [2/2] Inputs: 54
55. © 2019 CloudBees, Inc. All Rights Reserved. Custom WAR Packager. Usages Jenkins X – Serverless masters • CWP + JCasC + Jenkinsfile Runner • https://jenkins.io/blog/2018/09/12/speaker-blog-a-cloud-native-jenkins/ ci.jenkins.io-runner • https://github.com/jenkinsci/ci.jenkins.io-runner Demo packages 55
56. © 2019 CloudBees, Inc. All Rights Reserved. Takeaways 1. Dockerize Jenkins 2. Use & extend official images 3. Use static configurations 4. Use Configuration-as-Code 5. Use Custom WAR/Docker Packager 56
57. © 2019 CloudBees, Inc. All Rights Reserved. Useful links Docker image • https://github.com/jenkinsci/docker Docker Package demo (with JCasC) • https://github.com/oleg-nenashev/demo-jenkins-config-as -code/tree/casc-plugin Custom WAR Packager • https://github.com/jenkinsci/custom-war-packager 57
58. © 2019 CloudBees, Inc. All Rights Reserved. Want to know more? Want to present? Swiss Jenkins Meetup • https://www.meetup.com/Swiss-Jenkins-Area-Meetup CloudBees Days Zurich, Apr 10 • https://www.cloudbees.com/cloudbees-days#zurich • Discount code: CBM50 DevOps World | Jenkins World • https://www.cloudbees.com/devops-world • San Francisco: Aug 12-15 (CFP - Mar 24) • Lisbon: Dec 02-05 (CFP - May) • Discount code: JAM20 58
59. © 2019 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 59