Swiss JAM: State of Jenkins & Configuration as Code

A talk about recent changes and ongoing projects in the Jenkins community: "Jenkins: Shifting Gears" blogpost, Configuration-as-code plugin, Jenkins Evergreen, Jenkins X, and Cloud Native Jenkins.

There will be also a hands-on demo of the Configuration as Code plugin and Docker packaging.: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/tree/casc-plugin

Transcript

1. State of Jenkins & Configuration as Code Oleg Nenashev, CloudBees Swiss Jenkins Meetup Sep 11, 2018
2. > whoami @oleg_nenashev oleg-nenashev • Based in Neuchatel • Jenkins Architecture Team • CloudBees Core • CloudBees Jenkins Support 2
3. > whoami -jenkins Hudson/Jenkins user since 2008 Jenkins contributor since 2012 Now: • Maintainer: Core & plugins • Security Team member • Meetup organizer • Google Summer of Code org admin • Cloud Native SIG participant @oleg_nenashev oleg-nenashev 3
4. • Building local community • Sharing experiences • Looking for contributors Swiss Jenkins Area Meetup 4 https://www.meetup.com/Swiss-Jenkins-Area-Meetup https://twitter.com/JenkinsMeetupCH
5. Jenkins Area Meetups • 88 meetup groups • ~26000 participants • https://www.meetu p.com/pro/jenkins/ + Online Meetup https://jenkins.io/projects/jam/ 5
6. 6 About you ØWhat do you expect from this meetup?
7. 7 Today’s agenda…
8. My plan was… 8
9. 9https://digitalsavvy.com/elephant-in-the-room-2/
10. 10 Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ “Jenkins: Shifting Gears” slides by Kohsuke https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view
11. • Extensible • General Purpose • Community Shifting Gears: Our Amazing Success https://flic.kr/p/5FawSa 11 © 2018 CloudBees, Inc. All Rights Reserved
12. 1. Most popular CI/CD tool in the world 2. It’s open source 3. Big community 4. Plugin ecosystem 5. … Who is Mr. Jenkins? https://jenkins.io 12
13. Shifting Gears: Making Changes 13 © 2018 CloudBees, Inc. All Rights Reserved
14. • Service Instability • Brittle Configuration • Assembly Required • Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ 14 © 2018 CloudBees, Inc. All Rights Reserved
15. • Service Instability • Brittle Configuration • Assembly Required • Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ Oleg: Jenkins is an automation framework, needs maintenance. Users need a solution 15 © 2018 CloudBees, Inc. All Rights Reserved
16. Система Docs Plugins Configs Infra- structure Backend Samples Demos Framework =>System Infra- structure Infra- structure 16
17. • Service Instability • Brittle Configuration • Assembly Required • Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ Oleg: Developer Experiences == Contributor Experience. We need it 17 © 2018 CloudBees, Inc. All Rights Reserved
18. Cloud Native Jenkins https://flic.kr/p/75ncBF Jolt in Jenkins https://flic.kr/p/7MUAGk 18
19. • General purpose CI/CD engine that runs on Kubernetes that embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Shifting Gears: Cloud Native Jenkins 19 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
20. • With all the good things of Jenkins we’ve been pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG • Follow the footsteps of Jenkins X and converge with it Shifting Gears: Cloud Native Jenkins 20 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
21. 21 https://jenkins.io/sigs/
22. • Groups focusing on particular areas • Regular public meetings • Public chats and mailing lists • Venue for Jenkins Enhancement Proposal discussions • Examples: • Platform SIG, GSoC SIG, Chinese Localization • Cloud Native SIG Special Interest Groups 22 https://jenkins.io/sigs/
23. • https://jenkins.io/sigs/cloud-native/ • Focuses on making Jenkins cloud-native • Lead: Carlos Sanchez (Kubernetes Plugin maintainer) • Participants: • Individual Contributors • CloudBees, Google Cloud Platform, Microsoft Azure • My blog about the SIG: • https://jenkins.io/blog/2018/07/30/introducing-cloud-native-sig/ Cloud Native SIG 23
24. • Closer than you might think, because we have many of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Shifting Gears: Cloud Native Jenkins MVP 24 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved
25. • Continue the incremental path of Jenkins 2, BUT with a jolt • We need to get out of the local optimum • Better speed of development, better stability • Renegotiate “contract” with users • Move away from “forever compatibility” • Release model change, ala Java SE • Continuous delivery of Jenkins • Must make sense to users • be largely compatible, but can’t afford to be perfectly compatible Shifting Gears: Jolt in Jenkins https://flic.kr/p/65YARD 25 © 2018 CloudBees, Inc. All Rights Reserved
26. • Jenkins project already does incompatible changes • Security fixes and hardening • E.g. JEP-200: https://jenkins.io/blog/2018/03/15/jep-200-lts/ • Incompatible changes are fine if done properly • Now: upgrade guidelines and announcement blogposts • https://jenkins.io/doc/upgrade-guide/ Incompatible changes? FEAR NOT! 26
27. • Continuous Delivery infrastructure and tooling • API Deprecation policy & Engine • Feature flags (early adoption, A/B testing) • Tools to warn Jenkins users about incoming changes • Tools to warn plugin maintainers about changes • Telemetry API for better usage stats Jolt in Jenkins. What do we need? (IMHO) 27 // See Jenkins Evergreen below for CD infra and telemetry
28. • Evergreen • Configuration as Code • Continued evolution of Pipeline execution engine • Developer experience / onboarding improvements • Plugin spring cleaning • Table stakes service integration Shifting Gears: Viable Ideas CloudBees is backing up https://jenkins.io/blog/2018/08/31/shifting-gears/ 28 © 2018 CloudBees, Inc. All Rights Reserved
29. • We have to move away from local optimum, we need to change • This time, money is where the mouth is • Two initiatives, in broad stroke • Lots of details to be filled out • Will happen in SIGs and JEPs, in coming days • Looking forward for others to join Shifting Gears: Conclusions https://jenkins.io/blog/2018/08/31/shifting-gears/ 29 © 2018 CloudBees, Inc. All Rights Reserved
31. 31 Q: What do you think?
32. DevOps World | Jenkins World, Oct 22-25, Nice https://jenkins.io/blog/2018/08/21/contributor-summit-nice/ Want to know more? 32
33. So, what’s going on? 33
35. Cloud Native Jenkins 35 // Not an official logo
36. • Pluggable Storage • Cloud Native Jenkins itself • Replicated services, High Availability, zero downtime • Canary and rolling deployments • Integrations with Cloud Technologies, esp. Kubernetes • … • Configuration as Code Cloud Native SIG. Areas of interest 36
37. Pluggable Storage for Jenkins • Discussions started at Jenkins World 2016 • Main data types: • Artifacts • Logs • Configurations • Runs: metadata, test results, etc. • Fingerprints • … 37
38. 38 Current state https://jenkins.io/sigs/cloud-native/pluggable-storage/ 38
39. Artifacts • JEP-202 – External Artifact Storage • New API in the Jenkins core and Pipeline • Implementation: Artifact Manager for S3 • https://github.com/jenkinsci/artifact-manager-s3-plugin • Pipeline integration • Transparent support of archive/unarchive • Transparent support of stash/unstash 39
40. Logs in Jenkins 40 Build logs System logs Other • Agent logs • SCM polling • Multibranch indexing • … java.util.logging
41. Master Current state Agents Logs Build Console • All logs got through the master • Agents send logs to the master • Build console… comes from the disk as well 41
42. External Logs. Approach Agents Logstash Elasticsearch Master logs Agent logs Build Console Storage Master 42
43. • AWS CloudWatch Logs: https://github.com/jenkinsci/pipeline-log-fluentd- cloudwatch-plugin • Elasticsearch: https://github.com/jenkinsci/external- logging-elasticsearch-plugin • Old PoC: https://github.com/jenkinsci/custom-war- packager/tree/master/demo/external-logging- elasticsearch External Logs. Prototypes 43
44. Configurations • Configurations are stored differently L • hudson.XmlFile covers 95% of cases • Prior work: • API: https://github.com/jenkinsci/jenkins/pull/3393 • K8s resources: https://github.com/jstrachan/jenkins/tree/kubeify • JEP-213 – Current design (under review) • https://github.com/jenkinsci/jep/tree/master/jep/213 44
45. Cloud Native SIG. Plans To be discussed at contributor summits 45
46. More about Cloud Native Jenkins 46 http://sched.co/F9NT
47. http://jenkins-x.io/ 47
48. 48 48
49. Jenkins X. Value • Works out-of-the box • CLI for everything • Support of environments • Configuration management via GitOps • Extensible via Kubernetes 49
50. Jenkins X. Quickstart • > brew install jx • > jx install • > jx create quickstart • (or jx create spring, jx import) • Creates a Git repo • Registers web hooks • Creates images for Kubernetes • Runs the first Pipeline 50
51. Jenkins X. OOTB experience • Jenkins-master • Agents pool (Kubernetes plugin) • Nexus, chartmuseum, monocular • Environments: ▸ Local for Development ▸ Staging ▸ Integration with Production envs • IDE Integration 51
53. Links • https://jenkins-x.io/getting-started/ • https://jenkins-x.io/contribute/roadmap/ • https://github.com/rawlingsj/jenkins-x-presentations 53
54. More info about Jenkins X • Sep 12/13 – Workshops in Zurich • Cosmin Cojocar, Jenkins X: Continuous Delivery for Java services in Kubernetes ▸ https://workshoptage.ch/workshops/2018/jenkins-x-continuous-delivery-for-java- services-in-kubernetes/ • Paolo Carta, Matteo Baiguini. End-to-End Continuous Delivery on Kubernetes ▸ https://workshoptage.ch/workshops/2018/end-to-end-continuous-delivery-on- kubernetes/ • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes ▸ https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ 54
55. 55 Jenkins Evergreen Ø Distribution Ø Ecosystem for Continuous Delivery https://github.com/jenkins-infra/evergreen
56. https://jenkins.io/blog/2018/06/26/jenkins-essentials-at-eclipsecon-france/ 56
57. • Automated Sane Defaults • Cloud Native: AWS, Docker • Automatically Upgrading Distribution • And rollback… • Connected: Telemetry & Feedback • Obvious Path To User Success • Opinionated • Clear out obstacles • Documentation Jenkins Evergreen basics 57
59. Jenkins Evergreen. The four opens • Open Source • Open Design • Open Development • Open Community https://github.com/jenkins-infra/evergreen/#the-four-opens 59
60. Jenkins Evergreen Architecture Evergreen Instance evergreen- client registration authentication update manifest update level = 1. Client downloads WAR, plugins, client update, etc. installs, triggers restart then health-checks Backend Services evergreen.jenkins.io 60
61. CD for the core and plugins • Incrementals infrastructure (JEP-305) • Master, branches and pull requests go to Artifactory https://jenkins.io/blog/2018/05/15/incremental-deployment/ 61
62. Incrementals - usage • Not available in the standard Update Center • Can be downloaded from Artifactory • Support in official Docker images (jenkins/jenkins) # plugins.txt workflow-aggregator:2.5 workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.19-rc289.d09828a05a74 configuration-as-code:experimental 62
63. CI patches: essentialsTest() • Work-in-progress • New engine for plugin CI/CD in ci.jenkins.io • Pipeline Library • https://github.com/jenkins-infra/pipeline- library/blob/master/vars/essentialsTest.groovy • Integration tests during the build • Plugin Compat Tester • Acceptance Test Harness 63
64. 64 Jenkins Configuration-as-Code
66. Configuration as Code in Jenkins Jobs System configuration 66
67. Configuration as Code in Jenkins Jobs System configuration 67
68. “Infrastructure as Code” in Jenkins API Clients Jenkins CLI and REST API python- jenkins jenkins- client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Jenkins- internal solutions Groovy Boot Hooks Scriptler Plugin JCasC Plugin 68
69. State in 2016 – Stockholm JAM • No default solution recommended by the Jenkins Project • No active work on this front • System Config DSL, was abandoned • Configuration Management – outside Jenkins Project • Time lag in Features • Security fun • Compatibility (e.g. Jenkins 2) 69
70. Groovy! 70
72. Groovy Hooks [1/2] • Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • init, boot-failure – in the core https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 72
73. Groovy Hooks [2/2] • Script locations: • WEB-INF/HOOK.groovy in jenkins.war • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy • Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 73
74. Initialization flow JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() PLUGINS_STARTED . . . 74
75. Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code 75
76. More examples • https://github.com/oleg-nenashev/demo-jenkins- config-as-code • https://github.com/Praqma/JenkinsAsCodeReference • https://github.com/librecores/librecores-ci 76
77. Groovy! #ornot 77
78. Groovy Hook Limitations. General • Jenkins API is scattered and complex • Javadoc is not user-friendly • Public API is not enough sometimes • No export from UI Solution: • Configuration as Code Plugin 78
79. Groovy Hook Limitations. Missing Features • No class support • No shared libraries support • Error propagation: Jenkins does not stop on script error • Restart is required to apply changes There are workarounds 79
80. More about Groovy Hooks https://speakerdeck.com/onenashev/gbg-jam-managing-jenkins-with- groovy-hooks-and-jcasc • Examples • Tips & Tricks • Development and debugging 80
81. • Feb 2018 – first alpha • Aug 2018 – release candidates • Configuration via YAML • jenkinsci/configuration-as-code-plugin New plugin: Jenkins Configuration-as-Code 81
82. jenkins: systemMessage: "JCasC Demo" numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml 82
83. • YAML specification • Configuration export to YAML • CLI and REST API for configuration management • Pre-flight validations and dry-run • Support of many plugins out-of-the-box Features 83
84. Limitations (temporary?) • Not all plugins are supported • Forward/backward compatibility is not guaranteed • YAML - No dynamic logic / scripting • Startup Chicken&Egg for plugins • Race conditions for jobs • Handling of removed sections • Export from UI – many open issues 84
85. Initialization with JCasC JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . • Runs after the plugins get loaded • Runs in parallel with job loading • Can trigger restart on-demand 85
86. Comparison JCasC Plugin Groovy Hooks YAML – standard declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for config management Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support 86
87. 87 Why not both?
88. Combining JCasC and Groovy hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . • Groovy hooks run after JCasC • Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations • Use JCasC where possible 88
89. https://github.com/oleg- nenashev/demo-jenkins-config-as- code/tree/casc-plugin 89
90. Demo. Local Pipeline Development Env Intellij IDEA Filesystem SCM Plugin • Documentation, Syntax • Static analysis • Debug (only for hooks) Local Jenkins instance Source Code (local .git repos) • Configuration-as-Code • Same as production • Repos with libs • Jenkinsfile JENKINS_HOME (volume) 90
91. JCasC. Takeaways • Select an engine depending on your needs and expertise • New to Jenkins? Use JCasC • Combine engines if needed 91
92. What else? • Java 10+ Support • http://bit.ly/jenkins-java10-hackathon-summary • Experiments towards new Java support 92
93. What else? • Java 10+ Support • BlueOcean Extensibility • JEP-204 – BlueOcean Extensibility API • JEP-203 – BlueOcean Extensibility 93
94. What else? • Java 10+ Support • BlueOcean Extensibility • Google Summer of Code • Code Coverage API Plugin • Remoting over Apache Kafka • Simple Pull Request Job (aka Pipeline as YAML) https://jenkins.io/projects/gsoc/ 94
95. • Jenkins evolves. Key issues in the architecture and community are being addressed • There are new projects to try Takeaways != 95
96. Takeaways • It’s a great time to contribute ;) • https://jenkins.io/participate 96
97. 1. Join special interest groups (SIGs) 2. Let us know about your pain-points 3. Share your stories and experiences • Jenkins.io/blog and other blogs • Meetups Contribution != Code 97 Jenkins needs you!
98. Links Shifting Gears: • https://jenkins.io/blog/2018/08/31/shifting-gears/ Projects: • https://jenkins.io/sigs/cloud-native/ • https://jenkins-x.io/ • https://github.com/jenkins-infra/evergreen Configuration-as-Code Plugin • https://github.com/jenkinsci/configuration-as-code-plugin 98
99. Jenkins Events. What’s next? • Sep 16-19 - DevOps World | Jenkins World US • https://www.cloudbees.com/devops-world/san-francisco • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes • https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ • Oct 19 – Day of Jenkins [as code], Copenhagen • Oct 22-25 – DevOps World | Jenkins World Nice • https://www.cloudbees.com/devops-world/nice • Late Oct – Lausanne or Neuchatel? 99 DevOps World | Jenkins World 20% Discount Code JWJAMMEMBER
100. Contacts: E-mail: onenashev@cloudbees.com GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 100