Swiss JAM: State of Jenkins & Configuration as Code

Slide 1

Slide 1 text

State of Jenkins & Configuration as Code Oleg Nenashev, CloudBees Swiss Jenkins Meetup Sep 11, 2018

Slide 2

Slide 2 text

> whoami @oleg_nenashev oleg-nenashev • Based in Neuchatel • Jenkins Architecture Team • CloudBees Core • CloudBees Jenkins Support 2

Slide 3

Slide 3 text

> whoami -jenkins Hudson/Jenkins user since 2008 Jenkins contributor since 2012 Now: • Maintainer: Core & plugins • Security Team member • Meetup organizer • Google Summer of Code org admin • Cloud Native SIG participant @oleg_nenashev oleg-nenashev 3

Slide 4

Slide 4 text

• Building local community • Sharing experiences • Looking for contributors Swiss Jenkins Area Meetup 4 https://www.meetup.com/Swiss-Jenkins-Area-Meetup https://twitter.com/JenkinsMeetupCH

Slide 5

Slide 5 text

Jenkins Area Meetups • 88 meetup groups • ~26000 participants • https://www.meetu p.com/pro/jenkins/ + Online Meetup https://jenkins.io/projects/jam/ 5

Slide 6

Slide 6 text

6 About you ØWhat do you expect from this meetup?

Slide 7

Slide 7 text

7 Today’s agenda…

Slide 8

Slide 8 text

My plan was… 8

Slide 9

Slide 9 text

9https://digitalsavvy.com/elephant-in-the-room-2/

Slide 10

Slide 10 text

10 Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ “Jenkins: Shifting Gears” slides by Kohsuke https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view

Slide 11

Slide 11 text

Slide 12

Slide 12 text

1. Most popular CI/CD tool in the world 2. It’s open source 3. Big community 4. Plugin ecosystem 5. … Who is Mr. Jenkins? https://jenkins.io 12

Slide 13

Slide 13 text

Slide 14

Slide 14 text

• Service Instability • Brittle Configuration • Assembly Required • Slow Development Velocity • Vicious cycle in maintenance Shifting Gears: Challenges https://jenkins.io/blog/2018/08/31/shifting-gears/ 14 © 2018 CloudBees, Inc. All Rights Reserved

Slide 15

Slide 15 text

Slide 16

Slide 16 text

Система Docs Plugins Configs Infra- structure Backend Samples Demos Framework =>System Infra- structure Infra- structure 16

Slide 17

Slide 17 text

Slide 18

Slide 18 text

Cloud Native Jenkins https://flic.kr/p/75ncBF Jolt in Jenkins https://flic.kr/p/7MUAGk 18

Slide 19

Slide 19 text

• General purpose CI/CD engine that runs on Kubernetes that embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Shifting Gears: Cloud Native Jenkins 19 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved

Slide 20

Slide 20 text

• With all the good things of Jenkins we’ve been pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG • Follow the footsteps of Jenkins X and converge with it Shifting Gears: Cloud Native Jenkins 20 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved

Slide 21

Slide 21 text

21 https://jenkins.io/sigs/

Slide 22

Slide 22 text

• Groups focusing on particular areas • Regular public meetings • Public chats and mailing lists • Venue for Jenkins Enhancement Proposal discussions • Examples: • Platform SIG, GSoC SIG, Chinese Localization • Cloud Native SIG Special Interest Groups 22 https://jenkins.io/sigs/

Slide 23

Slide 23 text

• https://jenkins.io/sigs/cloud-native/ • Focuses on making Jenkins cloud-native • Lead: Carlos Sanchez (Kubernetes Plugin maintainer) • Participants: • Individual Contributors • CloudBees, Google Cloud Platform, Microsoft Azure • My blog about the SIG: • https://jenkins.io/blog/2018/07/30/introducing-cloud-native-sig/ Cloud Native SIG 23

Slide 24

Slide 24 text

• Closer than you might think, because we have many of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Shifting Gears: Cloud Native Jenkins MVP 24 https://jenkins.io/blog/2018/08/31/shifting-gears/ © 2018 CloudBees, Inc. All Rights Reserved

Slide 25

Slide 25 text

• Continue the incremental path of Jenkins 2, BUT with a jolt • We need to get out of the local optimum • Better speed of development, better stability • Renegotiate “contract” with users • Move away from “forever compatibility” • Release model change, ala Java SE • Continuous delivery of Jenkins • Must make sense to users • be largely compatible, but can’t afford to be perfectly compatible Shifting Gears: Jolt in Jenkins https://flic.kr/p/65YARD 25 © 2018 CloudBees, Inc. All Rights Reserved

Slide 26

Slide 26 text

• Jenkins project already does incompatible changes • Security fixes and hardening • E.g. JEP-200: https://jenkins.io/blog/2018/03/15/jep-200-lts/ • Incompatible changes are fine if done properly • Now: upgrade guidelines and announcement blogposts • https://jenkins.io/doc/upgrade-guide/ Incompatible changes? FEAR NOT! 26

Slide 27

Slide 27 text

• Continuous Delivery infrastructure and tooling • API Deprecation policy & Engine • Feature flags (early adoption, A/B testing) • Tools to warn Jenkins users about incoming changes • Tools to warn plugin maintainers about changes • Telemetry API for better usage stats Jolt in Jenkins. What do we need? (IMHO) 27 // See Jenkins Evergreen below for CD infra and telemetry

Slide 28

Slide 28 text

• Evergreen • Configuration as Code • Continued evolution of Pipeline execution engine • Developer experience / onboarding improvements • Plugin spring cleaning • Table stakes service integration Shifting Gears: Viable Ideas CloudBees is backing up https://jenkins.io/blog/2018/08/31/shifting-gears/ 28 © 2018 CloudBees, Inc. All Rights Reserved

Slide 29

Slide 29 text

• We have to move away from local optimum, we need to change • This time, money is where the mouth is • Two initiatives, in broad stroke • Lots of details to be filled out • Will happen in SIGs and JEPs, in coming days • Looking forward for others to join Shifting Gears: Conclusions https://jenkins.io/blog/2018/08/31/shifting-gears/ 29 © 2018 CloudBees, Inc. All Rights Reserved

Slide 30

Slide 30 text

No content

Slide 31

Slide 31 text

31 Q: What do you think?

Slide 32

Slide 32 text

DevOps World | Jenkins World, Oct 22-25, Nice https://jenkins.io/blog/2018/08/21/contributor-summit-nice/ Want to know more? 32

Slide 33

Slide 33 text

So, what’s going on? 33

Slide 34

Slide 34 text

Slide 35

Slide 35 text

Cloud Native Jenkins 35 // Not an official logo

Slide 36

Slide 36 text

• Pluggable Storage • Cloud Native Jenkins itself • Replicated services, High Availability, zero downtime • Canary and rolling deployments • Integrations with Cloud Technologies, esp. Kubernetes • … • Configuration as Code Cloud Native SIG. Areas of interest 36

Slide 37

Slide 37 text

Pluggable Storage for Jenkins • Discussions started at Jenkins World 2016 • Main data types: • Artifacts • Logs • Configurations • Runs: metadata, test results, etc. • Fingerprints • … 37

Slide 38

Slide 38 text

38 Current state https://jenkins.io/sigs/cloud-native/pluggable-storage/ 38

Slide 39

Slide 39 text

Artifacts • JEP-202 – External Artifact Storage • New API in the Jenkins core and Pipeline • Implementation: Artifact Manager for S3 • https://github.com/jenkinsci/artifact-manager-s3-plugin • Pipeline integration • Transparent support of archive/unarchive • Transparent support of stash/unstash 39

Slide 40

Slide 40 text

Logs in Jenkins 40 Build logs System logs Other • Agent logs • SCM polling • Multibranch indexing • … java.util.logging

Slide 41

Slide 41 text

Master Current state Agents Logs Build Console • All logs got through the master • Agents send logs to the master • Build console… comes from the disk as well 41

Slide 42

Slide 42 text

External Logs. Approach Agents Logstash Elasticsearch Master logs Agent logs Build Console Storage Master 42

Slide 43

Slide 43 text

• AWS CloudWatch Logs: https://github.com/jenkinsci/pipeline-log-fluentd- cloudwatch-plugin • Elasticsearch: https://github.com/jenkinsci/external- logging-elasticsearch-plugin • Old PoC: https://github.com/jenkinsci/custom-war- packager/tree/master/demo/external-logging- elasticsearch External Logs. Prototypes 43

Slide 44

Slide 44 text

Configurations • Configurations are stored differently L • hudson.XmlFile covers 95% of cases • Prior work: • API: https://github.com/jenkinsci/jenkins/pull/3393 • K8s resources: https://github.com/jstrachan/jenkins/tree/kubeify • JEP-213 – Current design (under review) • https://github.com/jenkinsci/jep/tree/master/jep/213 44

Slide 45

Slide 45 text

Cloud Native SIG. Plans To be discussed at contributor summits 45

Slide 46

Slide 46 text

More about Cloud Native Jenkins 46 http://sched.co/F9NT

Slide 47

Slide 47 text

http://jenkins-x.io/ 47

Slide 48

Slide 48 text

48 48

Slide 49

Slide 49 text

Jenkins X. Value • Works out-of-the box • CLI for everything • Support of environments • Configuration management via GitOps • Extensible via Kubernetes 49

Slide 50

Slide 50 text

Jenkins X. Quickstart • > brew install jx • > jx install • > jx create quickstart • (or jx create spring, jx import) • Creates a Git repo • Registers web hooks • Creates images for Kubernetes • Runs the first Pipeline 50

Slide 51

Slide 51 text

Jenkins X. OOTB experience • Jenkins-master • Agents pool (Kubernetes plugin) • Nexus, chartmuseum, monocular • Environments: ▸ Local for Development ▸ Staging ▸ Integration with Production envs • IDE Integration 51

Slide 52

Slide 52 text

Slide 53

Slide 53 text

Links • https://jenkins-x.io/getting-started/ • https://jenkins-x.io/contribute/roadmap/ • https://github.com/rawlingsj/jenkins-x-presentations 53

Slide 54

Slide 54 text

More info about Jenkins X • Sep 12/13 – Workshops in Zurich • Cosmin Cojocar, Jenkins X: Continuous Delivery for Java services in Kubernetes ▸ https://workshoptage.ch/workshops/2018/jenkins-x-continuous-delivery-for-java- services-in-kubernetes/ • Paolo Carta, Matteo Baiguini. End-to-End Continuous Delivery on Kubernetes ▸ https://workshoptage.ch/workshops/2018/end-to-end-continuous-delivery-on- kubernetes/ • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes ▸ https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ 54

Slide 55

Slide 55 text

55 Jenkins Evergreen Ø Distribution Ø Ecosystem for Continuous Delivery https://github.com/jenkins-infra/evergreen

Slide 56

Slide 56 text

https://jenkins.io/blog/2018/06/26/jenkins-essentials-at-eclipsecon-france/ 56

Slide 57

Slide 57 text

• Automated Sane Defaults • Cloud Native: AWS, Docker • Automatically Upgrading Distribution • And rollback… • Connected: Telemetry & Feedback • Obvious Path To User Success • Opinionated • Clear out obstacles • Documentation Jenkins Evergreen basics 57

Slide 58

Slide 58 text

Slide 59

Slide 59 text

Jenkins Evergreen. The four opens • Open Source • Open Design • Open Development • Open Community https://github.com/jenkins-infra/evergreen/#the-four-opens 59

Slide 60

Slide 60 text

Jenkins Evergreen Architecture Evergreen Instance evergreen- client registration authentication update manifest update level = 1. Client downloads WAR, plugins, client update, etc. installs, triggers restart then health-checks Backend Services evergreen.jenkins.io 60

Slide 61

Slide 61 text

CD for the core and plugins • Incrementals infrastructure (JEP-305) • Master, branches and pull requests go to Artifactory https://jenkins.io/blog/2018/05/15/incremental-deployment/ 61

Slide 62

Slide 62 text

Incrementals - usage • Not available in the standard Update Center • Can be downloaded from Artifactory • Support in official Docker images (jenkins/jenkins) # plugins.txt workflow-aggregator:2.5 workflow-support:incrementals;org.jenkins-ci.plugins.workflow;2.19-rc289.d09828a05a74 configuration-as-code:experimental 62

Slide 63

Slide 63 text

CI patches: essentialsTest() • Work-in-progress • New engine for plugin CI/CD in ci.jenkins.io • Pipeline Library • https://github.com/jenkins-infra/pipeline- library/blob/master/vars/essentialsTest.groovy • Integration tests during the build • Plugin Compat Tester • Acceptance Test Harness 63

Slide 64

Slide 64 text

64 Jenkins Configuration-as-Code

Slide 65

Slide 65 text

Slide 66

Slide 66 text

Configuration as Code in Jenkins Jobs System configuration 66

Slide 67

Slide 67 text

Configuration as Code in Jenkins Jobs System configuration 67

Slide 68

Slide 68 text

“Infrastructure as Code” in Jenkins API Clients Jenkins CLI and REST API python- jenkins jenkins- client (java) Configuration Management Ansible, Chef, … Docker, Docker Compose ... Jenkins- internal solutions Groovy Boot Hooks Scriptler Plugin JCasC Plugin 68

Slide 69

Slide 69 text

State in 2016 – Stockholm JAM • No default solution recommended by the Jenkins Project • No active work on this front • System Config DSL, was abandoned • Configuration Management – outside Jenkins Project • Time lag in Features • Security fun • Compatibility (e.g. Jenkins 2) 69

Slide 70

Slide 70 text

Groovy! 70

Slide 71

Slide 71 text

Slide 72

Slide 72 text

Groovy Hooks [1/2] • Standard Groovy engine • Triggered by Jenkins • Direct access to the runtime • Extensible hook engine • init, boot-failure – in the core https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 72

Slide 73

Slide 73 text

Groovy Hooks [2/2] • Script locations: • WEB-INF/HOOK.groovy in jenkins.war • WEB-INF/HOOK.groovy.d/*.groovy in jenkins.war • $JENKINS_HOME/HOOK.groovy • $JENKINS_HOME/HOOK.groovy.d/*.groovy • Execution in alphabetical order https://wiki.jenkins-ci.org/display/JENKINS/Groovy+Hook+Script 73

Slide 74

Slide 74 text

Initialization flow JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() PLUGINS_STARTED . . . 74

Slide 75

Slide 75 text

Example https://github.com/oleg-nenashev/demo-jenkins-config-as-code 75

Slide 76

Slide 76 text

More examples • https://github.com/oleg-nenashev/demo-jenkins- config-as-code • https://github.com/Praqma/JenkinsAsCodeReference • https://github.com/librecores/librecores-ci 76

Slide 77

Slide 77 text

Groovy! #ornot 77

Slide 78

Slide 78 text

Groovy Hook Limitations. General • Jenkins API is scattered and complex • Javadoc is not user-friendly • Public API is not enough sometimes • No export from UI Solution: • Configuration as Code Plugin 78

Slide 79

Slide 79 text

Groovy Hook Limitations. Missing Features • No class support • No shared libraries support • Error propagation: Jenkins does not stop on script error • Restart is required to apply changes There are workarounds 79

Slide 80

Slide 80 text

More about Groovy Hooks https://speakerdeck.com/onenashev/gbg-jam-managing-jenkins-with- groovy-hooks-and-jcasc • Examples • Tips & Tricks • Development and debugging 80

Slide 81

Slide 81 text

• Feb 2018 – first alpha • Aug 2018 – release candidates • Configuration via YAML • jenkinsci/configuration-as-code-plugin New plugin: Jenkins Configuration-as-Code 81

Slide 82

Slide 82 text

jenkins: systemMessage: "JCasC Demo" numExecutors: 1 scmCheckoutRetryCount: 4 mode: NORMAL securityRealm: local: allowsSignup: false users: - id: demoAdmin password: ${adminpw} jenkins.yaml 82

Slide 83

Slide 83 text

• YAML specification • Configuration export to YAML • CLI and REST API for configuration management • Pre-flight validations and dry-run • Support of many plugins out-of-the-box Features 83

Slide 84

Slide 84 text

Limitations (temporary?) • Not all plugins are supported • Forward/backward compatibility is not guaranteed • YAML - No dynamic logic / scripting • Startup Chicken&Egg for plugins • Race conditions for jobs • Handling of removed sections • Export from UI – many open issues 84

Slide 85

Slide 85 text

Initialization with JCasC JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . • Runs after the plugins get loaded • Runs in parallel with job loading • Can trigger restart on-demand 85

Slide 86

Slide 86 text

Comparison JCasC Plugin Groovy Hooks YAML – standard declarative definition Low barrier to entry Limited integrations Forward compatibility risks External rollback Apply changes w/o restart No debugging, YAGNI? Dry run mode + validation Export from UI CLI and REST API for config management Groovy – general-purpose OOP language High barrier to entry Can manage everything Compatible if API is stable External rollback Restart is required Debugging support 86

Slide 87

Slide 87 text

87 Why not both?

Slide 88

Slide 88 text

Combining JCasC and Groovy hooks JOBS_LOADED EXTENSIONS_AUGMENTED COMPLETED Groovy Hooks: init() JCasC PLUGINS_STARTED . . . • Groovy hooks run after JCasC • Hooks can fine-tune the configuration: • Missing integrations • Dynamic scripting • Job configurations and migrations • Use JCasC where possible 88

Slide 89

Slide 89 text

https://github.com/oleg- nenashev/demo-jenkins-config-as- code/tree/casc-plugin 89

Slide 90

Slide 90 text

Demo. Local Pipeline Development Env Intellij IDEA Filesystem SCM Plugin • Documentation, Syntax • Static analysis • Debug (only for hooks) Local Jenkins instance Source Code (local .git repos) • Configuration-as-Code • Same as production • Repos with libs • Jenkinsfile JENKINS_HOME (volume) 90

Slide 91

Slide 91 text

JCasC. Takeaways • Select an engine depending on your needs and expertise • New to Jenkins? Use JCasC • Combine engines if needed 91

Slide 92

Slide 92 text

What else? • Java 10+ Support • http://bit.ly/jenkins-java10-hackathon-summary • Experiments towards new Java support 92

Slide 93

Slide 93 text

What else? • Java 10+ Support • BlueOcean Extensibility • JEP-204 – BlueOcean Extensibility API • JEP-203 – BlueOcean Extensibility 93

Slide 94

Slide 94 text

What else? • Java 10+ Support • BlueOcean Extensibility • Google Summer of Code • Code Coverage API Plugin • Remoting over Apache Kafka • Simple Pull Request Job (aka Pipeline as YAML) https://jenkins.io/projects/gsoc/ 94

Slide 95

Slide 95 text

• Jenkins evolves. Key issues in the architecture and community are being addressed • There are new projects to try Takeaways != 95

Slide 96

Slide 96 text

Takeaways • It’s a great time to contribute ;) • https://jenkins.io/participate 96

Slide 97

Slide 97 text

1. Join special interest groups (SIGs) 2. Let us know about your pain-points 3. Share your stories and experiences • Jenkins.io/blog and other blogs • Meetups Contribution != Code 97 Jenkins needs you!

Slide 98

Slide 98 text

Links Shifting Gears: • https://jenkins.io/blog/2018/08/31/shifting-gears/ Projects: • https://jenkins.io/sigs/cloud-native/ • https://jenkins-x.io/ • https://github.com/jenkins-infra/evergreen Configuration-as-Code Plugin • https://github.com/jenkinsci/configuration-as-code-plugin 98

Slide 99

Slide 99 text

Jenkins Events. What’s next? • Sep 16-19 - DevOps World | Jenkins World US • https://www.cloudbees.com/devops-world/san-francisco • Oct 09 – Jenkins Meetup in Zurich • Cosmin Cojocar, Jenkins X: Continuous Applications Delivery for Kubernetes • https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/254221707/ • Oct 19 – Day of Jenkins [as code], Copenhagen • Oct 22-25 – DevOps World | Jenkins World Nice • https://www.cloudbees.com/devops-world/nice • Late Oct – Lausanne or Neuchatel? 99 DevOps World | Jenkins World 20% Discount Code JWJAMMEMBER

Slide 100

Slide 100 text

Contacts: E-mail: [email protected] GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 100