Share
Embed
Jenkins Contributor Summit 2018: Cloud Native Jenkins update
Cloud Native Jenkins update from the Jenkins contributor summit at DevOps World | Jenkins World (Sep 17, 2018). Cloud Native SIG, Pluggable Storage, Ephemeral Jenkins, and future plans.
Oleg Nenashev PRO
September 17, 2018
Transcript
33 slides
-
Cloud Native Jenkins update Jenkins Contributor Summit Sep 17, 2018
Oleg Nenashev Jesse Glick Cloud Native SIG members 1 -
Cloud Native SIG ? // No official logo so far
• SIG = Special Interest Group • Founded in July 2018 • https://jenkins.io/sigs/cloud-native/ • Goal: Improve Jenkins to run on Cloud environments as a "Cloud Native" application 2 -
Cloud Native Jenkins. Areas Pluggable storage Cloud Native Jenkins Architecture
Configuration as Code 3 -
Cloud Native Jenkins SIG. Members 36 participants, 3 company members
https://jenkins.io/sigs/cloud-native/ 4 . . . -
Pluggable storage • Jenkins stores everything in $JENKINS_HOME ◦ Well-known
scalability / performance bottleneck • Many plugins and workarounds • Limited support by Jenkins architecture ◦ Custom steps are required ◦ s3() or artifactory() • No hacks, we change the architecture 7 -
We want Pluggable Storage to be... • TRANSPARENT • No
configuration on user side • Standard steps work out-of-the-box ▸ archiveArtifacts 'giant.war' ▸ junit 'test-*.xml' ▸ ... 8 -
We want Pluggable Storage to be... • TRANSPARENT • SCALABLE
• Handle high parallel load • Upload large data straight from agent • Multi-master setups 9 -
We want Pluggable Storage to be... • TRANSPARENT • SCALABLE
• EXTENSIBLE • We provide API and reference implementations • We encourage contributors to create more implementations 10 -
Pluggable Storage: current state & priorities • Artifacts • Build
logs • Configuration • Credentials • Builds • Jobs • Test results Defined at the JW 2016 Contributor Summit • Code coverage • Static analysis • System logs • Task logs • Fingerprints • ... Legend done & available work in progress not started 11 -
External Artifact Manager Status: Available • JEP-202 - “External artifact
storage” • API patches in Jenkins 2.118+ Implementations • Artifact Manager for S3 • http://plugins.jenkins.io/artifact-manager-s3 • Available in Evergreen 12 -
Credentials Storage Status: Available API - Credentials Plugin 1.15+ (>2
years) • CredentialsProvider extension point • No API for hudson.Secret, no plans Implementations: • Kubernetes Credentials Provider 13 -
External Configurations Status: Work In Progress • Idea: Make XML
file (and other types) storage destinations pluggable • JEP-213 - Configuration Storage API in the core Implementations: • SQL - by Alex Nordlund (deepy) • K8s Resources (same Kubeify by James Strachan) config.xml 14 -
Test Results Status: Work in progress • API: Jenkins JUnit
plugin extensions • junit-plugin/pull/110 • Upload test results from agents • Details on-demand • Transparent for plugin features Implementations • SQL (e.g. Aurora Serverless) • Document storage (e.g. DynamoDB) • TMD (e.g. TestLink) 15 -
Jenkins master Build console External Build Logging Jenkins agent Execution
Run Log Storage CLI Standard Logging Systems Status: Work In Progress -
External Build Logging: Implementations 1. Compatibility: Filesystem-based storage 2. Elasticsearch
- generic implementation • jenkinsci/external-logging-elasticsearch-plugin 3. AWS CloudWatch Logs (perhaps via Fluentd) • jenkinsci/pipeline-log-fluentd-cloudwatch-plugin 17 https://youtu.be/9lTOtC9wA_I -
Pluggable Storage Future stories • Code Coverage storage (based on
Code Coverage API) • Static Analysis: FindBugs, Codestyle, etc. (based on Warnings Plugin 5.0) More info: https://jenkins.io/sigs/cloud-native/pluggable-storage/ 18 -
Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/
Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 21 -
• General purpose CI/CD engine that runs on Kubernetes that
embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view Already in the SIG 23 -
• With all the good things of Jenkins we’ve been
pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG, led by Carlos Sanchez • Follow the footsteps of Jenkins X and converge with it Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 24 -
• Closer than you might think, because we have many
of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Cloud Native Jenkins MVP // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 25 -
Oleg, Carlos and Jesse created a prototype (blog) • Single-shot
masters • Configuration as code • Jenkinsfile Runner • Custom WAR Packager Jenkins X 2.x uses the toolchain as well Presentation & demo: Tuesday, 2:30 PM Cloud Native Jenkins MVP. Prototype https://jenkins.io/blog/2018/09/12/speaker-b log-a-cloud-native-jenkins/ 26 -
• All activities in Jenkins X and Jenkins Evergreen •
Jenkinsfile Runner by Nicolas and Kohsuke • Cloud Native networking in Jenkins a. Remoting over Kafka project by Pham Vu Tuan b. (?) Kubernetes-native solutions Other ongoing activities 27 -
1. Define plans towards Cloud Native Jenkins 2. Continue Pluggable
storage work a. External Log Storage - GA b. Test Results c. JEP-213 - Configuration Storage for Jenkins 3. More alignment with Jenkins X & Evergreen 29 -
Cloud Native Jenkins SIG at the summit • This presentation
• 1.00PM-3:00PM - Cloud Native SIG Meeting (BoF table) ◦ All - Discuss Jenkins: Shifting Gears & Cloud Native SIG ◦ Pham Vu Tuan - Remoting over Kafka presentation ◦ Other topics (unconference) • 3.30PM - Ignite Talks ◦ Propose your own topic! 30 -
31 What can you do? 1. Join the Cloud Native
SIG to shape the future of Jenkins! 2. Try existing implementations and provide feedback 3. Create new External Storage Implementations https://jenkins.io/sigs/cloud-native/ -
Channels Gitter • https://gitter.im/jenkinsci/cloud-native-sig Mailing list • https://groups.google.com/forum/#!forum/jenkins-cloud-native-sig Meetings •
https://jenkins.io/sigs/cloud-native/#meetings 32