Slide 1

Slide 1 text

Cloud Native Jenkins update Jenkins Contributor Summit Sep 17, 2018 Oleg Nenashev Jesse Glick Cloud Native SIG members 1

Slide 2

Slide 2 text

Cloud Native SIG ? // No official logo so far ● SIG = Special Interest Group ● Founded in July 2018 ● https://jenkins.io/sigs/cloud-native/ ● Goal: Improve Jenkins to run on Cloud environments as a "Cloud Native" application 2

Slide 3

Slide 3 text

Cloud Native Jenkins. Areas Pluggable storage Cloud Native Jenkins Architecture Configuration as Code 3

Slide 4

Slide 4 text

Cloud Native Jenkins SIG. Members 36 participants, 3 company members https://jenkins.io/sigs/cloud-native/ 4 . . .

Slide 5

Slide 5 text

Company participants 5

Slide 6

Slide 6 text

Pluggable Storage 6

Slide 7

Slide 7 text

Pluggable storage ● Jenkins stores everything in $JENKINS_HOME ○ Well-known scalability / performance bottleneck ● Many plugins and workarounds ● Limited support by Jenkins architecture ○ Custom steps are required ○ s3() or artifactory() ● No hacks, we change the architecture 7

Slide 8

Slide 8 text

We want Pluggable Storage to be... • TRANSPARENT • No configuration on user side • Standard steps work out-of-the-box ▸ archiveArtifacts 'giant.war' ▸ junit 'test-*.xml' ▸ ... 8

Slide 9

Slide 9 text

We want Pluggable Storage to be... • TRANSPARENT • SCALABLE • Handle high parallel load • Upload large data straight from agent • Multi-master setups 9

Slide 10

Slide 10 text

We want Pluggable Storage to be... • TRANSPARENT • SCALABLE • EXTENSIBLE • We provide API and reference implementations • We encourage contributors to create more implementations 10

Slide 11

Slide 11 text

Pluggable Storage: current state & priorities • Artifacts • Build logs • Configuration • Credentials • Builds • Jobs • Test results Defined at the JW 2016 Contributor Summit • Code coverage • Static analysis • System logs • Task logs • Fingerprints • ... Legend done & available work in progress not started 11

Slide 12

Slide 12 text

External Artifact Manager Status: Available • JEP-202 - “External artifact storage” • API patches in Jenkins 2.118+ Implementations • Artifact Manager for S3 • http://plugins.jenkins.io/artifact-manager-s3 • Available in Evergreen 12

Slide 13

Slide 13 text

Credentials Storage Status: Available API - Credentials Plugin 1.15+ (>2 years) • CredentialsProvider extension point • No API for hudson.Secret, no plans Implementations: • Kubernetes Credentials Provider 13

Slide 14

Slide 14 text

External Configurations Status: Work In Progress • Idea: Make XML file (and other types) storage destinations pluggable • JEP-213 - Configuration Storage API in the core Implementations: • SQL - by Alex Nordlund (deepy) • K8s Resources (same Kubeify by James Strachan) config.xml 14

Slide 15

Slide 15 text

Test Results Status: Work in progress • API: Jenkins JUnit plugin extensions • junit-plugin/pull/110 • Upload test results from agents • Details on-demand • Transparent for plugin features Implementations • SQL (e.g. Aurora Serverless) • Document storage (e.g. DynamoDB) • TMD (e.g. TestLink) 15

Slide 16

Slide 16 text

Jenkins master Build console External Build Logging Jenkins agent Execution Run Log Storage CLI Standard Logging Systems Status: Work In Progress

Slide 17

Slide 17 text

External Build Logging: Implementations 1. Compatibility: Filesystem-based storage 2. Elasticsearch - generic implementation ● jenkinsci/external-logging-elasticsearch-plugin 3. AWS CloudWatch Logs (perhaps via Fluentd) ● jenkinsci/pipeline-log-fluentd-cloudwatch-plugin 17 https://youtu.be/9lTOtC9wA_I

Slide 18

Slide 18 text

Pluggable Storage Future stories • Code Coverage storage (based on Code Coverage API) • Static Analysis: FindBugs, Codestyle, etc. (based on Warnings Plugin 5.0) More info: https://jenkins.io/sigs/cloud-native/pluggable-storage/ 18

Slide 19

Slide 19 text

Cloud Native Jenkins 19

Slide 20

Slide 20 text

20

Slide 21

Slide 21 text

Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 21

Slide 22

Slide 22 text

Cloud Native Jenkins https://flic.kr/p/75ncBF Jolt in Jenkins https://flic.kr/p/7MUAGk 22

Slide 23

Slide 23 text

• General purpose CI/CD engine that runs on Kubernetes that embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view Already in the SIG 23

Slide 24

Slide 24 text

• With all the good things of Jenkins we’ve been pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG, led by Carlos Sanchez • Follow the footsteps of Jenkins X and converge with it Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 24

Slide 25

Slide 25 text

• Closer than you might think, because we have many of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Cloud Native Jenkins MVP // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 25

Slide 26

Slide 26 text

Oleg, Carlos and Jesse created a prototype (blog) ● Single-shot masters ● Configuration as code ● Jenkinsfile Runner ● Custom WAR Packager Jenkins X 2.x uses the toolchain as well Presentation & demo: Tuesday, 2:30 PM Cloud Native Jenkins MVP. Prototype https://jenkins.io/blog/2018/09/12/speaker-b log-a-cloud-native-jenkins/ 26

Slide 27

Slide 27 text

● All activities in Jenkins X and Jenkins Evergreen ● Jenkinsfile Runner by Nicolas and Kohsuke ● Cloud Native networking in Jenkins a. Remoting over Kafka project by Pham Vu Tuan b. (?) Kubernetes-native solutions Other ongoing activities 27

Slide 28

Slide 28 text

What’s next? 28

Slide 29

Slide 29 text

1. Define plans towards Cloud Native Jenkins 2. Continue Pluggable storage work a. External Log Storage - GA b. Test Results c. JEP-213 - Configuration Storage for Jenkins 3. More alignment with Jenkins X & Evergreen 29

Slide 30

Slide 30 text

Cloud Native Jenkins SIG at the summit ● This presentation ● 1.00PM-3:00PM - Cloud Native SIG Meeting (BoF table) ○ All - Discuss Jenkins: Shifting Gears & Cloud Native SIG ○ Pham Vu Tuan - Remoting over Kafka presentation ○ Other topics (unconference) ● 3.30PM - Ignite Talks ○ Propose your own topic! 30

Slide 31

Slide 31 text

31 What can you do? 1. Join the Cloud Native SIG to shape the future of Jenkins! 2. Try existing implementations and provide feedback 3. Create new External Storage Implementations https://jenkins.io/sigs/cloud-native/

Slide 32

Slide 32 text

Channels Gitter • https://gitter.im/jenkinsci/cloud-native-sig Mailing list • https://groups.google.com/forum/#!forum/jenkins-cloud-native-sig Meetings • https://jenkins.io/sigs/cloud-native/#meetings 32

Slide 33

Slide 33 text

Thank you! 33