Jenkins Contributor Summit 2018: Cloud Native Jenkins update

Cloud Native Jenkins update from the Jenkins contributor summit at DevOps World | Jenkins World (Sep 17, 2018). Cloud Native SIG, Pluggable Storage, Ephemeral Jenkins, and future plans.

Transcript

1. Cloud Native Jenkins update Jenkins Contributor Summit Sep 17, 2018 Oleg Nenashev Jesse Glick Cloud Native SIG members 1
2. Cloud Native SIG ? // No official logo so far • SIG = Special Interest Group • Founded in July 2018 • https://jenkins.io/sigs/cloud-native/ • Goal: Improve Jenkins to run on Cloud environments as a "Cloud Native" application 2
3. Cloud Native Jenkins. Areas Pluggable storage Cloud Native Jenkins Architecture Configuration as Code 3
4. Cloud Native Jenkins SIG. Members 36 participants, 3 company members https://jenkins.io/sigs/cloud-native/ 4 . . .
5. Company participants 5
6. Pluggable Storage 6
7. Pluggable storage • Jenkins stores everything in $JENKINS_HOME ◦ Well-known scalability / performance bottleneck • Many plugins and workarounds • Limited support by Jenkins architecture ◦ Custom steps are required ◦ s3() or artifactory() • No hacks, we change the architecture 7
8. We want Pluggable Storage to be... • TRANSPARENT • No configuration on user side • Standard steps work out-of-the-box ▸ archiveArtifacts 'giant.war' ▸ junit 'test-*.xml' ▸ ... 8
9. We want Pluggable Storage to be... • TRANSPARENT • SCALABLE • Handle high parallel load • Upload large data straight from agent • Multi-master setups 9
10. We want Pluggable Storage to be... • TRANSPARENT • SCALABLE • EXTENSIBLE • We provide API and reference implementations • We encourage contributors to create more implementations 10
11. Pluggable Storage: current state & priorities • Artifacts • Build logs • Configuration • Credentials • Builds • Jobs • Test results Defined at the JW 2016 Contributor Summit • Code coverage • Static analysis • System logs • Task logs • Fingerprints • ... Legend done & available work in progress not started 11
12. External Artifact Manager Status: Available • JEP-202 - “External artifact storage” • API patches in Jenkins 2.118+ Implementations • Artifact Manager for S3 • http://plugins.jenkins.io/artifact-manager-s3 • Available in Evergreen 12
13. Credentials Storage Status: Available API - Credentials Plugin 1.15+ (>2 years) • CredentialsProvider extension point • No API for hudson.Secret, no plans Implementations: • Kubernetes Credentials Provider 13
14. External Configurations Status: Work In Progress • Idea: Make XML file (and other types) storage destinations pluggable • JEP-213 - Configuration Storage API in the core Implementations: • SQL - by Alex Nordlund (deepy) • K8s Resources (same Kubeify by James Strachan) config.xml 14
15. Test Results Status: Work in progress • API: Jenkins JUnit plugin extensions • junit-plugin/pull/110 • Upload test results from agents • Details on-demand • Transparent for plugin features Implementations • SQL (e.g. Aurora Serverless) • Document storage (e.g. DynamoDB) • TMD (e.g. TestLink) 15
16. Jenkins master Build console External Build Logging Jenkins agent Execution Run Log Storage CLI Standard Logging Systems Status: Work In Progress
17. External Build Logging: Implementations 1. Compatibility: Filesystem-based storage 2. Elasticsearch - generic implementation • jenkinsci/external-logging-elasticsearch-plugin 3. AWS CloudWatch Logs (perhaps via Fluentd) • jenkinsci/pipeline-log-fluentd-cloudwatch-plugin 17 https://youtu.be/9lTOtC9wA_I
18. Pluggable Storage Future stories • Code Coverage storage (based on Code Coverage API) • Static Analysis: FindBugs, Codestyle, etc. (based on Warnings Plugin 5.0) More info: https://jenkins.io/sigs/cloud-native/pluggable-storage/ 18
19. Cloud Native Jenkins 19
21. Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ Slides: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 21
22. Cloud Native Jenkins https://flic.kr/p/75ncBF Jolt in Jenkins https://flic.kr/p/7MUAGk 22
23. • General purpose CI/CD engine that runs on Kubernetes that embraces fundamentally different architecture & extensibility mechanisms • Ingredients • Kubernetes as platform • New extensibility mechanism • Data on cloud managed data services • Configuration as Code • Evergreen • Secure by design Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view Already in the SIG 23
24. • With all the good things of Jenkins we’ve been pushing for • Great onboarding experiences, led by Jenkins 2 & Evergreen • Modern lovable UX led by Blue Ocean • Being a general purpose, usable for any software development • In the context of Cloud Native SIG, led by Carlos Sanchez • Follow the footsteps of Jenkins X and converge with it Cloud Native Jenkins // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 24
25. • Closer than you might think, because we have many of the ingredients already going • Basic picture • Webhook receiver • Build engine ala Jenkinsfile runner, sprinkled with configuration-as-code • Delivered continuously, ala Evergreen • Not in scope for MVP • No GUI • No long term data storage Cloud Native Jenkins MVP // source: https://drive.google.com/file/d/160LjRKw42XXuXT2n4j8BLCzsSL5z2m2_/view 25
26. Oleg, Carlos and Jesse created a prototype (blog) • Single-shot masters • Configuration as code • Jenkinsfile Runner • Custom WAR Packager Jenkins X 2.x uses the toolchain as well Presentation & demo: Tuesday, 2:30 PM Cloud Native Jenkins MVP. Prototype https://jenkins.io/blog/2018/09/12/speaker-b log-a-cloud-native-jenkins/ 26
27. • All activities in Jenkins X and Jenkins Evergreen • Jenkinsfile Runner by Nicolas and Kohsuke • Cloud Native networking in Jenkins a. Remoting over Kafka project by Pham Vu Tuan b. (?) Kubernetes-native solutions Other ongoing activities 27
28. What’s next? 28
29. 1. Define plans towards Cloud Native Jenkins 2. Continue Pluggable storage work a. External Log Storage - GA b. Test Results c. JEP-213 - Configuration Storage for Jenkins 3. More alignment with Jenkins X & Evergreen 29
30. Cloud Native Jenkins SIG at the summit • This presentation • 1.00PM-3:00PM - Cloud Native SIG Meeting (BoF table) ◦ All - Discuss Jenkins: Shifting Gears & Cloud Native SIG ◦ Pham Vu Tuan - Remoting over Kafka presentation ◦ Other topics (unconference) • 3.30PM - Ignite Talks ◦ Propose your own topic! 30
31. 31 What can you do? 1. Join the Cloud Native SIG to shape the future of Jenkins! 2. Try existing implementations and provide feedback 3. Create new External Storage Implementations https://jenkins.io/sigs/cloud-native/
32. Channels Gitter • https://gitter.im/jenkinsci/cloud-native-sig Mailing list • https://groups.google.com/forum/#!forum/jenkins-cloud-native-sig Meetings • https://jenkins.io/sigs/cloud-native/#meetings 32
33. Thank you! 33