Cloud Friendly(?) Jenkins. How we failed to make Jenkins cloud native and what we learned?

Key takeaway: Being Cloud Friendly is a must, Cloud Native is optional. Likely YAGNI in your projects

Jenkins has been one of the most popular automation servers in the world. It is widely used for CI/CD purposes in cloud environments. You may have heard that Jenkins is not cloud native, and of course it is true. You may have heard that Jenkins controllers are a legacy monolith from the 2000s. That is not true, it is quite **cloud friendly** if used right. And yes we tried to make Jenkins cloud native, but it didn’t quite work. Why?

I have been involved in Jenkins architecture projects for more than ten years, including pluggable storage, multi-tenant Jenkins, cloud native agents, breaking down the controller monolith, Kubernetes integrations, etc., etc. Some of these efforts worked, some didn’t. But it was a great learning experience that can be used by developers and maintainers of other projects that consider re-platforming them to make cloud- and Kubernetes-native. I'll talk about the technical assumptions and mistakes we made in Jenkins, and how it worked. What to keep in mind when doing the re-platforming? And do you even need it?

Transcript

2. > whoami @oleg_nenashev oleg-nenashev Dr. Nenashev / Mr. Jenkins Community builder Developer Tools Hacker #RussiansAgainstPutin #StandWithUkraine 2
3. Async Q&A: Jenkins Forum community.jenkins.io/t/cloud-friendly-jenkins/13426
4. My Open [and not so] Communities Time * Once a contributor, [almost] always a contributor Neuchâtel 4
5. We talk Jenkins! Disclaimer • My talk does not represent my employer’s opinion • It is not an official position of the Jenkins community • It is not related to the CloudBees announcements on HA Jenkins, but that’s a good stuff 5
6. Disclaimer: Taking a break in Jenkins Away: personal events, ongoing war I participate in governance and GSoC (Co-)Maintainers welcome! I will be back! CDF Statement We stand with the people of Ukraine. Please assist humanitarian efforts for the Ukrainian people and those affected by the military invasion of Ukraine by supporting international aid organizations, including UNICEF. 6
8. @birthmarkbart
10. Flew past like plywood over Paris - idiom in the Russian language 10 J'ai survolé Paris comme du contreplaqué
11. 11 • Having a baby • COVID • War in Ukraine • Solo Parenting Me missing Kubecon
12. 12 @oleg_nenashev
13. 13 TL;DR: for your projects Being Cloud Friendly is a must, Cloud Native is optional
14. Not talking about Gradle today Join me tomorrow, Observability Day 14
17. 17 My talks on evolution of CI/CD https://speakerdeck.com/onenashev/
18. Generations of CI/CD Tools (Oleg’s IMHO) IV III II I CRON …. 18
19. Generation III. New demands Shift left Generation III evolves … but not fast enough for early adopters 19
20. != Jenkins has CHANGED 20
21. Generations of CI/CD tools IV III Evolution R.I.P. ??? V ??? 21
22. Modern Jenkins • Pipeline-as-Code • Configuration-as-Code • New plugins and integrations • Modern packaging • Distributions for public clouds 22
23. Generations of CI/CD tools IV III Evolution R.I.P. ??? V ??? IV V ??? 23
24. CI/CD Tools. Generation V • Learns from experiences of previous generations ◦ Opinionated but no longer general purpose • High specialization ◦ Unix way: “Do One Thing and Do It Well” • Mostly cloud native • Mostly open source as a tool or as ecosystem V 24
25. 25 x.com/dastbe/status/1303858170155081728
26. CI/CD Tools. Generation V - Examples * Oleg’s CI tool generation definition 26
27. Example: Tekton Cloud Native Kubernetes Native Everything is container Steps and pluggability via containers tekton.dev 27
28. Getting out of Local Optimum “Jenkins: Shifting gears”, Kohsuke Kawaguchi, 2018 jenkins.io/blog/2018/08/31/shifting-gears Generation IV Generation V * Oleg’s CI tool generation definition 28
29. Cloud Native Jenkins “General purpose CI/CD engine that runs on Kubernetes and embraces fundamentally different architecture & extensibility mechanisms” 29 https://flic.kr/p/75ncBF Kohsuke Kawaguchi, 2018
30. 30 Being Cloud Native is not running on Kubernetes
31. Being Cloud Native • Best service for each need • Pay per use • “Infinite” scaling / Scale to zero • Easy to use • Easy to maintain • Fast to develop 31
32. 32 Permanent agents On-demand agents CLOUD NATIVE (c) Jenkins Controller Web UI
33. Controller as a SPoF Macroservice 33 Permanent agents On-demand agents from cloud providers Web UI Jenkins Controller
34. Cloud Native Special Interest Group • Founded in July 2018 • jenkins.io/sigs/cloud-native • R.I.P in 2022 Goal: Improve Jenkins to run on Cloud environments as a "Cloud Native" application 34 ?
35. Cloud Native SIG Projects 35 • Pluggable Storage • Jenkins X 1.x • Configuration-as-Code • OTel and Cloud Events Integrations • Multi-tenant Jenkins Controller • Operator for Kubernetes • Jenkinsfile Runner (portable Pipeline)
37. Jenkins X. Out of the Box experience • Build packs - Continuous Delivery • GitOps • Nexus, chartmuseum, monocular • Environments: Local, staging, production • IDE Integrations 37
38. Pipeline engines become a commodity 38 Static Controllers Serverless build engines Jenkinsfile Runner Tekton (default) Classic Jenkins https://jenkins-x.io/ Jenkins X 3.x
39. Too Opinionated? 39 Jenkins X Not a Jenkins replacement, Keeps evolving
40. Cloud Native Jenkins “General purpose CI/CD engine that runs on Kubernetes and embraces fundamentally different architecture & extensibility mechanisms” 40 https://flic.kr/p/75ncBF Kohsuke Kawaguchi, 2018
41. (┛ಠ_ಠ)┛彡┻━┻ What if Jenkins controller was “on-demand”? 41
42. Jenkinsfile Runner • Binary and base Docker image • Runs a Pipeline job and shuts down • No Web UI* • On-demand packaging as a container or JAR github.com/jenkinsci/jenkinsfile-runner 42
43. JFR TL;DR 43 STDOUT/STDERR Workspace External storage Workspace Jenkinsfile Configuration (WAR, plugins, settings)
45. • Not a CI/CD solution on its own Limitations Triggers Webhook receivers Log services Result Storage Web UI 45
46. • Not a CI/CD solution on its own • No Web UI • No Queue • No build throttling, cross-controller communication, etc. • Designed for a single-container use Limitations 46
47. Foundations were not ready 47 Not a Jenkins replacement, Keeps evolving
48. Pluggable Storage: current state • Artifacts • Build logs • Configuration • Credentials • Builds • Jobs • Test results • Code coverage • Static analysis • System logs • Task logs • Fingerprints • ... Legend done & available partial not started jenkins.io/doc/book/using/pluggable-storage/ 48
50. 50 jenkins.io/projects/jenkins-operator
51. 51 https://dev.to/webhookrelay/easy-a nd-secure-jenkins-operator-deploy ment-on-kubernetes-1jn8
52. Jenkins Operator for Kubernetes 52 • Actually, two operators from different vendors • Core Jenkins vendors were busy with other projects • We tried to set up community governance too late
55. All projects worked, but no breakthrough No Jenkins successor? 55 ?
56. Did we FAIL? 56 Not really, but we didn't succeed either
57. Users want to run CI, they don’t care about infra 57
58. What they need 58
59. 59 • Infra and CI maintenance are overhead • SaaS or service • Simple for simple cases • Affordable for complex
60. Система Docs Plugins Configs Infra- structure Backend Samples Demos Infra- structure Infra- structure 60 ? without proper maintenance What they don’t want
61. 61 TL;DR: for your projects Being Cloud Friendly is a must, Cloud Native is optional
62. Being Cloud Native • Best service for each need • Pay per use • “Infinite” scaling / Scale to zero • Easy to use • Easy to maintain • Fast to develop 62
63. Being Cloud Friendly • Best service for each need Best service for your needs • Pay per use • “Infinite” scaling / Scale to zero Scales as needed • Easy to use in the cloud • Easy to maintain in the cloud • Fast to develop in the cloud 63
64. Could Cloud Native Jenkins succeed? 64
65. There are many who can do cloud native, but only a few who can do shared context services 65 Viktor Farcic devopsparadox.com
66. Jenkins succeeded as a service, not as SaaS 66 * Until now?
67. 67 cloudbees.com/blog/revolutionize-y our-jenkins-environment-with-ha-hs Shared Context / HA / SaaS by CloudBees
68. 68 Jenkins Contributor Summit, 02 Feb 2024
70. 70 Interoperability First Choose & Combine the best tools * Unix way
71. OTel in Jenkins • Traces can go through different tools, not just a single one • OTel Tracing is a DAG Source: plugins.jenkins.io/opentelemetry/ 71
72. CloudEvents Plugin (GSOC ‘21) 72
73. Tekton Client Plugin Pipeline Engine is no a commodity Tekton provides the ability to create “k8s-style resources for declaring CI/CD-style pipelines”. It is the go to CI/CD system for creating Pipelines on Kubernetes. You can create Tekton Resources from Jenkins Recent v1 Release ! Blogpost : Easily reuse Tekton and Jenkins X from Jenkins 73
74. Backstage Integrations 74 https://roadie.io/backstage/plugins/jenkins/
75. Checks Per Commit 75
76. • Multiple checks • Status reporting • Summary and Details • GitHub Markdown support • Annotations https://github.com/jenkinsci/gradle- plugin/pull/382/checks?check_run_i d=19876936412 76
77. The future is ahead! “Jenkins: Shifting gears”, Kohsuke Kawaguchi, 2018 jenkins.io/blog/2018/08/31/shifting-gears Generation IV Generation V * Oleg’s CI tool generation definition 77
78. Takeaways • One doesn’t have to be Cloud Native. Think Cloud Friendly first • Scalability is not a goal on its own, scale what needs to scale • Focus on user and developer experience • Pay your architecture debt 78
79. It’s [always] great time to contribute ;) jenkins.io/participate 79
80. It is a great time to contribute! jenkins.io/participate 80
81. GitHub: oleg-nenashev Twitter: @oleg_nenashev Jenkins Community Channels Questions? CNCF Ambassador Office Hours jenkins.io/participate 81