Share
Embed
Slide 1
Slide 1 text
[R]Evolution of open source
CI/CD Tools
by Oleg Nenashev, CDF/Jenkins
Nov 08, 2021
Slide 2
Slide 2 text
> whoami
@oleg_nenashev
oleg-nenashev
Slide 3
Slide 3 text
3
Agenda
● Why do WE need CI/CD?
● Generations of CI/CD tools
● What’s now and what’s next?
● What can you do after the talk?
Ask me later:
● How do we adjust as a company?
● Jenkins roadmap
These slides:
@oleg_nenashev
community.jenkins.io/t/new-t
alk-evolution-of-open-source-
ci-cd-tools/207
Discussion
bit.ly/devoops2021-cicd
Slide 4
Slide 4 text
Disclaimer
● Opinions are my own, happy to discuss
● My talk does not represent opinions of my
employer, the Continuous Delivery Foundation,
or the Jenkins community
● I present only open source projects today
Slide 5
Slide 5 text
What do Continuous Integration and
Continuous Delivery mean?
5
Slide 6
Slide 6 text
CI or not CI?
Code
Pom.xml
Tests
git clone
mvn clean verify
$? != 0 : sendmail
SCM
CRONTAB
Your Email
Inbox
Slide 7
Slide 7 text
7
CI or CD?
???
Slide 8
Slide 8 text
8
CI or CD?
Automation . Marketing
CI
CD
…
Slide 9
Slide 9 text
9
Disclaimer - CI/CD Tool
${YourCICDTool} Logo
Slide 10
Slide 10 text
Why do we need
CI/CD?
Selfish* view(s) on CI/CD
* it’s fine!
Slide 11
Slide 11 text
✔ Fast issue discovery? – Yes…
✔ Increased product quality? – Yes…
✔ Project transparency? – Yes…
11
Why do WE need CI/CD?
Slide 12
Slide 12 text
Managers
• CI/CD works
• Less resources
spent by the team
• Cool data and
screenshots for
slides
Slide 13
Slide 13 text
13
http://imgs.xkcd.com/c
omics/compiling.png
WAITING FOR CI RESULTS
BUILDING!
Slide 14
Slide 14 text
14
We do NOT want
• Merging changes
• Reviewing broken code
• Reviewing codestyle
and code smells
• Manually checking
dependencies (licenses,
security, etc.)
Slide 15
Slide 15 text
CI/CD Horrors
15
• Many notifications
• Going through zillions of links
• Digging into reports
• Infrastructure failures
• Flaky test
• UI!
Slide 16
Slide 16 text
16
We do NOT want
• Complex UI and
reports
• Endless forms
Slide 17
Slide 17 text
17
An engineer does not want to
study Jenkins.
He wants “build successful”
What do developers want?
Luigi is an
automation engineer
Slide 18
Slide 18 text
18
We do NOT want
• Maintain CI/CD
infrastructure
• Troubleshoot infra
failures
• Think about the
infra at all
Slide 19
Slide 19 text
✔ Fast issue discovery? – Yes…
✔ Increased product quality? – Yes…
✔ Project transparency? – Yes…
✔ Contributor Experience (aka User
Experience aka Developer Experience)
19
CI and CD 2021
Slide 20
Slide 20 text
● Continuous Integration and Continuous
Delivery are about processes not tools
● CI and CD are about you => Developer
experience
● Tools are here to help with
implementing processes
● Tools are about automation
20
Takeaways
Slide 21
Slide 21 text
21
CI/CD Tools and
their evolution
Slide 22
Slide 22 text
22
Continuous Delivery Landscape (Short)
https://landscape.cd.foundation/
Slide 23
Slide 23 text
23
The reality...
https://twitter.com/dastbe/status/
1303858170155081728
Slide 24
Slide 24 text
24
Types of CI/CD tools
Frameworks “Semi-manufactures”
• Many features provided
out of the box
• Can be extended
• Maximum flexibility
• Needs configuration
“Ready to fly”
• Everything available
out of the box
• Opinionated
• Hard to customize
Slide 25
Slide 25 text
Generations of CI/CD Tools (Oleg’s IMHO)
25
III
II
I CRON
. . .
Slide 26
Slide 26 text
Система
Docs
Plugins Configs
Infra-
structure
Backend
Samples
Demos
Infra-
structure
Infra-
structure
26
?
without proper
maintenance
Generation III.
Complexity, mostly frameworks
Slide 27
Slide 27 text
Generation III. Complexity
Spaghetti Automation
• A lot of customization
• Duplication
• Complexity
• Lack of expertise
• Difficult to maintain
• Mistakes
27
Slide 28
Slide 28 text
28
Generation III.
New demands
Shift
left
Generation III evolves
… but not fast enough
for early adopters
Slide 29
Slide 29 text
Generations of CI/CD Tools (Oleg’s IMHO)
29
IV
III
II
I CRON
….
Slide 30
Slide 30 text
● Takes experiences of Generation III into account
● Targets new environments
● Scalability and public clouds in mind
● Opinionated
BUT:
● Has to catch-up feature wise
● Too opinionated for many
30
Generation IV
IV
Slide 31
Slide 31 text
31
Deployment, originally designed for
Cloud-Native
Bare-Metal (sic!)
SaaS, Private and
Public Clouds
Slide 32
Slide 32 text
[X] as code – part of CD and DevOps
32
[X] – Configuration,
Documentation,
Pipeline…
Everything!
Store [X] along with the project
Versioning changes
Testing [X] along with the project
Slide 33
Slide 33 text
Configuration as Code in CI/CD
33
Managing automation jobs
(Pipeline as Code)
Managing the system itself
(infrastructure as code)
Slide 34
Slide 34 text
Pipelines
34
Slide 35
Slide 35 text
Pipelines
35
• Just another
buzzword?
• All stages as code
• Various formats
Slide 36
Slide 36 text
36
Pipeline may be different
Pipeline
Imperative
Declarative
Other…
Slide 37
Slide 37 text
37
Declarative definitions
are easier to use
Slide 38
Slide 38 text
Example -
AppVeyor
38
Slide 39
Slide 39 text
39
TeamCity
KotlinDSL
Slide 40
Slide 40 text
40
Imperative definitions
are more universal and
flexible
Slide 41
Slide 41 text
41
Scripted Pipeline
#!groovy
def imageName = 'jenkinsciinfra/ircbot’
node('docker') {
checkout scm
// Немного магии для получения тэга Docker-образа
sh 'git rev-parse HEAD > GIT_COMMIT'
shortCommit = readFile('GIT_COMMIT').take(6)
def imageTag = "build${shortCommit}»
stage 'Build ircbot'
withMavenEnv
(["BUILD_NUMBER=${env.BUILD_NUMBER}:${shortCommit}"]) {
sh 'make bot' // Make вызывает Maven
}
stage 'Build container'
def whale = docker.build("${imageName}:${imageTag}”)
stage 'Deploy container'
whale.push()
} https://github.com/jenkins-infra/ircbot
Slide 42
Slide 42 text
42
Slide 43
Slide 43 text
43
Example - Jenkins
Job DSL
Jenkins Scripted Pipeline
Jenkins Job Builder
Groovy
Groovy
YAML
…
Jenkins Declarative Pipeline Groovy
Slide 44
Slide 44 text
44
Gen III CI/CD tools
evolved!
… or became legacy
Slide 45
Slide 45 text
Evolution of generation III
45
IV
III
Evolution
R.I.P.
???
Slide 46
Slide 46 text
!=
Jenkins Evolution Example
46
Slide 47
Slide 47 text
Modern Jenkins
• Pipeline-as-Code
• Configuration-as-Code
• New plugins and integrations
• Modern packaging
• Distributions for public clouds
47
Slide 48
Slide 48 text
48
CI/CD Tools
Generation V
V
Slide 49
Slide 49 text
CI/CD Tools. Generation V
49
● Learns from experiences of previous generations
○ Opinionated but no longer general purpose
● High specialization
○ Unix way: “Do One Thing and Do It Well”
● Mostly cloud native
● Mostly open source as a tool or as ecosystem
V
Slide 50
Slide 50 text
CI/CD Tools. Generation V - Examples
50
??? Both IV and V
Slide 51
Slide 51 text
Example
Keptn - Control-plane for DevOps automation
of cloud-native applications
51
https://keptn.sh
Slide 52
Slide 52 text
Keptn
// Sandbox project in the CNCF
● Control plane, admin frontend/CLI
● Observability, dashboards & alerting
● SLO-driven multi-stage delivery
● Operations & remediation
52
https://keptn.sh
Slide 53
Slide 53 text
Keptn
● Sandbox project in the CNCF
● Observability, dashboards & alerting
● SLO-driven multi-stage delivery
● Operations & remediation
53
https://keptn.sh
Slide 54
Slide 54 text
Oleg’s view
54
What’s now?
Slide 55
Slide 55 text
55
https://cd.foundation/blog/202
1/02/01/trends-that-will-define-
ci-cd-in-2021/
Slide 56
Slide 56 text
CI/CD Tools
● Battle of generations, continued
○ Generation 3 and 4 tools should keep evolving… or die
○ Generation 5 keeps emerging
56
Slide 57
Slide 57 text
Getting out of Local Optimum
57
“Jenkins: Shifting gears”, Kohsuke Kawaguchi, 2018
jenkins.io/blog/2018/08/31/shifting-gears
Generation IV Generation V
Slide 58
Slide 58 text
CI/CD Tools
● Battle of generations, continued
● Automation/Pipeline engines become a commodity
58
Slide 59
Slide 59 text
Pipeline engines become a commodity
59
Static
Controllers
Serverless
build engines
Jenkinsfile
Runner
Tekton
(default)
Classic
Jenkins
https://jenkins-x.io/
Jenkins X 3.x
Slide 60
Slide 60 text
CI/CD Tools
● Battle of generations, continued
● Automation/Pipeline engines become a commodity
● Focus on emerging environments and use-cases
○ Cloud native environments
○ Container environments
○ Kubernetes, Kubernetes, Kubernetes
60
Slide 61
Slide 61 text
Being Cloud Native
● Best service for each need
● Pay per use
● “Infinite” scaling
● Easy to use
● Easy to maintain
● Fast to develop
61
Slide 62
Slide 62 text
Example: Tekton
https://tekton.dev/
Tekton is a powerful and flexible
open-source framework for creating
CI/CD systems, allowing developers to
build, test, and deploy across cloud
providers and on-premise systems.
62
Slide 63
Slide 63 text
Example: Tekton
● Cloud Native
● Kubernetes Native
● Everything is in container
● Steps and pluggability via
containers
https://tekton.dev/
63
Slide 64
Slide 64 text
CI/CD Tools
● Battle of generations, continued
● Automation/Pipeline engines become a commodity
● Focus on emerging environments and use-cases
○ Cloud native environments
○ Container environments
○ Kubernetes, Kubernetes, Kubernetes
● Focus on developer experience
64
Slide 65
Slide 65 text
Focus on developer experience
65
Slide 66
Slide 66 text
Jenkins X. Out of the Box experience
• Build packs - Continuous Delivery
• GitOps
• Nexus, chartmuseum, monocular
• Environments: Local, staging, production
• IDE Integrations
66
Slide 67
Slide 67 text
CI/CD Tools
● Battle of generations, continued
● Automation/Pipeline engines become a commodity
● Focus on emerging environments and use-cases
○ Cloud native environments
○ Container environments
○ Kubernetes, Kubernetes, Kubernetes
● Focus on developer experience
● Software as a Service
67
Slide 68
Slide 68 text
Software as a Service (SaaS)
● Ongoing switch to SaaS
● SaaS does will no longer block
you from using local resources
○ GitHub Runner
○ Gitlab Runner
○ …
● Internal SaaS Platforms
○ Social coding and Inner Source
○ Same principles as SaaS
68
Slide 69
Slide 69 text
Change is not just about
tools. It is about culture
69
Slide 70
Slide 70 text
70
Culture shift
● Not just
developers
anymore
● All roles are
involved
● Cross functional
communications
● TeamWork 💪
Slide 71
Slide 71 text
Culture Impact of DevOps
● “Shift left”
○ Developers manage pipelines
○ Developers take more control
● Focus on developer experience
○ Continued evolution of Declarative syntax
○ Imperative syntax for advanced users
● Focus on interoperability
● Being open source is an advantage
71
Slide 72
Slide 72 text
Culture Impact of Inner Source
● Open Source culture gets adopted
internally by the companies
● Collaboration is key
● Internal code collaboration platforms
● CI/CD tools are critical to ensure
developer experience and quality
72
https://innersourcecommons.org/
Slide 73
Slide 73 text
Open Source Won!
73
Slide 74
Slide 74 text
Going alone is no longer an option!
Competing against communities, partnerships and foundations?
Good luck!
74
Source:
https://www.freshboxx.in/blog/technolo
gy/free-and-open-source-software-foss
-a-general-introduction
Slide 75
Slide 75 text
What’s next?
Oleg’s view
75
Slide 76
Slide 76 text
THIS CITY NEEDS A HERO?
76
Slide 77
Slide 77 text
THIS CITY NEEDS A HERO? BETTER TOGETHER
77
Slide 78
Slide 78 text
78
Continuous
Delivery
Foundation
is here to help!
Slide 79
Slide 79 text
A neutral home for
the next generation
of continuous
delivery
collaboration
Slide 80
Slide 80 text
80
https://cd.foundation/
CD Foundation is an open-source
community improving the world's ability to
deliver software with security and speed
Slide 81
Slide 81 text
CDF Mission
● CDF believes in the power of Continuous Delivery to empower developers
and teams and to produce high quality software more rapidly
● CDF believes in the open-source solutions collectively addressing the
whole Software Delivery LifeCycle
● CDF fosters and sustains the ecosystem of open-source, vendor neutral
projects through collaborations and interoperability
● CDF advocates this idea and encourages collaborations among
practitioners to share and improve their practices
81
Slide 82
Slide 82 text
82
CDF Interoperability SIG
● Focus on developer experience and
their goals, not particular tools
● How?
○ Clarify what interoperability means for CI/CD
○ Provide a neutral forum for projects
○ Highlight and promote the needs of the users
○ Explore synergies between, and enable
collaboration across, the CI/CD projects
○ Pursue solutions which are; loosely coupled,
scalable, flexible, and technology agnostic
○ Reduce the need to implement in-house solutions
○ Attract/assist projects focused on interoperability github.com/cdfoundation/sig-interoperability
Slide 83
Slide 83 text
CDF Events SIG
● Helping CI/CD systems to talk to each other
in a standardized way
● Goal: Provide interoperability through better
communication and abstraction
● How?
○ Facilitate adoption of communication standards
○ Promote adoption
○ CDEvents - new standard for CD events over
CloudEvents
83
https://github.com/cdfoundation/sig-events
github.com/eiffel-community
cloudevents.io
Slide 84
Slide 84 text
The End?
...almost
84
Slide 85
Slide 85 text
Takeaways
● Automation “is eating the world”
● Continuous Integration is an industry standard
● Continuous Delivery is an industry standard
● CI and CD are not just about tools
● Tools are there, and they are evolving
● The Continuous Delivery Foundation is here to help
Slide 86
Slide 86 text
Get ready to the era of open source tools
● Become a good open source citizen!
● Go beyond FOSS side projects, adapt your core business
● Start an Open Source Program Office
● Adopt InnerSource internally
● Participate in foundations
86
Slide 87
Slide 87 text
What to do after the talk?
1. Take an open source CI/CD tool
2. Automate one of your projects
3. Try out another tool!
4. ...
Slide 88
Slide 88 text
It’s a great time to contribute ;)
https://jenkins.io/participate
Slide 89
Slide 89 text
Contributing to the CDF
● Join special interest groups!
○ MLOps, Security,Events
○ Interoperability, Best Practices
● Spread the Word!
● Join Ambassador program
● Extend Landscape
● Contribute to the CI/CD projects,
not only member projects
89
Slide 90
Slide 90 text
Contributing to Jenkins
jenkins.io/participate
Slide 91
Slide 91 text
And Keptn too!
https://keptn.sh/community/contributing/
● Code
● Advocacy
● Adoption and Feedback
● Integrations
● CloudEvents Ecosystem
91
Slide 92
Slide 92 text
Contacts:
E-mail: [email protected]
GitHub: oleg-nenashev
Twitter: @oleg_nenashev
QUESTIONS?
Slide 93
Slide 93 text
93
Where is an enterprise
${COMPANY_NAME}?
${COMPANY_
NAME} logo
Slide 94
Slide 94 text
94
Risks for ${COMPANY_NAME}
● Perceived as classic enterprises focused
company, risk of becoming legacy
● Low adoption in the rapidly growing cloud native
domain (aka “high performers”)
● Increasing cost of traditional top-to-bottom sales
● Growing competition from FOSS and Cloud
Native vendors
● Higher risk of churn by DevOps adopters and
high performers (need to have control)
${COMPANY_
NAME} logo
Slide 95
Slide 95 text
Low adoption + Competition + Churn
95
* Not yet, but soon to become one for many companies
Slide 96
Slide 96 text
${COMPANY_NAME}. Getting out of Local Optimum
96
“Jenkins: Shifting gears”, Kohsuke Kawaguchi, 2018
jenkins.io/blog/2018/08/31/shifting-gears
Enterprises High performers
Slide 97
Slide 97 text
Going alone is no longer an option!
Competing against communities, partnerships and foundations?
Good luck!
97
Source:
https://www.freshboxx.in/blog/technolo
gy/free-and-open-source-software-foss
-a-general-introduction
Slide 98
Slide 98 text
98
Making ${COMPANY_NAME} a
first class open source citizen
Oleg’s Proposals
Slide 99
Slide 99 text
Key ideas
● Become a good open source citizen!
● Make ongoing open source participation visible
● Embrace open source participation by your employees
● Go beyond FOSS side projects, adapt your core business
● Start an Open Source Program Office
● Join and participate in foundations
● Adopt InnerSource internally
99
Slide 100
Slide 100 text
Too many initiatives?
Open Source Program Office!
100
Slide 101
Slide 101 text
101
Source:
https://mercan.mercari.com/
en/articles/25786/
Slide 102
Slide 102 text
OSPO - Projects and programs
102
● Define Open Source Usage Strategy
● Define Open Source Investment Strategy - public or not
● Open Source Stewardship
● Foundation memberships representation, e.g. CDF, CNCF, Linux
Foundation, OpenSSF, Apache Foundation, Eclipse Foundation
● Facilitate community marketing and developer advocacy
● Adopt open source culture and best practices for internal product
development and collaboration. Be InnerSource!
Slide 103
Slide 103 text
Open Source Strategy
● Public
○ Communicate the vision to customers, end users and the community
○ Improve the company’s image
○ Communicate the vision to potential employees
● Internal
○ FOSS Usage guidelines and Licensing strategy
○ Clear decision making and boundaries for FOSS work
○ Minimum obstacles for open source contribution when reasonable
○ ...
103
Slide 104
Slide 104 text
InnerSource at ${COMPANY_NAME}. Why?
● Share best open source practices and
knowledge with the teams
● Improve cross-functional and
cross-project communications
● Remove barriers and break down silos via
better collaboration
● Improve overall employee experiences.
Happy engineers do not quit (c)
104
innersourcecommons.org
Slide 105
Slide 105 text
InnerSource at ${COMPANY_NAME}. How?
● Create public channels within the company for inner
source and open source culture discussions
● Start internal InnerSource Best Practices knowledge base
● Start a regular internal newsletter with tips and links.
Could reuse materials from InnerSource Commons and
contribute back (see below)
● Start regular internal meetups / office hours
● Start a regular internal Hackergarten event for study and
experiences sharing. Focus on internal products
● ….
105
innersourcecommons.org
Slide 106
Slide 106 text
Social media is your friend!
● Dedicated ${COMPANY_NAME} Community channels
● Promote internal and external contributions
● Collaboration with marketing, cross-posting and guerilla marketing
● Life @ ${COMPANY_NAME} includes Open Source!
● Same for the careers site!
106
Slide 107
Slide 107 text
More CDF and project SIGs
● Security SIG
● MLOps SIG
● Best Practices SIG
● Project SIGs and working groups
○ jenkins.io/sigs
○ jenkins.io/projects
107
Slide 108
Slide 108 text
CDF. Other Entities
● Governing board
○ Budgeting, staffing, etc.
○ Infra sponsorships
● Technical Oversight Committee
○ Defining technical direction
● Outreach Committee
○ Marketing, events
○ Ambassador program
108