1
DevOps from a
Different Data Set
What 30 million workflows reveal about high performing teams
Michael Stahnke
VP Platform Engineering
@stahnma
Slide 2
Slide 2 text
2
Slide 3
Slide 3 text
3
The Setup
What are you talking about
and how does this work?
The Insights
Using the data, what can we
apply to current industry trends
and influences?
The Data
Here’s what the data tell us.
Slide 4
Slide 4 text
4
The Setup
What are you talking about
and how does this work?
Slide 5
Slide 5 text
5
Slide 6
Slide 6 text
6
Slide 7
Slide 7 text
Performance derived vs
performance described
7
Slide 8
Slide 8 text
44,000 orgs
8
Slide 9
Slide 9 text
160,000 projects
9
Slide 10
Slide 10 text
1000x larger than all
State of DevOps Surveys
10
Slide 11
Slide 11 text
What’s changed year
over year?
11
Slide 12
Slide 12 text
12
Year 2019 2020
Days in set 30 30
Orgs >40,000 >44,000
Projects >150,000 >160,000
Second Year for Analysis
Slide 13
Slide 13 text
13
Slide 14
Slide 14 text
14
State of DevOps Report
Metrics
Deployment Frequency
Lead time to Change
Change Failure Rate
MTTR
Description when
mapping to CI
How often you initiate a
pipeline
Pipeline duration
Pipeline failure rate
Time from red to green
Metric
Mapping Metrics
Throughput
Duration
Success Rate
Recovery Time
Slide 15
Slide 15 text
15
The Setup
What are you talking about
and how does this work?
The Insights
Using the data, what can we
apply to current industry trends
and influences?
The Data
Here’s what the data tell us.
Slide 16
Slide 16 text
16
The Data
Here’s what the data tell us.
Slide 17
Slide 17 text
17
Throughput
Slide 18
Slide 18 text
How often do you push
code that triggers CI?
18
Slide 19
Slide 19 text
Most projects configured
to run per push to git server
19
Slide 20
Slide 20 text
20
Percentile 2020 Value
5p 0.03
50p 0.70
90p 16.03
95p 32.125
Mean 8.22
Throughput
Slide 21
Slide 21 text
Most projects are not
deploying dozens of
times per day
21
Slide 22
Slide 22 text
Why is this different
from survey data?
22
Slide 23
Slide 23 text
“Primary application or
service you work on”
23
Slide 24
Slide 24 text
24
Percentile 2020 Value 2019 Value
5p 0.03 0.03
50p 0.70 0.80
90p 16.03 13.00
95p 32.125 25.47
Mean 8.22 5.76
Throughput
Slide 25
Slide 25 text
Those leveraging CI well,
are doing so even more
25
Slide 26
Slide 26 text
There are fewer
developers worldwide
pushing code
26
Slide 27
Slide 27 text
27
Duration
Slide 28
Slide 28 text
How long does it take
to get results?
28
Slide 29
Slide 29 text
5% of builds finish in
< 12 seconds
29
* That’s roughly 500,000 builds in this sample
Slide 30
Slide 30 text
30
Percentile 2020 Value
5p 12 sec
50p 3.96 min
90p 21.35 min
95p 34.01 min
Mean 24.6 min
Duration
Slide 31
Slide 31 text
Half of all builds finish
in under 4 minutes
31
Slide 32
Slide 32 text
32
Percentile 2020 Value 2019 Value
5p 12 sec 10 sec
50p 3.96 min 3.38 min
90p 21.35 min 19.18 min
95p 34.01 min 31.73 min
Mean 24.6 min 26.76 min
Duration delta in a year
Slide 33
Slide 33 text
All pipelines are
running longer
33
Slide 34
Slide 34 text
34
Percentile 2020 Value 2019 Value
5p 12 sec 10 sec
50p 3.96 min 3.38 min
90p 21.35 min 19.18 min
95p 34.01 min 31.73 min
Mean 24.6 min 26.76 min
Duration delta in a year
Slide 35
Slide 35 text
35
Success Rate
Slide 36
Slide 36 text
How often does your pipeline
complete with a green status?
36
Slide 37
Slide 37 text
37
Percentile 2020 Value
5p 0%
50p 61%
90p 100%
95p 100%
Mean 54%
Success Rate
Slide 38
Slide 38 text
Some of our sample dabbles
with CI, but doesn’t get a
working build
38
Slide 39
Slide 39 text
Some of our sample saw
no failures within a month
39
Slide 40
Slide 40 text
40
Percentile 2020 Value 2019 Value
5p 0% 0%
50p 61% 60%
90p 100% 100%
95p 100% 100%
Mean 54% 54%
Success Rate
Slide 41
Slide 41 text
41
Percentile 2020 Value 2019 Value
50p 61% 60%
75p 89% 86%
85p 100% 98%
Success Rate
Slide 42
Slide 42 text
42
Recovery Time
Slide 43
Slide 43 text
Time a pipeline sits
in a failure state
43
Slide 44
Slide 44 text
44
Percentile 2020 Value
5p 2.06 min
50p 55.11 min
90p 39 hours
95p 3.4 days
Mean 14.85 hours
Recovery Time
Slide 45
Slide 45 text
Quick Recovery Time can be
from multiple contributors
running in parallel
45
Slide 46
Slide 46 text
The gap between 50th and 75th percentiles
looks like it represents waiting until
tomorrow to fix a failed build
(from 55 min at 50p to 9.5 hours at 75p)
46
Slide 47
Slide 47 text
47
Percentile 2020 Value 2019 Value
5p 2.06 min 2.83 min
50p 55.11 min 52.5 min
90p 39 hours 47 hours
95p 3.4 days 3.93 days
Mean 14.85 hours 16.61 hours
Recovery Time
Slide 48
Slide 48 text
Fastest Recovery Times have
improved (10th and percentile
and lower) year over year
48
Slide 49
Slide 49 text
49
The Setup
What are you talking about
and how does this work?
The Insights
Using the data, what can we
apply to current industry trends
and influences?
The Data
Here’s what the data tell us.
Slide 50
Slide 50 text
50
The Insights
Using the data, what can we
apply to current industry trends
and influences?
Slide 51
Slide 51 text
51
What development practices
definitively work?
Slide 52
Slide 52 text
Success Rate does not
correlate with company size
52
Slide 53
Slide 53 text
Duration is longest
for teams of one
53
Slide 54
Slide 54 text
Recovery Time decreases with
increased team size (up to 200)
54
Slide 55
Slide 55 text
Longest Recovery Times
are from teams of one.
55
Slide 56
Slide 56 text
Performance is better with more
than one contributor as shown
by multiple indicators
56
Slide 57
Slide 57 text
Software is
collaborative
57
Slide 58
Slide 58 text
58
Is “Don’t Deploy on Friday” a
real thing?
Slide 59
Slide 59 text
70% less Throughput
on weekends
59
Slide 60
Slide 60 text
11% less Throughput
on Friday (UTC).
60
Slide 61
Slide 61 text
9% less Throughput
on Monday (UTC).
61
Slide 62
Slide 62 text
Conclusion: About the same amount of
work happens Monday or Friday. So
people not holding back on pushing code
on Fridays.
62
Slide 63
Slide 63 text
63
What Language Trends
emerge?
Slide 64
Slide 64 text
64
21.73% JavaScript 2.44% Vue
11.36% TypeScript 2.12% Kotlin
9.56% Python 1.70% HCL
9.04% Ruby 1.59% Swift
6.16% HTML 1.26% C++
5.37% Java 1.21% Dockerfile
4.92% PHP 1.08% C#
3.89% Go 1.00% TSQL
3.17% CSS 0.96% Jupyter Notebook
2.99% Shell 0.83% Elixir
Languages in our sample
Slide 65
Slide 65 text
65
Language Throughput
1 Ruby 11 PHP
2 TypeScript 12 Java
3 Go 13 C#
4 Python 14 Jupyter Notebook
5 Kotlin 15 Shell
6 Elixir 16 Vue
7 Swift 17 C++
8 HCL 18 HTML
9 JavaScript 19 CSS
10 TSQL 20 Dockerfile
Slide 66
Slide 66 text
66
Language Success Rate at 50p
1 Vue 11 Elixir
2 CSS 12 PHP
3 Shell 13 Jupyter Notebook
4 Dockerfile 14 Python
5 TSQL 15 Ruby
6 HTML 16 Java
7 HCL 17 Kotlin
8 Go 18 C#
9 TypeScript 19 C++
10 JavaScript 20 Swift
Slide 67
Slide 67 text
67
Language Recovery Time at 50p
1 Go 11 Vue
2 JavaScript 12 Jupyter Notebook
3 Elixir 13 Kotlin
4 HCL 14 Java
5 Shell 15 Scala
6 Python 16 Ruby
7 TypeScript 17 PHP
8 CSS 18 TSQL
9 C# 19 Swift
10 HTML 20 C++
Slide 68
Slide 68 text
68
Language Duration at 50p
1 Shell 11 PHP
2 HCL 12 TypeScript
3 CSS 13 Java
4 HTML 14 Elixir
5 Gherkin 15 TSQL
6 JavaScript 16 Kotlin
7 Vue 17 Scala
8 Go 18 Ruby
9 Jupyter Notebook 19 C++
10 Python 20 Swift
Slide 69
Slide 69 text
69
Branch Information
Slide 70
Slide 70 text
70
Slide 71
Slide 71 text
Did the use of
master branch
decrease?
71
Slide 72
Slide 72 text
Not in any significant
way….yet.
72
Slide 73
Slide 73 text
Teams are innovating and
experimenting on feature
branches
73
Slide 74
Slide 74 text
Success Rate on default branch
higher than on non-default
branches
74
Slide 75
Slide 75 text
Success Rate is 80% on the default
branch at 50th percentile and 100%
for 75th percentile and above
75
Slide 76
Slide 76 text
Success Rate at 50p is 80% for
default and 58% for non-default
branches
76
Slide 77
Slide 77 text
Duration on default branches
are faster at every percentile.
77
Slide 78
Slide 78 text
Recovery Time is lower on
default branch at every percentile.
78
Slide 79
Slide 79 text
79
How has the global pandemic
impacted team performance?
Slide 80
Slide 80 text
80
Throughput
Slide 81
Slide 81 text
Peak Throughput
was April 2020
81
Slide 82
Slide 82 text
After April,
Throughput falls a bit
82
Slide 83
Slide 83 text
83
Duration
Slide 84
Slide 84 text
For 75th percentile and above, Duration
increased in Feb, the increase accelerated
in March, decreased in April, and increased
again in August to longest Duration
84
Slide 85
Slide 85 text
Hypothesis: more tests were written
in March, driving up Duration. In April,
a concentrated effort on optimization
85
Slide 86
Slide 86 text
86
Success Rate
Slide 87
Slide 87 text
Success Rates were the
highest on record in April 2020
87
Slide 88
Slide 88 text
Hypothesis: people working
hard on core business stability
88
Slide 89
Slide 89 text
89
Recovery Time
Slide 90
Slide 90 text
Since April, Recovery
Time has been improving
90
Slide 91
Slide 91 text
Orgs with the longest Recovery
Times (75th percentile and above)
have improved significantly
91
Slide 92
Slide 92 text
Hypothesis: Fewer
distractions* working at home
92
*For some values of distraction.
Slide 93
Slide 93 text
93
Final Thoughts
Slide 94
Slide 94 text
When mapped against survey surveying
data, CI users at 50p show up between
medium and high performers at an org level
(vs project level).
94
Slide 95
Slide 95 text
95
Slide 96
Slide 96 text
96
Slide 97
Slide 97 text
If you are average at using a CI
platform, you’ll be right on the line
between medium and high performer.
97
Slide 98
Slide 98 text
Our most frequent CI users have
better outcomes on our four
critical metrics
98
Slide 99
Slide 99 text
More collaborators
means better outcomes
99
Slide 100
Slide 100 text
100
We’re hiring.
circleci.com/careers
Slide 101
Slide 101 text
101
Thank you
Michael Stahnke @stahnma
Special thanks to Ron Powell and Melissa Santos who gathered
this data and were able to answer my questions about it.