Slide 1

Slide 1 text

1 DevOps from a Different Data Set What 30 million workflows reveal about high performing teams Michael Stahnke VP Platform Engineering @stahnma

Slide 2

Slide 2 text

2

Slide 3

Slide 3 text

3 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.

Slide 4

Slide 4 text

4 The Setup What are you talking about and how does this work?

Slide 5

Slide 5 text

5

Slide 6

Slide 6 text

6

Slide 7

Slide 7 text

Performance derived vs performance described 7

Slide 8

Slide 8 text

44,000 orgs 8

Slide 9

Slide 9 text

160,000 projects 9

Slide 10

Slide 10 text

1000x larger than all State of DevOps Surveys 10

Slide 11

Slide 11 text

What’s changed year over year? 11

Slide 12

Slide 12 text

12 Year 2019 2020 Days in set 30 30 Orgs >40,000 >44,000 Projects >150,000 >160,000 Second Year for Analysis

Slide 13

Slide 13 text

13

Slide 14

Slide 14 text

14 State of DevOps Report Metrics Deployment Frequency Lead time to Change Change Failure Rate MTTR Description when mapping to CI How often you initiate a pipeline Pipeline duration Pipeline failure rate Time from red to green Metric Mapping Metrics Throughput Duration Success Rate Recovery Time

Slide 15

Slide 15 text

15 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.

Slide 16

Slide 16 text

16 The Data Here’s what the data tell us.

Slide 17

Slide 17 text

17 Throughput

Slide 18

Slide 18 text

How often do you push code that triggers CI? 18

Slide 19

Slide 19 text

Most projects configured to run per push to git server 19

Slide 20

Slide 20 text

20 Percentile 2020 Value 5p 0.03 50p 0.70 90p 16.03 95p 32.125 Mean 8.22 Throughput

Slide 21

Slide 21 text

Most projects are not deploying dozens of times per day 21

Slide 22

Slide 22 text

Why is this different from survey data? 22

Slide 23

Slide 23 text

“Primary application or service you work on” 23

Slide 24

Slide 24 text

24 Percentile 2020 Value 2019 Value 5p 0.03 0.03 50p 0.70 0.80 90p 16.03 13.00 95p 32.125 25.47 Mean 8.22 5.76 Throughput

Slide 25

Slide 25 text

Those leveraging CI well, are doing so even more 25

Slide 26

Slide 26 text

There are fewer developers worldwide pushing code 26

Slide 27

Slide 27 text

27 Duration

Slide 28

Slide 28 text

How long does it take to get results? 28

Slide 29

Slide 29 text

5% of builds finish in < 12 seconds 29 * That’s roughly 500,000 builds in this sample

Slide 30

Slide 30 text

30 Percentile 2020 Value 5p 12 sec 50p 3.96 min 90p 21.35 min 95p 34.01 min Mean 24.6 min Duration

Slide 31

Slide 31 text

Half of all builds finish in under 4 minutes 31

Slide 32

Slide 32 text

32 Percentile 2020 Value 2019 Value 5p 12 sec 10 sec 50p 3.96 min 3.38 min 90p 21.35 min 19.18 min 95p 34.01 min 31.73 min Mean 24.6 min 26.76 min Duration delta in a year

Slide 33

Slide 33 text

All pipelines are running longer 33

Slide 34

Slide 34 text

34 Percentile 2020 Value 2019 Value 5p 12 sec 10 sec 50p 3.96 min 3.38 min 90p 21.35 min 19.18 min 95p 34.01 min 31.73 min Mean 24.6 min 26.76 min Duration delta in a year

Slide 35

Slide 35 text

35 Success Rate

Slide 36

Slide 36 text

How often does your pipeline complete with a green status? 36

Slide 37

Slide 37 text

37 Percentile 2020 Value 5p 0% 50p 61% 90p 100% 95p 100% Mean 54% Success Rate

Slide 38

Slide 38 text

Some of our sample dabbles with CI, but doesn’t get a working build 38

Slide 39

Slide 39 text

Some of our sample saw no failures within a month 39

Slide 40

Slide 40 text

40 Percentile 2020 Value 2019 Value 5p 0% 0% 50p 61% 60% 90p 100% 100% 95p 100% 100% Mean 54% 54% Success Rate

Slide 41

Slide 41 text

41 Percentile 2020 Value 2019 Value 50p 61% 60% 75p 89% 86% 85p 100% 98% Success Rate

Slide 42

Slide 42 text

42 Recovery Time

Slide 43

Slide 43 text

Time a pipeline sits in a failure state 43

Slide 44

Slide 44 text

44 Percentile 2020 Value 5p 2.06 min 50p 55.11 min 90p 39 hours 95p 3.4 days Mean 14.85 hours Recovery Time

Slide 45

Slide 45 text

Quick Recovery Time can be from multiple contributors running in parallel 45

Slide 46

Slide 46 text

The gap between 50th and 75th percentiles looks like it represents waiting until tomorrow to fix a failed build (from 55 min at 50p to 9.5 hours at 75p) 46

Slide 47

Slide 47 text

47 Percentile 2020 Value 2019 Value 5p 2.06 min 2.83 min 50p 55.11 min 52.5 min 90p 39 hours 47 hours 95p 3.4 days 3.93 days Mean 14.85 hours 16.61 hours Recovery Time

Slide 48

Slide 48 text

Fastest Recovery Times have improved (10th and percentile and lower) year over year 48

Slide 49

Slide 49 text

49 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.

Slide 50

Slide 50 text

50 The Insights Using the data, what can we apply to current industry trends and influences?

Slide 51

Slide 51 text

51 What development practices definitively work?

Slide 52

Slide 52 text

Success Rate does not correlate with company size 52

Slide 53

Slide 53 text

Duration is longest for teams of one 53

Slide 54

Slide 54 text

Recovery Time decreases with increased team size (up to 200) 54

Slide 55

Slide 55 text

Longest Recovery Times are from teams of one. 55

Slide 56

Slide 56 text

Performance is better with more than one contributor as shown by multiple indicators 56

Slide 57

Slide 57 text

Software is collaborative 57

Slide 58

Slide 58 text

58 Is “Don’t Deploy on Friday” a real thing?

Slide 59

Slide 59 text

70% less Throughput on weekends 59

Slide 60

Slide 60 text

11% less Throughput on Friday (UTC). 60

Slide 61

Slide 61 text

9% less Throughput on Monday (UTC). 61

Slide 62

Slide 62 text

Conclusion: About the same amount of work happens Monday or Friday. So people not holding back on pushing code on Fridays. 62

Slide 63

Slide 63 text

63 What Language Trends emerge?

Slide 64

Slide 64 text

64 21.73% JavaScript 2.44% Vue 11.36% TypeScript 2.12% Kotlin 9.56% Python 1.70% HCL 9.04% Ruby 1.59% Swift 6.16% HTML 1.26% C++ 5.37% Java 1.21% Dockerfile 4.92% PHP 1.08% C# 3.89% Go 1.00% TSQL 3.17% CSS 0.96% Jupyter Notebook 2.99% Shell 0.83% Elixir Languages in our sample

Slide 65

Slide 65 text

65 Language Throughput 1 Ruby 11 PHP 2 TypeScript 12 Java 3 Go 13 C# 4 Python 14 Jupyter Notebook 5 Kotlin 15 Shell 6 Elixir 16 Vue 7 Swift 17 C++ 8 HCL 18 HTML 9 JavaScript 19 CSS 10 TSQL 20 Dockerfile

Slide 66

Slide 66 text

66 Language Success Rate at 50p 1 Vue 11 Elixir 2 CSS 12 PHP 3 Shell 13 Jupyter Notebook 4 Dockerfile 14 Python 5 TSQL 15 Ruby 6 HTML 16 Java 7 HCL 17 Kotlin 8 Go 18 C# 9 TypeScript 19 C++ 10 JavaScript 20 Swift

Slide 67

Slide 67 text

67 Language Recovery Time at 50p 1 Go 11 Vue 2 JavaScript 12 Jupyter Notebook 3 Elixir 13 Kotlin 4 HCL 14 Java 5 Shell 15 Scala 6 Python 16 Ruby 7 TypeScript 17 PHP 8 CSS 18 TSQL 9 C# 19 Swift 10 HTML 20 C++

Slide 68

Slide 68 text

68 Language Duration at 50p 1 Shell 11 PHP 2 HCL 12 TypeScript 3 CSS 13 Java 4 HTML 14 Elixir 5 Gherkin 15 TSQL 6 JavaScript 16 Kotlin 7 Vue 17 Scala 8 Go 18 Ruby 9 Jupyter Notebook 19 C++ 10 Python 20 Swift

Slide 69

Slide 69 text

69 Branch Information

Slide 70

Slide 70 text

70

Slide 71

Slide 71 text

Did the use of master branch decrease? 71

Slide 72

Slide 72 text

Not in any significant way….yet. 72

Slide 73

Slide 73 text

Teams are innovating and experimenting on feature branches 73

Slide 74

Slide 74 text

Success Rate on default branch higher than on non-default branches 74

Slide 75

Slide 75 text

Success Rate is 80% on the default branch at 50th percentile and 100% for 75th percentile and above 75

Slide 76

Slide 76 text

Success Rate at 50p is 80% for default and 58% for non-default branches 76

Slide 77

Slide 77 text

Duration on default branches are faster at every percentile. 77

Slide 78

Slide 78 text

Recovery Time is lower on default branch at every percentile. 78

Slide 79

Slide 79 text

79 How has the global pandemic impacted team performance?

Slide 80

Slide 80 text

80 Throughput

Slide 81

Slide 81 text

Peak Throughput was April 2020 81

Slide 82

Slide 82 text

After April, Throughput falls a bit 82

Slide 83

Slide 83 text

83 Duration

Slide 84

Slide 84 text

For 75th percentile and above, Duration increased in Feb, the increase accelerated in March, decreased in April, and increased again in August to longest Duration 84

Slide 85

Slide 85 text

Hypothesis: more tests were written in March, driving up Duration. In April, a concentrated effort on optimization 85

Slide 86

Slide 86 text

86 Success Rate

Slide 87

Slide 87 text

Success Rates were the highest on record in April 2020 87

Slide 88

Slide 88 text

Hypothesis: people working hard on core business stability 88

Slide 89

Slide 89 text

89 Recovery Time

Slide 90

Slide 90 text

Since April, Recovery Time has been improving 90

Slide 91

Slide 91 text

Orgs with the longest Recovery Times (75th percentile and above) have improved significantly 91

Slide 92

Slide 92 text

Hypothesis: Fewer distractions* working at home 92 *For some values of distraction.

Slide 93

Slide 93 text

93 Final Thoughts

Slide 94

Slide 94 text

When mapped against survey surveying data, CI users at 50p show up between medium and high performers at an org level (vs project level). 94

Slide 95

Slide 95 text

95

Slide 96

Slide 96 text

96

Slide 97

Slide 97 text

If you are average at using a CI platform, you’ll be right on the line between medium and high performer. 97

Slide 98

Slide 98 text

Our most frequent CI users have better outcomes on our four critical metrics 98

Slide 99

Slide 99 text

More collaborators means better outcomes 99

Slide 100

Slide 100 text

100 We’re hiring. circleci.com/careers

Slide 101

Slide 101 text

101 Thank you Michael Stahnke @stahnma Special thanks to Ron Powell and Melissa Santos who gathered this data and were able to answer my questions about it.