DevOps From a Different Data Set

What can we learn about DevOps best practices by looking at data from a SaaS with 900K dev users, 25K orgs and 30m+ builds a month, particularly when compared with surveys where users opt-in? Join CircleCI's VP of Platform, Mike Stahnke, to understand what this unique dataset uncovers about high performing engineering teams: - What trends stand out from the data? - Are they different than what is seen through surveys where responders opt-in to participating as compared to being aggregated through platform usage?

Transcript

1. 1 DevOps from a Different Data Set What 30 million workflows reveal about high performing teams Michael Stahnke VP Platform Engineering @stahnma
3. 3 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.
4. 4 The Setup What are you talking about and how does this work?
7. Performance derived vs performance described 7
8. 44,000 orgs 8
9. 160,000 projects 9
10. 1000x larger than all State of DevOps Surveys 10
11. What’s changed year over year? 11
12. 12 Year 2019 2020 Days in set 30 30 Orgs >40,000 >44,000 Projects >150,000 >160,000 Second Year for Analysis
14. 14 State of DevOps Report Metrics Deployment Frequency Lead time to Change Change Failure Rate MTTR Description when mapping to CI How often you initiate a pipeline Pipeline duration Pipeline failure rate Time from red to green Metric Mapping Metrics Throughput Duration Success Rate Recovery Time
15. 15 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.
16. 16 The Data Here’s what the data tell us.
17. 17 Throughput
18. How often do you push code that triggers CI? 18
19. Most projects configured to run per push to git server 19
20. 20 Percentile 2020 Value 5p 0.03 50p 0.70 90p 16.03 95p 32.125 Mean 8.22 Throughput
21. Most projects are not deploying dozens of times per day 21
22. Why is this different from survey data? 22
23. “Primary application or service you work on” 23
24. 24 Percentile 2020 Value 2019 Value 5p 0.03 0.03 50p 0.70 0.80 90p 16.03 13.00 95p 32.125 25.47 Mean 8.22 5.76 Throughput
25. Those leveraging CI well, are doing so even more 25
26. There are fewer developers worldwide pushing code 26
27. 27 Duration
28. How long does it take to get results? 28
29. 5% of builds finish in < 12 seconds 29 * That’s roughly 500,000 builds in this sample
30. 30 Percentile 2020 Value 5p 12 sec 50p 3.96 min 90p 21.35 min 95p 34.01 min Mean 24.6 min Duration
31. Half of all builds finish in under 4 minutes 31
32. 32 Percentile 2020 Value 2019 Value 5p 12 sec 10 sec 50p 3.96 min 3.38 min 90p 21.35 min 19.18 min 95p 34.01 min 31.73 min Mean 24.6 min 26.76 min Duration delta in a year
33. All pipelines are running longer 33
34. 34 Percentile 2020 Value 2019 Value 5p 12 sec 10 sec 50p 3.96 min 3.38 min 90p 21.35 min 19.18 min 95p 34.01 min 31.73 min Mean 24.6 min 26.76 min Duration delta in a year
35. 35 Success Rate
36. How often does your pipeline complete with a green status? 36
37. 37 Percentile 2020 Value 5p 0% 50p 61% 90p 100% 95p 100% Mean 54% Success Rate
38. Some of our sample dabbles with CI, but doesn’t get a working build 38
39. Some of our sample saw no failures within a month 39
40. 40 Percentile 2020 Value 2019 Value 5p 0% 0% 50p 61% 60% 90p 100% 100% 95p 100% 100% Mean 54% 54% Success Rate
41. 41 Percentile 2020 Value 2019 Value 50p 61% 60% 75p 89% 86% 85p 100% 98% Success Rate
42. 42 Recovery Time
43. Time a pipeline sits in a failure state 43
44. 44 Percentile 2020 Value 5p 2.06 min 50p 55.11 min 90p 39 hours 95p 3.4 days Mean 14.85 hours Recovery Time
45. Quick Recovery Time can be from multiple contributors running in parallel 45
46. The gap between 50th and 75th percentiles looks like it represents waiting until tomorrow to fix a failed build (from 55 min at 50p to 9.5 hours at 75p) 46
47. 47 Percentile 2020 Value 2019 Value 5p 2.06 min 2.83 min 50p 55.11 min 52.5 min 90p 39 hours 47 hours 95p 3.4 days 3.93 days Mean 14.85 hours 16.61 hours Recovery Time
48. Fastest Recovery Times have improved (10th and percentile and lower) year over year 48
49. 49 The Setup What are you talking about and how does this work? The Insights Using the data, what can we apply to current industry trends and influences? The Data Here’s what the data tell us.
50. 50 The Insights Using the data, what can we apply to current industry trends and influences?
51. 51 What development practices definitively work?
52. Success Rate does not correlate with company size 52
53. Duration is longest for teams of one 53
54. Recovery Time decreases with increased team size (up to 200) 54
55. Longest Recovery Times are from teams of one. 55
56. Performance is better with more than one contributor as shown by multiple indicators 56
57. Software is collaborative 57
58. 58 Is “Don’t Deploy on Friday” a real thing?
59. 70% less Throughput on weekends 59
60. 11% less Throughput on Friday (UTC). 60
61. 9% less Throughput on Monday (UTC). 61
62. Conclusion: About the same amount of work happens Monday or Friday. So people not holding back on pushing code on Fridays. 62
63. 63 What Language Trends emerge?
64. 64 21.73% JavaScript 2.44% Vue 11.36% TypeScript 2.12% Kotlin 9.56% Python 1.70% HCL 9.04% Ruby 1.59% Swift 6.16% HTML 1.26% C++ 5.37% Java 1.21% Dockerfile 4.92% PHP 1.08% C# 3.89% Go 1.00% TSQL 3.17% CSS 0.96% Jupyter Notebook 2.99% Shell 0.83% Elixir Languages in our sample
65. 65 Language Throughput 1 Ruby 11 PHP 2 TypeScript 12 Java 3 Go 13 C# 4 Python 14 Jupyter Notebook 5 Kotlin 15 Shell 6 Elixir 16 Vue 7 Swift 17 C++ 8 HCL 18 HTML 9 JavaScript 19 CSS 10 TSQL 20 Dockerfile
66. 66 Language Success Rate at 50p 1 Vue 11 Elixir 2 CSS 12 PHP 3 Shell 13 Jupyter Notebook 4 Dockerfile 14 Python 5 TSQL 15 Ruby 6 HTML 16 Java 7 HCL 17 Kotlin 8 Go 18 C# 9 TypeScript 19 C++ 10 JavaScript 20 Swift
67. 67 Language Recovery Time at 50p 1 Go 11 Vue 2 JavaScript 12 Jupyter Notebook 3 Elixir 13 Kotlin 4 HCL 14 Java 5 Shell 15 Scala 6 Python 16 Ruby 7 TypeScript 17 PHP 8 CSS 18 TSQL 9 C# 19 Swift 10 HTML 20 C++
68. 68 Language Duration at 50p 1 Shell 11 PHP 2 HCL 12 TypeScript 3 CSS 13 Java 4 HTML 14 Elixir 5 Gherkin 15 TSQL 6 JavaScript 16 Kotlin 7 Vue 17 Scala 8 Go 18 Ruby 9 Jupyter Notebook 19 C++ 10 Python 20 Swift
69. 69 Branch Information
71. Did the use of master branch decrease? 71
72. Not in any significant way….yet. 72
73. Teams are innovating and experimenting on feature branches 73
74. Success Rate on default branch higher than on non-default branches 74
75. Success Rate is 80% on the default branch at 50th percentile and 100% for 75th percentile and above 75
76. Success Rate at 50p is 80% for default and 58% for non-default branches 76
77. Duration on default branches are faster at every percentile. 77
78. Recovery Time is lower on default branch at every percentile. 78
79. 79 How has the global pandemic impacted team performance?
80. 80 Throughput
81. Peak Throughput was April 2020 81
82. After April, Throughput falls a bit 82
83. 83 Duration
84. For 75th percentile and above, Duration increased in Feb, the increase accelerated in March, decreased in April, and increased again in August to longest Duration 84
85. Hypothesis: more tests were written in March, driving up Duration. In April, a concentrated effort on optimization 85
86. 86 Success Rate
87. Success Rates were the highest on record in April 2020 87
88. Hypothesis: people working hard on core business stability 88
89. 89 Recovery Time
90. Since April, Recovery Time has been improving 90
91. Orgs with the longest Recovery Times (75th percentile and above) have improved significantly 91
92. Hypothesis: Fewer distractions* working at home 92 *For some values of distraction.
93. 93 Final Thoughts
94. When mapped against survey surveying data, CI users at 50p show up between medium and high performers at an org level (vs project level). 94
97. If you are average at using a CI platform, you’ll be right on the line between medium and high performer. 97
98. Our most frequent CI users have better outcomes on our four critical metrics 98
99. More collaborators means better outcomes 99
100. 100 We’re hiring. circleci.com/careers
101. 101 Thank you Michael Stahnke @stahnma Special thanks to Ron Powell and Melissa Santos who gathered this data and were able to answer my questions about it.