DevOps From a Different Data Set

What can you learn about DevOps and software delivery practices by looking at data from a platform with more than 300,000 developers, 25,000 organizations and 25+ million builds per month? I wanted to find out. As an author of State of DevOps Report in 2018, I was very interested in this new data set from a large SaaS developer platform. What items pop out from the data? Are they different than what is seen through surveys where responders opt-in to participating vs. being aggregated through platform usage? Additionally, does the data validate assertions from State of DevOps Reports of years past about batch sizes, trunk development, and team size?

I’ll cover a view into anonymized team data to share insights, behaviors, and metrics that help teams build better software, faster from millions of builds, and look into characteristics of success that we can measure through our data. Then I’ll overlay that onto what we’ve found through survey data and point out confirmed actions that improve delivery flow, and open up some counter points to what we’ve seen so far.

Transcript

1. 1 DevOps From a Different Data Set Michael Stahnke VP Platform @stahnma
2. @stahnma
3. @stahnma coverage things I didn’t need to know, but am now happy I do
4. @stahnma coverage things I didn’t need to know, but am now happy I do
5. @stahnma coverage things that don’t seem obvious at first, but then kind of are
6. @stahnma coverage things that don’t seem obvious at first, but then kind of are
7. @stahnma coverage things that make you feel better about your team and their performance
8. @stahnma coverage things that make you feel better about your team and their performance
9. @stahnma coverage things that conventional wisdom says are great, and just don’t get proven out.
10. @stahnma coverage things that conventional wisdom says are great, and just don’t get proven out.
11. @stahnma disclaimer I am not a data scientist
12. @stahnma disclaimer I am not a data scientist
13. @stahnma
14. @stahnma 1.6 million jobs per day
15. @stahnma more than 40,000 orgs
16. @stahnma > 150,000 projects
17. @stahnma > 30 million workflows
18. @stahnma
19. @stahnma 1000x larger than all State of DevOps Surveys
20. @stahnma hard hitting stats the most common length of a repository name is 17.6 characters
21. @stahnma hard hitting stats the most common project name is api, followed by terraform, infrastructure, and web
22. @stahnma hard hitting stats after environmental names (e.g dev, test, stage) the most common branch name is update-readme
23. @stahnma hard hitting stats if you take out GitHub-generated branch names, and environmental names, the most common branch name is fixes
24. @stahnma hard hitting stats 0.01% of workflows have English curse words in the name of the branch
25. @stahnma hard hitting stats 0.01% of workflows have English curse words in the name of the branch chances are, this isn’t you, as it was only 359 orgs, or 0.001% of orgs
26. @stahnma
27. @stahnma
28. @stahnma
30. @stahnma
31. @stahnma
32. @stahnma deployment frequency = how often you kick off a workflow mapping metrics
33. @stahnma mttr = time from red to green mapping metrics
34. @stahnma change failure rate = workflow failure rate mapping metrics
35. @stahnma delivery lead time = workflow duration mapping metrics
36. @stahnma
37. @stahnma HOMO DEVOPTICUS
38. @stahnma deployment frequency = how often you kick off a workflow
39. @stahnma workflow duration minimum: 2.1 seconds
40. @stahnma workflow duration maximum: 3.3 Days
41. @stahnma workflow duration 50th percentile 3min 27 seconds 95th percentile 28 minutes
42. @stahnma workflow duration Min 50p 95p Max 2.1s 3m27s 28m45s 3.3 days
43. 43 a sidebar on deployments
44. @stahnma 25% of workflows have deploy in the name deployment
45. @stahnma 25 number of runs per day per project number of projects building this many
 times per day deployment frequency
46. @stahnma 50p 95p 99p Max workflows per day per project 3 29 74 2488 workflows per org per day 6 85 250 3036 workflows per project on master branch per day 1 13 39 2443 deployment frequency
47. @stahnma 50p 95p 99p Max workflows per day per project 3 29 74 2488 workflows per org per day 6 85 250 3036 workflows per project on master branch per day 1 13 39 2443 deployment frequency
48. @stahnma mttr = time from red to green
49. @stahnma fastest recovery < 1 second mttr
50. @stahnma absolute max: 30 days mttr
51. @stahnma median workflow recovery time 1044 min (17.5 hours) mttr
52. @stahnma median workflow recovery time 1044 min (17.5 hours) mttr
53. @stahnma • fastest recovery < 1 second • absolute max: 30 days • median workflow recovery time 1044 min (17.5 hours) mttr
54. @stahnma Put stuff here
55. @stahnma change failure rate = workflow failure rate
56. @stahnma change failure rate
57. @stahnma change failure rate 27%
58. @stahnma change failure rate on master
59. @stahnma change failure rate on master 18%
60. @stahnma change failure rate on topic branches

61. @stahnma change failure rate on topic branches
 31%
62. 62 a sidebar on failure rate
63. @stahnma they seem antithetical to continuous anything manual approvals
64. @stahnma some workflows contain more than one approval manual approvals
65. @stahnma 3.5% of workflows contain a manual approval manual approvals
66. @stahnma 27% failure rate for all workflows manual approvals
67. @stahnma 3% workflow failure rate manual approvals
68. @stahnma median workflow duration 3 min 27 sec manual approvals
69. @stahnma median workflow duration with an approval 
 14 min 38 sec manual approvals
70. @stahnma scheduled job recovery time is much longer, 24 hours vs 17.5 hours. scheduled jobs
71. @stahnma standard recovery time 17.5 hours scheduled jobs
72. @stahnma scheduled job recovery time 24 hours scheduled jobs
73. 73 another sidebar on failure rates
74. @stahnma Language Fail rate on master Median branches Median users PHP 7.0% 5 2 JavaScript 9.5% 5 2 Python 10.6% 5 2 Ruby 10.9% 9 3 Go 12.4% 4 2 Java 13.1% 4 2
75. @stahnma Language Fail rate on master Median branches Median users PHP 7.0% 5 2 JavaScript 9.5% 5 2 Python 10.6% 5 2 Ruby 10.9% 9 3 Go 12.4% 4 2 Java 13.1% 4 2
76. @stahnma Language Fail rate on master Median branches Median users PHP 7.0% 5 2 JavaScript 9.5% 5 2 Python 10.6% 5 2 Ruby 10.9% 9 3 Go 12.4% 4 2 Java 13.1% 4 2
77. @stahnma
78. @stahnma NVS1
79. @stahnma NVS1
80. @stahnma
81. @stahnma Delivery Team Productivity
82. @stahnma master branch stability deployment frequency
83. @stahnma Disclaimer I am not a data scientist
84. @stahnma NVS1 Net Value Score
85. @stahnma NVS1 Velocity with negative NVS Total Builds NVS
86. @stahnma NVS1 Branches Branches
87. @stahnma NVS1 Active Developers NVS How many pizzas can you eat?
88. @stahnma NVS1 Every team has projects they care less about
89. @stahnma NVS1 What do anomalies indicate?
90. @stahnma NVS1 What do anomalies indicate?
91. @stahnma NVS by language
92. conclusions surveys are great, and data is great.
93. conclusions everybody is not doing this better than you.
94. conclusions few organizations look like what we hear in conference talks and see in cases studies and books.
95. conclusions if you’re actively using CI (or at least CircleCI), you’re likely a high medium/ high performer or better
96. conclusions The data show master/trunk based development tends to move faster
97. Summary • Surveys are great, and data is great. • Everybody is not doing this better than you. • Few organizations look like what we hear in conference talks and see in cases studies and books. • If you’re actively using CI, you’re likely a high medium/high performer or better • The data show master/trunk based development tends to move faster • You should be writing PHP
98. @stahnma Thank you.