How do you know your DevEx Efforts are working?

Investing in developer experience as part of your larger platform initiatives is a wise move. But how do you know if your investments are paying off? Traditional DevOps metrics are a good start, but they may not be enough to measure DevEx. To get a complete picture of DevEx, you need to look at it from different levels - team, area, and aggregate. In this talk, Michael Stahnke, co-author of the State of DevOps Report from 2018-2021, will share engineering metrics that truly work and help you determine whether your DevEx program is delivering a positive return on investment. We'll delve into metrics that work for individuals, teams, and departments. The insights shared in this talk are based on experience running sizable sections of engineering at Puppet, CircleCI, and Flox. The metrics and measurements discussed are designed to help any leader or champion of a DevEx program represent the outcomes effectively. While some of the metrics are easy to capture and measure, others require serious investment. Nonetheless, we'll cover them all. If you've been relying on DORA metrics for a while, it's high time you take the next step and explore what's next for measuring engineering effectiveness.

Transcript

1. How do you know your DevEx Efforts are working? Michael Stahnke VP Engineering at Flox (flox.dev) @stahnma
2. Edu Career 2 9 Years 8 Years 4 Years About a Year
3. 3 @stahnma
4. 4 1. … @stahnma
5. @stahnma
6. @stahnma
7. @stahnma
8. We are here. @stahnma
9. Availability Cost for Downtime COGS Cost for Services in support of Gross Margin Security Attach Rate and dollar value for compliance docs and security questionnaires Developer Productivity Hiring Tradeoffs, time to value, throughput Developer Sentiment Developer Adoption @stahnma
10. Developer Productivity Hiring Tradeoffs, time to value, throughput @stahnma
11. We are here. @stahnma
12. Do I believe this? 12 @stahnma
13. @stahnma
14. @stahnma
15. We are here. @stahnma
16. I don’t feel elite. 16 @stahnma
17. Why don’t I feel elite? 17 @stahnma
18. 18 What truly is elite? @stahnma
19. Moving fast with confidence. 19 @stahnma
21. What are the categories of things I care about? 21 @stahnma
22. Is engineering throughput good? 22 @stahnma
23. Is that throughput meaningful? 23 @stahnma
24. Is that throughput meaningful? 24 @stahnma Exec Team Cares about this
25. If I add more engineers, will we go faster? Can I prove that? 25 @stahnma
26. If I add more engineers, will we go faster? Can I prove that? 26 @stahnma CFO needs this
27. 27 Metric Thoughts Layer Cake Quality People Throughput @stahnma
28. 28 Throughput Measurements @stahnma
29. deployments / week 29 @stahnma
30. Does adding more developers make things go faster? 30 @stahnma
31. deploys / dev / week 31 @stahnma
32. deploys / dev / week 32 @stahnma
33. 33 1. … @stahnma
34. narrow vs wide work 34 @stahnma
35. 35 1. … @stahnma
36. 36 People @stahnma
37. Are our systems easy to work on? 37 @stahnma
38. 38 @stahnma
39. fungible engineering? 39 @stahnma
40. more ^ fungible engineering? 40 @stahnma
41. 41 @stahnma
42. normalization 42 @stahnma
43. normalization 43 @stahnma
44. % of code provided by the platform. 44 @stahnma
45. 45 @stahnma
46. How quickly are they productive? 46 @stahnma
47. 5th contribution (PR) 47 @stahnma
48. deploys / dev / week 48 @stahnma
49. 49 Quality @stahnma
50. successful deployments / week 50 @stahnma
51. rollbacks (or roll forwards) 51 @stahnma
52. understanding changes happening all around you 52 @stahnma
53. 53 @stahnma
54. 54 Metric Thoughts Layer Cake Quality People Throughput @stahnma
55. 55 Mental Model for Metrics Baseline DevOps Metrics I don’t feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions @stahnma People Metric Metric Metric Themes
56. 56 Mental Model for Metrics Baseline DevOps Metrics I don’t feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions @stahnma People Metric Metric Metric Themes
57. 57 Mental Model for Metrics Baseline DevOps Metrics I don’t feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions @stahnma People Metric Metric Metric Themes
58. Availability Cost for Downtime COGS Cost for Services in support of Gross Margin Security Attach Rate and dollar value for compliance docs and security questionnaires Developer Productivity Hiring Tradeoffs, time to value, throughput Developer Sentiment Developer Adoption @stahnma
59. Summary 59 @stahnma What do you want your metrics to tell you? Can you correlate the movement with intentional action and improved outcomes? Do you have baseline metrics already? How have you joined these up with product management? Deploys / Dev / Week is the first metric I would start with after traditional DORA/DevOps metric sets
60. 60 Twitter’s Engineering Effectiveness Let 1,000 Flowers Bloom - http:// www.gigamonkeys.com/flowers/ The SPACE of developer Productivity. Research paper from Nicole Forsgren, GitHub Margaret-Anne Storey, University of Victoria Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler, Microsoft Research https://queue.acm.org/detail.cfm?id=3454124 GitLab MR Rate https://about.gitlab.com/handbook/engineering/ merge-request-rate/ @stahnma
61. flox.dev