Metrics: The Good the Bad and the Ugly

Sharing business metrics can be a great enabler of collaboration between product, commercial, development and operations. They provide a common reference point and help sort fact from opinion. They inform our decisions and prompt useful conversations.

But are they always all they seem; and when do they stop being helpful and start causing harm?

In this session we will look at examples of metrics from development and operations and use them to explore issues of provenance, cleanliness and bias. We’ll look at how the sharing of metrics can have a different impact on different audiences.

We’ll share takeaways on how to establish and grow a metrics practice that is helpful to everyone in the business.

Chris is the co-author, with Mattia Battiston of Team Guide to Metrics for Business Decisions (sample chapter available at bizmetricsbook.com). He has been leading engineering teams since 1999. He was pivotal in the BBC’s adoption of agile in 2000-2001. Since then, he has worked for broadcasters and video platform owners including Dish Network in the US and Deutsche Telekom in Europe.

Transcript

1. Metrics The good, the bad and the ugly
3. Imagined Realities “We cooperate effectively with strangers because we believe in things like gods, nations, money and human rights. Yet none of these things exists outside the stories that people invent and tell one another.” Yuval Noah Harari https://www.ynharari.com/topic/power-and-imagination/ @worldofchris
4. Stories @worldofchris
5. Reality Abstracts from Impacts Stories @worldofchris
6. Reality Abstracts from Impacts Stories Quality Value @worldofchris
7. Part One: Quality
8. What do we mean by quality?
9. “Quality is Conformance to Requirements” Philip B. Crosby @worldofchris
10. So what are the requirements?
11. What’s the context?
12. Dev Ops Generic Model @worldofchris
13. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model @worldofchris
14. Dev Generic Model Toyota Lean Metrics • Throughput • Lead Time • Value / Failure Demand @worldofchris
15. Throughput @worldofchris
16. Lead Time Story Lead Time @worldofchris
17. Value vs Failure Demand @worldofchris
18. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model @worldofchris
19. Ops Generic Model IBM RAS Model • Reliability • Availability • Serviceability @worldofchris
20. Reliability - Defect discovery rate @worldofchris
21. Reliability - Number of test cases @worldofchris
22. Availability - How does it change under load? @worldofchris
23. Availability - How does it change with load? @worldofchris
24. Availability - When is it needed? @worldofchris
25. Serviceability - Who can fix it? HT @PapaChrisMatts for Liquidity Matrix @worldofchris
26. Let me tell you a story
27. Once upon a time there was a SaaS platform... Platform @worldofchris
28. Platform Admin Input Output Search And that platform had four services... @worldofchris
29. Platform Admin Input Output Search But one of these services had problems. @worldofchris
30. The Business Decision
31. Admin Input Output Search Sales wanted to add more features... Engineering wanted to fix the problems they had inherited @worldofchris
32. Making the case - Operational Metrics
33. @worldofchris
34. Reliability - Cumulative Defect discovery @worldofchris
35. Availability Search latency in seconds
36. Serviceability
37. Making the case - Development Metrics
38. Throughput @worldofchris
39. So how did the story end?
40. @worldofchris
41. @worldofchris
42. Deployed Backlog Open Do Next In Progress QA Queue QA Deployment Queue @worldofchris
43. Where does all this data come from?
44. Dev Ops Generic Model Toyota Lean Metrics IBM RAS Model @worldofchris
45. Quality of Measurement Practice Metrics Practice Dev Ops
46. Cleanliness
48. @worldofchris
49. Selection
50. @worldofchris
51. @worldofchris
52. @lisacrispin @janetgregoryca @worldofchris
53. Part Two: Value
54. Reality Abstracts from Impacts Stories Quality Value @worldofchris
55. Quality of Measurement Practice Metrics Practice Dev Ops @worldofchris
56. Business Value Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
57. “Quality is Conformance to Requirements” Philip B. Crosby @worldofchris
58. Value + “Quality is Conformance to Requirements” Philip B. Crosby @worldofchris
59. Business Need Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
60. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
61. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL @davemcclure Pirate Metrics https://500hats.typepad.com/500blogs/2007/09/startup-metrics.html @worldofchris
62. Let me tell you a another story
63. @worldofchris
64. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
65. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL GOAL: Higher revenue through retention of trial customer as a gym member @worldofchris
66. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
67. Lead Time @worldofchris
68. Throughput @worldofchris
69. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
70. Test cases @worldofchris
72. ACQUISITION ACTIVATION REVENUE RETENTION REFERRAL GOAL: Higher revenue through retention of trial customer as a gym member BONUS: Repeat Business for the agency @worldofchris
73. Business Need AARRR! - Pirate Metrics Metrics Practice Cleanliness, Bias, Selection Dev Throughput, Lead Time, Failure Demand Ops Reliability, Availability, Serviceability @worldofchris
74. Imagined Realities “We cooperate effectively with strangers because we believe in things like gods, nations, money and human rights. Yet none of these things exists outside the stories that people invent and tell one another.” Yuval Noah Harari https://www.ynharari.com/topic/power-and-imagination/ @worldofchris
75. ― https://www.goodreads.com/photo/author/3472.Margaret_Atwood @worldofchris