Don't fall in the metrics trap (J-Fall)

Many of us have experience with organisations where teams are assessed based on the number of story points or stories they complete. We also know what usually happens next: teams focus on delivering the “right” number of points to avoid scrutiny from management. This approach diminishes the metric’s value. When people optimise for the measurement rather than the underlying goal, the metric ceases to provide useful insights.

In this talk, we will provide insight into how to measure a value without rendering it meaningless. We will also share practical approaches for using metrics as a useful mechanism. Instead of steering teams toward higher numbers, we show how measurements can help start conversations, reveal bottlenecks, and support your hypotheses for improving your way of working.

Join us to learn how leaders can guide actual improvements rather than dashboards that only seem to suggest improvement.

1. z Maarten Mulders | Laura Kester Don’t fall in the metrics trap
2. z A typical survey at work… § How many of you are using AI to generate (parts of) their code? § Roughly how many tokens did you burn last week? (one week later) § Roughly how many tokens did you burn last week? (wow, wait a minute…) What just happened?!
3. z Goodhart’s Law § “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” “Problems of Monetary Management: The UK Experience” (Charles Goodhart, 1975) § Simplified: “When a measure becomes a target, it ceases to be a good measure.” “‘Improving ratings’: audit in the British University system” (Marilyn Strathern, 1997) James Franklin Gresham, CC BY-SA 3.0 Howardcorn33 (Wikimedia) CC BY-SA 3.0
4. z Goodhart’s Law in practice (1)
5. z Goodhart’s Law in practice (2)
6. z In IT, “we” are great at measuring things!
7. z Story point inflation Last year = Now =
8. z Estimations and planning become political
9. z Teams become careful
10. z Trust will erode
11. z The watermelon effect Watermelon by: Steven Long
12. z Diagnosis: an organisation problem
13. z So, we can’t measure anything?!
14. z Let’s talk about goals § Where you want to go § Product vision § Milestones
15. z Metrics are useful information, not a goal § Informs about direction § Trends over time § Often expressed in numbers
16. z Check if your metric use is problematic 1. Is your metric linked to a (business) goal? 2. Are you expected to improve this metric for its own sake? 3. Can you gain useful information from this metric as a team, even if nobody else is looking at it?
17. z The 6 cures
18. z #1: metrics as hypothesis test
19. z #2: penalise gaming
20. z #3: Favour trends and variation
21. z #4: Collaboration
22. z #5: Expand your view
23. z #6: Focus on the story
24. z #6: Focus on the story
25. z #6: Focus on the story
26. z #6: Focus on the story
27. z Let’s treat this as a typical retrospective… What should we do differently regarding metrics?
28. z Stop Using metrics to compare teams or people Relying on single metrics for complex systems
29. z Start Use metrics to support a hypothesis Looking at the bigger picture
30. z Continue Metrics as a conversation starter Metrics to inform your decisionmaking
31. z Call to action Pick one measure you know you’re gaming. Turn it into a conversation starter in a sprint review or retrospective.
32. z That’s a wrap! Architect, Trainer, Speaker /in/mthmulders @mthmulders https://maarten.mulders.it/ Product owner, UX designer, Speaker /in/laura-kester-lmk Plea s e ra the te our s Dev mio ession i n app !