What you measure is what you get. Stories of metrics gone wrong

Situational awareness is vital in any situation. If you are unaware of an issue, then you are in a poor position to fix it. However, there is a natural human response to the publication of metrics, which is to optimise for what is measured. Here are some cautionary tales of how performance metrics inadvertently created incentives for behaviour that was bad for the system.

Transcript

1. This Summer, don't miss: Metrics Gone Bad Featuring: Dave Neary dneary@redhat.com
2. How things go bad • Metrics should result in action • Metrics create incentives • Sometimes, metrics create destructive incentives • People are weird
3. Setting the wrong incentives
5. Measuring the wrong things
6. Myopia
7. Drawing the wrong conclusions
8. Conclusion • Be careful what you measure – metrics create incentives • Focus on business and community success measurements • Use qualitative and quantitative analysis to interpret metrics