Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What you measure is what you get. Stories of metrics gone wrong

FLOSS Community Metrics meeting
July 20, 2014
Technology
2
410

What you measure is what you get. Stories of metrics gone wrong

Situational awareness is vital in any situation. If you are unaware of an issue, then you are in a poor position to fix it. However, there is a natural human response to the publication of metrics, which is to optimise for what is measured. Here are some cautionary tales of how performance metrics inadvertently created incentives for behaviour that was bad for the system.

FLOSS Community Metrics meeting

July 20, 2014
Tweet

More Decks by FLOSS Community Metrics meeting

See All by FLOSS Community Metrics meeting
Can we compare communities or are they all unique snowflakes?
flossmetrics
1
85
ownCloud and Community Metrics
flossmetrics
0
120
Catwatch, Zalando's open-source projects dashboard
flossmetrics
1
130
Cauldron - The metrics behind GitHub repositories
flossmetrics
1
120
HyperKitty - A web interface to access GNU Mailman v3 archives
flossmetrics
0
200
Measuring Velocity With A Teaspoon
flossmetrics
0
68
Towards containerizing non-software initiatives at the FSF
flossmetrics
0
47
Debian contributors, a tool to measure and credit a diverse community
flossmetrics
1
83
Eclipse Community Metrics
flossmetrics
2
140

Other Decks in Technology

See All in Technology
Spring Boot 2 から 3 へバージョンアップしてみた
red_frasco
1
150
時系列データをChatGPTで読み解いてみる【IoT-Tech Meetup】
soracom
PRO
0
800
組織の立ち上げと体制変更の1年
tarappo
2
860
AI完全初心者でも最新の生成AIの仕組がわかった気になれるプレゼン
shimap_sampo
5
1.8k
ABEMAのRenderScript Intrinsics Replacement Toolkit 導入事例
takahana
0
280
javapを使ってクラスファイルを読んでみよう！
yujisoftware
1
190
ジェネレーティブAIとの共創
yuyakakizaki08
1
460
AWS Direct Connectと愉快なGWたちのおさらい
minorun365
4
780
Impressions of the Ruby Kaigi
suzukahr
1
5.3k
Autonomous Database Cloud 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
10
26k
データマイニングと機械学習 - ニューラルネットワーク
trycycle
0
130
全AWSエンジニアに捧ぐ、CloudWatch 設計・運用 虎の巻 / CloudWatch design and operation bible
iselegant
29
11k

Featured

See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
41
8.2k
Atom: Resistance is Futile
akmur
256
24k
StorybookのUI Testing Handbookを読んだ
zakiyama
9
3.5k
Typedesign – Prime Four
hannesfritz
34
1.6k
jQuery: Nuts, Bolts and Bling
dougneiner
57
6.7k
How to train your dragon (web standard)
notwaldorf
69
4.4k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
239
19k
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
13
1.4k
Building a Scalable Design System with Sketch
lauravandoore
451
31k
Gamification - CAS2011
davidbonilla
75
4.2k
The MySQL Ecosystem @ GitHub 2015
samlambert
240
11k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
183
15k

Transcript

  1. This Summer, don't miss:
    Metrics
    Gone
    Bad
    Featuring:
    Dave Neary
    [email protected]

    View Slide

  2. How things go bad
    ●
    Metrics should result in action
    ●
    Metrics create incentives
    ●
    Sometimes, metrics create destructive
    incentives
    ●
    People are weird

    View Slide

  3. Setting the wrong incentives

    View Slide

  4. View Slide

  5. Measuring
    the wrong
    things

    View Slide

  6. Myopia

    View Slide

  7. Drawing the wrong conclusions

    View Slide

  8. Conclusion
    ●
    Be careful what you measure – metrics create
    incentives
    ●
    Focus on business and community success
    measurements
    ●
    Use qualitative and quantitative analysis to
    interpret metrics

    View Slide