Your Brain Hates You (and Other Hazards of Metrics)

Transcript

1. Your Brain Hates You And other Hazards of Metrics jeff eaton - camp lullabot ignite talks - 07-19-12
2. Who recognizes this screen? It’s Fitbit! We use it to generate metrics that help us get healthy -- which is awesome! I know I’ve started getting a lot more active now that it’s being measured, because I can see how active I really am, and compare myself to others to get a feel for where I stand.
3. We measure how we’re using our time with Freckle; we measure our web traffic with Google Analytics; we measure how influential we are with Klout... And this goes back more than a hundred years - the science of ‘productivity’ came out of measuring assembly line workers to increase factory output.
4. I Want to canoe, but I won’t get FitBit points! Anyone remember this quote from the Team call? There’s a weird side effect that often emerges when we have a lot of metrics at our disposal. We start altering our behaviors to match the metrics, often in ways that would feel silly if we weren’t measuring things.
5. This plays out in lots of ways in modern culture. “The Information Diet” talks about how SEO and viewer-targeting has turned our news landscape into “junk food.” “The Filter Bubble” talks about how “magical” metrics-driven filtering on sites like Google reinforces our biases rather than improving the information we have.
6. The big problem is the nature of the human brain. We’re wired to respond to metrics and measurements in potentially destructive ways. In this talk we’re going to look at three well-documented “cognitive biases” -- ruts the brain falls into easily -- that can sabotage metrics-driven approaches to problem solving.
7. The first pitfall is the “Hawthorne Effect.” In 1924, Hawthorne Electric conducted a study to figure out how to make their assembly line workers more productive. They used bright lighting one week and dim lighting the next, and measured factory output.
8. ...And the result was ‘both.’ Factory output went up AND STAYED UP when they brightened the lights, stayed up when they dimmed them, and stayed up when they returned to normal. And when the experiment ENDED, output returned to normal levels.
9. we change when we know someone’s Measuring
10. The second pitfall is the “Availability heuristic.” This is Nancy Grace; she’s a TV personality who covers ‘hot’ scandalous stuff. Tropical vacation kidnappings, child abductions, stuff like that. She also yells an awful lot.
11. Turns out that people who watch her show BELIEVE THOSE EVENTS are more common than they really are. Nancy Grace fans are bad judges of risk. Why? Human brains are wired to believe that things we’ve seen, things we have already learned, have more weight than unknown stuff ‘out there.’
12. We assume availability means relevance When we measure stuff, it becomes ‘easily available information.’ We naturally start to assume that it’s relevant and useful BECAUSE we have it.
13. The third cognitive pitfall is “Goodwell’s Law.” In the 1970s, he and lots of other economists were trying to come up with good metrics for a nation’s economic health. It was important because they wanted to weigh different economic strategies and figure out which ones worked better.
14. The problem was that anything they measured turned into something that was gamed. Think of it as the SEO problem, writ large: Google measured ‘inbound links’, but when people realized that they gamed it. In isolation, it turned into a useless thing to measure.
15. When a Measurement becomes a target it becomes less useful
16. Let’s step back a little and look at how these come together. This is Ben Brown; we’ve worked with him on cool stuff! He also built a social dating site called ‘Consumating.’ He wanted interactions on the site to work better.
17. Every time you interacted with someone - emailing with them, chatting, etc - you could rate it. The idea was that people who were fun to interact with would bubble up to the top, and assholes would be shunned. Yay!
18. Oh, but no. Six months later, the site had turned into roving gangs, stalking the ‘most popular’ person and downvoting them to push the SECOND most popular person into the top slot. No shit, they had dating-site voting gangs.
19. hawthorne effect Availability Heuristic Goodwell’s law Members changed their behavior and gamed the system when they realized they were being measured. Other members assumed that the easy-to-access information (someone’s score) was the best measurement. The interesting metric became a target in and of itself, and became decoupled from the underlying values.
20. Know our True goals Question the Metrics Abandon Stale ones
21. The Value that can be measured is not the true Value