Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

Read the full paper here: http://gregborenstein.com/assets/order-matters.pdf

Final project for an Interactive Machine Learning course at the MIT Media Lab. Demonstrates the presence of Anchoring Bias, as described by Kahneman and Tversky, in sequential ratings interaction common to recommender systems like Netflix.

Transcript

1. Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems Greg Borenstein MAS.S62: Interactive Machine Learning
2. When “people assess the probability of an uncertain event or the value of an uncertain quantity” they “rely on a limited number of heuristic principles” that “sometimes lead to severe and systematic errors” Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Cognitive Bias
3. When “people assess the probability of an uncertain event or the value of an uncertain quantity” they “rely on a limited number of heuristic principles” that “sometimes lead to severe and systematic errors” Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Cognitive Bias
4. initial value + insufficient adjustment bias towards the initial values Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Anchoring Bias
5. Percentage of African Countries in the UN? Image courtesy of Wikimedia Foundation.
6. Netflix rating interface circa December 10, 2013
8. Evidence of anchoring bias
9. Evidence of anchoring bias Method for mitigating it
10. Evidence of anchoring bias
11. MovieLens rating interface circa December 10, 2013
13. Difference in frequency of pair occurrences
15. P(r , r ) = P(r ) * P(r ) 1 2 1 2
16. Ratings organized chronologically by user
17. Ratings organized chronologically by user
23. Evidence of anchoring bias: compare classifiers
24. Limit to ratings in user sequences ~44,000
25. Order-Ignorant Constant prediction that minimizes mean squared error
26. P(R=i ) * (r-i) i = 1 5 2 Σ argmin r Order-Ignorant Constant prediction that minimizes mean squared error
29. Order-Aware Treat previous ratings as labels
30. Order-Aware Adjusted(R ) = Original(R ) + k*W(R ) n n Treat previous ratings as labels n-1
31. Order-Aware Treat previous ratings as labels Adjusted(R ) = Original(R ) + k*W(R , dt?) n-1 n n
32. r - r Order-Aware Treat previous ratings as labels W(r ) = 1/n * i-1 i = 1 n Σ i i
33. Evaluation Mean Square Error Order-Ignorant: 1.4927 Order-Aware: 1.4137 k=0.4
34. Evaluation Better predictions k=0.4 323/506 63.8%
36. Evidence of anchoring bias
37. Evidence of anchoring bias Method for mitigating it
38. Adjusted(R ) = Original(R ) + k*W(R ) n n n-1
39. Experiment Design (waiting on COUHES)
42. Experiment Design challenges
45. Next steps integrate MovieLens recommendations
46. Next steps integrate MovieLens recommendations Adjusted(R ) = Original(R ) + k*W(R , dt?) n-1 n n
48. Thanks. Greg Borenstein MAS.S62: Interactive Machine Learning