Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

Read the full paper here: http://gregborenstein.com/assets/order-matters.pdf

Final project for an Interactive Machine Learning course at the MIT Media Lab. Demonstrates the presence of Anchoring Bias, as described by Kahneman and Tversky, in sequential ratings interaction common to recommender systems like Netflix.

Transcript

1. Order Matters:
   Accounting for Anchoring Bias on User Labeling
   in Recommendation Systems
   Greg Borenstein
   MAS.S62: Interactive Machine Learning
   

   View Slide

2. When “people assess the probability
   of an uncertain event or the value of an uncertain quantity”
   they “rely on a limited number of heuristic principles”
   that “sometimes lead to severe and systematic errors”
   Kahneman and Tversky
   Judgement Under Uncertainty: Heuristics and Biases, 1974
   Cognitive Bias
   

   View Slide

3. When “people assess the probability
   of an uncertain event or the value of an uncertain quantity”
   they “rely on a limited number of heuristic principles”
   that “sometimes lead to severe and systematic errors”
   Kahneman and Tversky
   Judgement Under Uncertainty: Heuristics and Biases, 1974
   Cognitive Bias
   

   View Slide

4. initial value
   + insufficient adjustment
   bias towards the initial values
   Kahneman and Tversky
   Judgement Under Uncertainty: Heuristics and Biases, 1974
   Anchoring Bias
   

   View Slide

5. Percentage of African Countries in the UN?
   Image courtesy of Wikimedia Foundation.
   

   View Slide

6. Netflix rating interface circa December 10, 2013
   

   View Slide

7. View Slide

8. Evidence of anchoring bias
   

   View Slide

9. Evidence of anchoring bias
   Method for mitigating it
   

   View Slide

10. Evidence of anchoring bias
    

    View Slide

11. MovieLens rating interface circa December 10, 2013
    

    View Slide

12. View Slide

13. Difference in frequency of pair occurrences
    

    View Slide

14. View Slide

15. P(r , r ) = P(r ) * P(r )
    1 2 1 2
    

    View Slide

16. Ratings organized chronologically by user
    

    View Slide

17. Ratings organized chronologically by user
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. Evidence of anchoring bias:
    compare classifiers
    

    View Slide

24. Limit to ratings in user sequences
    ~44,000
    

    View Slide

25. Order-Ignorant
    Constant prediction that
    minimizes mean squared error
    

    View Slide

26. P(R=i ) * (r-i)
    i = 1
    5
    2
    Σ
    argmin
    r
    Order-Ignorant
    Constant prediction that
    minimizes mean squared error
    

    View Slide

27. View Slide

28. View Slide

29. Order-Aware
    Treat previous ratings as labels
    

    View Slide

30. Order-Aware
    Adjusted(R ) = Original(R ) + k*W(R )
    n n
    Treat previous ratings as labels
    n-1
    

    View Slide

31. Order-Aware
    Treat previous ratings as labels
    Adjusted(R ) = Original(R ) + k*W(R , dt?)
    n-1
    n n
    

    View Slide

32. r - r
    Order-Aware
    Treat previous ratings as labels
    W(r ) = 1/n *
    i-1
    i = 1
    n
    Σ i
    i
    

    View Slide

33. Evaluation
    Mean Square Error
    Order-Ignorant: 1.4927
    Order-Aware: 1.4137
    k=0.4
    

    View Slide

34. Evaluation
    Better predictions
    k=0.4
    323/506
    63.8%
    

    View Slide

35. View Slide

36. Evidence of anchoring bias
    

    View Slide

37. Evidence of anchoring bias
    Method for mitigating it
    

    View Slide

38. Adjusted(R ) = Original(R ) + k*W(R )
    n n n-1
    

    View Slide

39. Experiment Design
    (waiting on COUHES)
    

    View Slide

40. View Slide

41. View Slide

42. Experiment Design
    challenges
    

    View Slide

43. View Slide

44. View Slide

45. Next steps
    integrate MovieLens recommendations
    

    View Slide

46. Next steps
    integrate MovieLens recommendations
    Adjusted(R ) = Original(R ) + k*W(R , dt?)
    n-1
    n n
    

    View Slide

47. View Slide

48. Thanks.
    Greg Borenstein
    MAS.S62: Interactive Machine Learning
    

    View Slide