$30 off During Our Annual Pro Sale. View Details »

Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

gregab
December 13, 2013

Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

Read the full paper here: http://gregborenstein.com/assets/order-matters.pdf

Final project for an Interactive Machine Learning course at the MIT Media Lab. Demonstrates the presence of Anchoring Bias, as described by Kahneman and Tversky, in sequential ratings interaction common to recommender systems like Netflix.

gregab

December 13, 2013
Tweet

More Decks by gregab

Other Decks in Research

Transcript

  1. Order Matters:
    Accounting for Anchoring Bias on User Labeling
    in Recommendation Systems
    Greg Borenstein
    MAS.S62: Interactive Machine Learning

    View Slide

  2. When “people assess the probability
    of an uncertain event or the value of an uncertain quantity”
    they “rely on a limited number of heuristic principles”
    that “sometimes lead to severe and systematic errors”
    Kahneman and Tversky
    Judgement Under Uncertainty: Heuristics and Biases, 1974
    Cognitive Bias

    View Slide

  3. When “people assess the probability
    of an uncertain event or the value of an uncertain quantity”
    they “rely on a limited number of heuristic principles”
    that “sometimes lead to severe and systematic errors”
    Kahneman and Tversky
    Judgement Under Uncertainty: Heuristics and Biases, 1974
    Cognitive Bias

    View Slide

  4. initial value
    + insufficient adjustment
    bias towards the initial values
    Kahneman and Tversky
    Judgement Under Uncertainty: Heuristics and Biases, 1974
    Anchoring Bias

    View Slide

  5. Percentage of African Countries in the UN?
    Image courtesy of Wikimedia Foundation.

    View Slide

  6. Netflix rating interface circa December 10, 2013

    View Slide

  7. View Slide

  8. Evidence of anchoring bias

    View Slide

  9. Evidence of anchoring bias
    Method for mitigating it

    View Slide

  10. Evidence of anchoring bias

    View Slide

  11. MovieLens rating interface circa December 10, 2013

    View Slide

  12. View Slide

  13. Difference in frequency of pair occurrences

    View Slide

  14. View Slide

  15. P(r , r ) = P(r ) * P(r )
    1 2 1 2

    View Slide

  16. Ratings organized chronologically by user

    View Slide

  17. Ratings organized chronologically by user

    View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. Evidence of anchoring bias:
    compare classifiers

    View Slide

  24. Limit to ratings in user sequences
    ~44,000

    View Slide

  25. Order-Ignorant
    Constant prediction that
    minimizes mean squared error

    View Slide

  26. P(R=i ) * (r-i)
    i = 1
    5
    2
    Σ
    argmin
    r
    Order-Ignorant
    Constant prediction that
    minimizes mean squared error

    View Slide

  27. View Slide

  28. View Slide

  29. Order-Aware
    Treat previous ratings as labels

    View Slide

  30. Order-Aware
    Adjusted(R ) = Original(R ) + k*W(R )
    n n
    Treat previous ratings as labels
    n-1

    View Slide

  31. Order-Aware
    Treat previous ratings as labels
    Adjusted(R ) = Original(R ) + k*W(R , dt?)
    n-1
    n n

    View Slide

  32. r - r
    Order-Aware
    Treat previous ratings as labels
    W(r ) = 1/n *
    i-1
    i = 1
    n
    Σ i
    i

    View Slide

  33. Evaluation
    Mean Square Error
    Order-Ignorant: 1.4927
    Order-Aware: 1.4137
    k=0.4

    View Slide

  34. Evaluation
    Better predictions
    k=0.4
    323/506
    63.8%

    View Slide

  35. View Slide

  36. Evidence of anchoring bias

    View Slide

  37. Evidence of anchoring bias
    Method for mitigating it

    View Slide

  38. Adjusted(R ) = Original(R ) + k*W(R )
    n n n-1

    View Slide

  39. Experiment Design
    (waiting on COUHES)

    View Slide

  40. View Slide

  41. View Slide

  42. Experiment Design
    challenges

    View Slide

  43. View Slide

  44. View Slide

  45. Next steps
    integrate MovieLens recommendations

    View Slide

  46. Next steps
    integrate MovieLens recommendations
    Adjusted(R ) = Original(R ) + k*W(R , dt?)
    n-1
    n n

    View Slide

  47. View Slide

  48. Thanks.
    Greg Borenstein
    MAS.S62: Interactive Machine Learning

    View Slide