Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

2f4faa539dc6a0ae688e58d6a329fce9?s=47 gregab
December 13, 2013

Order Matters: Accounting for Anchoring Bias on User Labeling in Recommendation Systems

Read the full paper here: http://gregborenstein.com/assets/order-matters.pdf

Final project for an Interactive Machine Learning course at the MIT Media Lab. Demonstrates the presence of Anchoring Bias, as described by Kahneman and Tversky, in sequential ratings interaction common to recommender systems like Netflix.

2f4faa539dc6a0ae688e58d6a329fce9?s=128

gregab

December 13, 2013
Tweet

Transcript

  1. Order Matters: Accounting for Anchoring Bias on User Labeling in

    Recommendation Systems Greg Borenstein MAS.S62: Interactive Machine Learning
  2. When “people assess the probability of an uncertain event or

    the value of an uncertain quantity” they “rely on a limited number of heuristic principles” that “sometimes lead to severe and systematic errors” Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Cognitive Bias
  3. When “people assess the probability of an uncertain event or

    the value of an uncertain quantity” they “rely on a limited number of heuristic principles” that “sometimes lead to severe and systematic errors” Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Cognitive Bias
  4. initial value + insufficient adjustment bias towards the initial values

    Kahneman and Tversky Judgement Under Uncertainty: Heuristics and Biases, 1974 Anchoring Bias
  5. Percentage of African Countries in the UN? Image courtesy of

    Wikimedia Foundation.
  6. Netflix rating interface circa December 10, 2013

  7. None
  8. Evidence of anchoring bias

  9. Evidence of anchoring bias Method for mitigating it

  10. Evidence of anchoring bias

  11. MovieLens rating interface circa December 10, 2013

  12. None
  13. Difference in frequency of pair occurrences

  14. None
  15. P(r , r ) = P(r ) * P(r )

    1 2 1 2
  16. Ratings organized chronologically by user

  17. Ratings organized chronologically by user

  18. None
  19. None
  20. None
  21. None
  22. None
  23. Evidence of anchoring bias: compare classifiers

  24. Limit to ratings in user sequences ~44,000

  25. Order-Ignorant Constant prediction that minimizes mean squared error

  26. P(R=i ) * (r-i) i = 1 5 2 Σ

    argmin r Order-Ignorant Constant prediction that minimizes mean squared error
  27. None
  28. None
  29. Order-Aware Treat previous ratings as labels

  30. Order-Aware Adjusted(R ) = Original(R ) + k*W(R ) n

    n Treat previous ratings as labels n-1
  31. Order-Aware Treat previous ratings as labels Adjusted(R ) = Original(R

    ) + k*W(R , dt?) n-1 n n
  32. r - r Order-Aware Treat previous ratings as labels W(r

    ) = 1/n * i-1 i = 1 n Σ i i
  33. Evaluation Mean Square Error Order-Ignorant: 1.4927 Order-Aware: 1.4137 k=0.4

  34. Evaluation Better predictions k=0.4 323/506 63.8%

  35. None
  36. Evidence of anchoring bias

  37. Evidence of anchoring bias Method for mitigating it

  38. Adjusted(R ) = Original(R ) + k*W(R ) n n

    n-1
  39. Experiment Design (waiting on COUHES)

  40. None
  41. None
  42. Experiment Design challenges

  43. None
  44. None
  45. Next steps integrate MovieLens recommendations

  46. Next steps integrate MovieLens recommendations Adjusted(R ) = Original(R )

    + k*W(R , dt?) n-1 n n
  47. None
  48. Thanks. Greg Borenstein MAS.S62: Interactive Machine Learning