How Community Feedback Shapes Online Behavior

How Community Feedback Shapes Online Behavior

Presented at ICWSM 2014.

Social media systems rely on user feedback and rating mechanisms for personalization, ranking, and content filtering. However, when users evaluate content contributed by fellow users (e.g., by liking a post or voting on a comment), these evaluations create complex social feedback effects. We investigate how ratings on a piece of content affect its author’s future behavior. By studying four large comment-based news communities, we find that negative feedback leads to significant behavioral changes that are detrimental to the community. Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such. Moreover, these authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community. In contrast, positive feedback does not carry similar effects, and neither encourages rewarded authors to write more, nor improves the quality of their posts. Interestingly, the authors that receive no feedback are most likely to leave a community. Furthermore, a structural analysis of the voter network reveals that evaluations polarize the community the most when positive and negative votes are equally split.

8480b47e733a040fba07c32da414b0e0?s=128

Justin Cheng

June 02, 2014
Tweet

Transcript

  1. How Community Feedback Shapes User Behavior Justin Cheng STANFORD Cristian

    Danescu-Niculescu-Mizil MAX PLANCK INSTITUTE SWS Jure Leskovec STANFORD
  2. People evaluate content all the time.

  3. None
  4. None
  5. None
  6. None
  7. When a user evaluates user-generated content, he/she is indirectly evaluating

    other users.
  8. Evaluations create social feedback loops that affect user behavior.

  9. None
  10. None
  11. None
  12. None
  13. How do people react to evaluations?

  14. How does positive/negative feedback influence subsequent behavior?

  15. Positively Evaluated Negatively Evaluated ? ?

  16. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  17. Related work includes Quantifying social influence (e.g., voting)
 Muchnik, L.,

    Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science. Predicting evaluations (e.g., helpfulness) Ghose, A., & Ipeirotis, P. G. (2011). Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. KDE. Examining what people talk about Chen, Z., & Berger, J. (2013). When, why, and how controversy causes conversation. JCR.
  18. How do people perceive community feedback? SIDE NOTE

  19. None
  20. A post receives P up-votes and N down-votes. How do

    we then define a positive or negative evaluation? SIDE NOTE
  21. Doesn’t account for down-votes 15 P Number of up-votes 0

    15 15 vs
  22. Doesn’t account for proportion of up-votes 5 P-N Difference in

    up-votes and down-votes 0 50 45 vs
  23. Doesn’t account for total number of votes 4 P/(P+N) Proportion

    of up-votes 1 40 10 vs
  24. To measure how users perceived votes, we asked crowd workers

    to rate, on a 7 point scale, if they felt more positive, or negative about receiving a certain number of up/down-votes.
  25. 7 = Positive 1 = Negative 0 20 20 #

    of up-votes # of down-votes 10 10 User ratings were mostly independent of the total number of votes
  26. P Number of up-votes P-N Difference in up/down-votes P/(P+N) Proportion

    of up-votes 0.410 0.879 0.920 R2
  27. A post is positively evaluated when the proportion of up-votes

    is higher than a given threshold (e.g., ≥75th percentile). P/(P+N) = 9/(9+1) = 0.9 ≥ High Threshold
  28. A post is negatively evaluated when the proportion of up-votes

    is lower than a given threshold (e.g., ≤25th percentile). P/(P+N) = 2/(2+8) = 0.2 ≤ Low Threshold
  29. How does positive/negative feedback influence subsequent behavior? defined by the

    proportion of up-votes
  30. Four large comment-based news communities with 1.2M articles, 1.8M registered

    users, 42M posts, and 140M votes. CNN IGN Breitbart allkpop
  31. How do evaluations change after a positive/negative evaluation?

  32. Do users improve? Operant conditioning predicts that feedback would guide

    authors towards better behavior (i.e. up-votes are “reward” stimuli, and down-votes are “punishment” stimuli). Skinner, B. F. (1938). The behavior of organisms: An experimental analysis.
  33. Or do they get worse? Feedback can have negative effects.

    People given only positive feedback tend to become complacent. Also, bad impressions are quicker to form and more resistant to disconfirmation. Brinko, K. T. (1993). The practice of giving feedback to improve teaching: what is effective? Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good.
  34. User A Positively Evaluated Negatively Evaluated User B ✦ Both

    positively/negatively evaluated posts have ≥ 10 votes.
  35. User A User B … … … … Before After

  36. … … Better? Worse? Are users evaluated better or worse

    after receiving a positive evaluation? Before After { {
  37. … … Better? Worse? Are users evaluated better or worse

    after receiving a negative evaluation? Before After { {
  38. Proportion of up-votes 0.4 0.45 0.5 0.55 0.6 { {

    * Negatively-evaluated users are evaluated worse in the future. … … Before After
  39. Proportion of up-votes 0.7 0.75 0.8 0.85 0.9 { {

    Positively-evaluated users are evaluated no better/worse in the future. … … Before After
  40. Feedback does not seem to improve future behavior. SO FAR…

  41. … … … What happens after you give the same

    user a positive, or a negative evaluation? Wish: randomized controlled trial
  42. … Compare similar pairs of users who were evaluated differently

    on similar content Solution: propensity score matching … ≈ ≈ … … Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects.
  43. How much is an evaluation due to textual, or community

    effects? Problem
  44. How much is an evaluation due to textual, or community

    effects? Problem i.e. down-voting because of the post content I hate Obama…
  45. How much is an evaluation due to textual, or community

    effects? Problem i.e. down-voting because the community dislikes the author We don’t like you. We don’t like you.
  46. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  47. Does feedback regulate post quality? Evaluations affect Post quality (How

    well you write)
  48. User A Positively Evaluated Negatively Evaluated User B

  49. User A User B Similar Text Quality } ✦ Text

    quality determined by training a machine learning model using text features, validated using crowd workers.
  50. User A User B … … { Similar History Number

    of posts, overall proportion of up-votes, etc.
  51. User A User B … … … …

  52. User A User B … … … … Compare Text

    Quality After }
  53. … … After a positive evaluation, Or do they write

    worse? do users write better? … …
  54. … … After a negative evaluation, Or do they write

    worse? do users write better? … …
  55. Post quality drops significantly after a negative evaluation, but not

    after a positive evaluation. p < 0.05 in all communities NEGATIVITY BIAS To learn more about these types of effects, see Kanouse, D. E., & Hanson Jr, L. R. (1987). Negativity in evaluations.
  56. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  57. How does community perception of a user change after an

    evaluation? Evaluations affect Community bias (How people perceive you)
  58. Community Bias Actual Evaluation P/(P+N) Text Quality Text Quality Up-votes

    Down-votes 0.9 0.8 0.9-0.8 = +0.1
  59. User A User B … … … … Compare Community

    Bias After }
  60. Does the community perceive a user worse after a negative,

    than a positive evaluation? User A User B … … … …
  61. Posts made after a negative evaluation were perceived worse than

    those made after a positive evaluation. p < 0.05 in all communities HALO EFFECT
  62. More Positive Positive Eval. Negative Eval.

  63. More Positive Similar
 Text Quality Positive Eval. Negative Eval.

  64. Before More Positive Similar
 Text Quality Similar History Positive Eval.

    Negative Eval.
  65. Before More Positive After Similar
 Text Quality Similar History Positive

    Eval. Negative Eval.
  66. Before More Positive After Similar
 Text Quality Similar History Positive

    Eval. Negative Eval. Worse Quality
  67. Before More Positive After Similar
 Text Quality Similar History Positive

    Eval. Negative Eval. Worse Quality
  68. Before More Positive After Similar
 Text Quality Similar History Positive

    Eval. Negative Eval. Worse Perception Worse Quality
  69. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  70. Posting frequency (How regularly you post) Does feedback regulate post

    quantity? Evaluations affect
  71. Do users post more frequently after a positive/negative evaluation? …

    … Before After Time
  72. Users who receive negative feedback post more frequently. No Feedback

    Positive Negative Median relative change in posting frequency 1 1.5 2 2.5 3 3.5
  73. Users who receive negative feedback post more frequently. No Feedback

    Positive Negative Times more frequent after vs. before 1 1.5 2 2.5 3 3.5 Don’t feed the trolls!
  74. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  75. Does feedback result in subsequent backlash? Evaluations affect Voting behavior

    (How you vote on others)
  76. Proportion of up-votes given 0.6 0.625 0.65 0.675 0.7 Before

    After Positive Negative * Users who receive negative feedback are more likely to down-vote others.
  77. Is there a downward spiral in online communities?

  78. The proportion of down-votes is increasing over time. Proportion of

    down-votes 0.17 0.19 0.21 0.23 Jan Feb Mar Apr May Jun Jul Aug 78
  79. The proportion of down-votes is increasing over time. Proportion of

    down-votes 0.17 0.19 0.21 0.23 Jan Feb Mar Apr May Jun Jul Aug 79 0.8m down-votes 1.7m down-votes
  80. 0.18 0.23 Jan Mar May Jul 80 0.10 0.20 Jan

    Mar May Jul 0 0.12 Jan Mar May Jul 0.05 0.11 Nov Jan Mar May Breitbart allkpop CNN IGN
  81. The effects of negative feedback are more pronounced than those

    of positive feedback.
  82. Negatively-evaluated users write worse (and more!), are themselves evaluated worse

    by the community, and evaluate other community members worse. Positively-evaluated users, on the other hand, don’t do any better.
  83. After Before

  84. Future work could look at 1. communities that support only

    positive feedback (e.g., likes), 2. the effects of user reputation, 3. deeper linguistic analysis, and 4. experiments to establish causality.
  85. Also in the paper are 1. details of the matching

    experiment, 2. differences within a single thread and across multiple threads, and 3. the organization of voting networks
  86. ≈ … … …

  87. ≈ … … … Negatively evaluated users write worse

  88. ≈ … … … No evidence of change for positively

    evaluated users
  89. ≈ … … … Negatively evaluated users also write more

  90. ≈ … … … Negatively evaluated users are perceived worse

    by the community
  91. How Community Feedback Shapes User Behavior Justin Cheng STANFORD Cristian

    Danescu-Niculescu-Mizil MAX PLANCK INSTITUTE SWS Jure Leskovec STANFORD