Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How Community Feedback Shapes Online Behavior

How Community Feedback Shapes Online Behavior

Presented at ICWSM 2014.

Social media systems rely on user feedback and rating mechanisms for personalization, ranking, and content filtering. However, when users evaluate content contributed by fellow users (e.g., by liking a post or voting on a comment), these evaluations create complex social feedback effects. We investigate how ratings on a piece of content affect its author’s future behavior. By studying four large comment-based news communities, we find that negative feedback leads to significant behavioral changes that are detrimental to the community. Not only do authors of negatively-evaluated content contribute more, but also their future posts are of lower quality, and are perceived by the community as such. Moreover, these authors are more likely to subsequently evaluate their fellow users negatively, percolating these effects through the community. In contrast, positive feedback does not carry similar effects, and neither encourages rewarded authors to write more, nor improves the quality of their posts. Interestingly, the authors that receive no feedback are most likely to leave a community. Furthermore, a structural analysis of the voter network reveals that evaluations polarize the community the most when positive and negative votes are equally split.

Justin Cheng

June 02, 2014
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. How Community Feedback Shapes User Behavior Justin Cheng STANFORD Cristian

    Danescu-Niculescu-Mizil MAX PLANCK INSTITUTE SWS Jure Leskovec STANFORD
  2. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  3. Related work includes Quantifying social influence (e.g., voting)
 Muchnik, L.,

    Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science. Predicting evaluations (e.g., helpfulness) Ghose, A., & Ipeirotis, P. G. (2011). Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics. KDE. Examining what people talk about Chen, Z., & Berger, J. (2013). When, why, and how controversy causes conversation. JCR.
  4. A post receives P up-votes and N down-votes. How do

    we then define a positive or negative evaluation? SIDE NOTE
  5. To measure how users perceived votes, we asked crowd workers

    to rate, on a 7 point scale, if they felt more positive, or negative about receiving a certain number of up/down-votes.
  6. 7 = Positive 1 = Negative 0 20 20 #

    of up-votes # of down-votes 10 10 User ratings were mostly independent of the total number of votes
  7. A post is positively evaluated when the proportion of up-votes

    is higher than a given threshold (e.g., ≥75th percentile). P/(P+N) = 9/(9+1) = 0.9 ≥ High Threshold
  8. A post is negatively evaluated when the proportion of up-votes

    is lower than a given threshold (e.g., ≤25th percentile). P/(P+N) = 2/(2+8) = 0.2 ≤ Low Threshold
  9. Four large comment-based news communities with 1.2M articles, 1.8M registered

    users, 42M posts, and 140M votes. CNN IGN Breitbart allkpop
  10. Do users improve? Operant conditioning predicts that feedback would guide

    authors towards better behavior (i.e. up-votes are “reward” stimuli, and down-votes are “punishment” stimuli). Skinner, B. F. (1938). The behavior of organisms: An experimental analysis.
  11. Or do they get worse? Feedback can have negative effects.

    People given only positive feedback tend to become complacent. Also, bad impressions are quicker to form and more resistant to disconfirmation. Brinko, K. T. (1993). The practice of giving feedback to improve teaching: what is effective? Baumeister, R. F., Bratslavsky, E., Finkenauer, C., & Vohs, K. D. (2001). Bad is stronger than good.
  12. User A Positively Evaluated Negatively Evaluated User B ✦ Both

    positively/negatively evaluated posts have ≥ 10 votes.
  13. … … Better? Worse? Are users evaluated better or worse

    after receiving a positive evaluation? Before After { {
  14. … … Better? Worse? Are users evaluated better or worse

    after receiving a negative evaluation? Before After { {
  15. Proportion of up-votes 0.4 0.45 0.5 0.55 0.6 { {

    * Negatively-evaluated users are evaluated worse in the future. … … Before After
  16. Proportion of up-votes 0.7 0.75 0.8 0.85 0.9 { {

    Positively-evaluated users are evaluated no better/worse in the future. … … Before After
  17. … … … What happens after you give the same

    user a positive, or a negative evaluation? Wish: randomized controlled trial
  18. … Compare similar pairs of users who were evaluated differently

    on similar content Solution: propensity score matching … ≈ ≈ … … Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects.
  19. How much is an evaluation due to textual, or community

    effects? Problem i.e. down-voting because of the post content I hate Obama…
  20. How much is an evaluation due to textual, or community

    effects? Problem i.e. down-voting because the community dislikes the author We don’t like you. We don’t like you.
  21. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  22. User A User B Similar Text Quality } ✦ Text

    quality determined by training a machine learning model using text features, validated using crowd workers.
  23. User A User B … … { Similar History Number

    of posts, overall proportion of up-votes, etc.
  24. … … After a positive evaluation, Or do they write

    worse? do users write better? … …
  25. … … After a negative evaluation, Or do they write

    worse? do users write better? … …
  26. Post quality drops significantly after a negative evaluation, but not

    after a positive evaluation. p < 0.05 in all communities NEGATIVITY BIAS To learn more about these types of effects, see Kanouse, D. E., & Hanson Jr, L. R. (1987). Negativity in evaluations.
  27. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  28. How does community perception of a user change after an

    evaluation? Evaluations affect Community bias (How people perceive you)
  29. Does the community perceive a user worse after a negative,

    than a positive evaluation? User A User B … … … …
  30. Posts made after a negative evaluation were perceived worse than

    those made after a positive evaluation. p < 0.05 in all communities HALO EFFECT
  31. Before More Positive After Similar
 Text Quality Similar History Positive

    Eval. Negative Eval. Worse Perception Worse Quality
  32. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  33. Users who receive negative feedback post more frequently. No Feedback

    Positive Negative Median relative change in posting frequency 1 1.5 2 2.5 3 3.5
  34. Users who receive negative feedback post more frequently. No Feedback

    Positive Negative Times more frequent after vs. before 1 1.5 2 2.5 3 3.5 Don’t feed the trolls!
  35. Evaluations affect Post quality (How well you write) Community bias

    (How people perceive you) Voting behavior (How you vote on others) Posting frequency (How regularly you post)
  36. Proportion of up-votes given 0.6 0.625 0.65 0.675 0.7 Before

    After Positive Negative * Users who receive negative feedback are more likely to down-vote others.
  37. The proportion of down-votes is increasing over time. Proportion of

    down-votes 0.17 0.19 0.21 0.23 Jan Feb Mar Apr May Jun Jul Aug 78
  38. The proportion of down-votes is increasing over time. Proportion of

    down-votes 0.17 0.19 0.21 0.23 Jan Feb Mar Apr May Jun Jul Aug 79 0.8m down-votes 1.7m down-votes
  39. 0.18 0.23 Jan Mar May Jul 80 0.10 0.20 Jan

    Mar May Jul 0 0.12 Jan Mar May Jul 0.05 0.11 Nov Jan Mar May Breitbart allkpop CNN IGN
  40. Negatively-evaluated users write worse (and more!), are themselves evaluated worse

    by the community, and evaluate other community members worse. Positively-evaluated users, on the other hand, don’t do any better.
  41. Future work could look at 1. communities that support only

    positive feedback (e.g., likes), 2. the effects of user reputation, 3. deeper linguistic analysis, and 4. experiments to establish causality.
  42. Also in the paper are 1. details of the matching

    experiment, 2. differences within a single thread and across multiple threads, and 3. the organization of voting networks
  43. How Community Feedback Shapes User Behavior Justin Cheng STANFORD Cristian

    Danescu-Niculescu-Mizil MAX PLANCK INSTITUTE SWS Jure Leskovec STANFORD