Antisocial Behavior in Online Discussion Communities

User contributions in the form of posts, comments, and votes are essential to the success of online communities. However, allowing user participation also invites undesirable behavior such as trolling. Here, we characterize antisocial behavior in three large online discussion communities by analyzing users who were banned from these communities. We find that such users tend to concentrate their efforts in a small number of threads, are more likely to post irrelevantly, and are more successful at garnering responses from other users. Studying the evolution of these users from the moment they join a community up to when they get banned, we find that not only do they write worse than other users over time, but they also become increasingly less tolerated by the community. Further, we discover that antisocial behavior is exacerbated when community feedback is overly harsh. Our analysis also reveals distinct groups of users with different levels of antisocial behavior that can change over time. We use these insights to identify antisocial users early on, a task of high practical importance to community maintainers.

Presented at ICWSM 2015.

Transcript

1. ANTISOCIAL BEHAVIOR IN
   ONLINE COMMUNITIES
   Justin Cheng STANFORD
   Cristian Danescu-Niculesu-Mizil CORNELL
   Jure Leskovec STANFORD
   

   View Slide

2. View Slide

3. Warning
   This talk contains user posts with strong writing

   (profanity, sexism, racism, and religious intolerance).
   !
   

   View Slide

4. View Slide

5. It also shows that Islam and Christianity teaching
   women to dress modest could be right afterall.
   

   View Slide

6. Religious nut alert
   Clearly that is the only logical conclusion to
   this article. Now if you'll excuse me, I need to
   iron my tarp. I have work on Monday, and I
   want to appear 'modest'.
   fail at life. go bomb yourself.
   It also shows that Islam and Christianity teaching
   women to dress modest could be right afterall.
   

   View Slide

7. Both of these little skanks are ugly to the bone.
   

   View Slide

8. Both of these little skanks are ugly to the bone.
   You’re a troll.
   

   View Slide

9. Trolls disrupt online discussions
   Baker, P. (2001); Donath, J. S. (1999); Herring, S., et al. (2011); Shachaf, P. and Hara, N. (2010)
   

   View Slide

10. View Slide

11. Characterizing trolls in online
    discussion communities
    

    View Slide

12. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from

    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    

    View Slide

13. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from

    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis
    Baker, P. (2001); Donath, J. S. (1999); Herring, S., et al. (2011); Shachaf, P. and Hara, N. (2010)
    

    View Slide

14. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from
    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis Longitudinal Study
    

    View Slide

15. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from
    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis Longitudinal Study Predictive Modeling
    Adler, B. T., et al. (2011); Wang, W. Y. and McKeown, K. R. (2010)
    

    View Slide

16. How do we define trolling?
    

    View Slide

17. What data are we using?
    18 months ~1.7M users ~40M posts ~100M votes
    

    View Slide

18. Post Deletions

    by moderators
    2.0%
    (>500k)
    2.3%
    (>180k)
    2.7%
    (>110k)
    User Bans

    by moderators
    3.3%
    (>37k)
    1.7%
    (>5k)
    2.2%
    (>5k)
    How common is trolling?
    

    View Slide

19. How do we define trolling?
    Engaging in negatively marked online behavior
    Taking pleasure in upsetting others
    Not following the rules
    Disrupting a group while staying undercover
    Donath, J. S. (1999); Hardaker, C. (2010); Kirman, B., et al. (2012); Schwartz, M. (2008)
    

    View Slide

20. How do we define trolling?
    Troll

    User banned in the future.
    

    View Slide

21. How do we define trolling?
    Troll

    User banned in the future.
    Non-troll

    User who was never banned.
    

    View Slide

22. How do we define trolling?
    Troll

    User banned in the future.
    Non-troll

    User who was never banned, but is similarly active.
    (matched)
    

    View Slide

23. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from

    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis Longitudinal Study Predictive Modeling
    

    View Slide

24. Penny, once again you show why you are one of the
    best of the league. Always a class-act…
    Fantastic story. Kudos to Penny Hardaway. This is
    what you call giving back.
    *sniff* This had me tearing up in the office. Great job
    Penny! You were always a class act…
    Ex-NBA star returns to inner city, brings hoop dreams
    

    View Slide

25. stop reading then. Just sayin.. CNN not making
    you read...you chose to.
    How many white NBA players grew up in the
    Why…do I not see any articles similar to this about
    white NBA basketball players?......every single touchy
    feely story is about a black ball player............YOU
    GUYS MAKE ME SICK AS A READER !
    what you call giving back.
    Less similar to previous posts

    9% less similar (cosine similarity), p<10-4
    Penny, once again you show why you are one of the
    best of the league. Always a class-act…
    

    View Slide

26. charitable efforts by white NBA players, but
    this isn't a story about a Black NBA player...it's
    a story about someone who came back to their
    roots to contribute.
    b\c young black men need to see examples
    from their own race. They need to see that
    even minorities can succeed and give back
    instead. They connect with people of their own
    race. It allows for mentorship and guidance.
    A story about a millionaire helping kids in his
    poor neighborhood personally, and being a
    positive role model makes you sick as a reader.
    Gotta love conservatives.
    Get more replies from other users

    Twice as many replies, p<10-2
    

    View Slide

27. If you claim you want less government but want to control the bedroom,you're a
    Republican; If you want to cut Education, you're a Republican; If you want to cut Social
    Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a
    If you claim you want less government but want to control the bedroom,you're a
    Republican; If you want to cut Education, you're a Republican; If you want to cut Social
    Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a
    If you claim you want less government but want to control the bedroom,you're a
    Republican; If you want to cut Education, you're a Republican; If you want to cut Social
    Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a
    If you claim you want less government but want to control the
    bedroom,you're a Republican; If you want to cut Education, you're a
    If you claim you want less government but want to control the
    bedroom,you're a Republican; If you want to cut Education, you're a
    If you claim you want less government but want to control the
    bedroom,you're a Republican; If you want to cut Education, you're a
    If you claim you want less government but want to control the bedroom,you're a
    Republican; If you want to cut Education, you're a Republican; If you want to cut Social
    Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a
    If you claim you want less government but want to control the bedroom,you're a
    Republican; If you want to cut Education, you're a Republican; If you want to cut Social
    Security, you're a Republican; If you want to cut Medicare and Medicaid, you're a
    If you claim you want less government but want to control the bedroom,you're a
    Post more per thread

    47% more posts per thread, p<10-2
    

    View Slide

28. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from
    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis Longitudinal Study Predictive Modeling
    

    View Slide

29. A troll’s posts are deleted more
    Post Deletion Rate
    0
    0.2
    0.4
    0.6
    0.8
    1
    Time (Normalized)
    0.1 0.3 0.5 0.7 0.9
    Trolls
    Non-Trolls
    

    View Slide

30. Why a troll’s posts are deleted more
    Trolls write worse posts over time.
    Trolls start out worse than non trolls, and worsen more over time.
    1 p<0.05
    

    View Slide

31. Why a troll’s posts are deleted more
    Trolls write worse posts over time.
    Trolls start out worse than non trolls, and worsen more over time.
    Communities become less tolerant of trolls.
    A troll’s posts are more likely to be deleted later in their life.
    1
    2
    p<0.05
    p<10-4
    

    View Slide

32. Why a troll’s posts are deleted more
    Trolls write worse posts over time.
    Trolls start out worse than non trolls, and worsen more over time.
    Communities become less tolerant of trolls.
    A troll’s posts are more likely to be deleted later in their life.
    Communities can exacerbate trolling.
    Unfairly deleting a user’s posts causes them to write worse later.
    1
    2
    3
    p<0.05
    p<10-4
    p<0.05
    

    View Slide

33. Why a troll’s posts are deleted more
    Trolls write worse posts over time.
    Trolls start out worse than non trolls, and worsen more over time.
    Communities become less tolerant of trolls.
    A troll’s posts are more likely to be deleted later in their life.
    Communities can exacerbate trolling.
    Unfairly deleting a user’s posts causes them to write worse later.
    1
    2
    3
    ICWSM 2014
    >
    p<0.05
    p<10-4
    p<0.05
    

    View Slide

34. Characterizing trolls in online
    discussion communities
    How do trolls
    differ from
    non-trolls?
    1
    How do trolls
    change over
    time?
    2
    How do we
    predict troll-like
    behavior?
    3
    Large-scale Analysis Longitudinal Study Predictive Modeling
    

    View Slide

35. Can we predict whether a user
    will get banned in the future?
    First 10 Posts Balanced Dataset of Trolls and Non-Trolls
    

    View Slide

36. Prediction results on CNN
    Bag of Words
    ROC AUC
    0.5 0.6 0.7 0.8 0.9
    0.70
    

    View Slide

37. Prediction results on CNN
    Bag of Words
    Post Deletion Rate
    ROC AUC
    0.5 0.6 0.7 0.8 0.9
    0.74
    0.70
    (Manual)
    

    View Slide

38. Prediction results on CNN
    Bag of Words
    Post Deletion Rate
    ROC AUC
    0.5 0.6 0.7 0.8 0.9
    0.83
    0.74
    0.70
    (Automatic)
    (Manual)
    Our Approach
    

    View Slide

39. Our automatic approach
    generalizes across communities.
    Uses Interaction Patterns, Not Language
    Cross-Domain AUC = 0.68
    

    View Slide

40. Conclusion
    Trolls differ from non-trolls in their language and
    behavior.
    Trolls change over time, and the community plays a
    role in exacerbating trolling.
    A user’s initial posting activity can effectively predict
    whether that user will be subsequently banned.
    

    View Slide

41. Social Media
    

    View Slide

42. Anti Social Media
    

    View Slide

43. END OF PART 1
    

    View Slide

44. END OF PART 1
    

    View Slide

45. ANTISOCIAL BEHAVIOR IN
    ONLINE COMMUNITIES
    Justin Cheng STANFORD
    Cristian Danescu-Niculesu-Mizil CORNELL
    Jure Leskovec STANFORD http://bit.ly/trolls-paper
    

    View Slide