Anyone Can Become a Troll

In online communities, antisocial behavior such as trolling disrupts constructive discussion. While prior work suggests that trolling behavior is confined to a vocal and antisocial minority, we demonstrate that ordinary people can engage in such behavior as well. We propose two primary trigger mechanisms: the individual's mood, and the surrounding context of a discussion (e.g., exposure to prior trolling behavior). Through an experiment simulating an online discussion, we find that both negative mood and seeing troll posts by others significantly increases the probability of a user trolling, and together double this probability. To support and extend these results, we study how these same mechanisms play out in the wild via a data-driven, longitudinal analysis of a large online news discussion community. This analysis reveals temporal mood effects, and explores long range patterns of repeated exposure to trolling. A predictive model of trolling behavior shows that mood and discussion context together can explain trolling behavior better than an individual's history of trolling. These results combine to suggest that ordinary people can, under the right circumstances, behave like trolls.

Presented at CSCW 2017.

Transcript

1. (even jcccf, msbernst, cristian, and jure)
   Anyone can become a troll.
   1
   

   View Slide

2. Time (2016)
   “How trolls are ruining the internet”
   2
   News headlines
   

   View Slide

3. The Atlantic (2016)
   “When will the internet be safe for women?”
   “How trolls are ruining the internet”
   3
   News headlines
   

   View Slide

4. Salon (2014)
   “Furious trolls are everywhere”
   “When will the internet be safe for women?”
   “How trolls are ruining the internet”
   4
   News headlines
   

   View Slide

5. Pew Research (2014)
   40% of online users have been harassed
   5
   

   View Slide

6. Popular Science (2013)
   “Why we’re shutting off our comments”
   6
   More headlines
   

   View Slide

7. The Verge (2013)
   “We’re turning comments off for a while”
   “Why we’re shutting off our comments”
   7
   More headlines
   

   View Slide

8. Chicago Sun-Times (2014)
   “Sick of internet comments? Us, too”
   “We’re turning comments off for a while”
   “Why we’re shutting off our comments”
   8
   More headlines
   

   View Slide

9. Why is trolling so prevalent?
   RQ
   9
   

   View Slide

10. Understanding trolling lets us design ❤ communities
    Implication
    10
    healthier prosocial
    

    View Slide

11. What is trolling?
    11
    

    View Slide

12. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    1. Engaging in negatively marked online behavior?
    What is trolling?
    12
    

    View Slide

13. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    13
    

    View Slide

14. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    3. Taking pleasure in upsetting others?
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    14
    

    View Slide

15. Trolling is behavior outside community norms.
    3. Taking pleasure in upsetting others?
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    15
    name-calling personal attacks threats hate speech ethnically/racially offensive material
    

    View Slide

16. Donath (1999); Hardaker (2010); Buckels, et al. (2014)
    Trolling is largely due to sociopaths
    Prior work
    16
    

    View Slide

17. Trolling is due to ordinary people
    This work
    17
    

    View Slide

18. How much do trolls troll?
    18
    

    View Slide

19. 16M posts on 16K articles from .com
    19
    Data
    

    View Slide

20. How much do trolls troll?
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    20
    

    View Slide

21. The distribution of trolls is bimodal
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    21
    

    View Slide

22. Are there two types of trolls?
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    Situational trolling?
    22
    Lifelong trolling?
    

    View Slide

23. How to show that trolling is situational?
    23
    Challenge
    

    View Slide

24. Observational data isn’t causal
    How to show that trolling is situational?
    24
    

    View Slide

25. Experiments are hard to generalize
    Observational data isn’t causal
    How to show that trolling is situational?
    25
    

    View Slide

26. Solution: online experiment + observational study
    Experiments are hard to generalize
    Observational data isn’t causal
    How to show that trolling is situational?
    26
    

    View Slide

27. Anyone can become a troll
    27
    Hypothesis
    

    View Slide

28. Zimbardo (1969); Wilson (1982)
    Theory 1: “Broken windows”
    28
    

    View Slide

29. Jones & Bogat (1978); Rotton & Frey (1985)
    Theory 2: Unpleasant stimuli increase aggression
    29
    

    View Slide

30. Online experiment simulating a discussion forum
    30
    N=667 (40% female)
    

    View Slide

31. Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    31
    

    View Slide

32. We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    32
    

    View Slide

33. The quiz was either easy or difficult
    We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    33
    

    View Slide

34. Discussion context was either positive or negative
    The quiz was either easy or difficult
    We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    34
    

    View Slide

35. Here’s the easy quiz (positive mood condition)
    35
    

    View Slide

36. Here’s the easy quiz (positive mood condition)
    36
    

    View Slide

37. Here’s the difficult quiz (negative mood condition)
    37
    

    View Slide

38. Here’s the positive discussion context condition
    38
    

    View Slide

39. Here’s negative discussion context condition
    39
    

    View Slide

40. People were in a worse mood after the difficult quiz
    Easy Quiz
    Difficult Quiz
    POMS Mood Disturbance
    0 10 20 30 40 50
    40
    Manipulation checks
    

    View Slide

41. People also perceived seed troll posts as worse
    People were in a worse mood after the difficult quiz
    Seed Troll
    Seed Non-Troll
    Proportion of Upvotes
    0 0.2 0.4 0.6 0.8 1
    41
    Manipulation checks
    

    View Slide

42. How much trolling was there in each condition?
    42
    posts independently labeled by two expert
    raters using community guidelines
    

    View Slide

43. How much trolling was there in each condition?
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    Negative
    Context
    % Troll Posts
    43
    

    View Slide

44. Trolling is lowest in the positive conditions…
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35%
    Negative
    Context
    % Troll Posts
    44
    

    View Slide

45. …increases with either negative condition…
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35% 49%
    Negative
    Context
    47%
    % Troll Posts
    45
    

    View Slide

46. …and almost doubles in the negative conditions
    (p < 0.05 using a mixed effects logistic regression model)
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35% 49%
    Negative
    Context
    47% 68%
    % Troll Posts
    46
    

    View Slide

47. Negative affect also triples
    (p < 0.05)
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    1.1% 1.4%
    Negative
    Context
    2.3% 2.9%
    % Neg. Affect Words (LIWC)
    47
    

    View Slide

48. Comment from the positive mood/context condition:
    48
    

    View Slide

49. “Hilary is a solid candidate. As a woman, I appreciate
    that she's a woman, but it's not the only reason I think
    she would do well in office.”
    Comment from the positive mood/context condition:
    49
    

    View Slide

50. Comment from the negative mood/context condition:
    50
    

    View Slide

51. “Anyone who votes for her is a complete idiot. These
    supporters are why this country is in such bad shape
    now. Uneducated people.”
    Comment from the negative mood/context condition:
    51
    

    View Slide

52. Bad mood and negative context increase trolling
    52
    

    View Slide

53. But does this generalize?
    53
    Bad mood and negative context increase trolling
    

    View Slide

54. Online experiment + Observational study
    54
    But does this generalize?
    Bad mood and negative context increase trolling
    .com
    

    View Slide

55. Golder & Macy (2011)
    Can trolling vary with the time of day or day of week?
    55
    Replicating mood
    Neg. Affect
    Time of Day
    ?
    

    View Slide

56. Can trolling vary with the time of day or day of week?
    56
    Proportion of Flagged Posts
    0.03
    0.033
    0.036
    0.039
    0.042
    Time of Day
    0 6 12 18 24
    

    View Slide

57. Trolling peaks in the evening…
    57
    Proportion of Flagged Posts
    0.03
    0.033
    0.036
    0.039
    0.042
    Time of Day
    0 6 12 18 24
    Golder & Macy
    Our Data
    

    View Slide

58. …and early in the work week
    58
    Proportion of Flagged Posts
    0.03
    0.032
    0.034
    0.036
    0.038
    0.04
    Day of Week
    Mon Tue Wed Thu Fri Sat Sun
    

    View Slide

59. Trolling peaks when moods are worse
    59
    Time of Day
    Proportion of
    Flagged Posts
    Negative Affect
    Proportion of
    Downvotes
    Day of Week
    

    View Slide

60. Does mood spill over from prior discussions?
    60
    Replicating mood
    

    View Slide

61. Does mood spill over from prior discussions?
    61
    (Discussion)
    …
    …
    

    View Slide

62. Does mood spill over from prior discussions?
    62
    (Discussion)
    …
    …
    …
    …
    (Unrelated Discussions)
    …
    …
    ?
    ?
    

    View Slide

63. Mood spills over from prior discussions
    63
    …
    …
    (Discussion)
    …
    …
    (Unrelated Discussions)
    …
    …
    

    View Slide

64. Trolling is twice as likely in unrelated discussions
    (p < 0.01)
    64
    …
    …
    (Discussion)
    …
    …
    (Unrelated Discussions)
    …
    …
    2x
    

    View Slide

65. Does the initial post affect subsequent trolling?
    65
    Replicating context
    

    View Slide

66. Does the initial post affect subsequent trolling?
    66
    (Separate discussions of same article)
    …
    …
    ? ? ?
    ? ? ?
    

    View Slide

67. An initial post increases later trolling by over 1.5x
    (p < 0.01)
    67
    (Separate discussions of same article)
    …
    …
    ? ?
    

    View Slide

68. Does increased trolling have an additive effect?
    68
    Replicating context
    

    View Slide

69. Does increased trolling have an additive effect?
    69
    1 2 3 4 5
    …
    (Discussion)
    

    View Slide

70. Does increased trolling have an additive effect?
    70
    1 2 3 4 5
    …
    (Discussion)
    

    View Slide

71. Does increased trolling have an additive effect?
    71
    1 2 3 4 5
    …
    (Discussion)
    ?
    

    View Slide

72. Does increased trolling have an additive effect?
    72
    1 2 3 4 5
    …
    (Discussion)
    ?
    

    View Slide

73. More initial trolling means more future trolling
    73
    Pr(5th Post is Flagged)
    0
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    # of Flagged Posts in First 4 Posts
    0 1 2 3 4
    New participant
    Posted previously
    

    View Slide

74. Can we predict trolling before it happens?
    74
    

    View Slide

75. Logistic regression on 120K posts
    75
    Can we predict trolling before it happens?
    

    View Slide

76. Three sets of features: mood, context, or user-specific
    76
    Trolling is situational Trolling is innate
    Logistic regression on 120K posts
    Can we predict trolling before it happens?
    

    View Slide

77. How predictable is trolling?
    77
    

    View Slide

78. How predictable is trolling?
    78
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66
    

    View Slide

79. How predictable is trolling?
    79
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66
    

    View Slide

80. How predictable is trolling?
    80
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66
    

    View Slide

81. 81
    Troll or not?
    User
    

    View Slide

82. 82
    Troll or not?
    User
    Mood
    

    View Slide

83. 83
    Troll or not?
    User
    Mood
    Other
    users
    { }
    

    View Slide

84. Couldn’t we just ban trolls?
    84
    

    View Slide

85. But many “trolls” are ordinary people!
    85
    Couldn’t we just ban trolls?
    

    View Slide

86. Important to also curb situational trolling:
    86
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?
    

    View Slide

87. Through inferring mood…
    87
    Important to also curb situational trolling:
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?
    Rate limiting Enabling “undo”
    

    View Slide

88. Mazar, Amir, & Ariely (2008)
    Or altering the context of a discussion.
    88
    Through inferring mood…
    Important to also curb situational trolling:
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?
    Prioritizing constructive comments Ethical/moral reminders
    

    View Slide

89. (even @jcccf, @msbernst, cristian, and @jure. read the paper: bit.ly/anyonepaper)
    Because trolling is situational,
    anyone can become a troll.
    89
    

    View Slide