Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Anyone Can Become a Troll

Justin Cheng
February 28, 2017

Anyone Can Become a Troll

In online communities, antisocial behavior such as trolling disrupts constructive discussion. While prior work suggests that trolling behavior is confined to a vocal and antisocial minority, we demonstrate that ordinary people can engage in such behavior as well. We propose two primary trigger mechanisms: the individual's mood, and the surrounding context of a discussion (e.g., exposure to prior trolling behavior). Through an experiment simulating an online discussion, we find that both negative mood and seeing troll posts by others significantly increases the probability of a user trolling, and together double this probability. To support and extend these results, we study how these same mechanisms play out in the wild via a data-driven, longitudinal analysis of a large online news discussion community. This analysis reveals temporal mood effects, and explores long range patterns of repeated exposure to trolling. A predictive model of trolling behavior shows that mood and discussion context together can explain trolling behavior better than an individual's history of trolling. These results combine to suggest that ordinary people can, under the right circumstances, behave like trolls.

Presented at CSCW 2017.

Justin Cheng

February 28, 2017
Tweet

More Decks by Justin Cheng

Other Decks in Research

Transcript

  1. (even jcccf, msbernst, cristian, and jure)
    Anyone can become a troll.
    1

    View Slide

  2. Time (2016)
    “How trolls are ruining the internet”
    2
    News headlines

    View Slide

  3. The Atlantic (2016)
    “When will the internet be safe for women?”
    “How trolls are ruining the internet”
    3
    News headlines

    View Slide

  4. Salon (2014)
    “Furious trolls are everywhere”
    “When will the internet be safe for women?”
    “How trolls are ruining the internet”
    4
    News headlines

    View Slide

  5. Pew Research (2014)
    40% of online users have been harassed
    5

    View Slide

  6. Popular Science (2013)
    “Why we’re shutting off our comments”
    6
    More headlines

    View Slide

  7. The Verge (2013)
    “We’re turning comments off for a while”
    “Why we’re shutting off our comments”
    7
    More headlines

    View Slide

  8. Chicago Sun-Times (2014)
    “Sick of internet comments? Us, too”
    “We’re turning comments off for a while”
    “Why we’re shutting off our comments”
    8
    More headlines

    View Slide

  9. Why is trolling so prevalent?
    RQ
    9

    View Slide

  10. Understanding trolling lets us design ❤ communities
    Implication
    10
    healthier prosocial

    View Slide

  11. What is trolling?
    11

    View Slide

  12. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    1. Engaging in negatively marked online behavior?
    What is trolling?
    12

    View Slide

  13. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    13

    View Slide

  14. Donath (1999); Hardaker (2010); Kirman (2012); Schwartz (2008)
    3. Taking pleasure in upsetting others?
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    14

    View Slide

  15. Trolling is behavior outside community norms.
    3. Taking pleasure in upsetting others?
    2. Not following the rules?
    1. Engaging in negatively marked online behavior?
    What is trolling?
    15
    name-calling personal attacks threats hate speech ethnically/racially offensive material

    View Slide

  16. Donath (1999); Hardaker (2010); Buckels, et al. (2014)
    Trolling is largely due to sociopaths
    Prior work
    16

    View Slide

  17. Trolling is due to ordinary people
    This work
    17

    View Slide

  18. How much do trolls troll?
    18

    View Slide

  19. 16M posts on 16K articles from .com
    19
    Data

    View Slide

  20. How much do trolls troll?
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    20

    View Slide

  21. The distribution of trolls is bimodal
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    21

    View Slide

  22. Are there two types of trolls?
    Proportion of Banned Users
    0
    0.1
    0.2
    0.3
    0.4
    Proportion of Deleted Posts
    0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
    Situational trolling?
    22
    Lifelong trolling?

    View Slide

  23. How to show that trolling is situational?
    23
    Challenge

    View Slide

  24. Observational data isn’t causal
    How to show that trolling is situational?
    24

    View Slide

  25. Experiments are hard to generalize
    Observational data isn’t causal
    How to show that trolling is situational?
    25

    View Slide

  26. Solution: online experiment + observational study
    Experiments are hard to generalize
    Observational data isn’t causal
    How to show that trolling is situational?
    26

    View Slide

  27. Anyone can become a troll
    27
    Hypothesis

    View Slide

  28. Zimbardo (1969); Wilson (1982)
    Theory 1: “Broken windows”
    28

    View Slide

  29. Jones & Bogat (1978); Rotton & Frey (1985)
    Theory 2: Unpleasant stimuli increase aggression
    29

    View Slide

  30. Online experiment simulating a discussion forum
    30
    N=667 (40% female)

    View Slide

  31. Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    31

    View Slide

  32. We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    32

    View Slide

  33. The quiz was either easy or difficult
    We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    33

    View Slide

  34. Discussion context was either positive or negative
    The quiz was either easy or difficult
    We manipulated quiz difficulty and discussion context
    Complete a quiz, then participate in a discussion
    Online experiment simulating a discussion forum
    34

    View Slide

  35. Here’s the easy quiz (positive mood condition)
    35

    View Slide

  36. Here’s the easy quiz (positive mood condition)
    36

    View Slide

  37. Here’s the difficult quiz (negative mood condition)
    37

    View Slide

  38. Here’s the positive discussion context condition
    38

    View Slide

  39. Here’s negative discussion context condition
    39

    View Slide

  40. People were in a worse mood after the difficult quiz
    Easy Quiz
    Difficult Quiz
    POMS Mood Disturbance
    0 10 20 30 40 50
    40
    Manipulation checks

    View Slide

  41. People also perceived seed troll posts as worse
    People were in a worse mood after the difficult quiz
    Seed Troll
    Seed Non-Troll
    Proportion of Upvotes
    0 0.2 0.4 0.6 0.8 1
    41
    Manipulation checks

    View Slide

  42. How much trolling was there in each condition?
    42
    posts independently labeled by two expert
    raters using community guidelines

    View Slide

  43. How much trolling was there in each condition?
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    Negative
    Context
    % Troll Posts
    43

    View Slide

  44. Trolling is lowest in the positive conditions…
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35%
    Negative
    Context
    % Troll Posts
    44

    View Slide

  45. …increases with either negative condition…
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35% 49%
    Negative
    Context
    47%
    % Troll Posts
    45

    View Slide

  46. …and almost doubles in the negative conditions
    (p < 0.05 using a mixed effects logistic regression model)
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    35% 49%
    Negative
    Context
    47% 68%
    % Troll Posts
    46

    View Slide

  47. Negative affect also triples
    (p < 0.05)
    Positive
    Mood
    Negative
    Mood
    Positive
    Context
    1.1% 1.4%
    Negative
    Context
    2.3% 2.9%
    % Neg. Affect Words (LIWC)
    47

    View Slide

  48. Comment from the positive mood/context condition:
    48

    View Slide

  49. “Hilary is a solid candidate. As a woman, I appreciate
    that she's a woman, but it's not the only reason I think
    she would do well in office.”
    Comment from the positive mood/context condition:
    49

    View Slide

  50. Comment from the negative mood/context condition:
    50

    View Slide

  51. “Anyone who votes for her is a complete idiot. These
    supporters are why this country is in such bad shape
    now. Uneducated people.”
    Comment from the negative mood/context condition:
    51

    View Slide

  52. Bad mood and negative context increase trolling
    52

    View Slide

  53. But does this generalize?
    53
    Bad mood and negative context increase trolling

    View Slide

  54. Online experiment + Observational study
    54
    But does this generalize?
    Bad mood and negative context increase trolling
    .com

    View Slide

  55. Golder & Macy (2011)
    Can trolling vary with the time of day or day of week?
    55
    Replicating mood
    Neg. Affect
    Time of Day
    ?

    View Slide

  56. Can trolling vary with the time of day or day of week?
    56
    Proportion of Flagged Posts
    0.03
    0.033
    0.036
    0.039
    0.042
    Time of Day
    0 6 12 18 24

    View Slide

  57. Trolling peaks in the evening…
    57
    Proportion of Flagged Posts
    0.03
    0.033
    0.036
    0.039
    0.042
    Time of Day
    0 6 12 18 24
    Golder & Macy
    Our Data

    View Slide

  58. …and early in the work week
    58
    Proportion of Flagged Posts
    0.03
    0.032
    0.034
    0.036
    0.038
    0.04
    Day of Week
    Mon Tue Wed Thu Fri Sat Sun

    View Slide

  59. Trolling peaks when moods are worse
    59
    Time of Day
    Proportion of
    Flagged Posts
    Negative Affect
    Proportion of
    Downvotes
    Day of Week

    View Slide

  60. Does mood spill over from prior discussions?
    60
    Replicating mood

    View Slide

  61. Does mood spill over from prior discussions?
    61
    (Discussion)


    View Slide

  62. Does mood spill over from prior discussions?
    62
    (Discussion)




    (Unrelated Discussions)


    ?
    ?

    View Slide

  63. Mood spills over from prior discussions
    63


    (Discussion)


    (Unrelated Discussions)


    View Slide

  64. Trolling is twice as likely in unrelated discussions
    (p < 0.01)
    64


    (Discussion)


    (Unrelated Discussions)


    2x

    View Slide

  65. Does the initial post affect subsequent trolling?
    65
    Replicating context

    View Slide

  66. Does the initial post affect subsequent trolling?
    66
    (Separate discussions of same article)


    ? ? ?
    ? ? ?

    View Slide

  67. An initial post increases later trolling by over 1.5x
    (p < 0.01)
    67
    (Separate discussions of same article)


    ? ?

    View Slide

  68. Does increased trolling have an additive effect?
    68
    Replicating context

    View Slide

  69. Does increased trolling have an additive effect?
    69
    1 2 3 4 5

    (Discussion)

    View Slide

  70. Does increased trolling have an additive effect?
    70
    1 2 3 4 5

    (Discussion)

    View Slide

  71. Does increased trolling have an additive effect?
    71
    1 2 3 4 5

    (Discussion)
    ?

    View Slide

  72. Does increased trolling have an additive effect?
    72
    1 2 3 4 5

    (Discussion)
    ?

    View Slide

  73. More initial trolling means more future trolling
    73
    Pr(5th Post is Flagged)
    0
    0.1
    0.2
    0.3
    0.4
    0.5
    0.6
    0.7
    # of Flagged Posts in First 4 Posts
    0 1 2 3 4
    New participant
    Posted previously

    View Slide

  74. Can we predict trolling before it happens?
    74

    View Slide

  75. Logistic regression on 120K posts
    75
    Can we predict trolling before it happens?

    View Slide

  76. Three sets of features: mood, context, or user-specific
    76
    Trolling is situational Trolling is innate
    Logistic regression on 120K posts
    Can we predict trolling before it happens?

    View Slide

  77. How predictable is trolling?
    77

    View Slide

  78. How predictable is trolling?
    78
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66

    View Slide

  79. How predictable is trolling?
    79
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66

    View Slide

  80. How predictable is trolling?
    80
    User-specific
    Mood
    Discussion Context
    Combined
    AUC
    0.00 0.20 0.40 0.60 0.80
    0.78
    0.74
    0.60
    0.66

    View Slide

  81. 81
    Troll or not?
    User

    View Slide

  82. 82
    Troll or not?
    User
    Mood

    View Slide

  83. 83
    Troll or not?
    User
    Mood
    Other
    users
    { }

    View Slide

  84. Couldn’t we just ban trolls?
    84

    View Slide

  85. But many “trolls” are ordinary people!
    85
    Couldn’t we just ban trolls?

    View Slide

  86. Important to also curb situational trolling:
    86
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?

    View Slide

  87. Through inferring mood…
    87
    Important to also curb situational trolling:
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?
    Rate limiting Enabling “undo”

    View Slide

  88. Mazar, Amir, & Ariely (2008)
    Or altering the context of a discussion.
    88
    Through inferring mood…
    Important to also curb situational trolling:
    But many “trolls” are ordinary people!
    Couldn’t we just ban trolls?
    Prioritizing constructive comments Ethical/moral reminders

    View Slide

  89. (even @jcccf, @msbernst, cristian, and @jure. read the paper: bit.ly/anyonepaper)
    Because trolling is situational,
    anyone can become a troll.
    89

    View Slide