Upgrade to Pro — share decks privately, control downloads, hide ads and more …

AI4BC 2021 | Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online

Ed09e933a899fcae158439f11f66fed0?s=47 Emaad Manzoor
February 11, 2021
15

AI4BC 2021 | Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online

15 minute talk at the AAAI 2021 workshop on AI for Behavioral Change (AI4BC): https://ai4bc.github.io/ai4bc21/

Video: https://youtu.be/7tSLWogEoWo
Website: http:/emaadmanzoor.com/ethos

Ed09e933a899fcae158439f11f66fed0?s=128

Emaad Manzoor

February 11, 2021
Tweet

Transcript

  1. Influence via Ethos On the Persuasive Power of Reputation in

    Deliberation Online Emaad Manzoor
  2. Joint work with George H. Chen Dokyun Lee Michael D.

    Smith emaadmanzoor.com/ethos/
  3. Persuasion is 30% of the US GDP (McCloskey & Klamer

    1995; Antioch 2013)
  4. Vote Click Buy Mask

  5. Most persuasion attempts are unsuccessful Some are successful but short-lived

    Few are successful with lasting effects
  6. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman

    & Tversky 1974]
  7. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman

    & Tversky 1974] Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc.
  8. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman

    & Tversky 1974] Heuristic Pathway Low effort, short- lived attitude change Eg. Brand, affiliation, etc. — reputation Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc.
  9. “Unthinking Compliance” “[…] the ever-accelerating pace and informational crush of

    modern life will make this particular form of unthinking compliance more and more prevalent in the future.” — [Cialdini 1984]
  10. Need to Encourage Elaboration “The fact that individuals rely on

    automatic thinking has significant implications for understanding development challenges and for designing the best policies to overcome them.”— (World Bank Group 2015)
  11. Q. Does reputation have persuasive power in the context of

    dialogue?
  12. Text-based & Tech-Mediated Dialogue Managing Churn Direct Sales Organizational Deliberation

  13. Challenge 1 of 2 Persuasion is rare, and rarely stated

    explicitly
  14. Conversations from ChangeMyView reddit.com/r/changemyview/ 1 million+ conversations, 800,000+ members 20+

    moderators enforce high-quality dialogue 2013 2019
  15. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Post

  16. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Responses Post

  17. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Responses Post Debates

  18. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈

    Responses Post Debates
  19. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈

    Not Persuaded Not Persuaded Responses Post
  20. Poster Responder Explicit indicator of persuasion

  21. Poster Explicit indicator of persuasion Prominent display of reputation Responder

  22. Response Text High effort Displayed Reputation Low effort Information Processing

    on ChangeMyView [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974]
  23. Challenge 2 of 2 Establishing causality with unstructured observational data

  24. Identification Strategy I. Approximate challenger “skill” II. Address endogenous post

    selection III. Instrument for reputation IV. “Control for” challengers’ response text
  25. Persuasion (Outcome) Response Quality I. “Skill” as a Confounder Causal

    Graph [Pearl, 2009] may/may not cause does not cause Responder Skill Reputation (Treatment) Skill is common cause of the treatment (reputation) and outcome (persuasion)
  26. Accounts for all responder characteristics correlated with persuasion until the

    debate immediately prior Laplace-smoothed in practice [Manning et al 2008] skill(debate t) = no. posters persuaded before t t − 1 I. “Skill” as a Confounder
  27. II. Post Selection as a Confounder Users endogenously select posts

    to respond to based on factors like: • Characteristics of the post (eg. post topic) • Characteristics of the poster (eg. malleability)
  28. Post fixed effects — exploit “within-post” panel structure Post Debates

    Δ II. Post Selection as a Confounder
  29. r1 r2 r3 Decreasing attention and argument space Post Mean

    past position of responder as an instrument for their present reputation III. Instrument for Reputation Later (larger position) response lower persuasion probability Reputation no. of posters persuaded previously → ≈
  30. r1 r2 r3 Decreasing attention and argument space Post Addressed

    by controlling for each challenger’s observed response position Instrument exogeneity will be violated by post selection based on anticipated response position III. Instrument for Reputation
  31. Instrument: Mean past position of challenger before the present debate

    First-stage F-statistic > 3000 III. Instrument for Reputation
  32. Persuasion Instrument Unobserved Confounder Response Text Confounders of reputation and

    instrument must affect persuasion through the response text Motivation Reputation IV. Controlling for Unstructured Text
  33. Persuasion Instrument Unobserved Confounder Response Text Inherently unstructured and high

    dimensional How to “control for” text? Reputation IV. Controlling for Unstructured Text
  34. Persuasion Unobserved Confounder b a c d Response Text View

    text as four conceptual components Sufficient to control for a to block causal pathways How to “control for” text? Instrument Reputation IV. Controlling for Unstructured Text
  35. Unobserved Confounder Instrument Reputation Dimensionality Reduction (feature engineering, LDA, etc.)

    1. No guarantee against omitted variable bias 2. Estimates sensitive to researcher choices Persuasion b a c d Response Text IV. Controlling for Unstructured Text
  36. Unobserved Confounder Instrument Reputation My approach 1. Measure association strength

    between text, treatment, outcome, instrument 2. Combine associations using double machine learning [Chernozhukov et. al. 2016] Persuasion b a c d Response Text IV. Controlling for Unstructured Text
  37. Estimation and Inference Double Machine Learning [Chernozhukov et. al., 2016]

    • General “recipe” to derive Neyman-orthogonal moment condition, solve for empirically • Neutralizes “regularization bias” and “overfitting bias” that arise from ML model estimation • Valid asymptotic inference, fast convergence rates despite slowly-converging ML models O( n)
  38. Estimation and Inference 𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ])

    − β1 (rpu − 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 Persuasion Reputation Position Skill Instrument Response Text Post Fixed-Effect Solve Neyman-orthogonal moment for and β1 , β2 β3
  39. 𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ]) − β1 (rpu

    − 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 If ML models converge at least as — use deep ReLU neural networks [Farrell et. al., 2020] O(n1/4) Estimation and Inference Fast convergence O( n)
  40. Main Result +10 reputation units +26% persuasion rate increase over

    the platform average persuasion rate ( 3.5%) → ≈ *** 0.91 (0.08) Reputation (10 units) Skill (%) Outcome: Persuasion Treatment: Reputation *** 0.16 (0.02) Position (std. dev) *** -0.88 (0.08) Estimated Local Average Treatment Effect (LATE) Controls: Skill, position, response text, opinion fixed-effects percentage points
  41. Moderation consistent with theory of reputation as a “reference cue”

    [Bilancini & Boncinelli 2018] Content as a Moderator Resp. Length quantile 1 82% 89% Resp. Length quantile 4 Post length quantile 1 90% 83% Post length quantile 4 Reputation effect-share (vs skill)
  42. Displayed reputation is persuasive in dialogue emaadmanzoor.com/ethos/ Text can be

    useful for causal identification Conversation design: Content, framing, pacing — future work Key Finding What else have we learned? What can we do about it?