Share
Embed
AI4BC 2021 | Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online
15 minute talk at the AAAI 2021 workshop on AI for Behavioral Change (AI4BC): https://ai4bc.github.io/ai4bc21/
Video: https://youtu.be/7tSLWogEoWo
Website: http:/emaadmanzoor.com/ethos
Emaad Manzoor
February 11, 2021
Transcript
42 slides
-
Influence via Ethos On the Persuasive Power of Reputation in
Deliberation Online Emaad Manzoor -
Joint work with George H. Chen Dokyun Lee Michael D.
Smith emaadmanzoor.com/ethos/ -
Persuasion is 30% of the US GDP (McCloskey & Klamer
1995; Antioch 2013) -
Most persuasion attempts are unsuccessful Some are successful but short-lived
Few are successful with lasting effects -
Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman
& Tversky 1974] Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc. -
Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman
& Tversky 1974] Heuristic Pathway Low effort, short- lived attitude change Eg. Brand, affiliation, etc. — reputation Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc. -
“Unthinking Compliance” “[…] the ever-accelerating pace and informational crush of
modern life will make this particular form of unthinking compliance more and more prevalent in the future.” — [Cialdini 1984] -
Need to Encourage Elaboration “The fact that individuals rely on
automatic thinking has significant implications for understanding development challenges and for designing the best policies to overcome them.”— (World Bank Group 2015) -
Conversations from ChangeMyView reddit.com/r/changemyview/ 1 million+ conversations, 800,000+ members 20+
moderators enforce high-quality dialogue 2013 2019 -
Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈
Responses Post Debates -
Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈
Not Persuaded Not Persuaded Responses Post -
Response Text High effort Displayed Reputation Low effort Information Processing
on ChangeMyView [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974] -
Identification Strategy I. Approximate challenger “skill” II. Address endogenous post
selection III. Instrument for reputation IV. “Control for” challengers’ response text -
Persuasion (Outcome) Response Quality I. “Skill” as a Confounder Causal
Graph [Pearl, 2009] may/may not cause does not cause Responder Skill Reputation (Treatment) Skill is common cause of the treatment (reputation) and outcome (persuasion) -
Accounts for all responder characteristics correlated with persuasion until the
debate immediately prior Laplace-smoothed in practice [Manning et al 2008] skill(debate t) = no. posters persuaded before t t − 1 I. “Skill” as a Confounder -
II. Post Selection as a Confounder Users endogenously select posts
to respond to based on factors like: • Characteristics of the post (eg. post topic) • Characteristics of the poster (eg. malleability) -
Post fixed effects — exploit “within-post” panel structure Post Debates
Δ II. Post Selection as a Confounder -
r1 r2 r3 Decreasing attention and argument space Post Mean
past position of responder as an instrument for their present reputation III. Instrument for Reputation Later (larger position) response lower persuasion probability Reputation no. of posters persuaded previously → ≈ -
r1 r2 r3 Decreasing attention and argument space Post Addressed
by controlling for each challenger’s observed response position Instrument exogeneity will be violated by post selection based on anticipated response position III. Instrument for Reputation -
Instrument: Mean past position of challenger before the present debate
First-stage F-statistic > 3000 III. Instrument for Reputation -
Persuasion Instrument Unobserved Confounder Response Text Confounders of reputation and
instrument must affect persuasion through the response text Motivation Reputation IV. Controlling for Unstructured Text -
Persuasion Instrument Unobserved Confounder Response Text Inherently unstructured and high
dimensional How to “control for” text? Reputation IV. Controlling for Unstructured Text -
Persuasion Unobserved Confounder b a c d Response Text View
text as four conceptual components Sufficient to control for a to block causal pathways How to “control for” text? Instrument Reputation IV. Controlling for Unstructured Text -
Unobserved Confounder Instrument Reputation Dimensionality Reduction (feature engineering, LDA, etc.)
1. No guarantee against omitted variable bias 2. Estimates sensitive to researcher choices Persuasion b a c d Response Text IV. Controlling for Unstructured Text -
Unobserved Confounder Instrument Reputation My approach 1. Measure association strength
between text, treatment, outcome, instrument 2. Combine associations using double machine learning [Chernozhukov et. al. 2016] Persuasion b a c d Response Text IV. Controlling for Unstructured Text -
Estimation and Inference Double Machine Learning [Chernozhukov et. al., 2016]
• General “recipe” to derive Neyman-orthogonal moment condition, solve for empirically • Neutralizes “regularization bias” and “overfitting bias” that arise from ML model estimation • Valid asymptotic inference, fast convergence rates despite slowly-converging ML models O( n) -
Estimation and Inference 𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ])
− β1 (rpu − 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 Persuasion Reputation Position Skill Instrument Response Text Post Fixed-Effect Solve Neyman-orthogonal moment for and β1 , β2 β3 -
𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ]) − β1 (rpu
− 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 If ML models converge at least as — use deep ReLU neural networks [Farrell et. al., 2020] O(n1/4) Estimation and Inference Fast convergence O( n) -
Main Result +10 reputation units +26% persuasion rate increase over
the platform average persuasion rate ( 3.5%) → ≈ *** 0.91 (0.08) Reputation (10 units) Skill (%) Outcome: Persuasion Treatment: Reputation *** 0.16 (0.02) Position (std. dev) *** -0.88 (0.08) Estimated Local Average Treatment Effect (LATE) Controls: Skill, position, response text, opinion fixed-effects percentage points -
Moderation consistent with theory of reputation as a “reference cue”
[Bilancini & Boncinelli 2018] Content as a Moderator Resp. Length quantile 1 82% 89% Resp. Length quantile 4 Post length quantile 1 90% 83% Post length quantile 4 Reputation effect-share (vs skill) -
Displayed reputation is persuasive in dialogue emaadmanzoor.com/ethos/ Text can be
useful for causal identification Conversation design: Content, framing, pacing — future work Key Finding What else have we learned? What can we do about it?