AI4BC 2021 | Influence via Ethos: On the Persuasive Power of Reputation in Deliberation Online

15 minute talk at the AAAI 2021 workshop on AI for Behavioral Change (AI4BC): https://ai4bc.github.io/ai4bc21/

Video: https://youtu.be/7tSLWogEoWo
Website: http:/emaadmanzoor.com/ethos

Transcript

1. Influence via Ethos On the Persuasive Power of Reputation in Deliberation Online Emaad Manzoor
2. Joint work with George H. Chen Dokyun Lee Michael D. Smith emaadmanzoor.com/ethos/
3. Persuasion is 30% of the US GDP (McCloskey & Klamer 1995; Antioch 2013)
4. Vote Click Buy Mask
5. Most persuasion attempts are unsuccessful Some are successful but short-lived Few are successful with lasting effects
6. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974]
7. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974] Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc.
8. Information Processing Pathways [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974] Heuristic Pathway Low effort, short- lived attitude change Eg. Brand, affiliation, etc. — reputation Systematic Pathway High effort, long- lasting attitude change Eg. Content veracity, relevance, etc.
9. “Unthinking Compliance” “[…] the ever-accelerating pace and informational crush of modern life will make this particular form of unthinking compliance more and more prevalent in the future.” — [Cialdini 1984]
10. Need to Encourage Elaboration “The fact that individuals rely on automatic thinking has significant implications for understanding development challenges and for designing the best policies to overcome them.”— (World Bank Group 2015)
11. Q. Does reputation have persuasive power in the context of dialogue?
12. Text-based & Tech-Mediated Dialogue Managing Churn Direct Sales Organizational Deliberation
13. Challenge 1 of 2 Persuasion is rare, and rarely stated explicitly
14. Conversations from ChangeMyView reddit.com/r/changemyview/ 1 million+ conversations, 800,000+ members 20+ moderators enforce high-quality dialogue 2013 2019
15. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Post
16. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Responses Post
17. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Responses Post Debates
18. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈ Responses Post Debates
19. Conversations from ChangeMyView reddit.com/r/changemyview/ 2013 2019 Persuaded ( 3.5%) ≈ Not Persuaded Not Persuaded Responses Post
20. Poster Responder Explicit indicator of persuasion
21. Poster Explicit indicator of persuasion Prominent display of reputation Responder
22. Response Text High effort Displayed Reputation Low effort Information Processing on ChangeMyView [Petty & Cacioppo 1980; Chaiken 1980; Kahneman & Tversky 1974]
23. Challenge 2 of 2 Establishing causality with unstructured observational data
24. Identification Strategy I. Approximate challenger “skill” II. Address endogenous post selection III. Instrument for reputation IV. “Control for” challengers’ response text
25. Persuasion (Outcome) Response Quality I. “Skill” as a Confounder Causal Graph [Pearl, 2009] may/may not cause does not cause Responder Skill Reputation (Treatment) Skill is common cause of the treatment (reputation) and outcome (persuasion)
26. Accounts for all responder characteristics correlated with persuasion until the debate immediately prior Laplace-smoothed in practice [Manning et al 2008] skill(debate t) = no. posters persuaded before t t − 1 I. “Skill” as a Confounder
27. II. Post Selection as a Confounder Users endogenously select posts to respond to based on factors like: • Characteristics of the post (eg. post topic) • Characteristics of the poster (eg. malleability)
28. Post fixed effects — exploit “within-post” panel structure Post Debates Δ II. Post Selection as a Confounder
29. r1 r2 r3 Decreasing attention and argument space Post Mean past position of responder as an instrument for their present reputation III. Instrument for Reputation Later (larger position) response lower persuasion probability Reputation no. of posters persuaded previously → ≈
30. r1 r2 r3 Decreasing attention and argument space Post Addressed by controlling for each challenger’s observed response position Instrument exogeneity will be violated by post selection based on anticipated response position III. Instrument for Reputation
31. Instrument: Mean past position of challenger before the present debate First-stage F-statistic > 3000 III. Instrument for Reputation
32. Persuasion Instrument Unobserved Confounder Response Text Confounders of reputation and instrument must affect persuasion through the response text Motivation Reputation IV. Controlling for Unstructured Text
33. Persuasion Instrument Unobserved Confounder Response Text Inherently unstructured and high dimensional How to “control for” text? Reputation IV. Controlling for Unstructured Text
34. Persuasion Unobserved Confounder b a c d Response Text View text as four conceptual components Sufficient to control for a to block causal pathways How to “control for” text? Instrument Reputation IV. Controlling for Unstructured Text
35. Unobserved Confounder Instrument Reputation Dimensionality Reduction (feature engineering, LDA, etc.) 1. No guarantee against omitted variable bias 2. Estimates sensitive to researcher choices Persuasion b a c d Response Text IV. Controlling for Unstructured Text
36. Unobserved Confounder Instrument Reputation My approach 1. Measure association strength between text, treatment, outcome, instrument 2. Combine associations using double machine learning [Chernozhukov et. al. 2016] Persuasion b a c d Response Text IV. Controlling for Unstructured Text
37. Estimation and Inference Double Machine Learning [Chernozhukov et. al., 2016] • General “recipe” to derive Neyman-orthogonal moment condition, solve for empirically • Neutralizes “regularization bias” and “overfitting bias” that arise from ML model estimation • Valid asymptotic inference, fast convergence rates despite slowly-converging ML models O( n)
38. Estimation and Inference 𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ]) − β1 (rpu − 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 Persuasion Reputation Position Skill Instrument Response Text Post Fixed-Effect Solve Neyman-orthogonal moment for and β1 , β2 β3
39. 𝔼[((Ypu − 𝔼[Ypu |Xpu , τp ]) − β1 (rpu − 𝔼[rpu |Xpu , τp ])) −β2 (spu − 𝔼[spu |Xpu , τp ])) − β3 (tpu − 𝔼[tpu |Xpu , τp ])) × (Zpu − 𝔼[Zpu |Xpu , τp ])] = 0 If ML models converge at least as — use deep ReLU neural networks [Farrell et. al., 2020] O(n1/4) Estimation and Inference Fast convergence O( n)
40. Main Result +10 reputation units +26% persuasion rate increase over the platform average persuasion rate ( 3.5%) → ≈ *** 0.91 (0.08) Reputation (10 units) Skill (%) Outcome: Persuasion Treatment: Reputation *** 0.16 (0.02) Position (std. dev) *** -0.88 (0.08) Estimated Local Average Treatment Effect (LATE) Controls: Skill, position, response text, opinion fixed-effects percentage points
41. Moderation consistent with theory of reputation as a “reference cue” [Bilancini & Boncinelli 2018] Content as a Moderator Resp. Length quantile 1 82% 89% Resp. Length quantile 4 Post length quantile 1 90% 83% Post length quantile 4 Reputation effect-share (vs skill)
42. Displayed reputation is persuasive in dialogue emaadmanzoor.com/ethos/ Text can be useful for causal identification Conversation design: Content, framing, pacing — future work Key Finding What else have we learned? What can we do about it?