Topic Sampling @ SPSA 2024

A talk at the 2024 meeting of the Southern Political Science Association in New Orleans.

Transcript

1. Topic Sampling Carlisle Rainey Associate Professor Florida State University crainey@fsu.edu Scott Clifford Associate Professor Texas A&M University scottclifford@tamu.edu http://www.carlislerainey.com/talks/ Slides and papers at How to Generalize from Particular Experiments to a Larger Collection of Possible Experiments
4. General Problem You want to test a general hypothesis. The Supreme Court can move public opinion. You can only test a particular hypothesis. The Supreme Court can move public opinion on af fi rmative action.
5. Even More General You can imagine many possible experiments to test your claim. 🤔 Pick one; 
 hope it doesn’t matter. 😬 Problem Solution??
7. As you may know, there has been some debate about allowing imported drugs from Canada lately. Republicans are more likely to favor allowing imported drugs from Canada, while Democrats are more likely to oppose allowing imported drugs from Canada. We’d like to know your opinion. Do you favor or oppose imported drugs from Canada? As you may know, there has been some debate about marijuana legaliza ti on lately. Democrats are more likely to favor marijuana legaliza ti on, while Republicans are more likely to oppose marijuana legaliza ti on. We’d like to know your opinion. Do you favor or oppose marijuana legaliza ti on? Example: Party Cues
9. Let’s abstract away from the details…
10. unusually large effect close to zero negative
11. “typical” treatment effect (across topics) 0.36
12. SD of effects give or take 0.25
13. The treatment effects are about 0.36 give or take 0.25 or so. (combined with a batch of estimates for particular topics) These are really hard problems.
14. Two Steps Enumerate the population of topics and take a random sample of about 25 to 50. Summarize the separate experiments using a hierarchical model. Step 2 Step 1
16. In a single-topic study, we’d fi t the model yi = α + δTi + ϵi yi[j] = αj + δj Ti[j] + ϵi[j] Here, with a sample of topics, we want the model
17. ( αj δj ) ∼ N ( μα μδ ), ( σ2 α ρσα σδ ρσα σδ σ2 δ ) yi[j] = αj + δj Ti[j] + ϵi[j] Here, with a sample of topics, we want the model
18. # fit model with REML library(lme4) fit_reml <- lmer(y ~ treatment + (1 + treatment | topic_id), data = sample) # fit model with Stan library(rstanarm) fit_stan <- stan_lmer(y ~ treatment + (1 + treatment | topic_id), data = sample)
26. How Good Are Our Summaries? Quartet Typical SD Truth Estimate 90% CI Truth Estimate 90% CI Heterogenous 0.36 0.38 [0.17, 0.42] 0.25 0.19 [0.11, 0.26] Homogenous 0.36 0.38 [0.34, 0.43] 0.05 0.04 [0.003, 0.09] Constant 0.36 0.38 [0.33, 0.42] 0.00 0.04 [0.004, 0.09] Two-Valued 0.36 0.31 [0.04, 0.57] 0.72 0.67 [0.51, 0.91]
27. Important Things Use about 25 to 50 topics. Increase total number of respondents by about 20% to 50%. Thing 2 Thing 1
28. drugs from Canada marijuana legalization
29. The Three Papers Conceptual overview of topic sampling with detailed empirical application. Coming out in Political Behavior. Technical details focused on the estimators (bias, RMSE, coverage). Paper 2 Paper 1 http://www.carlislerainey.com/research/ http://www.carlislerainey.com/talks/ Conceptual discussion of the statistical consequences of studying a single topic. Paper 3