Carlisle Rainey
March 02, 2023
260

# Topic Sampling

At talk at the 2023 Elections, Public Opinion, and Political Behavior Conference at FSU.

March 02, 2023

## Transcript

1. Topic Sampling
Carlisle Rainey
Associate Professor

Florida State University

[email protected]
Scott Clifford
Associate Professor

University of Houston

[email protected]

Thomas Leeper
Senior Visiting Fellow

London School of Economics

[email protected]

http://www.carlislerainey.com/talk
Slides and papers at
How to Generalize from Particular Experiments to a
Larger Collection of Possible Experiments

2. General Problem
You want to test a general hypothesis.
The Supreme Court can move public opinion.
You can only test a particular hypothesis.
The Supreme Court can move public opinion on af
fi
rmative action.

3. Even More General
You can imagine many possible
experiments to test your claim.
Pick one. Hope it doesn’t matter.

4. As you may know, there has been some debate about allowing imported drugs from

Republicans are more likely to favor allowing imported drugs from Canada, while
Democrats are more likely to oppose allowing imported drugs from Canada.

We’d like to know your opinion. Do you favor or oppose imported drugs from Canada?
As you may know, there has been some debate about marijuana legaliza
ti
on lately.

Democrats are more likely to favor marijuana legaliza
ti
on, while Republicans are more
likely to oppose marijuana legaliza
ti
on.

We’d like to know your opinion. Do you favor or oppose marijuana legaliza
ti
on?
Example: Party Cues

5. Let’s abstract away
from the details…

6. unusually

large effect
close to zero
negative

7. “typical” treatment effect

(across topics)
0.36

8. SD of effects
give or take

0.25

9. The treatment effects are

give or take 0.25 or so.
(combined with a batch of estimates
for particular topics)
These are really
hard problems.

10. Two Steps
Enumerate the population of topics and
take a random sample of about 25 to 50.
Summarize the separate experiments
using a hierarchical model.
Step 2
Step 1

11. In a single-topic study, we’d
fi
t the model
yi
= α + δTi
+ ϵi
yi[j]
= αj
+ δj
Ti[j]
+ ϵi[j]
Here, with a sample of topics, we want the model

12. (
αj
δj
) ∼ N (
μα
μδ
),
(
σ2
α
ρσα
σδ
ρσα
σδ
σ2
δ
)
yi[j]
= αj
+ δj
Ti[j]
+ ϵi[j]
Here, with a sample of topics, we want the model

13. # fit model with REML

library(lme4)

fit_reml <- lmer(y ~ treatment + (1 + treatment | topic_id),

data = sample)
# fit model with Stan

library(rstanarm)

fit_stan <- stan_lmer(y ~ treatment + (1 + treatment | topic_id),

data = sample)

14. How Good Are Our
Summaries?
Quartet
Typical SD
Truth Estimate 90% CI Truth Estimate 90% CI
Heterogenous 0.36 0.38 [0.17, 0.42] 0.25 0.19 [0.11, 0.26]
Homogenous 0.36 0.38 [0.34, 0.43] 0.05 0.04 [0.003, 0.09]
Constant 0.36 0.38 [0.33, 0.42] 0.00 0.04 [0.004, 0.09]
Two-Valued 0.36 0.31 [0.04, 0.57] 0.72 0.67 [0.51, 0.91]

15. Important Things
Use about 25 to 50 topics.
Increase total number of respondents by
about 20% to 50%.
Thing 2
Thing 1

16. drugs from

marijuana

legalization

17. The Two Papers
Conceptual overview of topic sampling with detailed
empirical application. Coming out in Political Behavior.
Technical details focused on the estimators (bias,
RMSE, coverage) and power calculations.
Paper 2
Paper 1
http://www.carlislerainey.com/research/
http://www.carlislerainey.com/talk