Topic Sampling

At talk at the 2023 Elections, Public Opinion, and Political Behavior Conference at FSU.

Transcript

1. Topic Sampling
   Carlisle Rainey
   Associate Professor
   
   
   Florida State University
   
   
   crainey@fsu.edu
   Scott Clifford
   Associate Professor
   
   
   University of Houston
   
   
   sclifford@uh.edu
   
   
   Thomas Leeper
   Senior Visiting Fellow
   
   
   London School of Economics
   
   
   thosjleeper@gmail.com
   
   
   http://www.carlislerainey.com/talk
   Slides and papers at
   How to Generalize from Particular Experiments to a
   Larger Collection of Possible Experiments
   

   View Slide

2. View Slide

3. View Slide

4. General Problem
   You want to test a general hypothesis.
   The Supreme Court can move public opinion.
   You can only test a particular hypothesis.
   The Supreme Court can move public opinion on af
   fi
   rmative action.
   

   View Slide

5. Even More General
   You can imagine many possible
   experiments to test your claim.
   Pick one. Hope it doesn’t matter.
   

   View Slide

6. As you may know, there has been some debate about allowing imported drugs from
   Canada lately.
   
   
   Republicans are more likely to favor allowing imported drugs from Canada, while
   Democrats are more likely to oppose allowing imported drugs from Canada.
   
   
   We’d like to know your opinion. Do you favor or oppose imported drugs from Canada?
   As you may know, there has been some debate about marijuana legaliza
   ti
   on lately.
   
   
   Democrats are more likely to favor marijuana legaliza
   ti
   on, while Republicans are more
   likely to oppose marijuana legaliza
   ti
   on.
   
   
   We’d like to know your opinion. Do you favor or oppose marijuana legaliza
   ti
   on?
   Example: Party Cues
   

   View Slide

7. View Slide

8. Let’s abstract away
   from the details…
   

   View Slide

9. unusually
   
   
   large effect
   close to zero
   negative
   

   View Slide

10. “typical” treatment effect
    
    
    (across topics)
    0.36
    

    View Slide

11. SD of effects
    give or take
    
    
    0.25
    

    View Slide

12. The treatment effects are
    about 0.36
    
    
    give or take 0.25 or so.
    (combined with a batch of estimates
    for particular topics)
    These are really
    hard problems.
    

    View Slide

13. Two Steps
    Enumerate the population of topics and
    take a random sample of about 25 to 50.
    Summarize the separate experiments
    using a hierarchical model.
    Step 2
    Step 1
    

    View Slide

14. View Slide

15. In a single-topic study, we’d
    fi
    t the model
    yi
    = α + δTi
    + ϵi
    yi[j]
    = αj
    + δj
    Ti[j]
    + ϵi[j]
    Here, with a sample of topics, we want the model
    

    View Slide

16. (
    αj
    δj
    ) ∼ N (
    μα
    μδ
    ),
    (
    σ2
    α
    ρσα
    σδ
    ρσα
    σδ
    σ2
    δ
    )
    yi[j]
    = αj
    + δj
    Ti[j]
    + ϵi[j]
    Here, with a sample of topics, we want the model
    

    View Slide

17. # fit model with REML
    
    
    library(lme4)
    
    
    fit_reml <- lmer(y ~ treatment + (1 + treatment | topic_id),
    
    
    data = sample)
    # fit model with Stan
    
    
    library(rstanarm)
    
    
    fit_stan <- stan_lmer(y ~ treatment + (1 + treatment | topic_id),
    
    
    data = sample)
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. How Good Are Our
    Summaries?
    Quartet
    Typical SD
    Truth Estimate 90% CI Truth Estimate 90% CI
    Heterogenous 0.36 0.38 [0.17, 0.42] 0.25 0.19 [0.11, 0.26]
    Homogenous 0.36 0.38 [0.34, 0.43] 0.05 0.04 [0.003, 0.09]
    Constant 0.36 0.38 [0.33, 0.42] 0.00 0.04 [0.004, 0.09]
    Two-Valued 0.36 0.31 [0.04, 0.57] 0.72 0.67 [0.51, 0.91]
    

    View Slide

26. Important Things
    Use about 25 to 50 topics.
    Increase total number of respondents by
    about 20% to 50%.
    Thing 2
    Thing 1
    

    View Slide

27. drugs from
    
    
    Canada
    marijuana
    
    
    legalization
    

    View Slide

28. The Two Papers
    Conceptual overview of topic sampling with detailed
    empirical application. Coming out in Political Behavior.
    Technical details focused on the estimators (bias,
    RMSE, coverage) and power calculations.
    Paper 2
    Paper 1
    http://www.carlislerainey.com/research/
    http://www.carlislerainey.com/talk
    

    View Slide