$30 off During Our Annual Pro Sale. View Details »

Topic Sampling

Topic Sampling

At talk at the 2023 Elections, Public Opinion, and Political Behavior Conference at FSU.

Carlisle Rainey

March 02, 2023
Tweet

More Decks by Carlisle Rainey

Other Decks in Science

Transcript

  1. Topic Sampling
    Carlisle Rainey
    Associate Professor


    Florida State University


    [email protected]
    Scott Clifford
    Associate Professor


    University of Houston


    [email protected]


    Thomas Leeper
    Senior Visiting Fellow


    London School of Economics


    [email protected]


    http://www.carlislerainey.com/talk
    Slides and papers at
    How to Generalize from Particular Experiments to a
    Larger Collection of Possible Experiments

    View Slide

  2. View Slide

  3. View Slide

  4. General Problem
    You want to test a general hypothesis.
    The Supreme Court can move public opinion.
    You can only test a particular hypothesis.
    The Supreme Court can move public opinion on af
    fi
    rmative action.

    View Slide

  5. Even More General
    You can imagine many possible
    experiments to test your claim.
    Pick one. Hope it doesn’t matter.

    View Slide

  6. As you may know, there has been some debate about allowing imported drugs from
    Canada lately.


    Republicans are more likely to favor allowing imported drugs from Canada, while
    Democrats are more likely to oppose allowing imported drugs from Canada.


    We’d like to know your opinion. Do you favor or oppose imported drugs from Canada?
    As you may know, there has been some debate about marijuana legaliza
    ti
    on lately.


    Democrats are more likely to favor marijuana legaliza
    ti
    on, while Republicans are more
    likely to oppose marijuana legaliza
    ti
    on.


    We’d like to know your opinion. Do you favor or oppose marijuana legaliza
    ti
    on?
    Example: Party Cues

    View Slide

  7. View Slide

  8. Let’s abstract away
    from the details…

    View Slide

  9. unusually


    large effect
    close to zero
    negative

    View Slide

  10. “typical” treatment effect


    (across topics)
    0.36

    View Slide

  11. SD of effects
    give or take


    0.25

    View Slide

  12. The treatment effects are
    about 0.36


    give or take 0.25 or so.
    (combined with a batch of estimates
    for particular topics)
    These are really
    hard problems.

    View Slide

  13. Two Steps
    Enumerate the population of topics and
    take a random sample of about 25 to 50.
    Summarize the separate experiments
    using a hierarchical model.
    Step 2
    Step 1

    View Slide

  14. View Slide

  15. In a single-topic study, we’d
    fi
    t the model
    yi
    = α + δTi
    + ϵi
    yi[j]
    = αj
    + δj
    Ti[j]
    + ϵi[j]
    Here, with a sample of topics, we want the model

    View Slide

  16. (
    αj
    δj
    ) ∼ N (
    μα
    μδ
    ),
    (
    σ2
    α
    ρσα
    σδ
    ρσα
    σδ
    σ2
    δ
    )
    yi[j]
    = αj
    + δj
    Ti[j]
    + ϵi[j]
    Here, with a sample of topics, we want the model

    View Slide

  17. # fit model with REML


    library(lme4)


    fit_reml <- lmer(y ~ treatment + (1 + treatment | topic_id),


    data = sample)
    # fit model with Stan


    library(rstanarm)


    fit_stan <- stan_lmer(y ~ treatment + (1 + treatment | topic_id),


    data = sample)

    View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. View Slide

  22. View Slide

  23. View Slide

  24. View Slide

  25. How Good Are Our
    Summaries?
    Quartet
    Typical SD
    Truth Estimate 90% CI Truth Estimate 90% CI
    Heterogenous 0.36 0.38 [0.17, 0.42] 0.25 0.19 [0.11, 0.26]
    Homogenous 0.36 0.38 [0.34, 0.43] 0.05 0.04 [0.003, 0.09]
    Constant 0.36 0.38 [0.33, 0.42] 0.00 0.04 [0.004, 0.09]
    Two-Valued 0.36 0.31 [0.04, 0.57] 0.72 0.67 [0.51, 0.91]

    View Slide

  26. Important Things
    Use about 25 to 50 topics.
    Increase total number of respondents by
    about 20% to 50%.
    Thing 2
    Thing 1

    View Slide

  27. drugs from


    Canada
    marijuana


    legalization

    View Slide

  28. The Two Papers
    Conceptual overview of topic sampling with detailed
    empirical application. Coming out in Political Behavior.
    Technical details focused on the estimators (bias,
    RMSE, coverage) and power calculations.
    Paper 2
    Paper 1
    http://www.carlislerainey.com/research/
    http://www.carlislerainey.com/talk

    View Slide