Testing Hypotheses of No Meaningful Effect

Transcript

1. Testing Hypotheses of
   No Meaningful Effect
   Carlisle Rainey
   

   View Slide

2. I'd like to convince you of
   three things
   

   View Slide

3. Important
   1
   

   View Slide

4. Our arguments are
   not compelling
   2
   

   View Slide

5. Our arguments can
   be compelling
   3
   

   View Slide

6. Important
   Step 1:
   

   View Slide

7. Hypotheses of no meaningful effect are
   crucial
   to complete evaluation of theories
   

   View Slide

8. Interaction
   

   View Slide

9. Social heterogeneity increases the
   number of parties, but only when electoral
   institutions are sufficiently permissive.
   “
   ”
   Clark and Golder (2006)
   

   View Slide

10. Adjudication
    

    View Slide

11. View Slide

12. View Slide

13. How often do these
    examples occur?
    

    View Slide

14. 30%
    

    View Slide

15. Our arguments are
    not compelling
    Step 2:
    

    View Slide

16. No American president since FDR has won
    a second term when the unemployment
    rate topped 7.2 percent...
    Obama must defy that
    trend to keep his job.
    “
    ”
    New York Times
    June 1, 2011
    “
    ”
    

    View Slide

17. View Slide

18. View Slide

19. The nation's unemployment
    status by itself is not going to
    affect Obama's.
    “
    ”
    Seth Masket
    June 2, 2011
    

    View Slide

20. My critique
    

    View Slide

21. Rule out implausible
    relationships
    

    View Slide

22. View Slide

23. View Slide

24. View Slide

25. View Slide

26. View Slide

27. View Slide

28. Insignificance can't
    be used to argue for
    “no effect.”
    

    View Slide

29. But doesn't everyone
    already know that?
    “
    ”
    A skeptic
    

    View Slide

30. Political scientists draw
    strong conclusions
    from insignificance.
    

    View Slide

31. Recessions have no effect
    on whether a democracy is
    consolidated.
    “
    ”
    Svolik (2008)
    

    View Slide

32. Our arguments
    can be compelling
    Step 3:
    

    View Slide

33. just like any
    other hypothesis
    

    View Slide

34. Argue against
    relationships inconsistent
    with the hypothesis
    

    View Slide

35. Define
    substantively
    meaningful
    Step 1:
    

    View Slide

36. Check the
    90% CI
    Step 2:
    

    View Slide

37. Only negativity about the respondent’s
    preferred candidate should have a significant
    demobilizing effect on voter turnout.
    “
    ”
    Krupnikov (2011)
    

    View Slide

38. What's a meaningful effect?
    1% 3% 5% 7% 9%
    

    View Slide

39. The demobilizing effect
    could be as large as
    9%
    

    View Slide

40. Those are the
    three things
    

    View Slide

41. Important
    1
    

    View Slide

42. Our arguments are
    not compelling
    2
    

    View Slide

43. Our arguments can
    be compelling
    3
    

    View Slide

44. But what should
    I do about this?
    “
    ”
    You
    

    View Slide

45. Go read the details
    crain.co/nme
    1
    

    View Slide

46. Think about your
    own work.
    2
    

    View Slide

47. Keep in mind when
    reviewing others' work
    3
    

    View Slide

48. View Slide