Statistical Rethinking 2023 - Lecture 06

Course: https://github.com/rmcelreath/stat_rethinking_2023

Transcript

1. Statistical Rethinking
   6. Good & Bad Controls
   2023
   

   View Slide

2. Avoid Being Clever At All Costs
   Being clever: unreliable, opaque
   Given a causal model, can use logic
   to derive implications
   Others can use same logic to verify
   & challenge your work
   Better than clever
   

   View Slide

3. X Z Y
   e Pipe
   X Z Y
   e Fork
   X Z Y
   e Collider
   X Z Y
   e Descendant
   A
   

   View Slide

4. X Z Y
   e Pipe
   X Z Y
   e Fork
   X Z Y
   e Collider
   X and Y associated
   unless stratify by Z
   X and Y associated
   unless stratify by Z
   X and Y not associated
   unless stratify by Z
   

   View Slide

5. X Y
   U
   treatment outcome
   confound
   

   View Slide

6. X Y
   U
   treatment outcome
   confound
   RANDOMIZE!
   R
   

   View Slide

7. X Y
   U
   treatment outcome
   confound
   randomize?
   R
   

   View Slide

8. Causal inking
   In an experiment, we cut causes of
   the treatment
   We randomize (we try at least)
   So how does causal inference
   without randomization ever work?
   Is there a statistical procedure that
   mimics randomization?
   X Y
   U
   Without randomization
   X Y
   U
   With randomization
   do(X)
   

   View Slide

9. Causal inking
   Is there a statistical procedure
   that mimics randomization? X Y
   U
   Without randomization
   P(Y|do(X)) = P(Y|?)
   do(X) means intervene on X
   Can analyze causal model to
   nd answer (if it exists)
   X Y
   U
   With randomization
   do(X)
   

   View Slide

10. Example: Simple Confound
    X Y
    U
    

    View Slide

11. Example: Simple Confound
    X Y
    U Non-causal path
    X Y
    Close the fork!
    Condition on U
    

    View Slide

12. Example: Simple Confound
    X Y
    U Non-causal path
    X Y
    Close the fork!
    Condition on U
    P(Y|do(X)) =
    ∑
    U
    P(Y|X, U)P(U) = E
    U
    P(Y|X, U)
    “ e distribution of Y, strati ed by X and U,
    averaged over the distribution of U.”
    

    View Slide

13. e causal e ect of X on Y is not (in
    general) the coe cient relating X to Y
    It is the distribution of Y when we change
    X, averaged over the distributions of the
    control variables (here U)
    P(Y|do(X)) =
    ∑
    U
    P(Y|X, U)P(U) = E
    U
    P(Y|X, U)
    “ e distribution of Y, strati ed by X and U,
    averaged over the distribution of U.”
    X Y
    U
    

    View Slide

14. Marginal E ects Example
    B G
    C
    cheetahs
    baboons gazelle
    

    View Slide

15. B G
    C
    cheetahs present
    

    View Slide

16. B G
    C
    B G
    C
    cheetahs absent
    Causal e ect of baboons depends upon distribution of cheetahs
    cheetahs present
    

    View Slide

17. do-calculus
    For DAGs, rules for nding
    P(Y|do(X)) known as do-calculus
    do-calculus says what is possible
    to say before picking functions
    Justi es graphical analysis
    Do calculus, not too much, mostly graphs
    

    View Slide

18. do-calculus
    do-calculus is worst case:
    additional assumptions o en
    allow stronger inference
    do-calculus is best case:
    if inference possible by do-
    calculus, does not depend on
    special assumptions
    Judea Pearl, father of do-calculus (1966)
    

    View Slide

19. Backdoor Criterion
    Backdoor Criterion is a shortcut to
    applying (some) results of do-calculus
    Can be performed with your eyeballs
    

    View Slide

20. Backdoor Criterion: Rule to nd a set of
    variables to stratify by to yield P(Y|do(X))
    (1) Identify all paths connecting the
    treatment (X) to the outcome (Y)
    (2) Paths with arrows entering X are
    backdoor paths (non-causal paths)
    (3) Find adjustment set that closes/blocks
    all backdoor paths
    

    View Slide

21. (1) Identify all paths connecting the
    treatment (X) to the outcome (Y)
    

    View Slide

22. (2) Paths with arrows entering X are
    backdoor paths (confounding paths)
    

    View Slide

23. (3) Find a set of control variables that
    close/block all backdoor paths
    Block the pipe: X ⫫ U | Z
    Z “knows” all of the association
    between X,Y that is due to U
    

    View Slide

24. (3) Find a set of control variables that
    close/block all backdoor paths
    P(Y|do(X)) =
    ∑
    z
    P(Y|X, Z)P(Z = z)
    μ
    i
    = α + β
    X
    X
    i
    + β
    Z
    Z
    i
    Y
    i
    ∼ Normal(μ
    i
    , σ)
    Block the pipe: X ⫫ U | Z
    

    View Slide

25. # simulate confounded Y
    N b_XY b_UY b_UZ b_ZX set.seed(10)
    U Z X Y d

    View Slide

26. # ignore U,Z
    m_YX alist(
    Y ~ dnorm( mu , sigma ),
    mu a ~ dnorm( 0 , 1 ),
    b_XY ~ dnorm( 0 , 1 ),
    sigma ~ dexp( 1 )
    ), data=d )
    # stratify by Z
    m_YXZ alist(
    Y ~ dnorm( mu , sigma ),
    mu a ~ dnorm( 0 , 1 ),
    c(b_XY,b_Z) ~ dnorm( 0 , 1 ),
    sigma ~ dexp( 1 )
    ), data=d )
    post post2 dens(post$b_XY,lwd=3,col=1,xlab="posterior
    b_XY",xlim=c(-0.3,0.3))
    dens(post2$b_XY,lwd=3,col=2,add=TRUE)
    Y|X,Z
    Y|X
    -0.3 -0.2 -0.1 0.0 0.1 0.2 0.3
    0 2 4 6 8
    posterior b_XY
    Density
    

    View Slide

27. # ignore U,Z
    m_YX alist(
    Y ~ dnorm( mu , sigma ),
    mu a ~ dnorm( 0 , 1 ),
    b_XY ~ dnorm( 0 , 1 ),
    sigma ~ dexp( 1 )
    ), data=d )
    # stratify by Z
    m_YXZ alist(
    Y ~ dnorm( mu , sigma ),
    mu a ~ dnorm( 0 , 1 ),
    c(b_XY,b_Z) ~ dnorm( 0 , 1 ),
    sigma ~ dexp( 1 )
    ), data=d )
    post post2 dens(post$b_XY,lwd=3,col=1,xlab="posterior
    b_XY",xlim=c(-0.3,0.3))
    dens(post2$b_XY,lwd=3,col=2,add=TRUE)
    > precis(m_YXZ)
    mean sd 5.5% 94.5%
    a -0.32 0.09 -0.47 -0.18
    b_XY -0.01 0.08 -0.13 0.11
    b_Z 0.24 0.11 0.06 0.42
    sigma 1.18 0.06 1.08 1.27
    Coe cient on Z means
    nothing. “Table 2 Fallacy”
    

    View Slide

28. X Y
    Z B
    List all the paths connecting X and Y.
    Which need to be closed to estimate
    e ect of X on Y?
    A
    C
    

    View Slide

29. View Slide

30. X Y
    Z B
    A
    C
    P(Y|do(X))
    X Y
    Z B
    A
    C
    

    View Slide

31. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    

    View Slide

32. Causal path, open
    X Y
    Z B
    A
    C
    

    View Slide

33. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    

    View Slide

34. Backdoor path, open
    Close with C
    X Y
    Z B
    A
    C
    

    View Slide

35. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C
    

    View Slide

36. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C
    

    View Slide

37. Backdoor path, open
    Close with Z
    X Y
    Z B
    A
    C
    

    View Slide

38. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C Z
    

    View Slide

39. Backdoor path, opened by Z
    A or B to close
    X Y
    Z B
    A
    C
    

    View Slide

40. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C Z
    A,B
    

    View Slide

41. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C Z
    A,B
    

    View Slide

42. Backdoor path, open
    Close with A or Z
    X Y
    Z B
    A
    C
    

    View Slide

43. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    C Z
    A,B
    

    View Slide

44. X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    X Y
    Z B
    A
    C
    Minimum adjustment set:
    C, Z, and either A or B
    (B is better choice)
    C Z
    A,B
    

    View Slide

45. www.dagitty.net
    

    View Slide

46. G P
    U
    C
    grandparent
    education
    parent
    education
    child
    education
    unobserved
    confound
    

    View Slide

47. U
    G P
    C
    P is a mediator
    Pipe: G –> P –> C
    

    View Slide

48. G P
    U
    C
    P is a collider
    Pipe: G –> P –> C
    Fork: C P
    

    View Slide

49. G P
    U
    C
    Can estimate total
    e ect of G on C
    Cannot estimate
    direct e ect
    G P
    U
    C
    C
    i
    ∼ Normal(μ
    i
    , σ)
    μ
    i
    = α + β
    G
    G
    i
    C
    i
    ∼ Normal(μ
    i
    , σ)
    μ
    i
    = α + β
    G
    G
    i
    + β
    P
    P
    i
    

    View Slide

50. Backdoor Criterion
    do-calc more than backdoors &
    adjustment sets
    Full Luxury Bayes: use all variables, but
    in separate sub-models instead of single
    regression
    do-calc less demanding: nds relevant
    variables; saves us having to make some
    assumptions; not always a regression
    

    View Slide

51. PAUSE
    

    View Slide

52. Good & Bad Controls
    “Control” variable: Variable introduced to
    an analysis so that a causal estimate is
    possible
    Common wrong heuristics for choosing
    control variables
    Anything in the spreadsheet YOLO!
    Any variables not highly collinear
    Any pre-treatment measurement (baseline)
    

    View Slide

53. X
    Cinelli, Forney, Pearl 2021 A Crash Course in Good and Bad Controls
    Y
    

    View Slide

54. X Y
    u v
    Z
    Cinelli, Forney, Pearl 2021 A Crash Course in Good and Bad Controls
    unobserved
    

    View Slide

55. X Y
    u v
    Z
    Cinelli, Forney, Pearl 2021 A Crash Course in Good and Bad Controls
    Health
    person 1
    Health
    person 2
    Hobbies
    person 1
    Hobbies
    person 2
    Friends
    

    View Slide

56. X Y
    u v
    Z
    (1) List the paths
    

    View Slide

57. X Y
    u v
    Z
    (1) List the paths
    X → Y
    

    View Slide

58. X Y
    u v
    Z
    (1) List the paths
    X → Y
    X ← u → Z ← v → Y
    

    View Slide

59. X Y
    u v
    Z
    (1) List the paths
    X → Y
    X ← u → Z ← v → Y
    frontdoor & open
    backdoor & closed
    (2) Find backdoors
    

    View Slide

60. X Y
    u v
    Z
    (1) List the paths
    X → Y
    X ← u → Z ← v → Y
    frontdoor & open
    backdoor & closed
    (2) Find backdoors
    

    View Slide

61. X Y
    u v
    Z
    (1) List the paths
    X → Y
    X ← u → Z ← v → Y
    frontdoor & open
    backdoor & closed
    (2) Find backdoors (3) Close backdoors
    

    View Slide

62. X Y
    u v
    Z
    What happens if you stratify by
    Z?
    Opens the backdoor path
    Z could be a pre-treatment
    variable
    Not safe to always control pre-
    treatment measurements
    Health
    person 1
    Health
    person 2
    Hobbies
    person 1
    Hobbies
    person 2
    Friends
    

    View Slide

63. X Y
    Z
    u
    

    View Slide

64. X Y
    Z
    u
    Win
    lottery
    Lifespan
    Happiness
    Contextual
    confounds
    

    View Slide

65. X Y
    Z
    u
    X → Z → Y
    X → Z ← u → Y
    No backdoor, no need
    to control for Z
    

    View Slide

66. X Y
    Z
    u
    f X u Z Y bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    1 1
    1 1
    

    View Slide

67. X Y
    Z
    u
    -1.0 0.0 0.5 1.0 1.5 2.0
    0.0 1.0 2.0
    posterior mean
    Density
    f X u Z Y bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    Y ~ X
    correct
    Y ~ X + Z
    wrong
    1 1
    1 1
    

    View Slide

68. X Y
    Z
    u
    Y ~ X
    correct
    Y ~ X + Z
    wrong
    Change bZY to zero
    f X u Z Y bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    -1.0 -0.5 0.0 0.5
    0.0 1.0 2.0
    posterior mean
    Density
    1 0
    1 1
    

    View Slide

69. X Y
    Z
    u
    X → Z → Y
    X → Z ← u → Y
    No backdoor, no need
    to control for Z
    Controlling for Z biases
    treatment estimate X
    Controlling for Z opens biasing
    path through u
    Can estimate e ect of X; Cannot
    estimate mediation e ect Z
    Win
    lottery
    Lifespan
    Happiness
    

    View Slide

70. X Y
    Z
    u
    Win
    lottery
    Lifespan
    Happiness
    Montgomery et al 2018 How Conditioning on Posttreatment Variables Can Ruin Your Experiment
    Regression with confounds
    Regression with post-
    treatment variables
    

    View Slide

71. X Y
    Z
    Do not touch the collider!
    

    View Slide

72. X Y
    Z u
    Colliders not always so obvious
    

    View Slide

73. X Y
    Z u
    education
    values
    income
    family
    

    View Slide

74. X Y
    Z
    Case-control bias
    (selection on outcome)
    

    View Slide

75. X Y
    Z
    Education Occupation
    Income
    Case-control bias
    (selection on outcome)
    

    View Slide

76. X Y Z
    f X Y Z bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    0.0 0.5 1.0 1.5
    0 1 2 3 4 5
    posterior mean
    Density
    Y ~ X
    correct
    Y ~ X + Z
    wrong
    1 1
    Case-control bias
    (selection on outcome)
    

    View Slide

77. X Y
    Z
    “Precision parasite”
    No backdoors
    But still not good to
    condition on Z
    

    View Slide

78. X Y
    Z
    “Precision parasite”
    f Z X Y bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    0.6 0.8 1.0 1.2 1.4
    0 1 2 3 4
    posterior mean
    Density
    Y ~ X
    correct
    Y ~ X + Z
    wrong
    

    View Slide

79. X Y
    Z
    u
    “Bias ampli cation”
    X and Y confounded by u
    Something truly awful happens
    when we add Z
    

    View Slide

80. f Z u X Y bX bXZ return( c(bX,bXZ) )
    }
    sim dens( sim[1,] , lwd=3 , xlab="posterior mean" )
    dens( sim[2,] , lwd=3 , col=2 , add=TRUE )
    X Y
    Z
    u
    -0.5 0.0 0.5 1.0
    0 1 2 3 4 5
    posterior mean
    Density
    Y ~ X
    biased
    Y ~ X + Z
    more bias
    true value
    is zero
    

    View Slide

81. X Y
    Z
    u
    -0.5 0.0 0.5 1.0
    0 1 2 3 4 5
    posterior mean
    Density
    Y ~ X
    biased
    Y ~ X + Z
    more bias
    true value
    is zero
    WHY?
    Covariation X & Y requires
    variation in their causes
    Within each level of Z, less
    variation in X
    Confound u relatively more
    important within each Z
    

    View Slide

82. -5 0 5 10
    -2 0 2 4
    X
    Y
    X Y
    Z
    u
    0
    + + +
    n Z u X Y Z = 0 Z = 1
    

    View Slide

83. X Y
    Z
    u
    education
    occupation income
    regional/cultural
    factors
    

    View Slide

84. Good & Bad Controls
    “Control” variable: Variable
    introduced to an analysis so that a
    causal estimate is possible
    Heuristics fail — adding control
    variables can be worse than omitting
    Make assumptions explicit
    MODEL
    ALL THE
    THINGS
    

    View Slide

85. Course Schedule
    Week 1 Bayesian inference Chapters 1, 2, 3
    Week 2 Linear models & Causal Inference Chapter 4
    Week 3 Causes, Confounds & Colliders Chapters 5 & 6
    Week 4 Over tting / MCMC Chapters 7, 8, 9
    Week 5 Generalized Linear Models Chapters 10, 11
    Week 6 Integers & Other Monsters Chapters 11 & 12
    Week 7 Multilevel models I Chapter 13
    Week 8 Multilevel models II Chapter 14
    Week 9 Measurement & Missingness Chapter 15
    Week 10 Generalized Linear Madness Chapter 16
    https://github.com/rmcelreath/stat_rethinking_2023
    

    View Slide

86. View Slide

87. BONUS
    

    View Slide

88. TABLE 2-ESTIMATED PROBIT MODELS
    FOR THE USE OF A SCREEN
    Finals
    Preliminaries blind blind
    (1) (2) (3)
    (Proportion female),_ 2.744 3.120 0.490
    (3.265) (3.271) (1.163)
    [0.006] [0.004] [0.011]
    (Proportion of orchestra -26.46 -28.13 -9.467
    personnel with <6 (7.314) (8.459) (2.787)
    years tenure),- 1 [-0.058] [-0.039] [-0.207]
    "Big Five" orchestra 0.367
    (0.452)
    [0.001]
    pseudo R2 0.178 0.193 0.050
    Number of observations 294 294 434
    who attende
    the names of
    For the preli
    advancemen
    round. Anoth
    the semifina
    of who won
    we recorded
    section, prin
    dition was h
    individual ha
    semifinal or
    occurs when
    above some
    compete in a
    recorded wh
    

    View Slide

89. Table 2 Fallacy
    Not all coe cients are causal
    e ects
    Statistical model designed to
    identify X –> Y will not also
    identify e ects of control
    variables
    Table 2 is dangerous
    Westreich & Greenland 2013 e Table 2 Fallacy
    724 THE AMERICAN
    EC
    TABLE 2-ESTIMATED PROBIT MODELS
    FOR THE USE OF A SCREEN
    Finals
    Preliminaries blind blind
    (1) (2) (3)
    (Proportion female),_ 2.744 3.120 0.490
    (3.265) (3.271) (1.163)
    [0.006] [0.004] [0.011]
    (Proportion of orchestra -26.46 -28.13 -9.467
    personnel with <6 (7.314) (8.459) (2.787)
    years tenure),- 1 [-0.058] [-0.039] [-0.207]
    "Big Five" orchestra 0.367
    (0.452)
    [0.001]
    pseudo R2 0.178 0.193 0.050
    Number of observations 294 294 434
    Notes: The dependent variable is 1 if the orchestra adopts a
    screen, 0 otherwise. Huber standard errors (with orchestra
    random effects) are in parentheses. All specifications in-
    clude a constant. Changes in probabilities are in brackets.
    

    View Slide

90. A
    X Y
    S
    Westreich & Greenland 2013 e Table 2 Fallacy
    Stroke
    HIV
    Smoking
    Age
    

    View Slide

91. View Slide

92. Use Backdoor Criterion
    A
    X Y
    S
    

    View Slide

93. Use Backdoor Criterion
    A
    X Y
    S X Y
    

    View Slide

94. Use Backdoor Criterion
    A
    X Y
    S X Y
    X Y
    S
    

    View Slide

95. Use Backdoor Criterion
    A
    X Y
    S X Y
    X Y
    S
    A
    X Y
    

    View Slide

96. Use Backdoor Criterion
    A
    X Y
    S X Y
    X Y
    S
    A
    X Y
    A
    X Y
    S
    

    View Slide

97. Use Backdoor Criterion
    A
    X Y
    S X Y
    X Y
    S
    A
    X Y
    A
    X Y
    S
    

    View Slide

98. Y
    i
    ∼ Normal(μ
    i
    , σ)
    μ
    i
    = α + β
    X
    X
    i
    + β
    S
    S
    i
    + β
    A
    A
    i
    A
    X Y
    S
    

    View Slide

99. A
    X Y
    S
    Confounded by A
    and S
    Unconditional
    X
    

    View Slide

100. Coe cient for X:
     E ect of X on Y
     (still must
     marginalize!)
     A
     X Y
     S
     Confounded by A
     and S
     A
     X Y
     S
     Unconditional Conditional on A and S
     X
     

     View Slide

101. A
     X Y
     S
     E ect of S
     confounded by A
     Unconditional
     S
     

     View Slide

102. Coe cient for S:
     Direct e ect of S on Y
     A
     X Y
     S
     E ect of S
     confounded by A
     Unconditional Conditional on A and X
     A
     X Y
     S
     S
     

     View Slide

103. A
     X Y
     S
     Total causal e ect
     of A on Y ows
     through all paths
     Unconditional
     A
     

     View Slide

104. Coe cient for A:
     Direct e ect of A on Y
     A
     X Y
     S
     Total causal e ect
     of A on Y ows
     through all paths
     Unconditional Conditional on X and S
     A
     X Y
     S
     A
     

     View Slide

105. A
     X Y
     S
     Stroke
     HIV
     Smoking
     Age
     u
     unobserved
     confound
     

     View Slide

106. Table 2 Fallacy
     Not all coe cients created equal
     So do not present them as equal
     Options:
     Do not present control coe cients
     Give explicit interpretation of each
     No interpretation without causal
     representation
     A
     X Y
     S
     u
     

     View Slide

107. View Slide