Causal Inference 2022 Week 10

Transcript

1. CAUSAL INFERENCE Case study William Lowe Hertie School Data Science Lab 2022-04-25
2. PLAN 1 Controversy and the research base Estimand warmup Case study: Fryer Selection trouble What the obvious comparisons identify Principal strati cation You can’t always get what you want, but sometimes you can get what you need Two approaches to not knowing something Behind the scenes
3. CONTROVERSY 2
4. AND WEAK RESEARCH 3 We conducted a set of regression models to test whether a person fatally shot was more likely to be Black (or Hispanic) than White in certain types of FOIS. [...] is provides an estimate of racial disparities across all shootings. (Johnson et al., ) Q → What are they estimating here? → Is that quantity a guide to the causal question? (Knox & Mummolo, a; PNAS, )
5. AND WEAK RESEARCH 3 We conducted a set of regression models to test whether a person fatally shot was more likely to be Black (or Hispanic) than White in certain types of FOIS. [...] is provides an estimate of racial disparities across all shootings. (Johnson et al., ) Q → What are they estimating here? → Is that quantity a guide to the causal question? (Knox & Mummolo, a; PNAS, ) One surprising claim from the paper P(shot minority civilian, white o cer, X) < P(shot minority civilian, minority o cer, X)
6. AND WEAK RESEARCH 3 We conducted a set of regression models to test whether a person fatally shot was more likely to be Black (or Hispanic) than White in certain types of FOIS. [...] is provides an estimate of racial disparities across all shootings. (Johnson et al., ) Q → What are they estimating here? → Is that quantity a guide to the causal question? (Knox & Mummolo, a; PNAS, ) One surprising claim from the paper P(shot minority civilian, white o cer, X) < P(shot minority civilian, minority o cer, X) B T is is the rst quantity P(minority civilian shot, white o cer, X) P(shot white o cer, X) P(minority civilian white o cer, X) But we have no idea what the denominator is!
7. CASE STUDY 4 (Fryer, )
8. CASE STUDY 5
9. CASE STUDY 6 P → Behaviour / suspicion (U) causes stops (M) and, conditional on M, possible use of force Y. → Questioning (M) always generates a line in the administrative data set D M Y U Last week’s notation changed to match Knox et al. ( )
11. EFFECTS 8 N Following the notation we introduced for mediation, in the paper: Mi = Mi(Di) = d Mi(d)I[Di = d] (med) Yi(Di , Mi(Di)) = d m Yi(d, m)I[Di = d, Mi = m] (out) I In (mediator) the indicator variable I picks out which Mi is realized, on the basis of i’s actual race In (outcome) the indicator variable picks out which outcome is realized, on the basis of i’s actual race, and whether i was actually stopped N Assume we can estimate the average di erence in Y by race among the stopped ∆ = E[Y D = , M = ] − E[Y D = , M = ] Is this the causal e ect of race?
12. TYPES 9 P Labelling Di = if i is white Di = if i is minority we can de ne types of person Mi( ) = , Mi( ) = (never stop) Mi( ) = , Mi( ) = (always stop) Mi( ) = , Mi( ) = (stop only if minority) Mi( ) = , Mi( ) = (stop only if white) As always, we cannot observe which strata each stopped person is in
13. WILL THE REAL ATE PLEASE STAND UP? 10 S e (unknown) proportion of the population in each race and stratum S is P(d, S) = P(Di = d race , Mi( ) = m, Mi( ) = m′ stratum ) For example, the proportion of minorities who are stopped because they are a minority is P( , S = stop if minority) = P(Di = , Mi( ) = , Mi( ) = ) S ere’s an local ATE for every race and stratum: LATES = E[Yi( , Mi( )) − Yi( , Mi( )) d, S] A So the overall ATE is a weighted average ATE = E[Y( , M( )) − Y( , M( ))] = D d S s LATEsP(d, S = s)
14. WILL THE REAL ATE PLEASE STAND UP? 10 S e (unknown) proportion of the population in each race and stratum S is P(d, S) = P(Di = d race , Mi( ) = m, Mi( ) = m′ stratum ) For example, the proportion of minorities who are stopped because they are a minority is P( , S = stop if minority) = P(Di = , Mi( ) = , Mi( ) = ) S ere’s an local ATE for every race and stratum: LATES = E[Yi( , Mi( )) − Yi( , Mi( )) d, S] A So the overall ATE is a weighted average ATE = E[Y( , M( )) − Y( , M( ))] = D d S s LATEsP(d, S = s) P We always see S = always stop, but → We never see S = never stop, so we’ve no idea what its P(d, S) is → Whether we see everyone else may depend on their race We can’t estimate this!
15. OTHER EFFECTS 11 Previously we’ve considered person speci c estimands, e.g. the ATT → ATE is a weighted average of ATT and ATC → Weight is the proportion of people treated is is not so hard to do because we observe who was treated How about the e ect of race on the stopped? Here, things get tricky...
16. OTHER EFFECTS 11 Previously we’ve considered person speci c estimands, e.g. the ATT → ATE is a weighted average of ATT and ATC → Weight is the proportion of people treated is is not so hard to do because we observe who was treated How about the e ect of race on the stopped? Here, things get tricky... F ATEM= = E[Y( , M( )) − Y( , M( )) M = ] ATTM= = E[Y( , M( )) − Y( , M( )) D = , M = ] CDEM= = E[Y( , ) − Y( , ) M = ] What’s the di erence? → ATEM= lets the stopping process to be determined by the person’s actual race → ATTM= lets the stopping process to be determined by the person’s minority status. → CDEM= imagines that the stop happened regardless of actual race is should remind you of NDEs and CDEs...
17. THE NAIVE APPROACH AGAIN 12 It is straightforward (but extremely tedious) to prove that the bias in the naive approach E[ ˆ ∆] = ATEM− + difference in effect size across strata E[Y( , ) − Y( , ) always stop] − E[Y( , ) − Y( , ) stop if minority] × racial stop proportion among stopped minorities P(M( ) = D = , M = )P(D = M = ) − difference in force across strata E[Y( , ) always stop] − E[Y( , ) stop if minority] × proportion of racially stopped minority P(M( ) = D = , M = ) Under reasonable assumptions, the bias is always negative
18. WHAT TO DO? 13 D More data won’t work, di erent data will Intuitively, if we can estimate P(D, M) then we have information about M = and D. → Back out principal strata sizes → Reconstruct the ATE S Assume a range of values for P(M( ) = D = , M = ) and check
19. COMMENTARY 14 Q is work came about because two puzzling results (Fryer, ) that fueled public arguments about police violence in the US → Minorities are stopped disproportionately (relative to their population proportions) → Force is (relatively) not much more likely to be used against minorities than against whites So is there race bias in policing or not?
20. COMMENTARY 14 Q is work came about because two puzzling results (Fryer, ) that fueled public arguments about police violence in the US → Minorities are stopped disproportionately (relative to their population proportions) → Force is (relatively) not much more likely to be used against minorities than against whites So is there race bias in policing or not? D Stop-question-frisk records are made available by NYPD (and some other large police forces). F Does it makes sense to think of race as a ‘treatment’? More on that next time. D e DAG you saw last week shows clear collider bias in a mediation structure, complicated by post-treatment sample selection. → Bias in simple comparisons is almost guaranteed Usefully summarised in (Bronner & Mithani, )
21. COMMENTARY 15 T What does unbiased look like? → OK, not the ATE then! → Controlled or ‘natural’ e ect? → How to identify this? How to de ne e ects when post-treatment events restructure the population? (Slough, ), e.g. → ‘truncation by death’ (Zhang & Rubin, ) and the SACE T → Identify what information is missing → Figure out what to do since we don’t have it
22. COMMENTARY 15 T What does unbiased look like? → OK, not the ATE then! → Controlled or ‘natural’ e ect? → How to identify this? How to de ne e ects when post-treatment events restructure the population? (Slough, ), e.g. → ‘truncation by death’ (Zhang & Rubin, ) and the SACE T → Identify what information is missing → Figure out what to do since we don’t have it A → Econometrica (non-blind reviews): Love the paper. No. → American Journal of Political Science: Is policing really politics? (Editor: yes, shut up Reviewer )
23. COMMENTARY 15 T What does unbiased look like? → OK, not the ATE then! → Controlled or ‘natural’ e ect? → How to identify this? How to de ne e ects when post-treatment events restructure the population? (Slough, ), e.g. → ‘truncation by death’ (Zhang & Rubin, ) and the SACE T → Identify what information is missing → Figure out what to do since we don’t have it A → Econometrica (non-blind reviews): Love the paper. No. → American Journal of Political Science: Is policing really politics? (Editor: yes, shut up Reviewer ) Won’t somebody think of the economists? A → Immediate pushback: Estimand? ‘well ackshually’ → an acrimonious public debate Usefully summarised on the Gelman blog [link] and in the Boston Review (Hu, )
24. POLICY IMPACT 16 Do the Police read the APSR? → No, but they do read F → Di erent estimands, e.g. Risk ratios (Zhao et al., ) → Di erent applications, e.g. ‘outcome tests’: (Ayres, ; Knox & Mummolo, b) → Using machine learning to compute bounds (Duarte et al., ) → Counterexamples? (Gaebler et al., n.d.)
25. REFERENCES 17 Ayres, I. ( ). Pervasive prejudice? Unconventional evidence of race and gender discrimination. Univ. of Chicago Press. Bronner, L., & Mithani, J. ( , June ). Why statistics don’t capture the full extent of the systemic bias in policing ( vethirtyeight.com). Duarte, G., Finkelstein, N., Knox, D., Mummolo, J., & Shpitser, I. ( , September ). An automated approach to causal inference in discrete settings ([cs, stat] No. . ). arXiv. Fryer, R. G. ( ). An empirical analysis of racial di erences in police use of force. Journal of Political Economy, ( ), – . Gaebler, J., Cai, W., & Basse, G. (n.d.). Deconstructing claims of post-treatment bias in observational studies of discrimination. Hu, L. ( , May ). Race, policing, and the limits of social science. Boston Review. Johnson, D. J., Tress, T., Burkel, N., Taylor, C., & Cesario, J. ( ). O cer characteristics and racial disparities in fatal o cer-involved shootings. Proceedings of the National Academy of Sciences, ( ), – . Knox, D., Lowe, W., & Mummolo, J. ( ). Administrative records mask racially biased policing. American Political Science Review, ( ), – .
26. REFERENCES 18 Knox, D., & Mummolo, J. ( a). Making inferences about racial disparities in police violence. Proceedings of the National Academy of Sciences, ( ), – . Knox, D., & Mummolo, J. ( b). Toward a general causal framework for the study of racial bias in policing. Journal of Political Institutions and Political Economy, ( ), – . PNAS. ( ). Retraction for Johnson et al., O cer characteristics and racial disparities in fatal o cer-involved shootings. Proceedings of the National Academy of Sciences, ( ), – . Slough, T. ( ). Phantom counterfactuals. American Journal of Political Science. Zhang, J. L., & Rubin, D. B. ( ). Estimation of causal e ects via principal strati cation when some outcomes are truncated by “death”. Journal of Educational and Behavioral Statistics, ( ), – . Zhao, Q., Keele, L. J., Small, D. S., & Jo e, M. M. ( , June ). A note on post-treatment selection in studying racial discrimination in policing.
27. LOOKING FORWARD... 19 Apparently not...
28. ETHICAL IMPLICATIONS 20 T How much are future (not yet existing) lives (and their quality) worth relative to current ones? P → e e ect of an action on the existing population (or its quality of life) t years in the future → e e ect on populations that will exist (or the quality of their lives) t years in the future Some of these people will not exist (or will exist di erently) depending on ‘treatment’s ese denominator problems are an issue in meta-ethics and policy!
29. PLAN 21 Controversy and the research base Estimand warmup Case study: Fryer Selection trouble What the obvious comparisons identify Principal strati cation You can’t always get what you want, but sometimes you can get what you need Two approaches to not knowing something Behind the scenes