Jake Hofman
April 26, 2019
520

Modeling Social Data, Lecture 12: Causality and Experiments

April 26, 2019

Transcript

2. Prediction Seeing: Make a forecast, leaving the world as it

is vs. Causation Doing: Anticipate what will happen when you make a change in the world
3. Prediction Seeing: Make a forecast, leaving the world as it

is (seeing my neighbor with an umbrella might predict rain) vs. Causation Doing: Anticipate what will happen when you make a change in the world (but handing my neighbor an umbrella doesn’t cause rain)
4. “Causes of effects” It’s tempting to ask “what caused Y”,

e.g. ◦ What makes an email spam? ◦ What caused my kid to get sick? ◦ Why did the stock market drop? This is ”reverse causal inference”, and is generally quite hard John Stuart Mill (1843)
5. “Effects of causes” Alternatively, we can ask “what happens if

we do X?”, e.g. ◦ How does education impact future earnings? ◦ What is the effect of advertising on sales? ◦ How does hospitalization affect health? This is “forward causal inference”: still hard, but less contentious! John Stuart Mill (1843)
6. Example: Hospitalization on health What’s wrong with estimating this model

from observational data? Health tomorrow Hospital visit today Effect? Arrow means “X causes Y”
7. Confounds The effect and cause might be confounded by a

common cause, and be changing together as a result Health tomorrow Hospital visit today Effect? Health today Dashed circle means “unobserved”
8. Confounds If we only get to observe them changing together,

we can’t estimate the effect of hospitalization changing alone Health tomorrow Hospital visit today Effect? Health today
9. A counterfactual (what-if) definition What if you would have acted

differently? E.g., how does the health of a hospitalized patient compare to their health if they would have stayed home? We only get to observe one of these outcomes, which is the fundamental problem of causal inference How does this differ from an observational estimate?
10. Observational estimates Let’s say all sick people in our dataset

went to the hospital today, and healthy people stayed home The observed difference in health tomorrow is: Δobs = (Sick and went to hospital) – (Healthy and stayed home)
11. Observational estimates Let’s say all sick people in our dataset

went to the hospital today, and healthy people stayed home The observed difference in health tomorrow is: Δobs = [(Sick and went to hospital) – (Sick if stayed home)] + [(Sick if stayed home) - (Healthy and stayed home)]
12. Selection bias Let’s say all sick people in our dataset

went to the hospital today, and healthy people stayed home The observed difference in health tomorrow is: Δobs = [(Sick and went to hospital) – (Sick if stayed home)] + [(Sick if stayed home) - (Healthy and stayed home)] Causal effect Selection bias (Baseline difference between those who opted in to the treatment and those who didn’t)
13. Basic identity of causal inference Let’s say all sick people

in our dataset went to the hospital today, and healthy people stayed home The observed difference in health tomorrow is: Observed difference = Causal effect – Selection bias Selection bias is likely negative here, making the observed difference an underestimate of the causal effect
14. Simpson’s paradox Selection bias can be so large that observational

and causal estimates give opposite effects (e.g., going to hospitals makes you less healthy) http://vudlab.com/simpsons
15. Simpson’s paradox So which is right, the aggregated or the

partitioned? It depends on the causal mechanism https://en.wikipedia.org/wiki/Simpson%27s_paradox
16. Simpson’s paradox So which is right, the aggregated or the

partitioned? It depends on the causal mechanism Morgan and Winship (2015) 108 Chapter 4. Models of Causal Exposure and Identiﬁcation Criteria Motivation SAT Rejected Admitted Applicants to a Hypothetical College Figure 4.2 Simulation of conditional dependence within values of a collider variable.
17. “To find out what happens when you change something, it

is necessary to change it.” -GEORGE BOX

19. Counterfactuals To isolate the causal effect, we have to change

one and only one thing (hospital visits), and compare outcomes + vs (what happened) Reality (what would have happened) Counterfactual
20. The ideal causal estimate CLONE EACH PERSON SEND ONE COPY

TO THE HOSPITAL, MAKE THE OTHER STAY HOME MEASURE THE DIFFERENCE IN HEALTH BETWEEN THE COPIES
21. But this might be confounded for various reasons---e.g., Mark has

a different diet than Scott
22. Counterfactuals We never get to observe what would have happened

if we did something else, so we have to estimate it + vs (what happened) Reality (what would have happened) Counterfactual
23. Random assignment We can use randomization to create two groups

that differ only in which treatment they receive, restoring symmetry + World 1 World 2 Heads Tails
24. Random assignment We can use randomization to create two groups

that differ only in which treatment they receive, restoring symmetry + World 1 World 2 Heads Tails
25. Random assignment We can use randomization to create two groups

that differ only in which treatment they receive, restoring symmetry + World 1 World 2
26. Basic identity of causal inference The observed difference is now

the causal effect: Observed difference = Causal effect – Selection bias = Causal effect Selection bias is zero, since there’s no difference, on average, between those who were hospitalized and those who weren’t
27. Hospital visit today Random assignment Random assignment determines the treatment

independent of any confounds Health tomorrow Effect? Health today Coin flip Double lines mean “intervention”

29. Experiments: Caveats / limitations Random assignment is the “gold standard”

for causal inference, but it has some limitations: ◦ Randomization often isn’t feasible and/or ethical ◦ Experiments are costly in terms of time and money ◦ It’s difficult to create convincing parallel worlds ◦ Effects in the lab can differ from real-world effects ◦ Inevitably people deviate from their random assignments
30. Validity of experiments INTERNAL VALIDITY Could anything other than the

treatment (i.e. a confound) have produced this outcome? Was the study double-blind? Did doctors give the experimental drug to some especially sick patients (breaking randomization) hoping that it would save them? Or treat patients differently based on whether they got the drug or not? EXTERNAL VALIDITY Do the results of the experiment hold in settings we care about? Would this medication be just as effective outside of a clinical trial, when usage is less rigorously monitored or when tried on a different population of patients? Slide thanks to Andrew Mao
31. Expanding the experiment design space Complexity, Realism Size, Scale Duration,

Participation Physical labs • Longer periods of time • Fewer constraints on location • More samples of data • Large-scale social interaction • Realistic vs. abstract, simple tasks • More precise instrumentation A software-based “virtual lab” with online participants Slide thanks to Andrew Mao

39. Natural experiments Sometimes we get lucky and nature effectively runs

experiments for us, e.g.: ◦ As-if random: People are randomly exposed to water sources ◦ Instrumental variables: A lottery influences military service ◦ Discontinuities: Star ratings get arbitrarily rounded ◦ Difference in differences: Minimum wage changes in just one state
40. Natural experiments Sometimes we get lucky and nature effectively runs

experiments for us, e.g.: ◦ As-if random: People are randomly exposed to water sources ◦ Instrumental variables: A lottery influences military service ◦ Discontinuities: Star ratings get arbitrarily rounded ◦ Difference in differences: Minimum wage changes in just one state Experiments happen all the time, we just have to notice them
41. As-if random Idea: Nature randomly assigns conditions Example: People are

randomly exposed to water sources (Snow, 1854) http://bit.ly/johnsnowmap
42. Instrumental variables Idea: An instrument independently shifts the distribution of

a treatment Example: A lottery influences military service (Angrist, 1990) Military service Future earnings Effect? Confounds Lottery
43. Figure 4: Average Revenue around Discontinuous Changes in Rating Notes:

Each restaurant’s log revenue is de-meaned to normalize a restaurant’s average log revenue to zero. Normalized log revenues are then averaged within bins based on how far the restaurant’s rating is from a rounding threshold in that quarter. The graph plots average log revenue as a function of how far the rating is from a rounding threshold. All points with a positive (negative) distance from a discontinuity are rounded up (down). Regression discontinuities Idea: Things change around an arbitrarily chosen threshold Example: Star ratings get arbitrarily rounded (Luca, 2011) http://bit.ly/yelpstars
44. Difference in differences Idea: Compare differences after a sudden change

with trends in a control group Example: Minimum wage changes in just one state (Card & Krueger, 1994) http://stats.stackexchange.com/a/125266
45. Natural experiments: Caveats Natural experiments are great, but: ◦ Good

natural experiments are hard to find ◦ They rely on many (untestable) assumptions ◦ The treated population may not be the one of interest
46. Closing thoughts Large-scale observational data is useful for building predictive

models of a static world
47. Closing thoughts But without appropriate random variation, it’s hard to

predict what happens when you change something in the world