Slide 1
Slide 1 text
Experiments in the digital age
Matthew J. Salganik
Department of Sociology, Princeton University
& Cornell Tech
IC2S2
June 25, 2016
Slide 2
Slide 2 text
Bit by Bit:
Social Research in the Digital Age
Social Scientists ←→ Data Scientists
Slide 3
Slide 3 text
Isn’t computational social science a fad?
Slide 4
Slide 4 text
Fenn and Raskino (2008)
Slide 5
Slide 5 text
Found data vs designed data
Slide 6
Slide 6 text
Observing behavior
Asking questions
Running experiments
Creating mass collaboration
Slide 7
Slide 7 text
Observing behavior
Asking questions
Running experiments
Creating mass collaboration
Slide 8
Slide 8 text
No content
Slide 9
Slide 9 text
No content
Slide 10
Slide 10 text
What’s the same?
recruiting participants
randomization treatment
delivering treatment and control
measuring outcomes
Slide 11
Slide 11 text
What’s the same?
recruiting participants
randomization treatment
delivering treatment and control
measuring outcomes
What’s different?
Fully digital experiment leads to zero variable cost data.
Slide 12
Slide 12 text
What’s the same?
recruiting participants
randomization treatment
delivering treatment and control
measuring outcomes
What’s different?
Fully digital experiment leads to zero variable cost data.
Constraint on size is not cost, it is ethics.
Slide 13
Slide 13 text
Lab
experiments
Field
experiments
Slide 14
Slide 14 text
Analog
Digital
Lab
experiments
Field
experiments
Slide 15
Slide 15 text
Moving beyond simple experiments
Slide 16
Slide 16 text
Research Article
The Constructive, Destructive,
and Reconstructive Power of
Social Norms
P. Wesley Schultz,1 Jessica M. Nolan,2 Robert B. Cialdini,3 Noah J. Goldstein,3 and Vladas Griskevicius3
1California State University, San Marcos; 2University of Arkansas; and 3Arizona State University
ABSTRACT—Despite a long tradition of effectiveness in
laboratory tests, normative messages have had mixed
success in changing behavior in field contexts, with some
studies showing boomerang effects. To test a theoretical
account of this inconsistency, we conducted a field exper-
iment in which normative messages were used to promote
household energy conservation. As predicted, a descriptive
normative message detailing average neighborhood usage
produced either desirable energy savings or the undesir-
able boomerang effect, depending on whether households
were already consuming at a low or high rate. Also as
predicted, adding an injunctive message (conveying social
approval or disapproval) eliminated the boomerang
effect. The results offer an explanation for the mixed
Tabanico, & Rendo
´n, in press). Such social-norms marketing
campaigns have emerged as an alternative to more traditional
approaches (e.g., information campaigns, moral exhortation, fear-
inducing messages) designed to reduce undesirable conduct
(Donaldson, Graham, Piccinin, & Hansen, 1995).
The rationale for the social-norms marketing approach is
based on two consistent findings: (a) The majority of individuals
overestimate the prevalence of many undesirable behaviors,
such as alcohol use among peers (e.g., Borsari & Carey, 2003;
Prentice & Miller, 1993), and (b) individuals use their percep-
tions of peer norms as a standard against which to compare their
own behaviors (e.g., Baer, Stacy, & Larimer, 1991; Clapp &
McDonell, 2000; Perkins & Berkowitz, 1986). Social-norms
marketing campaigns seek to reduce the occurrence of delete-
PSYCHOLOGICAL SCIENCE
Slide 17
Slide 17 text
Figures from Allcott (2011)
Slide 18
Slide 18 text
No content
Slide 19
Slide 19 text
No content
Slide 20
Slide 20 text
Figures from Allcott (2011)
Slide 21
Slide 21 text
Figures from Allcott (2011)
Slide 22
Slide 22 text
No content
Slide 23
Slide 23 text
If you want to move beyond simple experiments:
Heterogeneity of treatment effects
External validity
Mechanisms
Slide 24
Slide 24 text
Making it happen
Slide 25
Slide 25 text
No content
Slide 26
Slide 26 text
No content
Slide 27
Slide 27 text
No content
Slide 28
Slide 28 text
No content
Slide 29
Slide 29 text
Drive variable cost to 0
Slide 30
Slide 30 text
No content
Slide 31
Slide 31 text
No content
Slide 32
Slide 32 text
Main sources of variable costs:
staff time
participant payment
Slide 33
Slide 33 text
Main sources of variable costs:
staff time
participant payment
Solutions:
Automation (your experiments should run while you sleep)
Design enjoyable experiments
Slide 34
Slide 34 text
Music Lab: a zero variable cost experiment with Peter Dodds and
Duncan Watts
Slide 35
Slide 35 text
No content
Slide 36
Slide 36 text
No content
Slide 37
Slide 37 text
No content
Slide 38
Slide 38 text
No content
Slide 39
Slide 39 text
Subjects
condition
Independent
World 1
condition
World 8
World
Social influence
Slide 40
Slide 40 text
(a) Less social influence (b) More social influence
Slide 41
Slide 41 text
Results:
More social influence leads to more unpredictability
Slide 42
Slide 42 text
Results:
More social influence leads to more unpredictability
You can predict failure but you can’t predict success
Slide 43
Slide 43 text
No content
Slide 44
Slide 44 text
Zero variable cost is a means not an end.
Slide 45
Slide 45 text
Important difference between:
zero variable cost and zero variable cost to you
Slide 46
Slide 46 text
Ethics: The 3 R’s
Slide 47
Slide 47 text
http://www.kittenwar.com/kittens/69827/
Slide 48
Slide 48 text
The Principles of Humane Experimental Technique
by Russell and Burch (1959)
Replace
Refine
Reduce
Slide 49
Slide 49 text
No content
Slide 50
Slide 50 text
No content
Slide 51
Slide 51 text
No content
Slide 52
Slide 52 text
Replace experiments with less invasive methods
Slide 53
Slide 53 text
Replace experiments with less invasive methods
Slide 54
Slide 54 text
Refine treatments to make them less harmful
Slide 55
Slide 55 text
Refine treatments to make them less harmful
Rather than blocking posts, they could have boosted posts
Slide 56
Slide 56 text
Reduce the number of participants
Difference-in-difference estimator rather than a
difference-of-means estimator.
Would have cut the required sample size, perhaps by half
(based on Deng et al. (2013) & Xie and Aurisset (2016)).
Slide 57
Slide 57 text
When should we care about reducing the number of participants?
Slide 58
Slide 58 text
When should we care about reducing the number of participants?
1. uncertainty about whether the experiment will cause harm
2. the experiment was large
3. participation was not voluntary
Slide 59
Slide 59 text
No content
Slide 60
Slide 60 text
With great power there must also come
great responsibility
Slide 61
Slide 61 text
The 3 Rs shows that humane methods can be an opportunity:
Slide 62
Slide 62 text
The 3 Rs shows that humane methods can be an opportunity:
potentially more efficient than standard methods
Slide 63
Slide 63 text
The 3 Rs shows that humane methods can be an opportunity:
potentially more efficient than standard methods
stimulates interesting research (e.g., differential privacy)
Slide 64
Slide 64 text
1. Observing behavior
2. Asking questions
3. Running experiments
4. Creating mass collaboration
Slide 65
Slide 65 text
Bit by Bit:
Social Research in the Digital Age
Social Scientists ←→ Data Scientists
Slide 66
Slide 66 text
Bit by Bit:
Social Research in the Digital Age
Social Scientists ←→ Data Scientists
Open Review begins August 18
http://bitbybitbook.com