Meta-Analytic Method Meta-Analytic Methods for Cognitive Science CogSci Pre-Conference Workshop Molly Lewis 10 August 2016 11:00-12:00 

Meta-Analytic method Goal: Describe underlying true “population” effect Approach: Run many imperfect studies, then aggregate (1a) Collect studies (1b) Code (2) Aggregate ES = .6 

… Overview of the meta-analytic method 1. Calculate effect size for each study 2. Pool effect sizes across studies, weighting by sample size (+ look for moderators) Studies … … … Study characteristics Stats Effect Size + Var. d d_var 

Part 1: Effect Sizes 

Consider two experiments. Experiment 1: • 2AFC pointing • DV: number of points • 8 trials • 12 months Experiment 2: • 2AFC eye-tracking • DV: dwell time • 16 trials • 18 months How do we compare “success”? 

A common unit: Effect size Effect size as unit of analysis: Quantitative, scale-free measure of “success” 0.25 0.50 0.75 1.00 Prop. trials fixating novel object chance d Observed mean diff. between means standard dev. Effect Size = 

0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object Example: Mutual exclusivity effect size Where’s the dofa? Bion, et al. (2013) For 24 mo, mean proportion of trials fixating on novel object = .65 (SD = .13) chance .65 .13 d 

The two-sample case Control mean Observed mean d diff. between means standard dev. Effect Size = Plot from: http://www.realeducationreform.com/questions.html 

−1 0 1 2 3 Effect Size −1 0 1 2 3 Effect Size Effect size confidence interval n1 = 24 n2 = 24 

Interpreting Cohen’s d Size Description Cohen’s Intuition Psychological Example .2 “small” Diff. between the heights of 15 yo and 16 yo girls in the US Bouba-kiki effect in kids (~.15; Lammertink, et al. 2016) .5 “medium” Diff. between the heights of 14 yo and 18 yo girls. Cognitive behavioral therapy on anxiety (~ .4; (Belleville, et al., 2004) Sex difference in implicit math attitudes (~.5; Klein, et al., 2013) .8 “large” Diff. between the heights of 13 yo and 18 yo girls. Syntactic Priming (~.9; Mahowald, et al., under review) Mutual exclusivity ( ~1.0; Lewis & Frank, in prep) (Cohen, 1969) 

Interpreting Cohen’s d Cognitive (JEP:LMC) Social (JPSP) Psych. Science 0 1 2 0 1 2 3 0 1 2 3 0 1 2 3 absolute cohen's d M = .69 M = .19 M = .78 Estimating the Reproducibility of Psychological Science (OSF, Science, 2015) N = 97 Relatively “large” effects reported in cognitive psychology 

Effect sizes Prototype: Cohen’s d – Depends on aspect of design (e.g., within vs. between subject) – Many effect size metrics (Hedge’s g for small samples) – Can convert between ES metrics (compute.es package in R; AC Del Re, 2013) – Can calculate via different pieces of raw data Multiple Types: – the difference is between groups (t-test, d) – the relationship between variables (correlation, r) – the amount of variance accounted for by a factor (ANOVA, regression, f) – Associations between categorical variables (odds ratio, risk difference, log odds) – More generally, for any statistical test you conduct, can compute effect size (some more straight-forwardly than others) http://rpsychologist.com/d3/cohend/ 

Calculating Cohen’s d (Lakens, 2013) 

Part 2: Methods for pooling 

… Overview of the meta-analytic method 1. Calculate effect size for each study 2. Pool effect sizes across studies, weighting by sample size (+ look for moderators) Studies … … … Study characteristics Stats Effect Size + Var. d d_var 

Grand effect size −1.00 1.00 2.00 3.00 Effect size estimate 8. spiegel 7. markman 6. grassman 5. grassman 4. byers 3. bion 2. bion 1. bion 2011 1988 2010 2010 2009 2013 2013 2013 30 45 48 24 17 30 24 18 72 10 12 12 16 20 25 22 First author Year Age (m.) N Example: Mutual exclusivity meta-analysis Grand effect size estimate Pool effect sizes across studies, weighting by sample size 

Methods for pooling Analogous to logic in single study: – In a study, sample participants and pool to get estimate of effect in study (unweighted mean) – In meta-analysis, sample studies to get estimate of grand effect (weighted mean) Just as for models across participants, two models for pooling: – Fixed effect: One true population effect – Random effect: Random sample of many population effects, estimates mean 

Fixed Effect Model Slide from http://compare2what.blogspot.com/ (Keith Lohse, 2013) 

Fixed Effect Model 

Random Effect Model Random effect models recommended! 

(2) Byers-Heinlein & Werker (2009) Reported M = .12 t = 5.97 N = 16 SD = .079 baseline = 0 d = 1.49 Example meta-analysis with two studies (1) Markman and Wachtel (1988) Reported M = 4.9 t = 3.94 N = 10 baseline = 3 

(1) Calculate pooled and weighted estimate of population (2) Calculate variance of population estimate (3) Compute 95% confidence interval 

Testing for Moderators A good application for meta- analysis: – Testing for moderators requires more power than main effect (Button, et al., 2013) – Different levels may not be present within a single study – Theoretical progress In developmental research, a moderator of interest: age, vocabulary ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● r= 0.47 −1 0 1 2 20 30 40 50 60 Age (months) Cohen's d 

Meta-analytic methods in R Metafor: Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36(3), 1-48. URL: http://www.jstatsoft.org/v36/i03/ … Studies … … … Study characteristics Stats Effect Size + Var. d d_var 

Metalab 

Phenomena in MetaLab Prosody Communication Sounds Words 

Overall effect sizes Random effect models using metafor R package (Viechtbauer, 2010) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● Pointing and vocabulary Gaze following Online word recognition Concept−label advantage Mutual exclusivity Word segmentation Statistical sound learning Vowel discrimination (non−native) Vowel discrimination (native) Phonotactic learning IDS preference 0 1 2 3 Effect Size Phenomenon 

IDS preference Phonotactic learning Vowel discrimination (native) Vowel discrimination (non−native) Statistical sound learning Word segmentation Mutual exclusivity Concept−label advantage Online word recognition Gaze following Pointing and vocabulary −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 Age (years) Effect size (d)