BUCLD 14 November 2015 MetaLab: A tool for power analysis and experimental planning in developmental research Molly Lewis, Mika Braginsky, Christina Bergmann, Sho Tsuji, Alex Cristia, and Michael C. Frank 

Synthesizing language development research Kuhl (2004) 

Limitations of current syntheses 1) Categorical description 2) Lack of variability 3) Cross-domain comparisons difficult effect size 0 1 2 3 4 5 6 Time 

A practical reason for quantitative synthesis Findings from published research are not always reproducible (Ioannidis, 2005; Open Science Collaboration, 2013, 2015) – Maybe sample sizes too small? – But, effect sizes are typically unknown, so appropriate sample sizes difficult to determine prospectively Particularly problematic for language development research – Small sample sizes – Small effect sizes 

A Solution: Meta-analysis Aggregate across studies using quantitative methods Treat each study as a sample from a population of studies Get point estimate of the true effect size with measure of certainty By aggregating, can do more than with single study: – Look for moderators (age, method, etc.) – More precise estimate of effect size 

Conducting a meta-analysis 1. Select phenomenon of interest 2. Select papers via sampling strategy 3. Code statistics reported in papers into 4. Calculate effect sizes 5. Pool effect sizes across studies, weighting by sample size 

0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object Example: Mutual exclusivity meta-analysis Where’s the dofa? Bion, et al. (2013) For 24 mo, mean proportion of trials fixating on novel object = .65 (SD = .13) chance .65 .13 d 

Pool effect sizes across studies, weighting by sample size Grand effect size −1.00 1.00 2.00 3.00 Effect size estimate 8. spiegel 7. markman 6. grassman 5. grassman 4. byers 3. bion 2. bion 1. bion 2011 1988 2010 2010 2009 2013 2013 2013 30 45 48 24 17 30 24 18 72 10 12 12 16 20 25 22 First author Year Age (m.) N Example: Mutual exclusivity meta-analysis Grand effect size estimate 

MetaLab [metalab.stanford.edu] Web-based tool that aggregates meta-analyses across phenomena in language acquisition Publicly available Estimate effect sizes for particular phenomena, age, and method Summary visualizations of effect sizes and power calculator Random effect models using metafor R package (Viechtbauer, 2010) 

Current phenomena in MetaLab 1. Mutual exclusivity (Lewis & Frank, in prep) [N = 60 effect sizes] 2. Phonemic discrimination (Tsuji & Cristia, 2014) [N = 239] 3. Word segmentation (Bergmann & Cristia, 2015) [N = 196] 4. Infant directed speech preference (Dunst, Gorman & Hamby, 2012) [N = 50] 5. Pointing (Colonnesi, et al., 2010) [N = 30] 6. Label advantage in concept learning [N = 100] 



−1 0 1 2 0 10 20 30 Age (Months) Effect Size Synthesis across meta-analyses Infant directed speech Mutual exclusivity Label advantage Phonemic discrimination Pointing Word segmentation 

On-going meta-analyses Speed of processing ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 0.8 1.0 10 20 30 40 50 Age (months) reaction time (s) Sound Symbolism ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.00 0.25 0.50 0.75 1.00 10 15 20 25 Age (months) Effect size ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.00 0.25 0.50 0.75 1.00 10 15 20 25 Age (months) Effect size With Mathilde Fort (U Pompeu Fabra), Imme Lammertink (U Amsterdam), and Paula Fikkert (Radboud U) Gaze-following ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.00 0.25 0.50 0.75 1.00 10 15 20 25 Age (months) Prop. gaze following With Kyle MacDonald 

Efforts to scale-up Paper sampling Define sampling frame of papers for phenomenon Sample in systematic way from frame, and code Currently exploring different sampling approaches Teaching Provide students with meta-analysis primer Each student codes a paper(s) Training materials to share Community contributions (e.g. you!) 

Limitations Publication bias – available studies not randomly sampled from all studies conducted Magnitude of effect may be more related to method than phenomenon Non-representativeness of participant populations Limited number of similar studies in some domains 

Conclusion Practical tool for experiment planning Theoretical tool for a quantitative synthesis of development −1 0 1 2 0 10 20 30 Age (Months) Effect Size Infant directed speech preference Label advantage in concept learning Mutual exclusivity Phonemic discrimination Pointing Word segmentation 

Thanks! metalab.stanford.edu Bria Long (Harvard University), CSLI interns (Stanford University), Page Piccinini (UCSD)