Meta-analytic Method

4a8d62be623e1a3e3cde003a6a810c0b?s=47 mllewis
August 11, 2016
56

Meta-analytic Method

Cogsci Tutorial 2016 on meta-analytic method for cognitive scientists.

4a8d62be623e1a3e3cde003a6a810c0b?s=128

mllewis

August 11, 2016
Tweet

Transcript

  1. Meta-Analytic Method Meta-Analytic Methods for Cognitive Science CogSci Pre-Conference Workshop

    Molly Lewis 10 August 2016 11:00-12:00
  2. Meta-Analytic method Goal: Describe underlying true “population” effect Approach: Run

    many imperfect studies, then aggregate (1a) Collect studies (1b) Code (2) Aggregate ES = .6
  3. … Overview of the meta-analytic method 1. Calculate effect size

    for each study 2. Pool effect sizes across studies, weighting by sample size (+ look for moderators) Studies … … … Study characteristics Stats Effect Size + Var. d d_var
  4. Part 1: Effect Sizes

  5. Consider two experiments. Experiment 1: • 2AFC pointing • DV:

    number of points • 8 trials • 12 months Experiment 2: • 2AFC eye-tracking • DV: dwell time • 16 trials • 18 months How do we compare “success”?
  6. A common unit: Effect size Effect size as unit of

    analysis: Quantitative, scale-free measure of “success” 0.25 0.50 0.75 1.00 Prop. trials fixating novel object chance d Observed mean diff. between means standard dev. Effect Size =
  7. 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object

    0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object 0.00 0.25 0.50 0.75 1.00 Prop. trials fixating novel object Example: Mutual exclusivity effect size Where’s the dofa? Bion, et al. (2013) For 24 mo, mean proportion of trials fixating on novel object = .65 (SD = .13) chance .65 .13 d
  8. The two-sample case Control mean Observed mean d diff. between

    means standard dev. Effect Size = Plot from: http://www.realeducationreform.com/questions.html
  9. −1 0 1 2 3 Effect Size −1 0 1

    2 3 Effect Size Effect size confidence interval n1 = 24 n2 = 24
  10. Interpreting Cohen’s d Size Description Cohen’s Intuition Psychological Example .2

    “small” Diff. between the heights of 15 yo and 16 yo girls in the US Bouba-kiki effect in kids (~.15; Lammertink, et al. 2016) .5 “medium” Diff. between the heights of 14 yo and 18 yo girls. Cognitive behavioral therapy on anxiety (~ .4; (Belleville, et al., 2004) Sex difference in implicit math attitudes (~.5; Klein, et al., 2013) .8 “large” Diff. between the heights of 13 yo and 18 yo girls. Syntactic Priming (~.9; Mahowald, et al., under review) Mutual exclusivity ( ~1.0; Lewis & Frank, in prep) (Cohen, 1969)
  11. Interpreting Cohen’s d Cognitive (JEP:LMC) Social (JPSP) Psych. Science 0

    1 2 0 1 2 3 0 1 2 3 0 1 2 3 absolute cohen's d M = .69 M = .19 M = .78 Estimating the Reproducibility of Psychological Science (OSF, Science, 2015) N = 97 Relatively “large” effects reported in cognitive psychology
  12. Effect sizes Prototype: Cohen’s d – Depends on aspect of

    design (e.g., within vs. between subject) – Many effect size metrics (Hedge’s g for small samples) – Can convert between ES metrics (compute.es package in R; AC Del Re, 2013) – Can calculate via different pieces of raw data Multiple Types: – the difference is between groups (t-test, d) – the relationship between variables (correlation, r) – the amount of variance accounted for by a factor (ANOVA, regression, f) – Associations between categorical variables (odds ratio, risk difference, log odds) – More generally, for any statistical test you conduct, can compute effect size (some more straight-forwardly than others) http://rpsychologist.com/d3/cohend/
  13. Calculating Cohen’s d (Lakens, 2013)

  14. Part 2: Methods for pooling

  15. … Overview of the meta-analytic method 1. Calculate effect size

    for each study 2. Pool effect sizes across studies, weighting by sample size (+ look for moderators) Studies … … … Study characteristics Stats Effect Size + Var. d d_var
  16. Grand effect size −1.00 1.00 2.00 3.00 Effect size estimate

    8. spiegel 7. markman 6. grassman 5. grassman 4. byers 3. bion 2. bion 1. bion 2011 1988 2010 2010 2009 2013 2013 2013 30 45 48 24 17 30 24 18 72 10 12 12 16 20 25 22 First author Year Age (m.) N Example: Mutual exclusivity meta-analysis Grand effect size estimate Pool effect sizes across studies, weighting by sample size
  17. Methods for pooling Analogous to logic in single study: –

    In a study, sample participants and pool to get estimate of effect in study (unweighted mean) – In meta-analysis, sample studies to get estimate of grand effect (weighted mean) Just as for models across participants, two models for pooling: – Fixed effect: One true population effect – Random effect: Random sample of many population effects, estimates mean
  18. Fixed Effect Model Slide from http://compare2what.blogspot.com/ (Keith Lohse, 2013)

  19. Fixed Effect Model

  20. Random Effect Model Random effect models recommended!

  21. (2) Byers-Heinlein & Werker (2009) Reported M = .12 t

    = 5.97 N = 16 SD = .079 baseline = 0 d = 1.49 Example meta-analysis with two studies (1) Markman and Wachtel (1988) Reported M = 4.9 t = 3.94 N = 10 baseline = 3
  22. (1) Calculate pooled and weighted estimate of population (2) Calculate

    variance of population estimate (3) Compute 95% confidence interval
  23. Testing for Moderators A good application for meta- analysis: –

    Testing for moderators requires more power than main effect (Button, et al., 2013) – Different levels may not be present within a single study – Theoretical progress In developmental research, a moderator of interest: age, vocabulary • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • r= 0.47 −1 0 1 2 20 30 40 50 60 Age (months) Cohen's d
  24. Meta-analytic methods in R Metafor: Viechtbauer, W. (2010). Conducting meta-analyses

    in R with the metafor package. Journal of Statistical Software, 36(3), 1-48. URL: http://www.jstatsoft.org/v36/i03/ … Studies … … … Study characteristics Stats Effect Size + Var. d d_var
  25. Metalab

  26. Phenomena in MetaLab Prosody Communication Sounds Words

  27. Overall effect sizes Random effect models using metafor R package

    (Viechtbauer, 2010) • • • • • • • • • • • • • • • • • • • • • • Pointing and vocabulary Gaze following Online word recognition Concept−label advantage Mutual exclusivity Word segmentation Statistical sound learning Vowel discrimination (non−native) Vowel discrimination (native) Phonotactic learning IDS preference 0 1 2 3 Effect Size Phenomenon
  28. IDS preference Phonotactic learning Vowel discrimination (native) Vowel discrimination (non−native)

    Statistical sound learning Word segmentation Mutual exclusivity Concept−label advantage Online word recognition Gaze following Pointing and vocabulary −1 0 1 2 3 −1 0 1 2 3 −1 0 1 2 3 0 1 2 3 0 1 2 3 0 1 2 3 Age (years) Effect size (d)