Cardiff 12-5-2017

Invited Colloquium on Designing Efficient and Informative Studies

Transcript

1. Designing Efficient and Informative Studies Daniel Lakens @Lakens Eindhoven University of Technology
2. How do you determine the sample size for a new study?
3. Small samples have large variation, more Type 2 errors, and inaccurate estimates.
4. Schönbrodt & Perugini, 2013
6. Studies in psychology often have low power. Estimates average around 50%. Cohen, 1962; Fraley & Vazire, 2014
7. One reason for low power is that people use heuristics to plan their sample size.
8. You need to justify the sample size of a study. What goal do you want to achieve?
9. Goal according to JPSP:
10. Goal according to JESP:
11. Statistical power is the long-run probability of observing p < α with N participants, assuming a specific effect size.
12. But 1) You never know the true effect size, and the literature is biased, and 2) If you expect a true effect of 0, power is 0
13. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 Power Sample Size per condition in a independent t-test d=0.3 d=0.4 d=0.5 d=0.6 d=0.7 d=0.8
14. My department requires sample size justification before funding a study. One justification the IRB accepts is 90% power.
15. What is ‘evidence’?
16. Evidence is always relative. You want a higher likelihood of p<0.05 when H1 is true than when H0 is true.
17. High power leads to informative studies only when we control alpha levels.
18. What we have been doing wrong: Using previous studies as an effect size estimate
19. Distribution of η² for a medium effect size
20. Distribution of η² for a medium effect size
21. A pilot study does not provide a meaningful effect size estimate for planning subsequent studies. Leon, Davis, & Kraemer, 2011
22. Power analysis based on significant studies need to be based on a truncated F distribution. Taylor & Muller, 1996
23. Note the large variability
24. You can also take into account variability (‘assurance’) – e.g., using safeguard power. Perugini, Gallucci, & Constantini, 2014
25. Effect sizes from the published literature are always smaller than you expect, even when you take into account that effect sizes from the published literature are always smaller than you expect.
26. Plan for the change you would like to see in the world. Ask yourself: What is your smallest effect size of interest?
27. Requires you to specify H1! That’s a good thing. What does you theory predict, or what do you care about if H0 is false?
28. If we don’t, science becomes unfalsifiable. We can never ‘accept the null’.
29. But ‘I’m not interested in the size of the effect – the presence of any effect supports my theory!’ Really?
30. Detecting d = 0.001 requires 42 million people.
31. You make implicit choices about which effects are too small to matter all the time.
33. If you expect a ‘medium’ effect size and plan for 80% power, d<0.35 will never be significant.
34. If nothing else, the maximum sample you are willing to collect determines your SESOI.
35. Now you can also reject effects as large as, or larger than, your SESOI, using an equivalence test.
37. R package (“TOSTER”) & Excel
38. My prediction: Publishing a paper that say ‘p > 0.05, thus no effect’ will be difficult in 2019.
39. Extending your statistical tool kit with equivalence tests is an easy way to improve your inferences. Lakens, 2017
40. However, when the true effect size is larger than the SESOI, powering for it is inefficient (and possibly wasteful).
41. Social Sciences Replication Project
42. 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160 170 180 190 200 Power Sample Size per condition in a independent t-test d=0.3 d=0.4 d=0.5 d=0.6 d=0.7 d=0.8
44. When effect sizes are uncertain (=always), a better approach is sequential analyses.
45. Optional stopping: Collecting data until p < 0.05 inflates the Type 1 error.
46. A user of NHST could always obtain a significant result through optional stopping. Wagenmakers, 2007
48. Sequential analysis controls Type 1 error rates (e.g., Pocock correction).
49. Wald, 1945
51. Pocock Boundary Number of analyses p-value threshold 2 0.0294 3 0.0221 4 0.0182 5 0.0158
54. You also correct alpha levels for equivalence tests (and can calculate power for equivalence).
55. If you pre-register anyway, you can use one-sided tests (more logical & more efficient)
57. Use decision rules based on p-values or Bayes factors, but check Frequentist properties. Schonbrodt, Wagenmakers, Zehetleitner, & Perugini, 2015
58. The SESOI for the Higgs boson was not based on feasibility, but theory.
59. If you think the current reproducibility crisis was bad, wait till the theory crisis in psychology starts.
60. Thanks! @lakens https://www.coursera.org/learn/statistical-inferences