Statistics - We can do better!

A journey of a developer trying to learn how to apply statistics to an everyday problem.

Transcript

1. Statistics We can do better
2. Story Time
3. ! Programmers Need To Learn Statistics Or I Will Kill Them All http://zedshaw.com/essays/programmer_stats.html
4. Averages
8. > summary(a) Min. 1st Qu. Median Mean 3rd Qu. Max. 16.75 26.82 29.63 29.91 32.60 43.58 ! > summary(b) Min. 1st Qu. Median Mean 3rd Qu. Max. -15.48 16.90 31.07 30.12 43.42 80.86
9. > summary(a) Min. 1st Qu. Median Mean 3rd Qu. Max. 16.75 26.82 29.63 29.91 32.60 43.58 ! > summary(b) Min. 1st Qu. Median Mean 3rd Qu. Max. -15.48 16.90 31.07 30.12 43.42 80.86
10. > summary(a) Min. 1st Qu. Median Mean 3rd Qu. Max. 16.75 26.82 29.63 29.91 32.60 43.58 ! > summary(b) Min. 1st Qu. Median Mean 3rd Qu. Max. -15.48 16.90 31.07 30.12 43.42 80.86
11. > summary(a) Min. 1st Qu. Median Mean 3rd Qu. Max. 16.75 26.82 29.63 29.91 32.60 43.58 ! > summary(b) Min. 1st Qu. Median Mean 3rd Qu. Max. -15.48 16.90 31.07 30.12 43.42 80.86 ! > sd(a) [1] 4.787892 ! > sd(b) [1] 20.87684
12. Power “The Power of Ten Syndrome” - Zed A. Shaw
13. ! Definition: Power is the probability of rejecting the null hypothesis OpenIntro Statistics p. 195 The chance that your experiment is right (Almost!)
14. > power.t.test ! Power calculations for one and two sample t tests
 > power.prop.test Power Calculations for Two-Sample Test for Proportions
15. ! What’s a 2 sample T test? 2-Sample t-Test ! Use the 2-sample t-test to compare the averages between two groups and determine if there is a significant difference between them or if the observed difference is due instead to random chance.
16. > power.t.test Compute power of test, or determine parameters to obtain target power. ! power.t.test(n = NULL, delta = NULL, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE) ! Exactly one of the parameters n, delta, power, sd, and sig.level must be passed as NULL, and that parameter is determined from the others. Notice that the last two have non-NULL defaults so NULL must be explicitly passed if you want to compute them.
17. ! WHHAAAAAAAAAAAATTTTTT!!!!!!!!!
18. ! WOW 3% chance I’m right? > power.t.test(n = 50, delta = 0.01) ! Two-sample t test power calculation ! n = 50 delta = 0.01 sd = 1 sig.level = 0.05 power = 0.02803757 alternative = two.sided ! NOTE: n is number in *each* group
19. ! n = Sample Size power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
20. ! delta = True difference in means power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
21. power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE) ! sd = Standard deviation
22. ! sig.level = Significance level (Type I error probability) power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
23. ! power = Power of test (1 minus Type II error probability) power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
24. ! type = type of t test 2 sample, 1 sample or paired power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
25. ! alternative = One- or two-sided test power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
26. ! strict = Use strict interpretation in two- sided case power.t.test(n = NULL, delta = NULL, sd = 1, sig.level = 0.05, power = NULL, type = c(“two.sample”, “one.sample”, “paired”), alternative = c(“two.sided","one.sided"), strict = FALSE)
27. ! I DON’T KNOW WHAT ANY OF THAT MEANS
28. ! Let’s try something > power.t.test(n = 50, delta = 0.01, sd = 0.01) ! Two-sample t test power calculation ! n = 50 delta = 0.01 sd = 0.01 sig.level = 0.05 power = 0.9986074 alternative = two.sided ! NOTE: n is number in *each* group
29. ! 99.86% > power.t.test(n = 50, delta = 0.01, sd = 0.01) ! Two-sample t test power calculation ! n = 50 delta = 0.01 sd = 0.01 sig.level = 0.05 power = 0.9986074 alternative = two.sided ! NOTE: n is number in *each* group
30. ! NAILED IT !!!!
31. ! What is the standard deviation Here?
32. ! 70% not bad > power.t.test(n = 50, delta = 0.01, sd = 0.02) ! Two-sample t test power calculation ! n = 50 delta = 0.01 sd = 0.02 sig.level = 0.05 power = 0.6968888 alternative = two.sided ! NOTE: n is number in *each* group
33. ! 105 samples were needed > power.t.test(power=0.95, delta = 0.01, sd = 0.02) ! Two-sample t test power calculation ! n = 104.928 delta = 0.01 sd = 0.02 sig.level = 0.05 power = 0.95 alternative = two.sided ! NOTE: n is number in *each* group
34. ! So what do we know?
35. ! Sig.level
36. ! Sig.level ! Type 1 error probability
37. ! Power
38. ! Power ! 1 minus the type 2 error probability
39. ! What assumptions did we make?
40. ! What was power.prop.test?
41. Confounding If you want to measure something, then don't measure other shit. - Zed A. Shaw
42. Why does this matter?
43. ! http://www.statisticsdonewrong.com
45. ! Medical Trials and Power http://www.statisticsdonewrong.com/zbibliography.html#citation-tsang-2009iw
46. ! “The result was significant with p<0.05, so there’s only a 1 in 20 chance it’s a fluke!”, please beat them over the head with a statistics textbook for me. http://www.statisticsdonewrong.com/conclusion.html
48. ! When to retreat.
49. Resources: http://www.openintro.org/stat/textbook.php http://www.statisticsdonewrong.com/ Statistics with R (Dalgaard) http://zedshaw.com/essays/programmer_stats.html ! ! !