HBES 2015 Plenary

Plenary talk from the May 2015 meeting of the Human Behavior and Evolution Society

Transcript

1. The Evolution of Statistical Methods for Studying Human Evolution Richard McElreath Anthropology/Population Biology/Ecology/Animal Behavior University of California, Davis
2. * * * * * * * * * *
5. Null hypothesis testing Reject Bayesian statistics 1925
6.  0$7 ]D[LV 3DODQHWHVGVUEQ K OQWUUL SHULKHOLRQ LVWKHSRLQFORVHVWLWWDQH HL L RQKLVHOOSWLDORUEW L GHHULRQH E\aWHWP ISHULKHOLRQ FRPSXWHGYDOXHV ILQ HUO\VWXLHVRI ODQHWUPRWLQ 6SRHQZWD HNR WZ K R QHQ ),*85( 3DUDPHWHUV GHVFWLELQJ WKH SODQHWDU\ RUELW GLIIHUHQW SODQHV  ZKLFK GHWHUPLQHV WKH RULHQWDWLRQ RI WKH HOOLS SODQH 1RWH WKDW DSKHOLRQ LV WKH SRLQW RQ WKH RUELW IXUWKHVW IU SHULKHOLRQ LV WKH SRLQW FORVHVW WR WKH VXQ 7KH HOOLSVH LWVHOI LVG OHQJWK RI LWV VHPLPDMRU D[LV DQGHLWV HFFHQWULFLW\ )LQDOO\ WKH S Gauss 1809
7. Darwin 1837
8. (a) (b) Fig. 5. Depiction of the regression phenomenon, based on a figure from Galton (1889 (a) the expected landing position of pellets from a specific upper level compartment a point of origin of pellets landing in a specific lower level compartment There was a hint in the lecture that Galton was moving towards a mechanism his thinking. He seemed to approach Mendelian genetics when he speculated Francis Galton invents Bayesian conditional distributions (1873, 1889). See Stigler (2010): Darwin, Galton and the Statistical Enlightenment.
9. 150 500 Y chr MtDNA Karmin et al. 2015. Genome Research Thousands of Years Ago 0 50 100 150 0 50 100 Effective Population Size (thousands) Africa Andes Central Asia Europe Near-East & Caucasus Southeast & East Asia Siberia South Asia Region 10 Figure 2. Cumulative Bayesian skyline plots of Y chromosome and mtDNA diversit kya and 50 kya. Individual plots for each region are presented in Supplemental Figu Thousands of Years Ago 0 50 100 150 0 50 Effective Population Size (thousands) Thousands of Years Ago 0 100 200 300 400 500 0 50 100 Caucasus n 10 10
10. When I describe priming studies to audiences, the reaction is often disbelief . . . The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true. “ ” Daniel Kahneman Source: Thinking, Fast and Slow (page 57) Quoted in: Wagenmakers et al. “A skeptical eye on psi” Statistics in the replication crisis
11. estimates, 1b for partial eta-squared estimates. When available, the triangle indicates the effect size obtained in the original study (Elaboration Likelihood main effect estimate does not appear because it was extremely large, partial-eta square of .59). Large circles represent the aggregate effect size obtained across all participants. Error bars represent 99% noncentral confidence intervals around the effects. Small x’s represent the effect sizes obtained within each site. Stroop effect Metaphoric restructuring Availability heuristic Persistence Standardized treatment difference Power & perspective Weight embodiment Warmth perceptions original studies replications Disbelief is an option Many Labs 3: https://osf.io/ct89g/
12. (1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? _____ cents (1) A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost? _____ cents
13. Meyer et al. 2015. Disfluent Fonts Don’t Help People Solve Math Problems. Figure 2. The effect of disfluent font on Cognitive Reflection Test scores. The l each individual study’s sample size against its effect size. Error bars are bootstrap 9 right panel displays the bootstrap probability distribution of the effect size based DISFLUENT FONT AND ANALYTIC REASONING Original study (N=40)
14. The case against science is straight- forward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. “ ” Richard Horton 11 April 2015 issue of The Lancet Not just social science
15. “There are many more elemental ‘discoveries’ later shown to be false than there are entries in the present table.”
16. Ao 93 Ausonium Es 94 Hesperium Enrico Fermi (1901–1954) receiving 1938 Nobel Prize Enrico Fermi’s false-discovery elements
17. Evolution of statistics • Null models in evolution & ecology A tragedy in three acts Bayes and information criteria • Statistics in ecological context Why many published findings wrong Science as a population process
18. Act I
20. X → Y,Z Hypotheses Process models Statistical models
21. H0 “Evolution is neutral” P0A Neutral, equilibrium Hypotheses Process models Statistical models
22. H0 “Evolution is neutral” P0A Neutral, non-equilibrium P0B Neutral, equilibrium Hypotheses Process models Statistical models
23. H0 H1 “Evolution is neutral” “Selection matters” P0A Neutral, non-equilibrium P0B Neutral, equilibrium Hypotheses Process models Statistical models
24. H0 H1 “Evolution is neutral” “Selection matters” P0A Neutral, non-equilibrium P0B Neutral, equilibrium P1B Fluctuating selection P1A Constant selection Hypotheses Process models Statistical models
25. H0 H1 “Evolution is neutral” “Selection matters” P0A Neutral, non-equilibrium P0B Neutral, equilibrium P1B Fluctuating selection P1A Constant selection MI MII MIII Hypotheses Process models Statistical models
26. Neutral theory kinda wrong • Conceptual problems Null model not unique Selection can imitate null model • Empirically Neutrality important Selection structures genome Selection structures population Molecular clock not constant • Lesson: Compare multiple non-null models
28. JJ Harrison, “Phylidonyris novaehollandiae Bruny Island” Act II
29. A. “True” interaction strengths -3 -3 -1 - - + - - - B. Observed correlations C. Inferred partial correlations Figure 1: A. A small network of three competing species. The tree (top) tends not to co-occur Harris. 2015. Estimating species interactions from observational data with Markov networks http://dx.doi.org/10.1101/018861
30. A. “True” interaction strengths -3 -3 -1 - - + - - - B. Observed correlations C. Inferred partial correlations Figure 1: A. A small network of three competing species. The tree (top) tends not to co-occur Harris. 2015. Estimating species interactions from observational data with Markov networks http://dx.doi.org/10.1101/018861
31. A. “True” interaction strengths -3 -3 -1 - - + - - - B. Observed correlations C. Inferred partial correlations Figure 1: A. A small network of three competing species. The tree (top) tends not to co-occur Harris. 2015. Estimating species interactions from observational data with Markov networks http://dx.doi.org/10.1101/018861
32. 0 15 20 25 f Islands Shared marck bird species versus the number r example, there are 151 Bismarck 151)(150)/2=11,325 species pairs, of ributions (share zero islands), 3,208 give observed values; bars, predicted andomly assembled communities, as rloff method. Note that the observed cantIy (p < 10-s for the whole graph) SPECIES species per island ] SLANDS islands per A B C D E F G species a ii 11100 5 b 1 iii000 4 c Iii0000 3 d ii00000 2 e i01 O000 2 f i000000 1 6442100 17 SPECIES species per island ISLANDS ABCDEFG iiiii00 1111000 iii0000 d ii00000 e 0000011 f I000000 5432111 Fig. 2. Two examples of presence/absence matrices, each for six species on seven islands. "l"=present, "0"=absent. The matrices on the left and right have the same row sums and grand total, but Fig. 2 left is more nearly "nested" (see text p. 69 for definition) than is Fig. 2 right sult is a spectacular fit of the "null hypothesis" to the jaggedly observed graph for New Hebrides birds (Fig. 1 of Connor and Simberloff 1979), and acceptance at p>0.90 or p>0.99. This error did not arise in the analyses of West Indies birds and bats, for which Connor and Simberloff (1979, p. 1139, first para- graph of column 2) used a different computer aIgorithm. When we repeated the Connor-Simberloff test properly on the New islands per species 5 4 3 2 2 1 17 Daniel Simberloff
33. 0 15 20 25 f Islands Shared marck bird species versus the number r example, there are 151 Bismarck 151)(150)/2=11,325 species pairs, of ributions (share zero islands), 3,208 give observed values; bars, predicted andomly assembled communities, as rloff method. Note that the observed cantIy (p < 10-s for the whole graph) SPECIES species per island ] SLANDS islands per A B C D E F G species a ii 11100 5 b 1 iii000 4 c Iii0000 3 d ii00000 2 e i01 O000 2 f i000000 1 6442100 17 SPECIES species per island ISLANDS ABCDEFG iiiii00 1111000 iii0000 d ii00000 e 0000011 f I000000 5432111 Fig. 2. Two examples of presence/absence matrices, each for six species on seven islands. "l"=present, "0"=absent. The matrices on the left and right have the same row sums and grand total, but Fig. 2 left is more nearly "nested" (see text p. 69 for definition) than is Fig. 2 right sult is a spectacular fit of the "null hypothesis" to the jaggedly observed graph for New Hebrides birds (Fig. 1 of Connor and Simberloff 1979), and acceptance at p>0.90 or p>0.99. This error did not arise in the analyses of West Indies birds and bats, for which Connor and Simberloff (1979, p. 1139, first para- graph of column 2) used a different computer aIgorithm. When we repeated the Connor-Simberloff test properly on the New islands per species 5 4 3 2 2 1 17 Daniel Simberloff Jared Diamond Conor & Simberloff 1979 Diamond & Gilpin 1982
34. Null ecology • Multiple null models possible • Null uses observed frequencies, influenced by competition • Doesn’t indicate direction of interaction • Misses obvious interactions • Low power, high false-discovery (Gotelli & Ulrich 2010) 0 ECIES a b species per island ISLANDS A B C 1 0 1 0 1 0 1 1 i 1 islands per D species 0 2 i 2 g. 3. Example of a simple presence/absence matrix with a checker- ard distribution: four islands and two species ands and are absent from species-rich islands (see Fig. 2 right, ich has a supertramp as species e but has the same row ms and grand total as Fig. 2 left). Now compare the rearrangeability of the two matrices of g. 2. Recall that the rearrangement algorithm by which Connor d Simberloff generate simulated matrices seeks 2 by 2 subma- ces (not necessarily in adjacent rows or columns) of the form 10) 1)o (0, ?0 d changes them to the opposite form. This manipulation alters ither row nor column sums and hence maintains the con- aints of island species number and species occurrence fre- ency. For the matrix of Fig. 2 left, the sole possible such Islands 00101011110100001110 Ii010100001011110001 00101111001001001110 11010000110110110001 10110000000011111011 01001111111100000100 10011101011001001100 01100010100110110011 00101011100100110110 Species ii010100011011001001 11010111100001011000 00101000011110100111 i0000111100011001110 01111000011100110001 11001100100111000110 00110011011000111001 00010101101011101010 11101010010100010101 01101101010001010101 10010010101110101010 Fig. 4. Example of a iarge presence/absence matrix with distribution: 20 islands and 20 species, consisting of 10 the members of each pair having mutually exclusiv “Checkerboard” distribution
35. p < 10≠1000000), this increased to 17% (Appendix B). Figure 3 reflects the adjusted versio the results. 5 10 20 • • • • • • • • • • • • • • • • • • • • • • • • 0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 No environment Enironment 20 100 500 2500 20 100 500 2500 20 100 500 2500 Number of sites (log scale) R2 method • Markov network Partial covariance Covariance Null Figure 3: Proportion of variance in interaction coe cients explained by each method wit 10, or 20 species arrayed across varying numbers of sampled locations when environme filtering was absent (top row) or present (bottom row). A negative R 2 values implies t Harris. 2015. Estimating species interactions from observational data with Markov networks http://dx.doi.org/10.1101/018861 Accuracy Non-null models do better
36. interbreeding with Neandertals. The extent of LD between single nucleotide polymorphisms (SNPs) shared with Neandertals will thus reflect, at least in part, the time since Neandertals or their ancestors and modern humans or their ancestors last exchanged genes with each other. our inferences are qualitatively unaffected. Our methodology is based on the idea that if two a genetic distance x (expected number of crossover recomb events per meiosis) apart, arose on the Neandertal linea introgressed into modern humans at time tGF , the probabi these alleles have not been broken up by recombination sin The Date of Interbreeding between Neandertals and Modern Humans Sriram Sankararaman1,2*, Nick Patterson2, Heng Li2, Svante Pa ¨a ¨bo3*, David Reich1,2* 1 Department of Genetics, Harvard Medical School, Boston, Massachusetts, United States of America, 2 Broad Institute of MIT and Harvard, Cambridge, Massachusetts, United States of America, 3 Department of Evolutionary Genetics, Max Planck Institute for Evolutionary Anthropology, Leipzig, Germany Abstract Comparisons of DNA sequences between Neandertals and present-day humans have shown that Neandertals share more genetic variants with non-Africans than with Africans. This could be due to interbreeding between Neandertals and modern humans when the two groups met subsequent to the emergence of modern humans outside Africa. However, it could also be due to population structure that antedates the origin of Neandertal ancestors in Africa. We measure the extent of linkage disequilibrium (LD) in the genomes of present-day Europeans and find that the last gene flow from Neandertals (or their relatives) into Europeans likely occurred 37,000–86,000 years before the present (BP), and most likely 47,000–65,000 years ago. This supports the recent interbreeding hypothesis and suggests that interbreeding may have occurred when modern humans carrying Upper Paleolithic technologies encountered Neandertals as they expanded out of Africa. Citation: Sankararaman S, Patterson N, Li H, Pa ¨a ¨bo S, Reich D (2012) The Date of Interbreeding between Neandertals and Modern Humans. PLoS Genet 8(10): e1002947. doi:10.1371/journal.pgen.1002947 Editor: Joshua M. Akey, University of Washington, United States of America Received December 15, 2011; Accepted July 27, 2012; Published October 4, 2012 Copyright: ß 2012 Sankararaman et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: This work was supported by the Presidential Innovation Fund of the Max Planck Society, the Krekeler Foundation, and the National Science Foundation (HOMINID grant 1032255). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing Interests: The authors have declared that no competing interests exist. * E-mail: sankararaman@genetics.med.harvard.edu (SS); paabo@eva.mpg.de (SP); reich@genetics.med.harvard.edu (DR) Introduction A much-debated question in human evolution is the relationship between modern humans and Neandertals. Modern humans appear in the African fossil record about 200,000 years ago. Neandertals appear in the European fossil record about 230,000 years ago [1] and disappear about 30,000 year ago. They lived in Europe and western Asia with a range that extended as far east as Siberia [2] and as far south as the middle East. The overlap of Neandertals and modern humans in space and time suggests the possibility of interbreeding. Evidence, both for [3] and against interbreeding [4], have been put forth based on the analysis of modern human DNA. Although mitochondrial DNA from multiple Neandertals has shown that Neandertals fall outside the range of modern human variation [5,6,7,8,9,10], low-levels of gene flow cannot be excluded [10,11,12]. Analysis of the draft sequence of the Neandertal genome revealed that the Neandertal genome shares more alleles with non- African than with sub-Saharan African genomes [13]. One hypothesis that could explain this observation is a history of gene substructure in Africa is a plausible alternative to the hypothesis of recent gene flow. Today, sub-Saharan Africans harbor deep lineages that are consistent with a highly-structured ancestral population [17,18,19,20,21,22,23,24,25,26,27]. Evidence for an- cient structure in Africa has also been offered based on the substantial diversity in neurocranial geometry amongst early modern humans [28]. Thus, it is important to test formally whether substructure could explain the genetic evidence for Neandertals being more closely related to non-Africans than to Africans. A direct way to distinguish the hypothesis of recent gene flow from the hypothesis of ancient substructure is to infer the date for when the ancestors of Neandertals and a modern non-African population last exchanged genes. In the recent gene flow scenario, the date is not expected to be much older than 100,000 years ago, corresponding to the time of the earliest documented modern humans outside of Africa [29]. In the ancient substructure scenario, the date of last common ancestry is expected to be at least 230,000 years ago, since Neandertals must have separated from modern humans by that time based on the Neandertal fossil 2015 Act III
37. Non-null human evolution • Origins Speciation, admixture Introgression, sweeps • Behavior, development, diversity How kinship matters Non-null ontogeny Comparative methods Sex ratio (Towner & Luttbeg 2007)
38. Non-null, Bayesian data analysis • Bayesian inference Robust way to fit arbitrary models Focuses mind on generative models But won’t fix science • Information criteria AIC, DIC, WAIC Focus on multiple non-null models Focus on model uncertainty But won’t fix science Statistical Rethinking A Bayesian Course with R Examples Texts in Statistical Science Richard McElreath forthcoming in 2016
39. Quality of theory (base rate) Quality of measurement Quality of data analysis Quality of peer review Publication culture Funding culture Replication & meta-analysis Quality of post-pub peer review
40. Quality of theory (base rate) Quality of measurement Quality of data analysis Quality of peer review Publication culture Funding culture Replication & meta-analysis Quality of post-pub peer review
41. All about that base • Assume: Probability false positive finding is 5% Probability true positive finding is 50% Probability positive finding indicates true hypothesis? Most common answer: 0.95
42. All about that base • Assume: Probability false positive finding is 5% Probability true positive finding is 50% Probability positive finding indicates true hypothesis? Need base rate 1S(5|+) = 1S(+|5) 1S(5) 1S(+) = 1S(+|5) 1S(5) 1S(+|5) 1S(5) + 1S(+|') 1S(') 8 = /! O!O!O!O!O! 1S(5|+) = 1S(+|5) 1S(5) 1S(+) = 1S(+|5) 1S(5) 1S(+|5) 1S(5) + 1S(+|') 1S(')
43. “base rate”: proportion of hypotheses which are true true false The right junk in the right places
44. The right junk in the right places Positive Negative
45. The right junk in the right places Positive Negative Pr(true|+) = 0.5
46. The right junk in the right places Positive Negative
47. Base rate not known, often low • Chemistry: More than half of elemental discoveries false • Medicine: 8/10 published findings cannot be repeated • Psychology: Replication rate not good • Neuroscience: Many published associations cannot be repeated • Genomics: Most gene-phenotype associations cannot be replicated • Anthropology & Ecology: No one knows, no one cares? • Base rate is lower than rate of true published findings
48. Why is base rate low? • Science is hard; the truth is rare • Null hypothesis syndrome • Poor theorizing: intuition, metaphor, rhetoric, bad math • HARKing: Hypothesizing After Results are Known These asterisks are the keys to the universe!
49. Evolution and base rate • Evolutionary theory hugely successful • Base rate higher? • Problems: Explanatory, often not predictive Historically contingent Complex dynamics Success sum of many studies With Darwin on my side, what could go wrong?
50. Quality of theory (base rate) Quality of measurement Quality of data analysis Quality of peer review Publication culture Funding culture Replication & meta-analysis Quality of post-pub peer review
51. Quality of theory (base rate) Quality of measurement Quality of data analysis Quality of peer review Publication culture Funding culture Replication & meta-analysis Quality of post-pub peer review
52. 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – positive results! negative results! False (T)! Negative (–)! General case! General case (+ or –)! F! Science is a hot mess McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
53. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
54. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
55. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
56. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
57. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
58. 1. Hypothesis Selection! Novel hypotheses! Tested hypotheses! A previously tested hypothesis is selected for replication with probability r, otherwise a novel (untested) hypothesis is selected. Novel hypotheses are true with probability b. ! 1 – r! r! 2. Investigation! T! Real truth of hypothesis! Probability of result! 1 – β α β 1 – α + – 3. Communication! Experimental results are communicated to the scientific community with a probability that depends upon both the experimental result (+, –) and whether the hypothesis was novel (N) or a replication (R). Communicated results join the set of tested hypotheses. Uncommunicated replications revert to their prior status.! 1 – C N– C N– positive results! negative results! 1 – C R+ C R+ New result communicated! New result not communicated! 1 – C R– C R– File drawer! novel! replic.! novel! replic.! True (T)! False (T)! KEY! Interior = true epistemic state ! Exterior = experimental evidence! Unknown! Positive (+)! Negative (–)! General case! General case (+ or –)! F! %:/".*$4 0' 4$*&/5*'*$ %*4$07&3: 3*$)"3% .$&-3&"5)Ŵ,ŵ "/% 1"6- 4."-%*/0Ŵ 4VQQMFNFOUBM %BUF "QSJM    %IJĿĶŃĮŁĶļĻ ļij ijłĹĹ ĺļıIJĹ ńĶŁĵ ĿĮĻıļĺ ĿIJĽĹĶİĮŁĶļĻ -FU G5,T = O5,T/O CF UIF GSFRVFODZ PG USVF IZQPUIFTFT XJUI UBMMZ T 6OEFS UIF BTTVNQUJP E EFĕOJUJPOT TVQQMJFE JO UIF NBJO UFYU UIF GVMM SFDVSTJPO GPS O′ 5,T JT HJWFO CZ O′ 5,T = O5,T + BOS − G5,T(D3+ (Ŵ − β) + D3− β) + G5,T−Ŵ(Ŵ − β)D3+ + G5,T+ŴβD3− S T OPU FRVBM UP  PS −Ŵ *O UIPTF DBTFT UIFSF JT BO BEEJUJPOBM UFSN 'PS T = Ŵ O′ 5,Ŵ = O5,Ŵ + BOS − G5,Ŵ(D3+ (Ŵ − β) + D3− β) + G5,ų(Ŵ − β)D3+ + G5,ŵβD3− + BO(Ŵ − S)C(Ŵ − β) ćF BO(Ŵ − S)C(Ŵ − β) UFSN BDDPVOUT GPS JOĘPX PG OPWFM QPTJUJWF ĕOEJOHT BMM PG XIJDI B WBSJBCMFT GJ,T GPS J ∈ {5, '} ćJT BQQSPBDI JT QSPCBCMZ UIF NPTU TUSBJHIUGPSXBSE 4FDPOE JU DBO CF TPMWFE UP BOZ MFWFM PG BQQSPYJNBUJPO EFTJSFE CZ JUFSBUJWFMZ TPMWJOH UIF TZTUFN P FRVBUJPOT PVUXBSE GSPN T = ų #PUI BQQSPBDIFT ZJFME TPMVUJPOT UIBU UBLF UIF GPSN PG DMPTVSFT PG JOĕOJUF HFPNFUSJD TFSJF FYQSFTTJPOT 6TJOH UIFTF TPMVUJPOT XF GPVOE UIF VOCPVOEFE JOĕOJUF TFSJFT TPMVUJPO CBTFE VQPO JOUVJUJPO‰BOTBU[ JT XIBU PVS NBUIFNBUJDT JOTUSVDUPST VTFE UP DBMM JU 4JODF UIF TPMV UJPOT GSPN UIF CSVUFGPSDF BQQSPBDI MPPLFE MJLF DMPTVSFT PG JOĕOJUF TFSJFT BOE UIF TJNVMBUJPO SFTVMUT QSPEVDFE XIBU SFTFNCMFE B NJYUVSF PG HFPNFUSJD TFSJFT XF HVFTTFE UIF VOEFSMZJOH MJNJUJOH EJTUSJCVUJPO 8F UIFO WFSJĕFE PVS BOTBU[ TPMVUJPO CZ QMVHHJOH JU CBDL JOUP UIF SF DVSTJPOT BOE BMTP CZ DPNQBSJOH JU UP OVNFSJDBM SFTVMUT BOE PVS QSFWJPVT TPMVUJPOT 'JOBMMZ XF JOEVDFE UIF JOĕOJUF TFSJFT SFQSFTFOUBUJPO CZ DPOTUSVDUJOH 5BZMPS TFSJFT FYQBOTJPOT PG UIF DMPTFE TFSJFT FYQSFTTJPOT ZJFMEJOH UIF TFRVFOUJBM UFSNT PG UIF TPMVUJPO FYQSFTTJPO JO UIF OFY TFDUJPO  'VMM DPNNVOJDBUJPO TPMVUJPO )FSF XF SFQFBU UIF TJNQMFTU TVDI TPMVUJPO GSPN UIF NBJO UFYU BOE UIFO NPUJWBUF JUT KVTUJĕDBUJPO ćF TUFBEZ TUBUF QSPQPSUJPO PG IZQPUIFTFT UIB BSF CPUI USVF BOE IBWF UBMMZ T XIFO BMM ĕOEJOHT BSF DPNNVOJDBUFE JT HJWFO CZ ˆ Q5,T = C(Ŵ − S) ∞ N=Ŵ SN−Ŵ, N, (N + T)/ŵ (Ŵ − β) Ŵ ŵ (N+T)β Ŵ ŵ (N−T)  XIFSF ,(N, (N+T)/ŵ) JT UIF OVNCFS PG XBZT UP HFU (N+T)/ŵ QPTJUJWF ĕOEJOHT JO N JOWFTUJ HBUJPOT PG UIF TBNF IZQPUIFTJT ćJT JT TJNQMF UIF CJOPNJBM DIPPTFS CVU JNQMJDJUMZ FWBMVBUJOH UP [FSP XIFOFWFS (N+T)/ŵ JT OPU BO JOUFHFS 4JODF T JT UIF EJČFSFODF CFUXFFO UIF OVNCFS P Recursions: Solutions: McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
59.  0.001 0.1 0.5 0 0.2 0.5 0.8 1 0 0.1 0.3 0.5 0 0.2 0.5 0.8 1 0.5 0.8 0.99 0 0.2 0.5 0.8 1 0.05 0.1 0.15 0.2 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0 0.25 0.5 0.75 1 0 0.2 0 0.25 0.5 0.75 1 0 0.2 Proportion true Proportion true base rate replication rate power false-positive rate communicate neg. rep. communicate pos. rep. communicate neg. new 1 3 5 (a) (b) (c) (d) (e) (f) (g) 5 5 5 5 5 5 Pro communicate neg. rep. communicate pos. rep. communicate neg. new Optimistic Pessimistic scenario 3 3 3 3 3 3 0 0 0 Proportion true hypotheses at different numbers of positive findings McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
60.  0.001 0.1 0.5 0 0.2 0.5 0.8 1 0 0.1 0.3 0.5 0 0.2 0.5 0.8 1 0.5 0.8 0.99 0 0.2 0.5 0.8 1 0.05 0.1 0.15 0.2 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0.5 0.8 1 0 0.25 0.5 0.75 1 0 0.2 0 0.25 0.5 0.75 1 0 0.2 0 0.25 0.5 0.75 1 0 0.2 Proportion true Proportion true base rate replication rate power false-positive rate communicate neg. rep. communicate pos. rep. communicate neg. new 1 3 5 (a) (b) (c) (d) (e) (f) (g) 5 5 5 5 5 5 Pro communicate neg. rep. communicate pos. rep. communicate neg. new Optimistic Pessimistic scenario 3 3 3 3 3 3 0 0 0 Proportion true hypotheses at different numbers of positive findings Base rate and false-positive rate always important McElreath & Smaldino. 2015. Replication, communication, and the population dynamics of scientific discovery.
61. Two evolutionary trends • Decline of the null hypothesis Lack of unique predictions Silent on what actually happened Need generative models Null will live on • Rise of Bayesian statistical inference Focus on generative models Focus on multiple non-null models Easy to include diverse sources of error Works with experiments & fieldwork
62. “When asked [...] what can be done in observational studies to clarify [...] causation, Sir Ronald Fisher replied: ‘Make your theories elaborate.’ What Sir Ronald meant, as subsequent discussion showed, was that [...] one should envision as many different consequences of its truth as possible, and plan observational studies to discover whether each of these consequences is found to hold.” William Cochran (1965) Human science observational
63. BREAKING: SCIENCE IS HARD • Even non-null models mostly wrong • Bayes just a way to extract information from data • Individual contributions cannot live up to legends • Must cultivate population dynamics, “evolutionary epistemology” Donald T. Campbell (1916–1996)