Reproducibility & Minimizing Bias

Transcript

1. Reproducibility & Minimizing Bias Research Reproducibility 2018: Short Course Science University of Utah, Salt Lake City 14 June 2018 Hilda Bastian @hildabast hildabastian.net
2. “Statistical workers who fail to scrutinize the goodness of their observed data and carry through a satisfactory analysis upon poor observations, will end up with ridiculous conclusions which cannot be maintained”. Raymond Pearl, 1919 Via Doug Altman, Iveta Simera: http://www.jameslindlibrary.org/articles/a- history-of-the-evolution-of-guidelines-for-reporting-medical-research-the- long-road-to-the-equator-network/
3. •  Methods reproducibility: enough detail to be able to repeat it •  Results reproducibility: it replicates •  Inferential reproducibility: similar conclusions drawn about it Defining research reproducibility Goodman, Fanelli, Ioannidis (2016). Science Translational Medicine; 8 (341) ps12 —  But even this still doesn’t cover everything in this debate
4. “Reproducibility” is a euphemism
5. Specialization & complexity has grown 1850 Beaver & Rosen (2005). Scientometrics; 1(3). http:// www.akademiai.com/doi/abs/10.1007/BF02016308 1st co-authored article (1870) 1st article with >1,000 authors (2004) INTERNET Wikipedia (2001) 40% of papers have co-authors (1940)
7. “The idealized expert- generated, one-way, authoritative reign of science is over.” Debating science’s problems in 2016: now it’s everybody’s problem Sobo et al (2016). Medical Anthropology. 35 (6): 529-546.
9. •  Research on scientific methods – across the full spectrum of activity and use •  Detailed registered protocols for as much research as possible
10. •  Collaborative practice •  Optimizing fidelity
11. •  Good data, analysis, and reporting practice
12. —  Communicating results in ways that can be used by non-specialists
13. •  The Yin & Yang of post- publication activity •  Valuing quality, replication and validation studies
14. •  Strengthen culture of constructive criticism •  Consequences for authors of non-response to important questions & criticism •  More accountability and consequences for editors, reviewers, and journals (Südhof (2016). PLOS Biology 14(8): e1002547.)
17. Knowing what we (don’t) know before doing another study
18. Systematic review •  Pre-specified/registered protocol •  Clear, precise question(s) •  Search strategy •  Inclusion/exclusion criteria •  Data extraction method •  Quality assessment •  Pre-specified analyses & methods
19. Systematic reviews – gateway to meta-research
20. Studies are a collection of parts of uneven quality
21. Keep up with what are “questionable research practices” & how to avoid the pitfalls https://replicationindex.wordpress.com/2015/01/24/questionable-research- practices-definition-detect-and-recommendations-for-better-practices/
22. Skills: design critical appraisal
23. Bias prevention beyond research design •  Diversity in participation and areas of study •  Cognitive biases e.g. confirmation bias, motivated reasoning
24. Everyday choices build culture Virtuous spiral of trust
25. Values & Integrity Research & Analytical Skills Life Skills Cognitive Skills
26. 7 Principles of Public Life https://en.wikipedia.org/wiki/Committee_on_Standards_in_Public_Life#The_Seven_Principles_of_Public_life
29. It’s not enough to mean well: principles & effects matter
30. Principles & effects matter…
31. Values & Integrity Research & Analytical Skills Life Skills Cognitive Skills
32. Principles can clash – & they can be hard to live up to
33. Take critics seriously…
34. There can be a fine line between idealism & becoming ideological
35. Arthur Miller Timebends: a life. 1987; New York: Grove Press.
36. Charisma & skill at manipulation are a risky combination
37. Values & Integrity Research & Analytical Skills Life Skills Cognitive Skills
38. Types of biases… •  Statistical •  Ideological •  Cognitive •  Commercial …
40. “[We] have made far more progress in cataloguing cognitive biases than in finding ways to correct them.” Lilienfield et al (2009). Perspectives Psychol Sci. 4 (4): 390-398.
41. Build bias minimization skills – & slow down
42. Thanks!