Reproducibility & Minimizing Bias

Transcript

1. Reproducibility & Minimizing Bias Research Reproducibility 2018: Short Course Science

   University of Utah, Salt Lake City 14 June 2018 Hilda Bastian @hildabast hildabastian.net

2. “Statistical workers who fail to scrutinize the goodness of their

   observed data and carry through a satisfactory analysis upon poor observations, will end up with ridiculous conclusions which cannot be maintained”. Raymond Pearl, 1919 Via Doug Altman, Iveta Simera: http://www.jameslindlibrary.org/articles/a- history-of-the-evolution-of-guidelines-for-reporting-medical-research-the- long-road-to-the-equator-network/

3. •  Methods reproducibility: enough detail to be able to repeat

   it •  Results reproducibility: it replicates •  Inferential reproducibility: similar conclusions drawn about it Defining research reproducibility Goodman, Fanelli, Ioannidis (2016). Science Translational Medicine; 8 (341) ps12 —  But even this still doesn’t cover everything in this debate

4. “Reproducibility” is a euphemism

5. Specialization & complexity has grown 1850 Beaver & Rosen (2005).

   Scientometrics; 1(3). http:// www.akademiai.com/doi/abs/10.1007/BF02016308 1st co-authored article (1870) 1st article with >1,000 authors (2004) INTERNET Wikipedia (2001) 40% of papers have co-authors (1940)
6. None

7. “The idealized expert- generated, one-way, authoritative reign of science is

   over.” Debating science’s problems in 2016: now it’s everybody’s problem Sobo et al (2016). Medical Anthropology. 35 (6): 529-546.
8. None

9. •  Research on scientific methods – across the full spectrum

   of activity and use •  Detailed registered protocols for as much research as possible

10. •  Collaborative practice •  Optimizing fidelity

11. •  Good data, analysis, and reporting practice

12. —  Communicating results in ways that can be used by

    non-specialists

13. •  The Yin & Yang of post- publication activity • 

    Valuing quality, replication and validation studies

14. •  Strengthen culture of constructive criticism •  Consequences for authors

    of non-response to important questions & criticism •  More accountability and consequences for editors, reviewers, and journals (Südhof (2016). PLOS Biology 14(8): e1002547.)
15. None
16. None

17. Knowing what we (don’t) know before doing another study

18. Systematic review •  Pre-specified/registered protocol •  Clear, precise question(s) • 

    Search strategy •  Inclusion/exclusion criteria •  Data extraction method •  Quality assessment •  Pre-specified analyses & methods

19. Systematic reviews – gateway to meta-research

20. Studies are a collection of parts of uneven quality

21. Keep up with what are “questionable research practices” & how

    to avoid the pitfalls https://replicationindex.wordpress.com/2015/01/24/questionable-research- practices-definition-detect-and-recommendations-for-better-practices/

22. Skills: design critical appraisal

23. Bias prevention beyond research design •  Diversity in participation and

    areas of study •  Cognitive biases e.g. confirmation bias, motivated reasoning

24. Everyday choices build culture Virtuous spiral of trust

25. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills

26. 7 Principles of Public Life https://en.wikipedia.org/wiki/Committee_on_Standards_in_Public_Life#The_Seven_Principles_of_Public_life

27. None
28. None

29. It’s not enough to mean well: principles & effects matter

30. Principles & effects matter…

31. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills

32. Principles can clash – & they can be hard to

    live up to

33. Take critics seriously…

34. There can be a fine line between idealism & becoming

    ideological

35. Arthur Miller Timebends: a life. 1987; New York: Grove Press.

36. Charisma & skill at manipulation are a risky combination

37. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills

38. Types of biases… •  Statistical •  Ideological •  Cognitive • 

    Commercial …
39. None

40. “[We] have made far more progress in cataloguing cognitive biases

    than in finding ways to correct them.” Lilienfield et al (2009). Perspectives Psychol Sci. 4 (4): 390-398.

41. Build bias minimization skills – & slow down

42. Thanks!