Reproducibility & Minimizing Bias

Reproducibility & Minimizing Bias

B7cccea0f2a6ef54318ec7ce197a438f?s=128

Hilda Bastian

June 14, 2018
Tweet

Transcript

  1. Reproducibility & Minimizing Bias Research Reproducibility 2018: Short Course Science

    University of Utah, Salt Lake City 14 June 2018 Hilda Bastian @hildabast hildabastian.net
  2. “Statistical workers who fail to scrutinize the goodness of their

    observed data and carry through a satisfactory analysis upon poor observations, will end up with ridiculous conclusions which cannot be maintained”. Raymond Pearl, 1919 Via Doug Altman, Iveta Simera: http://www.jameslindlibrary.org/articles/a- history-of-the-evolution-of-guidelines-for-reporting-medical-research-the- long-road-to-the-equator-network/
  3. •  Methods reproducibility: enough detail to be able to repeat

    it •  Results reproducibility: it replicates •  Inferential reproducibility: similar conclusions drawn about it Defining research reproducibility Goodman, Fanelli, Ioannidis (2016). Science Translational Medicine; 8 (341) ps12 —  But even this still doesn’t cover everything in this debate
  4. “Reproducibility” is a euphemism

  5. Specialization & complexity has grown 1850 Beaver & Rosen (2005).

    Scientometrics; 1(3). http:// www.akademiai.com/doi/abs/10.1007/BF02016308 1st co-authored article (1870) 1st article with >1,000 authors (2004) INTERNET Wikipedia (2001) 40% of papers have co-authors (1940)
  6. None
  7. “The idealized expert- generated, one-way, authoritative reign of science is

    over.” Debating science’s problems in 2016: now it’s everybody’s problem Sobo et al (2016). Medical Anthropology. 35 (6): 529-546.
  8. None
  9. •  Research on scientific methods – across the full spectrum

    of activity and use •  Detailed registered protocols for as much research as possible
  10. •  Collaborative practice •  Optimizing fidelity

  11. •  Good data, analysis, and reporting practice

  12. —  Communicating results in ways that can be used by

    non-specialists
  13. •  The Yin & Yang of post- publication activity • 

    Valuing quality, replication and validation studies
  14. •  Strengthen culture of constructive criticism •  Consequences for authors

    of non-response to important questions & criticism •  More accountability and consequences for editors, reviewers, and journals (Südhof (2016). PLOS Biology 14(8): e1002547.)
  15. None
  16. None
  17. Knowing what we (don’t) know before doing another study

  18. Systematic review •  Pre-specified/registered protocol •  Clear, precise question(s) • 

    Search strategy •  Inclusion/exclusion criteria •  Data extraction method •  Quality assessment •  Pre-specified analyses & methods
  19. Systematic reviews – gateway to meta-research

  20. Studies are a collection of parts of uneven quality

  21. Keep up with what are “questionable research practices” & how

    to avoid the pitfalls https://replicationindex.wordpress.com/2015/01/24/questionable-research- practices-definition-detect-and-recommendations-for-better-practices/
  22. Skills: design critical appraisal

  23. Bias prevention beyond research design •  Diversity in participation and

    areas of study •  Cognitive biases e.g. confirmation bias, motivated reasoning
  24. Everyday choices build culture Virtuous spiral of trust

  25. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills
  26. 7 Principles of Public Life https://en.wikipedia.org/wiki/Committee_on_Standards_in_Public_Life#The_Seven_Principles_of_Public_life

  27. None
  28. None
  29. It’s not enough to mean well: principles & effects matter

  30. Principles & effects matter…

  31. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills
  32. Principles can clash – & they can be hard to

    live up to
  33. Take critics seriously…

  34. There can be a fine line between idealism & becoming

    ideological
  35. Arthur Miller Timebends: a life. 1987; New York: Grove Press.

  36. Charisma & skill at manipulation are a risky combination

  37. Values & Integrity Research & Analytical Skills Life Skills Cognitive

    Skills
  38. Types of biases… •  Statistical •  Ideological •  Cognitive • 

    Commercial …
  39. None
  40. “[We] have made far more progress in cataloguing cognitive biases

    than in finding ways to correct them.” Lilienfield et al (2009). Perspectives Psychol Sci. 4 (4): 390-398.
  41. Build bias minimization skills – & slow down

  42. Thanks!