Reproducibility & Minimizing Bias

Transcript

1. Reproducibility & Minimizing Bias
   Research Reproducibility 2018: Short
   Course Science
   University of Utah, Salt Lake City
   14 June 2018
   Hilda Bastian
   @hildabast
   hildabastian.net
   

   View Slide

2. “Statistical workers who fail to scrutinize the
   goodness of their observed data and carry
   through a satisfactory analysis upon poor
   observations, will end up with ridiculous
   conclusions which cannot be maintained”.
   Raymond Pearl, 1919
   Via Doug Altman, Iveta Simera: http://www.jameslindlibrary.org/articles/a-
   history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-
   long-road-to-the-equator-network/
   

   View Slide

3. •  Methods reproducibility: enough detail to be able to
   repeat it
   •  Results reproducibility: it replicates
   •  Inferential reproducibility: similar conclusions drawn
   about it
   Defining research reproducibility
   Goodman, Fanelli, Ioannidis (2016). Science Translational Medicine; 8 (341) ps12
   —  But even this still
   doesn’t cover
   everything in this
   debate
   

   View Slide

4. “Reproducibility” is a euphemism
   

   View Slide

5. Specialization & complexity has grown
   1850
   Beaver & Rosen (2005). Scientometrics; 1(3). http://
   www.akademiai.com/doi/abs/10.1007/BF02016308
   1st co-authored
   article (1870)
   1st article with >1,000
   authors (2004)
   INTERNET
   Wikipedia
   (2001)
   40% of papers
   have co-authors
   (1940)
   

   View Slide

6. View Slide

7. “The idealized expert-
   generated, one-way,
   authoritative reign of
   science is over.”
   Debating science’s problems in 2016:
   now it’s everybody’s problem
   Sobo et al (2016). Medical Anthropology.
   35 (6): 529-546.
   

   View Slide

8. View Slide

9. •  Research on scientific
   methods – across the
   full spectrum of
   activity and use
   •  Detailed registered
   protocols for as much
   research as possible
   

   View Slide

10. •  Collaborative practice
    •  Optimizing fidelity
    

    View Slide

11. •  Good data,
    analysis, and
    reporting
    practice
    

    View Slide

12. —  Communicating results in ways that can be used
    by non-specialists
    

    View Slide

13. •  The Yin & Yang of post-
    publication activity
    •  Valuing quality,
    replication and validation
    studies
    

    View Slide

14. •  Strengthen culture of constructive criticism
    •  Consequences for authors of non-response to
    important questions & criticism
    •  More accountability and consequences for editors,
    reviewers, and journals (Südhof (2016). PLOS Biology 14(8): e1002547.)
    

    View Slide

15. View Slide

16. View Slide

17. Knowing what we (don’t) know before
    doing another study
    

    View Slide

18. Systematic review
    •  Pre-specified/registered
    protocol
    •  Clear, precise question(s)
    •  Search strategy
    •  Inclusion/exclusion
    criteria
    •  Data extraction method
    •  Quality assessment
    •  Pre-specified analyses &
    methods
    

    View Slide

19. Systematic reviews –
    gateway to meta-research
    

    View Slide

20. Studies are a collection of parts of
    uneven quality
    

    View Slide

21. Keep up with what are “questionable
    research practices” & how to avoid the
    pitfalls
    https://replicationindex.wordpress.com/2015/01/24/questionable-research-
    practices-definition-detect-and-recommendations-for-better-practices/
    

    View Slide

22. Skills: design critical appraisal
    

    View Slide

23. Bias prevention beyond research design
    •  Diversity in
    participation and
    areas of study
    •  Cognitive biases
    e.g. confirmation
    bias, motivated
    reasoning
    

    View Slide

24. Everyday choices build culture
    Virtuous spiral of trust
    

    View Slide

25. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills
    

    View Slide

26. 7 Principles of Public Life
    https://en.wikipedia.org/wiki/Committee_on_Standards_in_Public_Life#The_Seven_Principles_of_Public_life
    

    View Slide

27. View Slide

28. View Slide

29. It’s not enough to mean well:
    principles & effects matter
    

    View Slide

30. Principles & effects matter…
    

    View Slide

31. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills
    

    View Slide

32. Principles can clash –
    & they can be hard to live up to
    

    View Slide

33. Take critics seriously…
    

    View Slide

34. There can be a fine line between
    idealism & becoming ideological
    

    View Slide

35. Arthur Miller
    Timebends: a life. 1987; New York: Grove Press.
    

    View Slide

36. Charisma & skill at manipulation
    are a risky combination
    

    View Slide

37. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills
    

    View Slide

38. Types of biases…
    •  Statistical
    •  Ideological
    •  Cognitive
    •  Commercial …
    

    View Slide

39. View Slide

40. “[We] have made far
    more progress in
    cataloguing cognitive
    biases than in finding
    ways to correct them.”
    Lilienfield et al (2009). Perspectives
    Psychol Sci. 4 (4): 390-398.
    

    View Slide

41. Build bias minimization skills –
    & slow down
    

    View Slide

42. Thanks!
    

    View Slide