Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reproducibility & Minimizing Bias

Reproducibility & Minimizing Bias

Hilda Bastian

June 14, 2018
Tweet

More Decks by Hilda Bastian

Other Decks in Research

Transcript

  1. Reproducibility & Minimizing Bias
    Research Reproducibility 2018: Short
    Course Science
    University of Utah, Salt Lake City
    14 June 2018
    Hilda Bastian
    @hildabast
    hildabastian.net

    View Slide

  2. “Statistical workers who fail to scrutinize the
    goodness of their observed data and carry
    through a satisfactory analysis upon poor
    observations, will end up with ridiculous
    conclusions which cannot be maintained”.
    Raymond Pearl, 1919
    Via Doug Altman, Iveta Simera: http://www.jameslindlibrary.org/articles/a-
    history-of-the-evolution-of-guidelines-for-reporting-medical-research-the-
    long-road-to-the-equator-network/

    View Slide

  3. •  Methods reproducibility: enough detail to be able to
    repeat it
    •  Results reproducibility: it replicates
    •  Inferential reproducibility: similar conclusions drawn
    about it
    Defining research reproducibility
    Goodman, Fanelli, Ioannidis (2016). Science Translational Medicine; 8 (341) ps12
    —  But even this still
    doesn’t cover
    everything in this
    debate

    View Slide

  4. “Reproducibility” is a euphemism

    View Slide

  5. Specialization & complexity has grown
    1850
    Beaver & Rosen (2005). Scientometrics; 1(3). http://
    www.akademiai.com/doi/abs/10.1007/BF02016308
    1st co-authored
    article (1870)
    1st article with >1,000
    authors (2004)
    INTERNET
    Wikipedia
    (2001)
    40% of papers
    have co-authors
    (1940)

    View Slide

  6. View Slide

  7. “The idealized expert-
    generated, one-way,
    authoritative reign of
    science is over.”
    Debating science’s problems in 2016:
    now it’s everybody’s problem
    Sobo et al (2016). Medical Anthropology.
    35 (6): 529-546.

    View Slide

  8. View Slide

  9. •  Research on scientific
    methods – across the
    full spectrum of
    activity and use
    •  Detailed registered
    protocols for as much
    research as possible

    View Slide

  10. •  Collaborative practice
    •  Optimizing fidelity

    View Slide

  11. •  Good data,
    analysis, and
    reporting
    practice

    View Slide

  12. —  Communicating results in ways that can be used
    by non-specialists

    View Slide

  13. •  The Yin & Yang of post-
    publication activity
    •  Valuing quality,
    replication and validation
    studies

    View Slide

  14. •  Strengthen culture of constructive criticism
    •  Consequences for authors of non-response to
    important questions & criticism
    •  More accountability and consequences for editors,
    reviewers, and journals (Südhof (2016). PLOS Biology 14(8): e1002547.)

    View Slide

  15. View Slide

  16. View Slide

  17. Knowing what we (don’t) know before
    doing another study

    View Slide

  18. Systematic review
    •  Pre-specified/registered
    protocol
    •  Clear, precise question(s)
    •  Search strategy
    •  Inclusion/exclusion
    criteria
    •  Data extraction method
    •  Quality assessment
    •  Pre-specified analyses &
    methods

    View Slide

  19. Systematic reviews –
    gateway to meta-research

    View Slide

  20. Studies are a collection of parts of
    uneven quality

    View Slide

  21. Keep up with what are “questionable
    research practices” & how to avoid the
    pitfalls
    https://replicationindex.wordpress.com/2015/01/24/questionable-research-
    practices-definition-detect-and-recommendations-for-better-practices/

    View Slide

  22. Skills: design critical appraisal

    View Slide

  23. Bias prevention beyond research design
    •  Diversity in
    participation and
    areas of study
    •  Cognitive biases
    e.g. confirmation
    bias, motivated
    reasoning

    View Slide

  24. Everyday choices build culture
    Virtuous spiral of trust

    View Slide

  25. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills

    View Slide

  26. 7 Principles of Public Life
    https://en.wikipedia.org/wiki/Committee_on_Standards_in_Public_Life#The_Seven_Principles_of_Public_life

    View Slide

  27. View Slide

  28. View Slide

  29. It’s not enough to mean well:
    principles & effects matter

    View Slide

  30. Principles & effects matter…

    View Slide

  31. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills

    View Slide

  32. Principles can clash –
    & they can be hard to live up to

    View Slide

  33. Take critics seriously…

    View Slide

  34. There can be a fine line between
    idealism & becoming ideological

    View Slide

  35. Arthur Miller
    Timebends: a life. 1987; New York: Grove Press.

    View Slide

  36. Charisma & skill at manipulation
    are a risky combination

    View Slide

  37. Values &
    Integrity
    Research &
    Analytical
    Skills
    Life Skills
    Cognitive
    Skills

    View Slide

  38. Types of biases…
    •  Statistical
    •  Ideological
    •  Cognitive
    •  Commercial …

    View Slide

  39. View Slide

  40. “[We] have made far
    more progress in
    cataloguing cognitive
    biases than in finding
    ways to correct them.”
    Lilienfield et al (2009). Perspectives
    Psychol Sci. 4 (4): 390-398.

    View Slide

  41. Build bias minimization skills –
    & slow down

    View Slide

  42. Thanks!

    View Slide