20190712-iqbio.pdf

Transcript

1. Replicability Hackathon 2019
   IQ BIO REU, UPR-RP
   Data Sciences, Dana-Farber Cancer Institute
   Biostatistics, Harvard TH Chan School of Public Health
   Patrick Kimes, PhD Kelly Street, PhD
   

   View Slide

2. HEALTHCARE
   INNOVATION
   REPLICATHON
   12 y 13 de julio de 2019
   

   View Slide

3. Hi, I’m Kelly
   

   View Slide

4. Hi, I’m Patrick
   PhD, Statistics
   BA, Mathematics
   

   View Slide

5. I’d like to thank the Academy…
   Keegan
   Korthauer, PhD
   Alejandro
   Reyes, PhD
   for making this
   Patricia Ordóñez, PhD
   Juan S. Ramírez-Lugo, PhD
   Rafael Irizarry, PhD
   for making this possible
   

   View Slide

6. Code of Conduct
   https://github.com/pkimes/PR2019replicathon/blob/master/code_of_conduct.md
   Assume competence in the people you interact with.
   There are no stupid questions.
   Be considerate in speech and actions, and actively seek
   to acknowledge and respect the boundaries of fellow
   community members.
   Take care of each other. Alert one of the organizers or
   facilitators if you notice a dangerous situation, someone
   in distress, or a potential violation of this Code of
   Conduct, even if it seems inconsequential.
   We do not tolerate harassment in any form.
   

   View Slide

7. Replicability Hackathon 2019
   IQ BIO REU, UPR-RP
   Data Sciences, Dana-Farber Cancer Institute
   Biostatistics, Harvard TH Chan School of Public Health
   Patrick Kimes, PhD Kelly Street, PhD
   

   View Slide

8. http://jtleek.com/talks.html
   

   View Slide

9. View Slide

10. “many scientific studies are difficult or impossible to
    replicate or reproduce.”
    

    View Slide

11. https://doi.org/10.1371/journal.pmed.0020124
    

    View Slide

12. https://doi.org/10.1371/journal.pmed.0020124
    “Simulations show that for most study designs and settings,
    it is more likely for a research claim to be false than true.”
    

    View Slide

13. “… replications of 100 experimental
    and correlational studies …”
    https://www.ncbi.nlm.nih.gov/pubmed/26315443
    

    View Slide

14. “39% of effects were subjectively
    rated to have replicated the
    original results.”
    “… replications of 100 experimental
    and correlational studies …”
    https://www.ncbi.nlm.nih.gov/pubmed/26315443
    

    View Slide

15. 39%??
    https://www.ncbi.nlm.nih.gov/pubmed/26315443
    

    View Slide

16. http://jtleek.com/talks.html
    

    View Slide

17. http://jtleek.com/talks.html
    

    View Slide

18. http://jtleek.com/talks.html
    

    View Slide

19. http://jtleek.com/talks.html
    

    View Slide

20. http://jtleek.com/talks.html
    

    View Slide

21. http://jtleek.com/talks.html
    

    View Slide

22. http://jtleek.com/talks.html
    

    View Slide

23. what’s going on?
    

    View Slide

24. let’s clarify some language
    reproducibility
    replicability
    

    View Slide

25. let’s clarify some language
    reproducibility
    replicability
    the ability to take the original data
    and the computer code used to
    analyze the data and reproduce all of
    the numerical findings from the study
    https://simplystatistics.org/2016/08/24/replication-crisis/
    

    View Slide

26. let’s clarify some language
    reproducibility
    replicability
    the ability to take the original data
    and the computer code used to
    analyze the data and reproduce all of
    the numerical findings from the study
    https://simplystatistics.org/2016/08/24/replication-crisis/
    

    View Slide

27. let’s clarify some language
    reproducibility
    replicability
    the ability to repeat an entire study,
    independent of the original investigator
    without the use of original data
    https://simplystatistics.org/2016/08/24/replication-crisis/
    

    View Slide

28. let’s clarify some language
    reproducibility
    replicability
    the ability to repeat an entire study,
    independent of the original investigator
    without the use of original data
    https://simplystatistics.org/2016/08/24/replication-crisis/
    

    View Slide

29. let’s clarify some language
    reproducibility
    replicability
    

    View Slide

30. replicability
    

    View Slide

31. what’s going on?
    replicability
    

    View Slide

32. replicability
    crisis:
    experiments are replicated.
    results not so much.
    

    View Slide

33. should we expect
    scientific results to
    always replicate?
    replicability
    

    View Slide

34. Psychology studies reproducibility 2
    https://www.nytimes.com/2015/09/01/opinion/psychology-is-not-in-crisis.html?
    “But the failure to replicate is not a cause for alarm;
    in fact, it is a normal part of how science works.”
    

    View Slide

35. https://simplystatistics.org/2016/08/24/replication-crisis/
    “the replication crisis in science is largely attributable to
    a mismatch in our expectations of how often findings
    should replicate and how difficult it is to actually
    discover true findings in certain fields.”
    

    View Slide

36. https://simplystatistics.org/2013/08/01/the-roc-curves-of-science/
    “…I argue that the rate of
    discoveries is higher in
    biomedical research than in
    physics. But, to achieve this higher
    true positive rate, biomedical
    research has to tolerate a higher
    false positive rate.”
    

    View Slide

37. should we expect
    scientific results to
    always replicate?
    replicability
    not always
    

    View Slide

38. crisis:
    experiments are replicated.
    results not so much.
    or maybe that’s science?
    replicability
    

    View Slide

39. 39%??
    

    View Slide

40. replicability crisis
    opportunity
    

    View Slide

41. Replicability Hackathon 2019
    IQ BIO REU, UPR-RP
    so what about …
    … ?
    

    View Slide

42. our challenge
    https://www.ncbi.nlm.nih.gov/pubmed/22460902
    https://www.ncbi.nlm.nih.gov/pubmed/22460905
    

    View Slide

43. our challenge
    https://www.ncbi.nlm.nih.gov/pubmed/22460902
    https://www.ncbi.nlm.nih.gov/pubmed/22460905
    

    View Slide

44. drug sensitivity in
    cancer cell lines
    

    View Slide

45. drug sensitivity in
    cancer cell lines
    what is a cell line?
    

    View Slide

46. drug sensitivity in
    cancer cell lines
    cell line
    cell culture from a
    single cell that can
    grow indefinitely given
    appropriate conditions
    • easily grown
    • relatively inexpensive
    • amenable to high-
    throughput testing
    https://www.ncbi.nlm.nih.gov/pubmed/26248648
    

    View Slide

47. drug sensitivity in
    cancer cell lines
    what do we mean by
    drug sensitivity?
    

    View Slide

48. drug sensitivity in
    cancer cell lines
    viability
    relative measure of
    cell line abundance
    after treatment
    

    View Slide

49. drug sensitivity in
    cancer cell lines
    dose response curve
    model fit to viability
    as a function of drug
    concentration
    

    View Slide

50. drug sensitivity in
    cancer cell lines
    IC50
    IC50
    concentration at
    which cell growth is
    inhibited 50%
    

    View Slide

51. drug sensitivity in
    cancer cell lines
    AUC
    area under the activity
    curve
    AUC
    

    View Slide

52. drug sensitivity in
    cancer cell lines
    got it!
    … but why?
    

    View Slide

53. drug sensitivity in
    cancer cell lines
    clinical
    https://www.ncbi.nlm.nih.gov/pubmed/26248648
    

    View Slide

54. drug sensitivity in
    cancer cell lines
    important, impactful
    research!
    

    View Slide

55. our challenge
    https://www.ncbi.nlm.nih.gov/pubmed/22460902
    https://www.ncbi.nlm.nih.gov/pubmed/22460905
    

    View Slide

56. https://www.ncbi.nlm.nih.gov/pubmed/24284626
    

    View Slide

57. https://www.ncbi.nlm.nih.gov/pubmed/24284626
    

    View Slide

58. https://www.ncbi.nlm.nih.gov/pubmed/27905415
    

    View Slide

59. “Upon careful analysis of the same data,
    we have come to quite different and
    much more positive conclusions.”
    https://www.ncbi.nlm.nih.gov/pubmed/27905415
    

    View Slide

60. View Slide

61. crisis:
    experiments are replicated.
    results not so much.
    or maybe yes?
    replicability
    opportunity
    

    View Slide

62. enough with the what,
    on to the data
    

    View Slide

63. our challenge
    https://www.github.com/pkimes/PR2019replicathon
    

    View Slide

64. raw data
    

    View Slide

65. summarized data
    

    View Slide

66. summarized data
    

    View Slide

67. model
    summarized data
    

    View Slide

68. our challenge
    datasets
    

    View Slide

69. our challenge
    datasets
    

    View Slide

70. our challenge
    datasets
    > setwd(‘path/to/PR2019replicathon’)
    > raw <- readRDS(‘data/rawPharmacoData.rds’)
    > ls()
    [1] “raw”
    

    View Slide

71. our challenge
    template
    

    View Slide

72. our challenge
    template
    

    View Slide

73. our challenge
    template
    

    View Slide

74. our challenge
    tutorials
    

    View Slide

75. our challenge
    tutorials
    

    View Slide

76. our challenge
    do these
    studies agree?
    

    View Slide

77. Let’s Hack!
    

    View Slide

78. getting setup with
    

    View Slide

79. View Slide

80. View Slide

81. View Slide

82. feel free to specify a
    different location
    

    View Slide

83. ready to go!
    

    View Slide

84. https://pkimes.github.io/PR2019replicathon/
    

    View Slide

85. Let’s Hack!
    Gigaprobe
    learnlispandRTFM
    

    View Slide