Slide 1

Slide 1 text

Improve your work fl ow for reproducible science Mine Çetinkaya-Rundel Duke University + RStudio mine-cetinkaya-rundel [email protected] @minebocek 🔗 bit.ly/improve-repro-workflow

Slide 2

Slide 2 text

The results in Table 1 don’t seem to correspond to those in Figure 2!

Slide 3

Slide 3 text

61 3 44 94 12 4 45 20

Slide 4

Slide 4 text

70 have tried and failed to reproduce another scientist's experiments more than percent Monya Baker,. "1,500 scientists li ft the lid on reproducibility." Nature News 533.7604 (2016): 452.

Slide 5

Slide 5 text

50 have tried and failed to reproduce their own experiments more than percent Monya Baker. "1,500 scientists li ft the lid on reproducibility." Nature News 533.7604 (2016): 452.

Slide 6

Slide 6 text

1070 Google Scholar Search, 28 October 2021. results containing the term reproducibility crisis just in 2021 Google Scholar yields

Slide 7

Slide 7 text

1992 Jon Claerbout and Martin Karrenbach. "Electronic documents give reproducible research a new meaning." SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, 1992. 601-604. earliest reference reproducibility research* that I could find…

Slide 8

Slide 8 text

Jon Claerbout and Martin Karrenbach. "Electronic documents give reproducible research a new meaning." SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, 1992. 601-604.

Slide 9

Slide 9 text

Jon Claerbout and Martin Karrenbach. "Electronic documents give reproducible research a new meaning." SEG Technical Program Expanded Abstracts 1992. Society of Exploration Geophysicists, 1992. 601-604.

Slide 10

Slide 10 text

Michelle Lewis, et al. "Replication Study: Transcriptional amplification in tumor cells with elevated c-Myc." Elife 7 (2018): e30274.

Slide 11

Slide 11 text

Photo by Alexander Dummer on Unsplash]. setting the stage

Slide 12

Slide 12 text

replicability reproducibility same research question same research question same results same results new data same data

Slide 13

Slide 13 text

term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length. Figure 2. Relationship between bill depth and flipper length. e.g.

Slide 14

Slide 14 text

term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length. Figure 2. Relationship between bill depth and flipper length. e.g.

Slide 15

Slide 15 text

analysis report term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length.

Slide 16

Slide 16 text

analysis report term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length. Figure 2. Relationship between bill depth and flipper length.

Slide 17

Slide 17 text

analysis report Figure 2. Relationship between bill depth and flipper length. term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length.

Slide 18

Slide 18 text

term estimate std.error statistic p.value (Intercept) 33.6 1.25 27.0 1.39e-86 flipper_length_mm -0.0820 0.00618 -13.3 1.23e-32 Table 1. Regression output for predicting bill depth from flipper length. Figure 2. Relationship between bill depth and flipper length.

Slide 19

Slide 19 text

making research reproducible

Slide 20

Slide 20 text

raw data code & documentation to reproduce the analysis specifications of your computational environment make available and accessible Peng, Roger. "The reproducibility crisis in science: A statistical counterattack." Significance 12.3 (2015): 30-32. Gentleman, Robert, and Duncan Temple Lang. "Statistical analyses and reproducible research." Journal of Computational and Graphical Statistics 16.1 (2007): 1-23.

Slide 21

Slide 21 text

– Keith Baggerly “The most important tool is the mindset, when starting, that the end product will be reproducible.”

Slide 22

Slide 22 text

nobody, not even yourself, can recreate any part of your analysis push button reproducibility in published work 💃 🎯

Slide 23

Slide 23 text

“There’s no one-size-fits-all solution for computational reproducibility.” Perkel, Je ff rey M. "A toolkit for data transparency takes shape." Nature 560 (2018): 513-515.

Slide 24

Slide 24 text

but the following might help… 8 principles

Slide 25

Slide 25 text

organize your project 1

Slide 26

Slide 26 text

level of organization

Slide 27

Slide 27 text

simpler analysis raw - data processed - data manuscript | - manuscript.Rmd more complex analysis raw - data processed - data scripts manuscript f i gures | - manuscript.Rmd stick with the conventions of your peers

Slide 28

Slide 28 text

write READMEs liberally 2

Slide 29

Slide 29 text

raw - data processed - data scripts manuscript f i gures | - README.md | - airports.csv | - flights.csv | - planes.csv | - weather.csv # README This folder contains the raw data for the project. All datasets were downloaded from openflights.org/data.html on 2019-04-01. -airlines: Airline names -airports: Airports metadata -flights: F l ight data -planes: Plane metadata -weather: Hourly weather data | - airlines.csv

Slide 30

Slide 30 text

keep data tidy & machine readable 3

Slide 31

Slide 31 text

Student Exam Grade Name 1 2 Major Barney Donaldson 89 76 Data Science, Public Policy Clay Whelan 67 83 Public Policy Simran Bass 82 90 Statistics Chante Munro 45 72 Political Science, Statistics Gabrielle Cherry 32 79 . Kush Piper 98 sick Statistics Faizan Ratliff 82 75 Data Science Torin Ruiz 70 80 Sociology, Statistics Reiss Richardson missed exam 34 Neuroscience Ajwa Cochran 50 65 Data Science Low participation name exam_1 exam_2 f i rst_major second_major participation Barney Donaldson 89 76 Data Science Public Policy ok Clay Whelan 67 83 Public Policy NA ok Simran Bass 82 90 Statistics NA ok Chante Munro 45 72 Political Science Statistics low Gabrielle Cherry 32 79 NA NA ok Kush Piper 98 NA Statistics NA ok Faizan Ratliff 82 75 Data Science NA ok Torin Ruiz 70 80 Sociology Statistics ok Reiss Richardson NA 34 Neuroscience NA low Ajwa Cochran 50 65 Data Science NA low record code + document non-code steps + write tests Broman, Karl W., and Kara H. Woo. "Data organization in spreadsheets." The American Statistician 72.1 (2018): 2-10.

Slide 32

Slide 32 text

comment your code 4

Slide 33

Slide 33 text

🤷

Slide 34

Slide 34 text

use literate programming 5

Slide 35

Slide 35 text

No content

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

demo rmarkdown

Slide 38

Slide 38 text

‣ Learn more about R Markdown: ‣ Documentation: rmarkdown.rstudio.com ‣ Book: bookdown.org/yihui/rmarkdown ‣ Book: bookdown.org/yihui/rmarkdown-cookbook ‣ Learn more about the visual editor: ‣ Documentation: rstudio.github.io/visual-markdown-editing more resources…

Slide 39

Slide 39 text

use version control 6

Slide 40

Slide 40 text

changes tracked by hosted on

Slide 41

Slide 41 text

2 Git work fl ows GitHub fi rst Local fi rst

Slide 42

Slide 42 text

GitHub fi rst Today I start a new project! So I’ll do the right thing and create a repo first. ‣ Step 1: Create a new repo on GitHub ‣ Step 2: Copy the repo URL ‣ Step 3: Clone it using RStudio ‣ Step 4: Make changes locally ‣ Step 6: Commit and push to GitHub ‣ Step 7: Confirm your changes have propagated to GitHub

Slide 43

Slide 43 text

Local fi rst I have been working on a project for a while, and now I’m realising I should have been tracking it with git. ‣ Step 1: Create an RStudio Project from existing directory (if an .Rproj file doesn’t already exist) ‣ Step 2: usethis::use_git() and follow instructions ‣ Step 3: usethis::use_github() and follow instructions

Slide 44

Slide 44 text

demo git & github

Slide 45

Slide 45 text

‣ Learn more about using Git and GitHub with R: ‣ Book: happygitwithr.com ‣ Learn more about Git setup: ‣ Documentation: usethis.r-lib.org/articles/articles/usethis-setup.html more resources…

Slide 46

Slide 46 text

automate your process 7

Slide 47

Slide 47 text

raw - data processed - data scripts manuscript f i gures | - 01-load - packages.R | - 03-clean - data.R | - 04-explore.R | - 05-model.R | - 06-summarise.R | - 02-load - data.R | - 00-analyse.R

Slide 48

Slide 48 text

Karl Broman. “Minimal Make”, kbroman.org/minimal_make. 📖 recommended reading

Slide 49

Slide 49 text

Will Landau. “The targets R Package User Manual”, books.ropensci.org/targets. 📖 recommended reading

Slide 50

Slide 50 text

share computing environment 8

Slide 51

Slide 51 text

No content

Slide 52

Slide 52 text

1 organize your project 2 write READMEs liberally 3 keep data tidy & machine readable 4 comment your code 5 use literate programming 6 use version control 7 automate your process 8 share computing environment

Slide 53

Slide 53 text

Greg Wilson, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, Tracy K. Teal “Good enough practices in scientific computing." PLoS computational biology 13.6 (2017): e1005510.

Slide 54

Slide 54 text

Improve your work fl ow for reproducible science mine-cetinkaya-rundel [email protected] @minebocek 🔗 bit.ly/improve-repro-workflow