A minimum standard for publishing computational results

This paper was presented at the AMOS National Conference, Brisbane Convention and Exhibition Centre, Brisbane, Queensland.

A more detailed account of the work can be found at Authorea: https://www.authorea.com/users/5641/articles/15874/_show_article

Transcript

1. Damien Irving
   AMOS Conference, Brisbane 2015
   A minimum standard for publishing
   computational results in the weather
   and climate sciences
   

   View Slide

2. Overview
   1.  The reproducibility crisis
   2.  A reproducible paper
   3.  A new minimum standard
   

   View Slide

3. The reproducibility crisis
   •  Our field has rapidly transitioned to a
   computational science
   •  Conventions around communicating our
   methods have hardly changed
   –  Have you ever seen a paper provide code and
   software details?
   •  It’s impossible to replicate the results
   presented in journal papers today
   

   View Slide

4. The crisis response
   •  Funding agencies + journals1
   –  Some progress on dataset disclosure
   •  Most weather/climate journals have policies
   •  Not consistently enforced
   –  Weak or non-existent code requirements
   •  It’s not their fault
   –  No examples to base new standards on
   –  I set about addressing this deficiency…
   1. Stodden et al. 2013. PLoS ONE, 8, e67111
   

   View Slide

5. A reproducible paper - rationale
   •  Procedure needs to:
   –  Minimise the time involved1
   –  Minimise complexity of required tools
   –  Be consistent with computational best
   practices
   1. Stodden (2010). doi:10.2139/ssrn.1550193
   

   View Slide

6. A reproducible paper - components
   •  Irving D, Simmonds I (in press). A novel
   approach to diagnosing Southern
   Hemisphere planetary wave activity and
   its influence on regional climate variability.
   Journal of Climate. doi:10.1175/JCLI-
   D-15-0287.1
   –  Preprint:
   https://www.authorea.com/users/5641/
   articles/12197/_show_article
   –  Includes a brief computation section…
   

   View Slide

7. •  Cites key software packages
   •  Points to supplementary material
   –  Software description, code repository, log files
   –  Hosted at GitHub & Figshare:
   http://dx.doi.org/10.6084/m9.figshare.
   1385387
   Computation section
   

   View Slide

8. Software Description
   •  Name, version number, release date,
   institution and DOI or URL
   

   View Slide

9. Code repository
   •  Consistent with computational best
   practice1
   –  Write scripts
   –  Modularise, don’t copy/paste -> code library
   –  Use version control
   •  Your everyday repository is fine
   https://github.com/DamienIrving/climate-
   analysis
   1. Wilson et al. 2014. PLoS Biol, 12, e1001745
   

   View Slide

10. Log files
    •  Follow the NCO / CDO approach…
    –  Can generate timestamps with any language
    –  Features:
    •  Simple
    •  Read/writeable by anyone
    •  Easy to regenerate (no manual editing)
    

    View Slide

11. A new minimum standard
    •  Authors must include brief computation
    section which cites software and points to
    supplementary materials:
    –  Software description
    –  Code repository (public, version controlled)
    –  Log files
    •  Authors not obliged to provide assistance
    •  Reviews only need to check availability
    

    View Slide

12. Aim higher!
    •  Minimum standard is reproducible, but not
    very comprehensible
    –  Encouraging to see initiatives like the CWSLab
    workflow tool: http://cwslab.nci.org.au/
    

    View Slide

13. Conclusion
    •  There is a reproducibility crisis in weather/
    climate/ocean research
    •  This can be solved by adding a brief
    computation section to papers which
    points to supplementary materials:
    –  Software description
    –  Code repository (public, version controlled)
    –  Log files
    •  Journals could adopt this framework as a
    formal minimum standard
    

    View Slide

14. Look out for the BAMS essay!
    https://www.authorea.com/users/
    5641/articles/15874/_show_article
    

    View Slide