Journal of Open Source Software: When collaborative open source meets peer review

03e2e7de45b193cac192ae7ea071e5ff?s=47 Arfon Smith
October 16, 2019

Journal of Open Source Software: When collaborative open source meets peer review

This is a presentation I gave at FORCE2019 (https://www.force11.org/meetings/force2019) about the Journal of Open Source Software.

In this presentation I try and explain how JOSS works, and how we're leveraging open source conventions to build a low cost, open access journal.

03e2e7de45b193cac192ae7ea071e5ff?s=128

Arfon Smith

October 16, 2019
Tweet

Transcript

  1. Journal of Open Source Software Arfon M. Smith (@arfon), Tania

    Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J. Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano, Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson, Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake Vanderplas, Bruce E. Wilson, Yo Yehudi https://joss.theoj.org When collaborative open source meets peer review
  2. None
  3. Software is important

  4. Software is everywhere

  5. Software turns a theoretical model into quantitative predictions; software controls

    an experiment; and software extracts from raw data evidence supporting or rejecting a theory. Gaël Varoquaux – scikit-learn creator
  6. None
  7. None
  8. None
  9. None
  10. Software is eating the world* * Mark Andreessen

  11. the skills required to be a successful scientific researcher are

    increasingly indistinguishable from the skills required to be successful in industry. Jake VanderPlas (Google)
  12. We must find a way to legitimize software as a

    form of scholarship. Phil Bourne, University of Virginia
  13. Software isn’t a creditable research activity

  14. Challenges Technical Cultural Software isn’t easily citable ✔ ❌ Software

    citations aren’t allowed ❌ ✔ Software citations aren’t indexed ✔ ✔ Software isn’t peer reviewed ❌ ✔ Software can’t cite other software ✔ ❌
  15. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions?
  16. Develop mechanisms for native software citation – see FORCE11 software

    (and data) citation working groups. Make your software ‘citable’, and make it easy for people to cite you. Lobby/lean-on/encourage indexers to count software citations. Find some way to fit software into the current system
  17. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
  18. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
  19. Develop mechanisms for citing all of the constituent parts of

    a research endeavour e.g. data, software, methods/ protocols, results… One potential solution: Transitive Credit (Katz, 2014, https://doi.org/10.5334/ jors.be) Evolving beyond one- dimensional credit model
  20. Smith et al WMAP PLANK Jones et al Smith et

    al astropy scikit-learn Transitive Credit Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2
  21. Smith et al WMAP astropy PLANK Jones et al Smith

    et al scikit-learn Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2 Whyte et al numpy scipy 0.5 0.25 0.25
  22. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions? Really hard
  23. Individual change: Authors, reviewers, editors. Group/community change: Editorial teams, journals,

    publishers. Ecosystem change: Journals, publishers, indexers. Requires action across the whole scholarly ecosystem
  24. What if we just wrote papers about software?

  25. Software papers Gives us something easy to cite No changes

    required to existing infrastructure Publishing in existing journals raises profile of software within a community
  26. Software papers Writing another paper can be a ton of

    work Many journals don’t accept software papers For long-lived software packages, static authorship presents major issues Many papers about the same software may lead to citation dilution
  27. AAS Journals welcome articles which describe the design and function

    of software of relevance to research in astronomy and astrophysics. Such articles should contain a description of the software, its novel features and its intended use. Such articles need not include research results produced using the software, although including examples of applications can be helpful. There is no minimum length requirement for software articles.
  28. What if we made it as easy as possible to

    write and publish a software paper? Embracing the hack
  29. A developer friendly journal* for research software packages. Paper preparation

    (and submission) for well-documented software should take no more than an hour. The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. * Other venues exist for publishing papers about software
  30. Short – generally under two pages. A summary describing the

    high-level functionality and purpose of the software for a diverse, non-specialist audience. A clear Statement of Need that illustrates the research purpose of the software. Acknowledgement of any financial support. Reference to related work, e.g. research results derived using the software. Not permitted to contain novel results. Paper format
  31. Be as conventional as possible when it comes to integrating

    with the scholarly ecosystem GOAL ORCID for logins. Deposit metadata and issue DOIs with Crossref. Archive papers (and reviews) with Portico. Archive reviewed software with Dryad/ figshare/Zenodo. Get indexed (Google Scholar, Scopus etc.)
  32. Follow best practices where they exist GOAL Open access –

    papers licensed CC-BY (authors retain copyright). Enforce the Open Source Definition. Clear ethics, ownerships, business model and governance statements. Rigorous, transparent peer review process. Register with community directories (e.g. DOAJ).
  33. Provide a valuable service to authors GOAL Minimize author effort

    beyond existing software documentation they already have. Offer a constructive peer review process by experts. Focus on improving submissions, not rejecting. Transparent, achievable review criteria. Actually review the software… Zero APCs, open access.
  34. Leverage the best parts of open source where possible GOAL

    Carry out reviews in a familiar environment (on GitHub). Foster a collaborative, welcoming community. Transparent editorial decision making process. Leverage open source tools where possible (e.g. Pandoc) Automate all the things! Use robots to automate repetitive or labour-intensive tasks where possible.
  35. Submission and review process

  36. 2 https://joss.readthedocs.io/en/latest/index.html

  37. None
  38. None
  39. None
  40. Automated paper compilation Our editorial robot: Whedon Detecting programming language

  41. Editor agreeing Managing editor asking editors if they can take

    submission Author suggesting reviewers Managing editor asking Whedon to assign editor
  42. Reviewer #1 volunteering to review Editor asking Whedon to assign

    Reviewer #1 as reviewer for paper
  43. Reviewer #2 volunteering to review Editor asking Whedon to assign

    Reviewer #2 as reviewer for paper
  44. Editor asking Whedon to check the references for missing DOIs

    Whedon obliging
  45. Editor asking Whedon to start the main review

  46. Metadata (editor, author, reviewers) Badge to include on e.g. GitHub

  47. Checklist for each reviewer

  48. Reviewer opening issues on software repository

  49. Author asking Whedon to recompile paper

  50. Editor associating Zenodo software archive with paper Editor updating software

    version associated with review Editor tagging EiC team to accept
  51. Whedon produces final proofs of paper and Crossref metadata Managing

    editor asks Whedon to do a ‘dry run’ of accepting paper
  52. Whedon tweets link to paper Managing editor asks Whedon to

    accept paper …and confirms when DOI registered with Crossref
  53. Managing editor thanks reviewers and editor Whedon closes review

  54. None
  55. Whedon

  56. None
  57. None
  58. None
  59. https://peerj.com/articles/cs-147/

  60. None
  61. None
  62. None
  63. 0 200 400 600 800 2016-07-01 2017-01-01 2017-07-01 2018-01-01 2018-07-01

    2019-01-01 2019-07-01 Year 1: 104 (8.6 papers/month) Year 2: 181 (15.3 papers/month) Year 3: 281 (23.4 papers/month) Year 4: 148 (30.3 papers/month)
  64. 1000th paper in ~June 2020 500 papers/year in 2021 0

    500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
  65. https://blog.joss.theoj.org/2019/06/cost-models-for-running-an-online-open-journal

  66. Fixed costs we have to pay Annual Crossref membership: $275/year

    JOSS paper DOIs: $1/accepted paper JOSS website hosting: $19/month JOSS domain name registration: $10/year Portico fees: $250/year Total running costs: $1,063 $3.54/paper Assuming 300 papers published per year
  67. Including costs that we don’t currently pay Costs we now

    pay: $1,063/year Services for which we don’t currently pay: estimated at $20,600/year Volunteer editor services: 1.85 FTE effort or $370,000 (@$200,000/FTE) Reviewer time (not paid for in almost all journals, nor costed by JOSS) $1305/paper Assuming 300 papers published per year, excluding reviewer efforts
  68. But actually… Given that many journals don’t seem to pay

    editors much (if anything). If we include a modest editor stipend of $10,000 Total theoretical running costs: $31,663 $105/paper Assuming 300 papers published per year, excluding reviewer efforts
  69. Collaborative open source meets peer review

  70. The right to use, modify, and share not to contribute

    Open Source
  71. Open Collaborations: a highly collaborative development process and are receptive

    to contributions of code, documentation, discussion, etc. from anyone who shows competent interest. Karl Fogel – Producing Open Source Software
  72. None
  73. None
  74. Code first, permission later

  75. None
  76. None
  77. None
  78. None
  79. None
  80. None
  81. 200M Pull Requests ~70M In the last 12 months https://octoverse.github.com

  82. Receptive to contributions of code, documentation, discussion… https://opensource.com/life/16/5/growing-contributor-base-modern-open-source

  83. Collaborative open source Create more users through education. Encourage users

    to contribute through Liberal Contribution Policies. Enable more leaders through participatory governance.
  84. Open source contributor funnel: Mike McQuaid

  85. Open discussions

  86. Open discussions

  87. Receptive to contributions of code, documentation, discussion…

  88. Hacktoberfest: Global effort focused on contributions

  89. Reviewers Authors Editors Contributor flow @ JOSS

  90. Reviewers Authors Editors Contributor flow @ JOSS

  91. Bot-assisted communities

  92. Powered by ‘webhooks’

  93. Powered by ‘webhooks’

  94. None
  95. None
  96. None
  97. None
  98. None
  99. None
  100. What kind of journal is JOSS?

  101. A multi-disciplinary perspective on emergent and future innovations in peer

    review, Tennant et al., 2017
  102. continuous Post-publication commenting Informal discussion of published research, independent of

    any formal peer review that may have already occurred Can be performed on third-party platforms, anyone can contribute, public Comments can be rude or of low quality, comments across multiple platforms lack inter-operability, low visibility, low uptake PubMed Commons, PeerJ, PLOS, BMJ Collaborative A combination of referees, editors and external readers participate in the assessment of scientific manuscripts through interactive comments, often to reach a consensus decision, and a single set of revisions Iterative, transparent, editors sign reports, can be integrated with formal process, deters low quality submissions Can be additionally time-consuming, discussion quality variable, peer pressure and influence can tilt the balance eLife, Frontiers series, Copernicus journals, BMJ Open Science Portable Authors can take referee reports to multiple consecutive venues, often administered by a third-party service Reduces redundancy or duplication, saves time Low uptake by authors, low acceptance by journals, high cost BioMed Central journals, NPRC, Rubriq, Peerage of Science, MECA Recommendation services Post-publication evaluation and recommendation of significant articles, often through a peer- nominated consortium Crowd-sourced literature discovery, time saving, “prestige” factor when inside a consortium Paid services (subscription only), time consuming on recommender side, exclusive F1000 Prime, CiteULike Decoupled post-publication (annotation services) Comments or highlights added directly to highlighted sections of the work. Added notes can be private or public Rapid, crowd-sourced and collaborative, cross-publisher, low threshold for entry Non-interoperable, multiple venues, effort duplication, relatively unused, genuine critiques reserved PubPeer, Hypothesis, PaperHive, PeerLibrary or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic Advantages and disadvantages of the different approaches to peer review.
  103. is published Optional open peer review As single blind peer

    review, except that the referees are given the option to make their review and their name public Increased transparency Gives an unclear pictures of the review process if not all reviews are made public PeerJ, Nature Communications Pre-publication open peer review Referees are identified to authors pre-publication, and if the article is published, the full peer review history together with the names of the associated referees is made public Transparency, increased integrity of reviews Fear: referees may decline to review, or be unwilling to come across too critically or positively The medical BMC-series journals, The BMJ Post-publication open peer review The referee reports and the names of the referees are always made public regardless of the outcome of their review Fast publication, transparent process Fear: referees may decline to review, or be unwilling to come across too critically or positively F1000Research, ScienceOpen, PubPub, Publons Peer review by endorsement (PRE) Pre-arranged and invited, with referees providing a “stamp of approval” on publications Transparent, cost- effective, rapid, accountable Low uptake, prone to selection bias, not viewed as credible RIO Journal comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final- version commenting (Ross-Hellauer, 2017). The various parts of examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality Pros and cons of different approaches to anonymity in peer review.
  104. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  105. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  106. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  107. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  108. Some observations It seems to be working (i.e. we’re meeting

    a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
  109. Some observations People like the robot*… * Does cause occasional

    confusion
  110. Some observations People like the robot*… * Does cause occasional

    confusion
  111. Scaling JOSS Most of our challenges are about scaling people

    processes: • AEiC/managing editor rotations. • More editors. • Term limits for editors (to avoid burnout). Technology improvements: • Smarter reviewer assignments. • Better monitoring tools for editors. • Tools to help authors prepare their submissions. 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
  112. Unexpected consequences of working openly Semi-regular emails from people annoyed

    they haven’t been asked to review yet. Generally need relatively small number of invites to identify reviewers (~2 invites per reviewer). Vanity software package ‘pile-on’. For high- profile open source projects, often have many reviewers volunteering.
  113. Some awesome things about working openly Zero privileged information in

    the system: Reviewer reports, editorial decisions available to all. Increase transparency: • Public declarations of potential conflicts. • Editorial decisions documented in the open. • Clear expectations of authors. Reduces complexity of infrastructure. People can link to their reviews.
  114. Zero privileged information in the system: So sometimes authors chase

    reviewers, editors etc. Good reviewers become well known quickly potentially leading to reviewer burnout. Potential cultural barriers to entry for some and negative dynamics for junior staff. Some not-so-awesome things about working openly
  115. JOSS is a collaboration between author, editor and reviewer

  116. None
  117. McGonagle-O’Connell & Ratan: https://doi.org/10.1002/leap.1215

  118. Thanks! @arfon