Journal of Open Source Software: When collaborative open source meets peer review

This is a presentation I gave at FORCE2019 (https://www.force11.org/meetings/force2019) about the Journal of Open Source Software.

In this presentation I try and explain how JOSS works, and how we're leveraging open source conventions to build a low cost, open access journal.

Transcript

1. Journal of Open Source Software Arfon M. Smith (@arfon), Tania Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J. Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano, Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson, Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake Vanderplas, Bruce E. Wilson, Yo Yehudi https://joss.theoj.org When collaborative open source meets peer review
3. Software is important
4. Software is everywhere
5. Software turns a theoretical model into quantitative predictions; software controls an experiment; and software extracts from raw data evidence supporting or rejecting a theory. Gaël Varoquaux – scikit-learn creator
10. Software is eating the world* * Mark Andreessen
11. the skills required to be a successful scientific researcher are increasingly indistinguishable from the skills required to be successful in industry. Jake VanderPlas (Google)
12. We must find a way to legitimize software as a form of scholarship. Phil Bourne, University of Virginia
13. Software isn’t a creditable research activity
14. Challenges Technical Cultural Software isn’t easily citable ✔ ❌ Software citations aren’t allowed ❌ ✔ Software citations aren’t indexed ✔ ✔ Software isn’t peer reviewed ❌ ✔ Software can’t cite other software ✔ ❌
15. 1. Find some way to fit software into current (paper/book-centric) system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions?
16. Develop mechanisms for native software citation – see FORCE11 software (and data) citation working groups. Make your software ‘citable’, and make it easy for people to cite you. Lobby/lean-on/encourage indexers to count software citations. Find some way to fit software into the current system
17. To show we’ve done our research. To credit the work of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
18. To show we’ve done our research. To credit the work of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?
19. Develop mechanisms for citing all of the constituent parts of a research endeavour e.g. data, software, methods/ protocols, results… One potential solution: Transitive Credit (Katz, 2014, https://doi.org/10.5334/ jors.be) Evolving beyond one- dimensional credit model
20. Smith et al WMAP PLANK Jones et al Smith et al astropy scikit-learn Transitive Credit Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2
21. Smith et al WMAP astropy PLANK Jones et al Smith et al scikit-learn Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2 Whyte et al numpy scipy 0.5 0.25 0.25
22. 1. Find some way to fit software into current (paper/book-centric) system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions? Really hard
23. Individual change: Authors, reviewers, editors. Group/community change: Editorial teams, journals, publishers. Ecosystem change: Journals, publishers, indexers. Requires action across the whole scholarly ecosystem
24. What if we just wrote papers about software?
25. Software papers Gives us something easy to cite No changes required to existing infrastructure Publishing in existing journals raises profile of software within a community
26. Software papers Writing another paper can be a ton of work Many journals don’t accept software papers For long-lived software packages, static authorship presents major issues Many papers about the same software may lead to citation dilution
27. AAS Journals welcome articles which describe the design and function of software of relevance to research in astronomy and astrophysics. Such articles should contain a description of the software, its novel features and its intended use. Such articles need not include research results produced using the software, although including examples of applications can be helpful. There is no minimum length requirement for software articles.
28. What if we made it as easy as possible to write and publish a software paper? Embracing the hack
29. A developer friendly journal* for research software packages. Paper preparation (and submission) for well-documented software should take no more than an hour. The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. * Other venues exist for publishing papers about software
30. Short – generally under two pages. A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. A clear Statement of Need that illustrates the research purpose of the software. Acknowledgement of any financial support. Reference to related work, e.g. research results derived using the software. Not permitted to contain novel results. Paper format
31. Be as conventional as possible when it comes to integrating with the scholarly ecosystem GOAL ORCID for logins. Deposit metadata and issue DOIs with Crossref. Archive papers (and reviews) with Portico. Archive reviewed software with Dryad/ figshare/Zenodo. Get indexed (Google Scholar, Scopus etc.)
32. Follow best practices where they exist GOAL Open access – papers licensed CC-BY (authors retain copyright). Enforce the Open Source Definition. Clear ethics, ownerships, business model and governance statements. Rigorous, transparent peer review process. Register with community directories (e.g. DOAJ).
33. Provide a valuable service to authors GOAL Minimize author effort beyond existing software documentation they already have. Offer a constructive peer review process by experts. Focus on improving submissions, not rejecting. Transparent, achievable review criteria. Actually review the software… Zero APCs, open access.
34. Leverage the best parts of open source where possible GOAL Carry out reviews in a familiar environment (on GitHub). Foster a collaborative, welcoming community. Transparent editorial decision making process. Leverage open source tools where possible (e.g. Pandoc) Automate all the things! Use robots to automate repetitive or labour-intensive tasks where possible.
35. Submission and review process
36. 2 https://joss.readthedocs.io/en/latest/index.html
40. Automated paper compilation Our editorial robot: Whedon Detecting programming language
41. Editor agreeing Managing editor asking editors if they can take submission Author suggesting reviewers Managing editor asking Whedon to assign editor
42. Reviewer #1 volunteering to review Editor asking Whedon to assign Reviewer #1 as reviewer for paper
43. Reviewer #2 volunteering to review Editor asking Whedon to assign Reviewer #2 as reviewer for paper
44. Editor asking Whedon to check the references for missing DOIs Whedon obliging
45. Editor asking Whedon to start the main review
46. Metadata (editor, author, reviewers) Badge to include on e.g. GitHub
47. Checklist for each reviewer
48. Reviewer opening issues on software repository
49. Author asking Whedon to recompile paper
50. Editor associating Zenodo software archive with paper Editor updating software version associated with review Editor tagging EiC team to accept
51. Whedon produces final proofs of paper and Crossref metadata Managing editor asks Whedon to do a ‘dry run’ of accepting paper
52. Whedon tweets link to paper Managing editor asks Whedon to accept paper …and confirms when DOI registered with Crossref
53. Managing editor thanks reviewers and editor Whedon closes review
55. Whedon
59. https://peerj.com/articles/cs-147/
63. 0 200 400 600 800 2016-07-01 2017-01-01 2017-07-01 2018-01-01 2018-07-01 2019-01-01 2019-07-01 Year 1: 104 (8.6 papers/month) Year 2: 181 (15.3 papers/month) Year 3: 281 (23.4 papers/month) Year 4: 148 (30.3 papers/month)
64. 1000th paper in ~June 2020 500 papers/year in 2021 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
65. https://blog.joss.theoj.org/2019/06/cost-models-for-running-an-online-open-journal
66. Fixed costs we have to pay Annual Crossref membership: $275/year JOSS paper DOIs: $1/accepted paper JOSS website hosting: $19/month JOSS domain name registration: $10/year Portico fees: $250/year Total running costs: $1,063 $3.54/paper Assuming 300 papers published per year
67. Including costs that we don’t currently pay Costs we now pay: $1,063/year Services for which we don’t currently pay: estimated at $20,600/year Volunteer editor services: 1.85 FTE effort or $370,000 (@$200,000/FTE) Reviewer time (not paid for in almost all journals, nor costed by JOSS) $1305/paper Assuming 300 papers published per year, excluding reviewer efforts
68. But actually… Given that many journals don’t seem to pay editors much (if anything). If we include a modest editor stipend of $10,000 Total theoretical running costs: $31,663 $105/paper Assuming 300 papers published per year, excluding reviewer efforts
69. Collaborative open source meets peer review
70. The right to use, modify, and share not to contribute Open Source
71. Open Collaborations: a highly collaborative development process and are receptive to contributions of code, documentation, discussion, etc. from anyone who shows competent interest. Karl Fogel – Producing Open Source Software
74. Code first, permission later
81. 200M Pull Requests ~70M In the last 12 months https://octoverse.github.com
82. Receptive to contributions of code, documentation, discussion… https://opensource.com/life/16/5/growing-contributor-base-modern-open-source
83. Collaborative open source Create more users through education. Encourage users to contribute through Liberal Contribution Policies. Enable more leaders through participatory governance.
84. Open source contributor funnel: Mike McQuaid
85. Open discussions
86. Open discussions
87. Receptive to contributions of code, documentation, discussion…
88. Hacktoberfest: Global effort focused on contributions
89. Reviewers Authors Editors Contributor flow @ JOSS
90. Reviewers Authors Editors Contributor flow @ JOSS
91. Bot-assisted communities
92. Powered by ‘webhooks’
93. Powered by ‘webhooks’
100. What kind of journal is JOSS?
101. A multi-disciplinary perspective on emergent and future innovations in peer review, Tennant et al., 2017
102. continuous Post-publication commenting Informal discussion of published research, independent of any formal peer review that may have already occurred Can be performed on third-party platforms, anyone can contribute, public Comments can be rude or of low quality, comments across multiple platforms lack inter-operability, low visibility, low uptake PubMed Commons, PeerJ, PLOS, BMJ Collaborative A combination of referees, editors and external readers participate in the assessment of scientific manuscripts through interactive comments, often to reach a consensus decision, and a single set of revisions Iterative, transparent, editors sign reports, can be integrated with formal process, deters low quality submissions Can be additionally time-consuming, discussion quality variable, peer pressure and influence can tilt the balance eLife, Frontiers series, Copernicus journals, BMJ Open Science Portable Authors can take referee reports to multiple consecutive venues, often administered by a third-party service Reduces redundancy or duplication, saves time Low uptake by authors, low acceptance by journals, high cost BioMed Central journals, NPRC, Rubriq, Peerage of Science, MECA Recommendation services Post-publication evaluation and recommendation of significant articles, often through a peer- nominated consortium Crowd-sourced literature discovery, time saving, “prestige” factor when inside a consortium Paid services (subscription only), time consuming on recommender side, exclusive F1000 Prime, CiteULike Decoupled post-publication (annotation services) Comments or highlights added directly to highlighted sections of the work. Added notes can be private or public Rapid, crowd-sourced and collaborative, cross-publisher, low threshold for entry Non-interoperable, multiple venues, effort duplication, relatively unused, genuine critiques reserved PubPeer, Hypothesis, PaperHive, PeerLibrary or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic Advantages and disadvantages of the different approaches to peer review.
103. is published Optional open peer review As single blind peer review, except that the referees are given the option to make their review and their name public Increased transparency Gives an unclear pictures of the review process if not all reviews are made public PeerJ, Nature Communications Pre-publication open peer review Referees are identified to authors pre-publication, and if the article is published, the full peer review history together with the names of the associated referees is made public Transparency, increased integrity of reviews Fear: referees may decline to review, or be unwilling to come across too critically or positively The medical BMC-series journals, The BMJ Post-publication open peer review The referee reports and the names of the referees are always made public regardless of the outcome of their review Fast publication, transparent process Fear: referees may decline to review, or be unwilling to come across too critically or positively F1000Research, ScienceOpen, PubPub, Publons Peer review by endorsement (PRE) Pre-arranged and invited, with referees providing a “stamp of approval” on publications Transparent, cost- effective, rapid, accountable Low uptake, prone to selection bias, not viewed as credible RIO Journal comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final- version commenting (Ross-Hellauer, 2017). The various parts of examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality Pros and cons of different approaches to anonymity in peer review.
104. Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
105. Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
106. Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
107. Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
108. Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.
109. Some observations People like the robot*… * Does cause occasional confusion
110. Some observations People like the robot*… * Does cause occasional confusion
111. Scaling JOSS Most of our challenges are about scaling people processes: • AEiC/managing editor rotations. • More editors. • Term limits for editors (to avoid burnout). Technology improvements: • Smarter reviewer assignments. • Better monitoring tools for editors. • Tools to help authors prepare their submissions. 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01
112. Unexpected consequences of working openly Semi-regular emails from people annoyed they haven’t been asked to review yet. Generally need relatively small number of invites to identify reviewers (~2 invites per reviewer). Vanity software package ‘pile-on’. For high- profile open source projects, often have many reviewers volunteering.
113. Some awesome things about working openly Zero privileged information in the system: Reviewer reports, editorial decisions available to all. Increase transparency: • Public declarations of potential conflicts. • Editorial decisions documented in the open. • Clear expectations of authors. Reduces complexity of infrastructure. People can link to their reviews.
114. Zero privileged information in the system: So sometimes authors chase reviewers, editors etc. Good reviewers become well known quickly potentially leading to reviewer burnout. Potential cultural barriers to entry for some and negative dynamics for junior staff. Some not-so-awesome things about working openly
115. JOSS is a collaboration between author, editor and reviewer
117. McGonagle-O’Connell & Ratan: https://doi.org/10.1002/leap.1215
118. Thanks! @arfon