Journal of Open Source Software: When collaborative open source meets peer review

Transcript

1. Journal of Open Source Software Arfon M. Smith (@arfon), Tania

   Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J. Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano, Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson, Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake Vanderplas, Bruce E. Wilson, Yo Yehudi https://joss.theoj.org When collaborative open source meets peer review
2. None

3. Software is important

4. Software is everywhere

5. Software turns a theoretical model into quantitative predictions; software controls

   an experiment; and software extracts from raw data evidence supporting or rejecting a theory. Gaël Varoquaux – scikit-learn creator
6. None
7. None
8. None
9. None

10. Software is eating the world* * Mark Andreessen

11. the skills required to be a successful scientific researcher are

    increasingly indistinguishable from the skills required to be successful in industry. Jake VanderPlas (Google)

12. We must find a way to legitimize software as a

    form of scholarship. Phil Bourne, University of Virginia

13. Software isn’t a creditable research activity

14. Challenges Technical Cultural Software isn’t easily citable ✔ ❌ Software

    citations aren’t allowed ❌ ✔ Software citations aren’t indexed ✔ ✔ Software isn’t peer reviewed ❌ ✔ Software can’t cite other software ✔ ❌

15. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions?

16. Develop mechanisms for native software citation – see FORCE11 software

    (and data) citation working groups. Make your software ‘citable’, and make it easy for people to cite you. Lobby/lean-on/encourage indexers to count software citations. Find some way to fit software into the current system

17. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?

18. To show we’ve done our research. To credit the work

    of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?

19. Develop mechanisms for citing all of the constituent parts of

    a research endeavour e.g. data, software, methods/ protocols, results… One potential solution: Transitive Credit (Katz, 2014, https://doi.org/10.5334/ jors.be) Evolving beyond one- dimensional credit model

20. Smith et al WMAP PLANK Jones et al Smith et

    al astropy scikit-learn Transitive Credit Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2

21. Smith et al WMAP astropy PLANK Jones et al Smith

    et al scikit-learn Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2 Whyte et al numpy scipy 0.5 0.25 0.25

22. 1. Find some way to fit software into current (paper/book-centric)

    system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions? Really hard

23. Individual change: Authors, reviewers, editors. Group/community change: Editorial teams, journals,

    publishers. Ecosystem change: Journals, publishers, indexers. Requires action across the whole scholarly ecosystem

24. What if we just wrote papers about software?

25. Software papers Gives us something easy to cite No changes

    required to existing infrastructure Publishing in existing journals raises profile of software within a community

26. Software papers Writing another paper can be a ton of

    work Many journals don’t accept software papers For long-lived software packages, static authorship presents major issues Many papers about the same software may lead to citation dilution

27. AAS Journals welcome articles which describe the design and function

    of software of relevance to research in astronomy and astrophysics. Such articles should contain a description of the software, its novel features and its intended use. Such articles need not include research results produced using the software, although including examples of applications can be helpful. There is no minimum length requirement for software articles.

28. What if we made it as easy as possible to

    write and publish a software paper? Embracing the hack

29. A developer friendly journal* for research software packages. Paper preparation

    (and submission) for well-documented software should take no more than an hour. The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. * Other venues exist for publishing papers about software

30. Short – generally under two pages. A summary describing the

    high-level functionality and purpose of the software for a diverse, non-specialist audience. A clear Statement of Need that illustrates the research purpose of the software. Acknowledgement of any financial support. Reference to related work, e.g. research results derived using the software. Not permitted to contain novel results. Paper format

31. Be as conventional as possible when it comes to integrating

    with the scholarly ecosystem GOAL ORCID for logins. Deposit metadata and issue DOIs with Crossref. Archive papers (and reviews) with Portico. Archive reviewed software with Dryad/ figshare/Zenodo. Get indexed (Google Scholar, Scopus etc.)

32. Follow best practices where they exist GOAL Open access –

    papers licensed CC-BY (authors retain copyright). Enforce the Open Source Definition. Clear ethics, ownerships, business model and governance statements. Rigorous, transparent peer review process. Register with community directories (e.g. DOAJ).

33. Provide a valuable service to authors GOAL Minimize author effort

    beyond existing software documentation they already have. Offer a constructive peer review process by experts. Focus on improving submissions, not rejecting. Transparent, achievable review criteria. Actually review the software… Zero APCs, open access.

34. Leverage the best parts of open source where possible GOAL

    Carry out reviews in a familiar environment (on GitHub). Foster a collaborative, welcoming community. Transparent editorial decision making process. Leverage open source tools where possible (e.g. Pandoc) Automate all the things! Use robots to automate repetitive or labour-intensive tasks where possible.

35. Submission and review process

36. 2 https://joss.readthedocs.io/en/latest/index.html

37. None
38. None
39. None

40. Automated paper compilation Our editorial robot: Whedon Detecting programming language

41. Editor agreeing Managing editor asking editors if they can take

    submission Author suggesting reviewers Managing editor asking Whedon to assign editor

42. Reviewer #1 volunteering to review Editor asking Whedon to assign

    Reviewer #1 as reviewer for paper

43. Reviewer #2 volunteering to review Editor asking Whedon to assign

    Reviewer #2 as reviewer for paper

44. Editor asking Whedon to check the references for missing DOIs

    Whedon obliging

45. Editor asking Whedon to start the main review

46. Metadata (editor, author, reviewers) Badge to include on e.g. GitHub

47. Checklist for each reviewer

48. Reviewer opening issues on software repository

49. Author asking Whedon to recompile paper

50. Editor associating Zenodo software archive with paper Editor updating software

    version associated with review Editor tagging EiC team to accept

51. Whedon produces final proofs of paper and Crossref metadata Managing

    editor asks Whedon to do a ‘dry run’ of accepting paper

52. Whedon tweets link to paper Managing editor asks Whedon to

    accept paper …and confirms when DOI registered with Crossref

53. Managing editor thanks reviewers and editor Whedon closes review

54. None

55. Whedon

56. None
57. None
58. None

59. https://peerj.com/articles/cs-147/

60. None
61. None
62. None

63. 0 200 400 600 800 2016-07-01 2017-01-01 2017-07-01 2018-01-01 2018-07-01

    2019-01-01 2019-07-01 Year 1: 104 (8.6 papers/month) Year 2: 181 (15.3 papers/month) Year 3: 281 (23.4 papers/month) Year 4: 148 (30.3 papers/month)

64. 1000th paper in ~June 2020 500 papers/year in 2021 0

    500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01

65. https://blog.joss.theoj.org/2019/06/cost-models-for-running-an-online-open-journal

66. Fixed costs we have to pay Annual Crossref membership: $275/year

    JOSS paper DOIs: $1/accepted paper JOSS website hosting: $19/month JOSS domain name registration: $10/year Portico fees: $250/year Total running costs: $1,063 $3.54/paper Assuming 300 papers published per year

67. Including costs that we don’t currently pay Costs we now

    pay: $1,063/year Services for which we don’t currently pay: estimated at $20,600/year Volunteer editor services: 1.85 FTE effort or $370,000 (@$200,000/FTE) Reviewer time (not paid for in almost all journals, nor costed by JOSS) $1305/paper Assuming 300 papers published per year, excluding reviewer efforts

68. But actually… Given that many journals don’t seem to pay

    editors much (if anything). If we include a modest editor stipend of $10,000 Total theoretical running costs: $31,663 $105/paper Assuming 300 papers published per year, excluding reviewer efforts

69. Collaborative open source meets peer review

70. The right to use, modify, and share not to contribute

    Open Source

71. Open Collaborations: a highly collaborative development process and are receptive

    to contributions of code, documentation, discussion, etc. from anyone who shows competent interest. Karl Fogel – Producing Open Source Software
72. None
73. None

74. Code first, permission later

75. None
76. None
77. None
78. None
79. None
80. None

81. 200M Pull Requests ~70M In the last 12 months https://octoverse.github.com

82. Receptive to contributions of code, documentation, discussion… https://opensource.com/life/16/5/growing-contributor-base-modern-open-source

83. Collaborative open source Create more users through education. Encourage users

    to contribute through Liberal Contribution Policies. Enable more leaders through participatory governance.

84. Open source contributor funnel: Mike McQuaid

85. Open discussions

86. Open discussions

87. Receptive to contributions of code, documentation, discussion…

88. Hacktoberfest: Global effort focused on contributions

89. Reviewers Authors Editors Contributor flow @ JOSS

90. Reviewers Authors Editors Contributor flow @ JOSS

91. Bot-assisted communities

92. Powered by ‘webhooks’

93. Powered by ‘webhooks’

94. None
95. None
96. None
97. None
98. None
99. None

100. What kind of journal is JOSS?

101. A multi-disciplinary perspective on emergent and future innovations in peer

     review, Tennant et al., 2017

102. continuous Post-publication commenting Informal discussion of published research, independent of

     any formal peer review that may have already occurred Can be performed on third-party platforms, anyone can contribute, public Comments can be rude or of low quality, comments across multiple platforms lack inter-operability, low visibility, low uptake PubMed Commons, PeerJ, PLOS, BMJ Collaborative A combination of referees, editors and external readers participate in the assessment of scientific manuscripts through interactive comments, often to reach a consensus decision, and a single set of revisions Iterative, transparent, editors sign reports, can be integrated with formal process, deters low quality submissions Can be additionally time-consuming, discussion quality variable, peer pressure and influence can tilt the balance eLife, Frontiers series, Copernicus journals, BMJ Open Science Portable Authors can take referee reports to multiple consecutive venues, often administered by a third-party service Reduces redundancy or duplication, saves time Low uptake by authors, low acceptance by journals, high cost BioMed Central journals, NPRC, Rubriq, Peerage of Science, MECA Recommendation services Post-publication evaluation and recommendation of significant articles, often through a peer- nominated consortium Crowd-sourced literature discovery, time saving, “prestige” factor when inside a consortium Paid services (subscription only), time consuming on recommender side, exclusive F1000 Prime, CiteULike Decoupled post-publication (annotation services) Comments or highlights added directly to highlighted sections of the work. Added notes can be private or public Rapid, crowd-sourced and collaborative, cross-publisher, low threshold for entry Non-interoperable, multiple venues, effort duplication, relatively unused, genuine critiques reserved PubPeer, Hypothesis, PaperHive, PeerLibrary or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic Advantages and disadvantages of the different approaches to peer review.

103. is published Optional open peer review As single blind peer

     review, except that the referees are given the option to make their review and their name public Increased transparency Gives an unclear pictures of the review process if not all reviews are made public PeerJ, Nature Communications Pre-publication open peer review Referees are identified to authors pre-publication, and if the article is published, the full peer review history together with the names of the associated referees is made public Transparency, increased integrity of reviews Fear: referees may decline to review, or be unwilling to come across too critically or positively The medical BMC-series journals, The BMJ Post-publication open peer review The referee reports and the names of the referees are always made public regardless of the outcome of their review Fast publication, transparent process Fear: referees may decline to review, or be unwilling to come across too critically or positively F1000Research, ScienceOpen, PubPub, Publons Peer review by endorsement (PRE) Pre-arranged and invited, with referees providing a “stamp of approval” on publications Transparent, cost- effective, rapid, accountable Low uptake, prone to selection bias, not viewed as credible RIO Journal comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final- version commenting (Ross-Hellauer, 2017). The various parts of examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality Pros and cons of different approaches to anonymity in peer review.

104. Some observations It seems to be working (i.e. we’re meeting

     a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

105. Some observations It seems to be working (i.e. we’re meeting

     a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

106. Some observations It seems to be working (i.e. we’re meeting

     a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

107. Some observations It seems to be working (i.e. we’re meeting

     a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

108. Some observations It seems to be working (i.e. we’re meeting

     a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

109. Some observations People like the robot*… * Does cause occasional

     confusion

110. Some observations People like the robot*… * Does cause occasional

     confusion

111. Scaling JOSS Most of our challenges are about scaling people

     processes: • AEiC/managing editor rotations. • More editors. • Term limits for editors (to avoid burnout). Technology improvements: • Smarter reviewer assignments. • Better monitoring tools for editors. • Tools to help authors prepare their submissions. 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01

112. Unexpected consequences of working openly Semi-regular emails from people annoyed

     they haven’t been asked to review yet. Generally need relatively small number of invites to identify reviewers (~2 invites per reviewer). Vanity software package ‘pile-on’. For high- profile open source projects, often have many reviewers volunteering.

113. Some awesome things about working openly Zero privileged information in

     the system: Reviewer reports, editorial decisions available to all. Increase transparency: • Public declarations of potential conflicts. • Editorial decisions documented in the open. • Clear expectations of authors. Reduces complexity of infrastructure. People can link to their reviews.

114. Zero privileged information in the system: So sometimes authors chase

     reviewers, editors etc. Good reviewers become well known quickly potentially leading to reviewer burnout. Potential cultural barriers to entry for some and negative dynamics for junior staff. Some not-so-awesome things about working openly

115. JOSS is a collaboration between author, editor and reviewer

116. None

117. McGonagle-O’Connell & Ratan: https://doi.org/10.1002/leap.1215

118. Thanks! @arfon