Tweet
Tweet

Slide 1

Slide 1 text

Journal of Open Source Software Arfon M. Smith (@arfon), Tania Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J. Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano, Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson, Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake Vanderplas, Bruce E. Wilson, Yo Yehudi https://joss.theoj.org When collaborative open source meets peer review

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

Software is important

Slide 4

Slide 4 text

Software is everywhere

Slide 5

Slide 5 text

Software turns a theoretical model into quantitative predictions; software controls an experiment; and software extracts from raw data evidence supporting or rejecting a theory. Gaël Varoquaux – scikit-learn creator

Slide 6

Slide 6 text

No content

Slide 7

Slide 7 text

No content

Slide 8

Slide 8 text

No content

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Software is eating the world* * Mark Andreessen

Slide 11

Slide 11 text

the skills required to be a successful scientific researcher are increasingly indistinguishable from the skills required to be successful in industry. Jake VanderPlas (Google)

Slide 12

Slide 12 text

We must find a way to legitimize software as a form of scholarship. Phil Bourne, University of Virginia

Slide 13

Slide 13 text

Software isn’t a creditable research activity

Slide 14

Slide 14 text

Challenges Technical Cultural Software isn’t easily citable ✔ ❌ Software citations aren’t allowed ❌ ✔ Software citations aren’t indexed ✔ ✔ Software isn’t peer reviewed ❌ ✔ Software can’t cite other software ✔ ❌

Slide 15

Slide 15 text

1. Find some way to fit software into current (paper/book-centric) system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions?

Slide 16

Slide 16 text

Develop mechanisms for native software citation – see FORCE11 software (and data) citation working groups. Make your software ‘citable’, and make it easy for people to cite you. Lobby/lean-on/encourage indexers to count software citations. Find some way to fit software into the current system

Slide 17

Slide 17 text

To show we’ve done our research. To credit the work of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?

Slide 18

Slide 18 text

To show we’ve done our research. To credit the work of others. To enrich the scholarly record. To avoid plagiarism. * Other reasons exist of course… Why do we cite things*?

Slide 19

Slide 19 text

Develop mechanisms for citing all of the constituent parts of a research endeavour e.g. data, software, methods/ protocols, results… One potential solution: Transitive Credit (Katz, 2014, https://doi.org/10.5334/ jors.be) Evolving beyond one- dimensional credit model

Slide 20

Slide 20 text

Smith et al WMAP PLANK Jones et al Smith et al astropy scikit-learn Transitive Credit Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2

Slide 21

Slide 21 text

Smith et al WMAP astropy PLANK Jones et al Smith et al scikit-learn Dan Katz 0.3 0.1 0.1 0.1 0.2 0.2 Whyte et al numpy scipy 0.5 0.25 0.25

Slide 22

Slide 22 text

1. Find some way to fit software into current (paper/book-centric) system. 2. Evolve beyond one-dimensional credit model. How to better recognize software contributions? Really hard

Slide 23

Slide 23 text

Individual change: Authors, reviewers, editors. Group/community change: Editorial teams, journals, publishers. Ecosystem change: Journals, publishers, indexers. Requires action across the whole scholarly ecosystem

Slide 24

Slide 24 text

What if we just wrote papers about software?

Slide 25

Slide 25 text

Software papers Gives us something easy to cite No changes required to existing infrastructure Publishing in existing journals raises profile of software within a community

Slide 26

Slide 26 text

Software papers Writing another paper can be a ton of work Many journals don’t accept software papers For long-lived software packages, static authorship presents major issues Many papers about the same software may lead to citation dilution

Slide 27

Slide 27 text

AAS Journals welcome articles which describe the design and function of software of relevance to research in astronomy and astrophysics. Such articles should contain a description of the software, its novel features and its intended use. Such articles need not include research results produced using the software, although including examples of applications can be helpful. There is no minimum length requirement for software articles.

Slide 28

Slide 28 text

What if we made it as easy as possible to write and publish a software paper? Embracing the hack

Slide 29

Slide 29 text

A developer friendly journal* for research software packages. Paper preparation (and submission) for well-documented software should take no more than an hour. The primary purpose of a JOSS paper is to enable citation credit to be given to authors of research software. * Other venues exist for publishing papers about software

Slide 30

Slide 30 text

Short – generally under two pages. A summary describing the high-level functionality and purpose of the software for a diverse, non-specialist audience. A clear Statement of Need that illustrates the research purpose of the software. Acknowledgement of any financial support. Reference to related work, e.g. research results derived using the software. Not permitted to contain novel results. Paper format

Slide 31

Slide 31 text

Be as conventional as possible when it comes to integrating with the scholarly ecosystem GOAL ORCID for logins. Deposit metadata and issue DOIs with Crossref. Archive papers (and reviews) with Portico. Archive reviewed software with Dryad/ figshare/Zenodo. Get indexed (Google Scholar, Scopus etc.)

Slide 32

Slide 32 text

Follow best practices where they exist GOAL Open access – papers licensed CC-BY (authors retain copyright). Enforce the Open Source Definition. Clear ethics, ownerships, business model and governance statements. Rigorous, transparent peer review process. Register with community directories (e.g. DOAJ).

Slide 33

Slide 33 text

Provide a valuable service to authors GOAL Minimize author effort beyond existing software documentation they already have. Offer a constructive peer review process by experts. Focus on improving submissions, not rejecting. Transparent, achievable review criteria. Actually review the software… Zero APCs, open access.

Slide 34

Slide 34 text

Leverage the best parts of open source where possible GOAL Carry out reviews in a familiar environment (on GitHub). Foster a collaborative, welcoming community. Transparent editorial decision making process. Leverage open source tools where possible (e.g. Pandoc) Automate all the things! Use robots to automate repetitive or labour-intensive tasks where possible.

Slide 35

Slide 35 text

Submission and review process

Slide 36

Slide 36 text

2 https://joss.readthedocs.io/en/latest/index.html

Slide 37

Slide 37 text

No content

Slide 38

Slide 38 text

No content

Slide 39

Slide 39 text

No content

Slide 40

Slide 40 text

Automated paper compilation Our editorial robot: Whedon Detecting programming language

Slide 41

Slide 41 text

Editor agreeing Managing editor asking editors if they can take submission Author suggesting reviewers Managing editor asking Whedon to assign editor

Slide 42

Slide 42 text

Reviewer #1 volunteering to review Editor asking Whedon to assign Reviewer #1 as reviewer for paper

Slide 43

Slide 43 text

Reviewer #2 volunteering to review Editor asking Whedon to assign Reviewer #2 as reviewer for paper

Slide 44

Slide 44 text

Editor asking Whedon to check the references for missing DOIs Whedon obliging

Slide 45

Slide 45 text

Editor asking Whedon to start the main review

Slide 46

Slide 46 text

Metadata (editor, author, reviewers) Badge to include on e.g. GitHub

Slide 47

Slide 47 text

Checklist for each reviewer

Slide 48

Slide 48 text

Reviewer opening issues on software repository

Slide 49

Slide 49 text

Author asking Whedon to recompile paper

Slide 50

Slide 50 text

Editor associating Zenodo software archive with paper Editor updating software version associated with review Editor tagging EiC team to accept

Slide 51

Slide 51 text

Whedon produces final proofs of paper and Crossref metadata Managing editor asks Whedon to do a ‘dry run’ of accepting paper

Slide 52

Slide 52 text

Whedon tweets link to paper Managing editor asks Whedon to accept paper …and confirms when DOI registered with Crossref

Slide 53

Slide 53 text

Managing editor thanks reviewers and editor Whedon closes review

Slide 54

Slide 54 text

No content

Slide 55

Slide 55 text

Whedon

Slide 56

Slide 56 text

No content

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

https://peerj.com/articles/cs-147/

Slide 60

Slide 60 text

No content

Slide 61

Slide 61 text

No content

Slide 62

Slide 62 text

No content

Slide 63

Slide 63 text

0 200 400 600 800 2016-07-01 2017-01-01 2017-07-01 2018-01-01 2018-07-01 2019-01-01 2019-07-01 Year 1: 104 (8.6 papers/month) Year 2: 181 (15.3 papers/month) Year 3: 281 (23.4 papers/month) Year 4: 148 (30.3 papers/month)

Slide 64

Slide 64 text

1000th paper in ~June 2020 500 papers/year in 2021 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01

Slide 65

Slide 65 text

https://blog.joss.theoj.org/2019/06/cost-models-for-running-an-online-open-journal

Slide 66

Slide 66 text

Fixed costs we have to pay Annual Crossref membership: $275/year JOSS paper DOIs: $1/accepted paper JOSS website hosting: $19/month JOSS domain name registration: $10/year Portico fees: $250/year Total running costs: $1,063 $3.54/paper Assuming 300 papers published per year

Slide 67

Slide 67 text

Including costs that we don’t currently pay Costs we now pay: $1,063/year Services for which we don’t currently pay: estimated at $20,600/year Volunteer editor services: 1.85 FTE effort or $370,000 (@$200,000/FTE) Reviewer time (not paid for in almost all journals, nor costed by JOSS) $1305/paper Assuming 300 papers published per year, excluding reviewer efforts

Slide 68

Slide 68 text

But actually… Given that many journals don’t seem to pay editors much (if anything). If we include a modest editor stipend of $10,000 Total theoretical running costs: $31,663 $105/paper Assuming 300 papers published per year, excluding reviewer efforts

Slide 69

Slide 69 text

Collaborative open source meets peer review

Slide 70

Slide 70 text

The right to use, modify, and share not to contribute Open Source

Slide 71

Slide 71 text

Open Collaborations: a highly collaborative development process and are receptive to contributions of code, documentation, discussion, etc. from anyone who shows competent interest. Karl Fogel – Producing Open Source Software

Slide 72

Slide 72 text

No content

Slide 73

Slide 73 text

No content

Slide 74

Slide 74 text

Code first, permission later

Slide 75

Slide 75 text

No content

Slide 76

Slide 76 text

No content

Slide 77

Slide 77 text

No content

Slide 78

Slide 78 text

No content

Slide 79

Slide 79 text

No content

Slide 80

Slide 80 text

No content

Slide 81

Slide 81 text

200M Pull Requests ~70M In the last 12 months https://octoverse.github.com

Slide 82

Slide 82 text

Receptive to contributions of code, documentation, discussion… https://opensource.com/life/16/5/growing-contributor-base-modern-open-source

Slide 83

Slide 83 text

Collaborative open source Create more users through education. Encourage users to contribute through Liberal Contribution Policies. Enable more leaders through participatory governance.

Slide 84

Slide 84 text

Open source contributor funnel: Mike McQuaid

Slide 85

Slide 85 text

Open discussions

Slide 86

Slide 86 text

Open discussions

Slide 87

Slide 87 text

Receptive to contributions of code, documentation, discussion…

Slide 88

Slide 88 text

Hacktoberfest: Global effort focused on contributions

Slide 89

Slide 89 text

Reviewers Authors Editors Contributor flow @ JOSS

Slide 90

Slide 90 text

Reviewers Authors Editors Contributor flow @ JOSS

Slide 91

Slide 91 text

Bot-assisted communities

Slide 92

Slide 92 text

Powered by ‘webhooks’

Slide 93

Slide 93 text

Powered by ‘webhooks’

Slide 94

Slide 94 text

No content

Slide 95

Slide 95 text

No content

Slide 96

Slide 96 text

No content

Slide 97

Slide 97 text

No content

Slide 98

Slide 98 text

No content

Slide 99

Slide 99 text

No content

Slide 100

Slide 100 text

What kind of journal is JOSS?

Slide 101

Slide 101 text

A multi-disciplinary perspective on emergent and future innovations in peer review, Tennant et al., 2017

Slide 102

Slide 102 text

continuous Post-publication commenting Informal discussion of published research, independent of any formal peer review that may have already occurred Can be performed on third-party platforms, anyone can contribute, public Comments can be rude or of low quality, comments across multiple platforms lack inter-operability, low visibility, low uptake PubMed Commons, PeerJ, PLOS, BMJ Collaborative A combination of referees, editors and external readers participate in the assessment of scientific manuscripts through interactive comments, often to reach a consensus decision, and a single set of revisions Iterative, transparent, editors sign reports, can be integrated with formal process, deters low quality submissions Can be additionally time-consuming, discussion quality variable, peer pressure and influence can tilt the balance eLife, Frontiers series, Copernicus journals, BMJ Open Science Portable Authors can take referee reports to multiple consecutive venues, often administered by a third-party service Reduces redundancy or duplication, saves time Low uptake by authors, low acceptance by journals, high cost BioMed Central journals, NPRC, Rubriq, Peerage of Science, MECA Recommendation services Post-publication evaluation and recommendation of significant articles, often through a peer- nominated consortium Crowd-sourced literature discovery, time saving, “prestige” factor when inside a consortium Paid services (subscription only), time consuming on recommender side, exclusive F1000 Prime, CiteULike Decoupled post-publication (annotation services) Comments or highlights added directly to highlighted sections of the work. Added notes can be private or public Rapid, crowd-sourced and collaborative, cross-publisher, low threshold for entry Non-interoperable, multiple venues, effort duplication, relatively unused, genuine critiques reserved PubPeer, Hypothesis, PaperHive, PeerLibrary or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic Advantages and disadvantages of the different approaches to peer review.

Slide 103

Slide 103 text

is published Optional open peer review As single blind peer review, except that the referees are given the option to make their review and their name public Increased transparency Gives an unclear pictures of the review process if not all reviews are made public PeerJ, Nature Communications Pre-publication open peer review Referees are identified to authors pre-publication, and if the article is published, the full peer review history together with the names of the associated referees is made public Transparency, increased integrity of reviews Fear: referees may decline to review, or be unwilling to come across too critically or positively The medical BMC-series journals, The BMJ Post-publication open peer review The referee reports and the names of the referees are always made public regardless of the outcome of their review Fast publication, transparent process Fear: referees may decline to review, or be unwilling to come across too critically or positively F1000Research, ScienceOpen, PubPub, Publons Peer review by endorsement (PRE) Pre-arranged and invited, with referees providing a “stamp of approval” on publications Transparent, cost- effective, rapid, accountable Low uptake, prone to selection bias, not viewed as credible RIO Journal comprising seven different traits of OPR: participation, identity, reports, interaction, platforms, pre-review manuscripts, and final- version commenting (Ross-Hellauer, 2017). The various parts of examined in more detail below. Each of these feed into the wider core issues in peer review of incentivizing engagement, providing appropriate recognition and certification, and quality Pros and cons of different approaches to anonymity in peer review.

Slide 104

Slide 104 text

Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

Slide 105

Slide 105 text

Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

Slide 106

Slide 106 text

Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

Slide 107

Slide 107 text

Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

Slide 108

Slide 108 text

Some observations It seems to be working (i.e. we’re meeting a demand that exists)… People enjoy editing, reviewing, and being reviewed at JOSS.

Slide 109

Slide 109 text

Some observations People like the robot*… * Does cause occasional confusion

Slide 110

Slide 110 text

Some observations People like the robot*… * Does cause occasional confusion

Slide 111

Slide 111 text

Scaling JOSS Most of our challenges are about scaling people processes: • AEiC/managing editor rotations. • More editors. • Term limits for editors (to avoid burnout). Technology improvements: • Smarter reviewer assignments. • Better monitoring tools for editors. • Tools to help authors prepare their submissions. 0 500 1000 1500 2017-01-01 2018-01-01 2019-01-01 2020-01-01 2021-01-01

Slide 112

Slide 112 text

Unexpected consequences of working openly Semi-regular emails from people annoyed they haven’t been asked to review yet. Generally need relatively small number of invites to identify reviewers (~2 invites per reviewer). Vanity software package ‘pile-on’. For high- profile open source projects, often have many reviewers volunteering.

Slide 113

Slide 113 text

Some awesome things about working openly Zero privileged information in the system: Reviewer reports, editorial decisions available to all. Increase transparency: • Public declarations of potential conflicts. • Editorial decisions documented in the open. • Clear expectations of authors. Reduces complexity of infrastructure. People can link to their reviews.

Slide 114

Slide 114 text

Zero privileged information in the system: So sometimes authors chase reviewers, editors etc. Good reviewers become well known quickly potentially leading to reviewer burnout. Potential cultural barriers to entry for some and negative dynamics for junior staff. Some not-so-awesome things about working openly

Slide 115

Slide 115 text

JOSS is a collaboration between author, editor and reviewer

Slide 116

Slide 116 text

No content

Slide 117

Slide 117 text

McGonagle-O’Connell & Ratan: https://doi.org/10.1002/leap.1215

Slide 118

Slide 118 text

Thanks! @arfon