Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Journal of Open Source Software: When collaborative open source meets peer review

Arfon Smith
October 16, 2019

Journal of Open Source Software: When collaborative open source meets peer review

This is a presentation I gave at FORCE2019 (https://www.force11.org/meetings/force2019) about the Journal of Open Source Software.

In this presentation I try and explain how JOSS works, and how we're leveraging open source conventions to build a low cost, open access journal.

Arfon Smith

October 16, 2019
Tweet

More Decks by Arfon Smith

Other Decks in Research

Transcript

  1. Journal of Open Source Software
    Arfon M. Smith (@arfon), Tania Allard, Lorena A Barba, Katy Barnhart, Juanjo Bazán, Monica Bobra, Jed
    Brown, Jason Clark, Abigail Cabunoc Mayes, George Githinji, Olivia Guest, Roman Valls Guimera, Melissa
    Gymrek, Alex Hanna, Lindsey Heagy, Kathryn Huff, Mark A. Jensen, Daniel S. Katz, Vincent Knight, Thomas J.
    Leeper, Christopher R. Madan, Melissa Weber Mendonça, Kevin M. Moerman, Kyle Niemeyer, Lorena Pantano,
    Viviane Pons, Jack Poulson, Pjotr Prins, Karthik Ram, Marie E. Rognes, Ariel Rokem, Charlotte Soneson,
    Matthew Sottile, Yuan Tang, Tracy Teal, George K. Thiruvathukal, Kristen Thyng, Leonardo Uieda, Jake
    Vanderplas, Bruce E. Wilson, Yo Yehudi
    https://joss.theoj.org
    When collaborative open source meets peer review

    View full-size slide

  2. Software is important

    View full-size slide

  3. Software is everywhere

    View full-size slide

  4. Software turns a theoretical model into
    quantitative predictions; software controls an
    experiment; and software extracts from raw data
    evidence supporting or rejecting a theory.
    Gaël Varoquaux – scikit-learn creator

    View full-size slide

  5. Software is eating the world*
    * Mark Andreessen

    View full-size slide

  6. the skills required to be a
    successful scientific researcher
    are increasingly indistinguishable
    from the skills required to be
    successful in industry.
    Jake VanderPlas (Google)

    View full-size slide

  7. We must find a way to
    legitimize software as a
    form of scholarship.
    Phil Bourne, University of Virginia

    View full-size slide

  8. Software isn’t a creditable
    research activity

    View full-size slide

  9. Challenges Technical Cultural
    Software isn’t easily citable ✔ ❌
    Software citations aren’t allowed ❌ ✔
    Software citations aren’t indexed ✔ ✔
    Software isn’t peer reviewed ❌ ✔
    Software can’t cite other software ✔ ❌

    View full-size slide

  10. 1. Find some way to fit software into
    current (paper/book-centric)
    system.
    2. Evolve beyond one-dimensional
    credit model.
    How to better
    recognize
    software
    contributions?

    View full-size slide

  11. Develop mechanisms for native software
    citation – see FORCE11 software (and data)
    citation working groups.
    Make your software ‘citable’, and make it easy
    for people to cite you.
    Lobby/lean-on/encourage indexers to count
    software citations.
    Find some way
    to fit software
    into the current
    system

    View full-size slide

  12. To show we’ve done our research.
    To credit the work of others.
    To enrich the scholarly record.
    To avoid plagiarism.
    * Other reasons exist of course…
    Why do we cite things*?

    View full-size slide

  13. To show we’ve done our research.
    To credit the work of others.
    To enrich the scholarly record.
    To avoid plagiarism.
    * Other reasons exist of course…
    Why do we cite things*?

    View full-size slide

  14. Develop mechanisms for citing all of the
    constituent parts of a research
    endeavour e.g. data, software, methods/
    protocols, results…
    One potential solution: Transitive Credit
    (Katz, 2014, https://doi.org/10.5334/
    jors.be)
    Evolving
    beyond one-
    dimensional
    credit model

    View full-size slide

  15. Smith et al
    WMAP PLANK
    Jones et al
    Smith et al astropy scikit-learn
    Transitive Credit
    Dan Katz
    0.3 0.1 0.1 0.1 0.2 0.2

    View full-size slide

  16. Smith et al
    WMAP astropy
    PLANK
    Jones et al
    Smith et al scikit-learn
    Dan Katz
    0.3 0.1 0.1 0.1 0.2 0.2
    Whyte et al numpy scipy
    0.5 0.25 0.25

    View full-size slide

  17. 1. Find some way to fit software into
    current (paper/book-centric)
    system.
    2. Evolve beyond one-dimensional
    credit model.
    How to better
    recognize
    software
    contributions?
    Really hard

    View full-size slide

  18. Individual change: Authors, reviewers,
    editors.
    Group/community change: Editorial
    teams, journals, publishers.
    Ecosystem change: Journals, publishers,
    indexers.
    Requires action
    across the
    whole scholarly
    ecosystem

    View full-size slide

  19. What if we just wrote
    papers about software?

    View full-size slide

  20. Software
    papers
    Gives us something easy to cite
    No changes required to existing
    infrastructure
    Publishing in existing journals raises
    profile of software within a community

    View full-size slide

  21. Software
    papers
    Writing another paper can be a ton of
    work
    Many journals don’t accept software
    papers
    For long-lived software packages, static
    authorship presents major issues
    Many papers about the same software
    may lead to citation dilution

    View full-size slide

  22. AAS Journals welcome articles which describe the design
    and function of software of relevance to research in
    astronomy and astrophysics. Such articles should contain
    a description of the software, its novel features and its
    intended use. Such articles need not include research
    results produced using the software, although including
    examples of applications can be helpful. There is no
    minimum length requirement for software articles.

    View full-size slide

  23. What if we made it as easy as
    possible to write and publish a
    software paper?
    Embracing the hack

    View full-size slide

  24. A developer friendly journal* for research software
    packages.
    Paper preparation (and submission) for well-documented
    software should take no more than an hour.
    The primary purpose of a JOSS paper is to enable citation
    credit to be given to authors of research software.
    * Other venues exist for publishing papers about software

    View full-size slide

  25. Short – generally under two pages.
    A summary describing the high-level functionality and purpose of the
    software for a diverse, non-specialist audience.
    A clear Statement of Need that illustrates the research purpose of the
    software.
    Acknowledgement of any financial support.
    Reference to related work, e.g. research results derived using the
    software.
    Not permitted to contain novel results.
    Paper format

    View full-size slide

  26. Be as
    conventional as
    possible when
    it comes to
    integrating with
    the scholarly
    ecosystem
    GOAL ORCID for logins.
    Deposit metadata and issue DOIs with
    Crossref.
    Archive papers (and reviews) with Portico.
    Archive reviewed software with Dryad/
    figshare/Zenodo.
    Get indexed (Google Scholar, Scopus etc.)

    View full-size slide

  27. Follow best
    practices where
    they exist
    GOAL Open access – papers licensed CC-BY
    (authors retain copyright).
    Enforce the Open Source Definition.
    Clear ethics, ownerships, business model
    and governance statements.
    Rigorous, transparent peer review process.
    Register with community directories (e.g.
    DOAJ).

    View full-size slide

  28. Provide a
    valuable service
    to authors
    GOAL Minimize author effort beyond existing
    software documentation they already have.
    Offer a constructive peer review process
    by experts.
    Focus on improving submissions, not
    rejecting.
    Transparent, achievable review criteria.
    Actually review the software…
    Zero APCs, open access.

    View full-size slide

  29. Leverage the
    best parts of
    open source
    where possible
    GOAL Carry out reviews in a familiar environment
    (on GitHub).
    Foster a collaborative, welcoming
    community.
    Transparent editorial decision making
    process.
    Leverage open source tools where possible
    (e.g. Pandoc)
    Automate all the things! Use robots to
    automate repetitive or labour-intensive
    tasks where possible.

    View full-size slide

  30. Submission and review process

    View full-size slide

  31. 2
    https://joss.readthedocs.io/en/latest/index.html

    View full-size slide

  32. Automated paper
    compilation
    Our editorial robot:
    Whedon
    Detecting
    programming
    language

    View full-size slide

  33. Editor agreeing
    Managing editor
    asking editors if
    they can take
    submission
    Author suggesting
    reviewers
    Managing editor
    asking Whedon to
    assign editor

    View full-size slide

  34. Reviewer #1
    volunteering to
    review
    Editor asking
    Whedon to assign
    Reviewer #1 as
    reviewer for paper

    View full-size slide

  35. Reviewer #2
    volunteering to
    review
    Editor asking
    Whedon to assign
    Reviewer #2 as
    reviewer for paper

    View full-size slide

  36. Editor asking
    Whedon to check
    the references for
    missing DOIs
    Whedon obliging

    View full-size slide

  37. Editor asking
    Whedon to start
    the main review

    View full-size slide

  38. Metadata (editor,
    author, reviewers)
    Badge to include
    on e.g. GitHub

    View full-size slide

  39. Checklist for each
    reviewer

    View full-size slide

  40. Reviewer opening
    issues on software
    repository

    View full-size slide

  41. Author asking
    Whedon to
    recompile paper

    View full-size slide

  42. Editor associating
    Zenodo software
    archive with paper
    Editor updating
    software version
    associated with
    review
    Editor tagging EiC
    team to accept

    View full-size slide

  43. Whedon produces
    final proofs of
    paper and Crossref
    metadata
    Managing editor
    asks Whedon to do
    a ‘dry run’ of
    accepting paper

    View full-size slide

  44. Whedon tweets
    link to paper
    Managing editor
    asks Whedon to
    accept paper
    …and confirms
    when DOI
    registered with
    Crossref

    View full-size slide

  45. Managing editor
    thanks reviewers
    and editor
    Whedon closes
    review

    View full-size slide

  46. https://peerj.com/articles/cs-147/

    View full-size slide

  47. 0
    200
    400
    600
    800
    2016-07-01
    2017-01-01
    2017-07-01
    2018-01-01
    2018-07-01
    2019-01-01
    2019-07-01
    Year 1: 104 (8.6 papers/month)
    Year 2: 181 (15.3 papers/month)
    Year 3: 281 (23.4 papers/month)
    Year 4: 148 (30.3 papers/month)

    View full-size slide

  48. 1000th paper in ~June 2020
    500 papers/year in 2021
    0
    500
    1000
    1500
    2017-01-01
    2018-01-01
    2019-01-01
    2020-01-01
    2021-01-01

    View full-size slide

  49. https://blog.joss.theoj.org/2019/06/cost-models-for-running-an-online-open-journal

    View full-size slide

  50. Fixed costs we
    have to pay
    Annual Crossref membership: $275/year
    JOSS paper DOIs: $1/accepted paper
    JOSS website hosting: $19/month
    JOSS domain name registration: $10/year
    Portico fees: $250/year
    Total running costs: $1,063
    $3.54/paper
    Assuming 300 papers published per year

    View full-size slide

  51. Including
    costs that we
    don’t currently
    pay
    Costs we now pay: $1,063/year
    Services for which we don’t currently pay:
    estimated at $20,600/year
    Volunteer editor services: 1.85 FTE effort
    or $370,000 (@$200,000/FTE)
    Reviewer time (not paid for in almost all
    journals, nor costed by JOSS)
    $1305/paper
    Assuming 300 papers published per year,
    excluding reviewer efforts

    View full-size slide

  52. But actually…
    Given that many journals don’t seem to
    pay editors much (if anything).
    If we include a modest editor stipend of
    $10,000
    Total theoretical running costs: $31,663
    $105/paper
    Assuming 300 papers published per
    year, excluding reviewer efforts

    View full-size slide

  53. Collaborative open source
    meets peer review

    View full-size slide

  54. The right to use, modify, and share
    not to contribute
    Open Source

    View full-size slide

  55. Open Collaborations: a highly
    collaborative development process and
    are receptive to contributions of code,
    documentation, discussion, etc. from
    anyone who shows competent interest.
    Karl Fogel – Producing Open Source Software

    View full-size slide

  56. Code first, permission later

    View full-size slide

  57. 200M
    Pull Requests
    ~70M
    In the last 12 months
    https://octoverse.github.com

    View full-size slide

  58. Receptive to contributions of code, documentation, discussion…
    https://opensource.com/life/16/5/growing-contributor-base-modern-open-source

    View full-size slide

  59. Collaborative
    open source
    Create more users through education.
    Encourage users to contribute
    through Liberal Contribution Policies.
    Enable more leaders through participatory
    governance.

    View full-size slide

  60. Open source contributor funnel: Mike McQuaid

    View full-size slide

  61. Open discussions

    View full-size slide

  62. Open discussions

    View full-size slide

  63. Receptive to contributions of code, documentation, discussion…

    View full-size slide

  64. Hacktoberfest: Global effort focused on contributions

    View full-size slide

  65. Reviewers
    Authors
    Editors
    Contributor
    flow @ JOSS

    View full-size slide

  66. Reviewers
    Authors
    Editors
    Contributor
    flow @ JOSS

    View full-size slide

  67. Bot-assisted communities

    View full-size slide

  68. Powered by ‘webhooks’

    View full-size slide

  69. Powered by ‘webhooks’

    View full-size slide

  70. What kind of journal is JOSS?

    View full-size slide

  71. A multi-disciplinary perspective on emergent and future innovations in peer review, Tennant et al., 2017

    View full-size slide

  72. continuous
    Post-publication
    commenting
    Informal discussion of published
    research, independent of any
    formal peer review that may have
    already occurred
    Can be performed on
    third-party platforms,
    anyone can contribute,
    public
    Comments can be
    rude or of low quality,
    comments across
    multiple platforms lack
    inter-operability, low
    visibility, low uptake
    PubMed Commons,
    PeerJ, PLOS, BMJ
    Collaborative A combination of referees, editors
    and external readers participate
    in the assessment of scientific
    manuscripts through interactive
    comments, often to reach a
    consensus decision, and a single
    set of revisions
    Iterative, transparent,
    editors sign reports,
    can be integrated
    with formal process,
    deters low quality
    submissions
    Can be additionally
    time-consuming,
    discussion quality
    variable, peer pressure
    and influence can tilt the
    balance
    eLife, Frontiers
    series, Copernicus
    journals, BMJ Open
    Science
    Portable Authors can take referee reports
    to multiple consecutive venues,
    often administered by a third-party
    service
    Reduces redundancy
    or duplication, saves
    time
    Low uptake by authors,
    low acceptance by
    journals, high cost
    BioMed Central
    journals, NPRC,
    Rubriq, Peerage of
    Science, MECA
    Recommendation
    services
    Post-publication evaluation and
    recommendation of significant
    articles, often through a peer-
    nominated consortium
    Crowd-sourced
    literature discovery,
    time saving, “prestige”
    factor when inside a
    consortium
    Paid services (subscription
    only), time consuming
    on recommender side,
    exclusive
    F1000 Prime, CiteULike
    Decoupled
    post-publication
    (annotation services)
    Comments or highlights added
    directly to highlighted sections
    of the work. Added notes can be
    private or public
    Rapid, crowd-sourced
    and collaborative,
    cross-publisher, low
    threshold for entry
    Non-interoperable,
    multiple venues, effort
    duplication, relatively
    unused, genuine
    critiques reserved
    PubPeer, Hypothesis,
    PaperHive, PeerLibrary
    or deceptive publishing cast a shadow of doubt on the validity of versus careerism versus output measurement), and an academic
    Advantages and disadvantages of the different approaches to peer review.

    View full-size slide

  73. is published
    Optional open peer
    review
    As single blind peer review,
    except that the referees are
    given the option to make
    their review and their name
    public
    Increased transparency Gives an unclear
    pictures of the review
    process if not all reviews
    are made public
    PeerJ, Nature
    Communications
    Pre-publication
    open peer review
    Referees are identified to
    authors pre-publication,
    and if the article is
    published, the full peer
    review history together
    with the names of the
    associated referees is
    made public
    Transparency, increased
    integrity of reviews
    Fear: referees may
    decline to review, or be
    unwilling to come across
    too critically or positively
    The medical BMC-series
    journals, The BMJ
    Post-publication
    open peer review
    The referee reports and
    the names of the referees
    are always made public
    regardless of the outcome
    of their review
    Fast publication,
    transparent process
    Fear: referees may
    decline to review, or be
    unwilling to come across
    too critically or positively
    F1000Research,
    ScienceOpen, PubPub,
    Publons
    Peer review by
    endorsement (PRE)
    Pre-arranged and invited,
    with referees providing a
    “stamp of approval” on
    publications
    Transparent, cost-
    effective, rapid,
    accountable
    Low uptake, prone
    to selection bias, not
    viewed as credible
    RIO Journal
    comprising seven different traits of OPR: participation, identity,
    reports, interaction, platforms, pre-review manuscripts, and final-
    version commenting (Ross-Hellauer, 2017). The various parts of
    examined in more detail below. Each of these feed into the
    wider core issues in peer review of incentivizing engagement,
    providing appropriate recognition and certification, and quality
    Pros and cons of different approaches to anonymity in peer review.

    View full-size slide

  74. Some
    observations
    It seems to be working (i.e. we’re meeting
    a demand that exists)…
    People enjoy editing, reviewing, and being
    reviewed at JOSS.

    View full-size slide

  75. Some
    observations
    It seems to be working (i.e. we’re meeting
    a demand that exists)…
    People enjoy editing, reviewing, and being
    reviewed at JOSS.

    View full-size slide

  76. Some
    observations
    It seems to be working (i.e. we’re meeting
    a demand that exists)…
    People enjoy editing, reviewing, and being
    reviewed at JOSS.

    View full-size slide

  77. Some
    observations
    It seems to be working (i.e. we’re meeting
    a demand that exists)…
    People enjoy editing, reviewing, and being
    reviewed at JOSS.

    View full-size slide

  78. Some
    observations
    It seems to be working (i.e. we’re meeting
    a demand that exists)…
    People enjoy editing, reviewing, and being
    reviewed at JOSS.

    View full-size slide

  79. Some
    observations
    People like the robot*…
    * Does cause occasional confusion

    View full-size slide

  80. Some
    observations
    People like the robot*…
    * Does cause occasional confusion

    View full-size slide

  81. Scaling JOSS
    Most of our challenges are about scaling
    people processes:
    • AEiC/managing editor rotations.
    • More editors.
    • Term limits for editors (to avoid burnout).
    Technology improvements:
    • Smarter reviewer assignments.
    • Better monitoring tools for editors.
    • Tools to help authors prepare their
    submissions.
    0
    500
    1000
    1500
    2017-01-01
    2018-01-01
    2019-01-01
    2020-01-01
    2021-01-01

    View full-size slide

  82. Unexpected
    consequences
    of working
    openly
    Semi-regular emails from people annoyed
    they haven’t been asked to review yet.
    Generally need relatively small number of
    invites to identify reviewers (~2 invites per
    reviewer).
    Vanity software package ‘pile-on’. For high-
    profile open source projects, often have
    many reviewers volunteering.

    View full-size slide

  83. Some awesome
    things about
    working openly
    Zero privileged information in the system:
    Reviewer reports, editorial decisions available
    to all.
    Increase transparency:
    • Public declarations of potential conflicts.
    • Editorial decisions documented in the open.
    • Clear expectations of authors.
    Reduces complexity of infrastructure.
    People can link to their reviews.

    View full-size slide

  84. Zero privileged information in the system: So
    sometimes authors chase reviewers, editors
    etc.
    Good reviewers become well known quickly
    potentially leading to reviewer burnout.
    Potential cultural barriers to entry for some
    and negative dynamics for junior staff.
    Some
    not-so-awesome
    things about
    working openly

    View full-size slide

  85. JOSS is a collaboration
    between author, editor and
    reviewer

    View full-size slide

  86. McGonagle-O’Connell & Ratan: https://doi.org/10.1002/leap.1215

    View full-size slide

  87. Thanks!
    @arfon

    View full-size slide