$30 off During Our Annual Pro Sale. View Details »

A look inside the black box of scientific publishing

Matteo Cavalleri
February 16, 2021

A look inside the black box of scientific publishing

Publishing the results of one’s research is an integral part of the scientific process, yet scholarly journals are often seen as black boxes by researchers. What happens to a paper after it is submitted? Who is deciding on its fate? What is the role of the journal editor and the editorial office? How does the peer-review process work, and are its core principles still relevant in today’s changing publishing landscape? In this talk I will discuss these questions in an attempt to de-mystify the peer review process from an editor's perspective, and cover the whats, the hows and the whys of peer review.

Matteo Cavalleri

February 16, 2021
Tweet

More Decks by Matteo Cavalleri

Other Decks in Science

Transcript

  1. Wiley
    A look inside the black box of scientific publishing
    Dr Matteo Cavalleri, Publisher, Materials Science & Physics
    @physicsteo linkedin.com/teowaits

    View Slide

  2. Who I am
    M.S. in Chemistry (1999)
    PhD in Chemical Physics (2005)
    PostDoc 2005-2008

    View Slide

  3. Publishing is a career for PhDs
    Peer-Review Editor (2008-2010), Berlin
    Associate Editor (2010-2012), NYC
    Editor-in-Chief (2012-2020), NYC
    Executive Editor (2017-2020) & Publisher (Present)

    View Slide

  4. Where I work
    HOBOKEN, NJ
    • Founded in 1807 in NYC
    • Headquarter in Hoboken, NJ
    • Publicly listed in NYSE
    • ~5000 staff worldwide
    • ~1600 journals
    • ~9000 books
    …in partnership with 1085 organizations
    (865 scholarly societies, + institutes,
    universities, goverments,…)

    View Slide

  5. Where I work
    HOBOKEN, NJ
    • Founded in 1807 in NYC
    • Headquarter in Hoboken, NJ
    • Publicly listed in NYSE
    • ~5000 staff worldwide
    • ~1600 journals
    • ~9000 books
    …in partnership with 1085 organizations
    (865 scholarly societies, + institutes,
    universities, goverments,…)
    View from the office (not my office)
    #WorkFromHome

    View Slide

  6. What I do all day Publisher (Present)
    • International in-house editorial team of 7, based in East Coast of US
    • Lead US in-house editorial team (co-founded in 2010)
    • Publisher of the Materials Science & Physics team
    • Portfolio includes OA & subscription titles
    !"#$%&

    View Slide

  7. Why publish?
    •Fame
    •Recognition by peers
    •Fortune
    •Promotions
    •Grant applications
    •Establish precedence
    •Responsibility
    •Taxpayer-funded research
    • Making your research
    public
    • “If your research does
    not generate papers,
    it might just as well
    not have been done.”
    –George Whitesides
    • Papers provide the
    shoulders that others
    can stand on
    The “Publish or Perish” culture responsible for an
    environment where (some) scientific journals are de facto
    gatekeepers of science

    View Slide

  8. Why journals?
    Dissemination
    Spreading the word
    through publishing
    platforms
    But also indexing and
    generally organizing
    knowledge
    Registration
    Precedence of discovery is
    established based on
    article submission date to
    a journal
    Archival
    Safeguarding and
    preserving knowledge
    Publishers play an
    important role preserving
    the scientific record
    Certification
    Peer-review is still the
    gold standard for
    certifying articles
    This is not the same as
    quality-control!
    Peer-review management, Curation, Infrastructure, Ethics, & much, much more. Here’s a list of 96 things publishers do: https://bit.ly/2UW3rKX

    View Slide

  9. How do journals work?

    View Slide

  10. How do journals work?
    By Nick Kim (www.nearingzero.net); used with permission

    View Slide

  11. What is the peer-review process?
    “Peer review is the critical assessment of
    manuscripts submitted to journals by
    experts who are not part of the editorial
    staff”-International Committee of Medical Journals Editors
    WHAT IT CANNOT DO (*)
    WHAT IT SHOULD DO
    -Filter out bad/uninteresting work
    -Make as sure as possible the work
    is reported correctly
    -Make sure results are interpreted
    correctly, and convincingly
    -Improve the quality of publication
    -Detect fabrication
    -Prevent duplicate publication
    -Pick the most interesting papers
    -Ensure quality
    -Ensure the article is right for the
    journal
    (*) AUTOMATICALLY

    View Slide

  12. How did we get here?
    “We (Mr. Rosen and I) had sent you our manuscript for publication and had not
    authorised you to show it to specialists before it is printed. I see no reason to address
    the – in any case erroneous – comments of your anonymous expert.”
    1665: Publication of Journal des Sçavans and Philosophical Transactions of the Royal Society.
    1930s-1940s: Peer-review emerges, not very popular at first
    1960s: Peer-review becomes the institutional standard (Nature, 1967)
    Today: Enjoy overwhelming support despite issues*, alternatives emerge
    * Slow and costly, systemic biases, still misses fraudulent papers

    View Slide

  13. Peer-Review Types
    -SINGLE BLIND: Most common
    -DOUBLE BLIND: Medical journals
    -OPEN: Atmospheric Chemistry and Physics
    -SIGNED: Non-anonymous referees, BMJ Open
    -TECHNICAL PEER-REVIEW ONLY: PLoS One, Scientific Reports, PeerJ,…
    -MIX OF THE ABOVE: Independent/Interactive, “Frontiers In”, EMBO…
    -NONE: Evaluation by community post-publication, arXiv, F1000,…
    -INDIPENDENT FROM JOURNAL: Rubiq, Peerage of Science
    -PORTABLE: Within the same publishing house, portfolio of journals
    -TRANSPARENT: Report published with accepted article

    View Slide

  14. View Slide

  15. The editorial office
    EXTERNAL EDITORS
    IN-HOUSE EDITORS
    …all ACS, T&F, OUP, most
    Springer-Nature, Elsevier, Wiley
    journals…
    …+ some APS, RSC, IOP titles,
    Cell Press, Science, PLOS, The
    Lancet…

    View Slide

  16. The editorial office
    EXTERNAL EDITORS
    IN-HOUSE EDITORS
    Have PhDs, often PostDoc
    experience
    Work full time on journal – can
    dedicate more time and
    resources on new
    developments
    General view
    Have own research group
    Expert in specific field
    BOTH: peer-review, decision making, dealing with appeals,
    commissioning, conference participation and lab visits, writing news
    stories, contributing to “input” marketing …

    View Slide

  17. What editors look for?
    MOST JOURNALS
    -Novelty
    -Importance (in specific field / in related disciplines)
    -Interest
    ALL JOURNALS
    -Scope
    -Format (Communication, full paper, review…)
    -Understandability
    -Compliance to guidelines, ethical behavior
    Editors are not always qualified to evaluate the technical merits of manuscripts.
    This is the job of the referees.

    View Slide

  18. How are referees chosen?
    -Editors’ knowledge & experience
    -From related papers:
    - cited manuscripts
    - literature search
    -Additional research:
    - conference/lab visits
    - web search (good ‘ol Google)
    -Reviewer database:
    - keywords, interest, history…

    View Slide

  19. Referee suggestions are welcome
    -Not just the big names, please
    -No collaborators, previous advisors, grant co-applicants, …
    -Tell us about circumstances that may prevent impartial review:
    - close competitors, who may “scoop” you
    - other conflicts
    …within reason…
    Follow the 3W rule when suggesting reviewers: Tell the editor WHO, WHERE, WHY

    View Slide

  20. Accept, reject, or revise?
    -REJECTION
    - Without external referee reports (Editor)
    - Based on reports
    -REVISION
    - Reconsideration or resubmission
    possible after major revisions
    -ACCEPTANCE
    - Without changes (rare)
    - With minor changes
    The decision is the Editor’s job…the reviewer ‘s recommendation is not a vote -- it’s advice!

    View Slide

  21. Revision
    -REVISION
    • Carefully consider referee
    comments
    • Not all changes have to be
    made…
    • …but need convincing
    arguments for changes not
    made
    • Prepare revision
    • Revise manuscript
    • Highlight changes in
    manuscript
    • Point-by-point response to
    all referee criticisms
    • Changes made
    • Why changes not made
    • Response may go back to
    referees!
    • Need to convince editor
    and referees
    The peer-review process is not a private conversation between authors and
    referees. Try to work your answers to the reviewers in the revised manuscript!

    View Slide

  22. Rejection – not the end of the world!
    -REJECTION
    • Most scientists have been rejected– do not take it personally
    • Try to understand why the paper was rejected
    • Note that you have received the benefit of the Editors and reviewers’ time: take their advice seriously!
    • Re-evaluate your work
    • If you resubmit, begin as if you are going to write a new article
    • Consider offers to transfer for your manuscript to another related journal

    View Slide

  23. Reject and Transfer
    -REJECTION
    -Sharing referee reports and communication
    among editors allow research to find the right
    audience
    -Fast publication, often without further peer-
    review (if rejected on reports)
    -40% of submissions/year (~1.3M manuscripts!)
    are rejected after peer-review (Rubriq, 2013).
    Often by the same reviewers.
    Impact Factor
    12.256
    Impact Factor
    3.075

    View Slide

  24. Appeal
    USUALLY, NO
    -Risk of long time of publication
    -Good papers are found and cited
    -Editors & referees know journal well
    OCCASIONALLY, YES
    -Importance/novelty missed by editor/referees
    -Factual error in referee reports that lead to rejection
    -Need more clarification of decision
    Be calm, argumentative and give scientific justification for reassessment
    It’s false that Editors hate appeals. So should you appeal a rejection?
    Dear Dr. *****:
    Regarding your
    decision on my
    submission **-*****,
    I am afraid you and
    the referees are
    crazy. Please
    reconsider your
    decision.
    Best wishes
    -REJECTION

    View Slide

  25. We are pleased to inform you…

    View Slide

  26. Congratulations!
    -ACCEPTANCE
    • Production data checklist
    • Text format
    • Figure preparation
    • Proofs!
    • Return quickly!
    • Check CAREFULLY copyediting
    changes/queries
    • Ask a friend to check as well
    • NOW pop the prosecco bottle!
    Author
    Correction
    Early View
    Online
    Publication
    Issue Build
    and
    checking
    Issue
    Publishing
    and
    Distribution
    Typesetting
    Submission Peer review
    Copy-
    editing
    In the editorial office:

    View Slide

  27. Welcome to the future…

    View Slide

  28. Publishing is changing
    Science has changed a lot since the 17th century, but research dissemination not too much
    • Print is on the way out, how about the PDF?
    • Open Science & Open Access
    • Data sets and other forms of scientific output (codes!) as
    citable units
    • Referee’s reports are part of the discourse
    • Article metrics >> Journal metrics
    • Articles are discovered outside of issues
    • Demographic shift of authorship
    • Correct identification of authors is paramount!
    https://onlinelibrary.wiley.com/page/journal/1097461x/homepage/interactivearticle_vi.html

    View Slide

  29. Search Engine Optimization (SEO) for your article
    • Search-Engine friendly Title/Abstract
    • Use keywords throughout the article
    • Be consistent with authors names
    • In-bound links rule Google. Link your article
    across social media, networking and
    institutional sites
    • Network, highlight/elevate your colleagues,
    they will do the same for you!
    • Share data, code. Open science leads to
    greater collaboration, increased confidence
    in results and goodwill between researchers
    • Most journals welcome preprints!
    HELP PEOPLE FIND YOU
    https://authorservices.wiley.com/author-resources/Journal-Authors/Prepare/writing-for-seo.html

    View Slide

  30. Open Access
    A growing, global, movement
    800+ OA policies in place globally
    Grounded in shared Declarations Supported by policy changes
    0
    20
    40
    60
    80
    100
    120
    2011 2012 2013 2014 2015 2016 2017 2018 2019
    Substantiated by a growing OA portfolio

    View Slide

  31. Be identified, be credited
    Persistent digital identifier that
    distinguishes researchers from each
    other. http://orcid.org
    High-level taxonomy that can be used to represent the roles
    typically played by contributors to scientific scholarly output.
    https://casrai.org/credit.html

    View Slide

  32. Tl;dr THIS IS NOT HOW IT
    WORKS!
    Dr Matteo Cavalleri @physicsteo linkedin.com/teowaits
    [email protected]

    View Slide