A look inside the black box of scientific publishing

Publishing the results of one’s research is an integral part of the scientific process, yet scholarly journals are often seen as black boxes by researchers. What happens to a paper after it is submitted? Who is deciding on its fate? What is the role of the journal editor and the editorial office? How does the peer-review process work, and are its core principles still relevant in today’s changing publishing landscape? In this talk I will discuss these questions in an attempt to de-mystify the peer review process from an editor's perspective, and cover the whats, the hows and the whys of peer review.

Transcript

1. Wiley
   A look inside the black box of scientific publishing
   Dr Matteo Cavalleri, Publisher, Materials Science & Physics
   @physicsteo linkedin.com/teowaits
   

   View Slide

2. Who I am
   M.S. in Chemistry (1999)
   PhD in Chemical Physics (2005)
   PostDoc 2005-2008
   

   View Slide

3. Publishing is a career for PhDs
   Peer-Review Editor (2008-2010), Berlin
   Associate Editor (2010-2012), NYC
   Editor-in-Chief (2012-2020), NYC
   Executive Editor (2017-2020) & Publisher (Present)
   

   View Slide

4. Where I work
   HOBOKEN, NJ
   • Founded in 1807 in NYC
   • Headquarter in Hoboken, NJ
   • Publicly listed in NYSE
   • ~5000 staff worldwide
   • ~1600 journals
   • ~9000 books
   …in partnership with 1085 organizations
   (865 scholarly societies, + institutes,
   universities, goverments,…)
   

   View Slide

5. Where I work
   HOBOKEN, NJ
   • Founded in 1807 in NYC
   • Headquarter in Hoboken, NJ
   • Publicly listed in NYSE
   • ~5000 staff worldwide
   • ~1600 journals
   • ~9000 books
   …in partnership with 1085 organizations
   (865 scholarly societies, + institutes,
   universities, goverments,…)
   View from the office (not my office)
   #WorkFromHome
   

   View Slide

6. What I do all day Publisher (Present)
   • International in-house editorial team of 7, based in East Coast of US
   • Lead US in-house editorial team (co-founded in 2010)
   • Publisher of the Materials Science & Physics team
   • Portfolio includes OA & subscription titles
   !"#$%&
   

   View Slide

7. Why publish?
   •Fame
   •Recognition by peers
   •Fortune
   •Promotions
   •Grant applications
   •Establish precedence
   •Responsibility
   •Taxpayer-funded research
   • Making your research
   public
   • “If your research does
   not generate papers,
   it might just as well
   not have been done.”
   –George Whitesides
   • Papers provide the
   shoulders that others
   can stand on
   The “Publish or Perish” culture responsible for an
   environment where (some) scientific journals are de facto
   gatekeepers of science
   

   View Slide

8. Why journals?
   Dissemination
   Spreading the word
   through publishing
   platforms
   But also indexing and
   generally organizing
   knowledge
   Registration
   Precedence of discovery is
   established based on
   article submission date to
   a journal
   Archival
   Safeguarding and
   preserving knowledge
   Publishers play an
   important role preserving
   the scientific record
   Certification
   Peer-review is still the
   gold standard for
   certifying articles
   This is not the same as
   quality-control!
   Peer-review management, Curation, Infrastructure, Ethics, & much, much more. Here’s a list of 96 things publishers do: https://bit.ly/2UW3rKX
   

   View Slide

9. How do journals work?
   

   View Slide

10. How do journals work?
    By Nick Kim (www.nearingzero.net); used with permission
    

    View Slide

11. What is the peer-review process?
    “Peer review is the critical assessment of
    manuscripts submitted to journals by
    experts who are not part of the editorial
    staff”-International Committee of Medical Journals Editors
    WHAT IT CANNOT DO (*)
    WHAT IT SHOULD DO
    -Filter out bad/uninteresting work
    -Make as sure as possible the work
    is reported correctly
    -Make sure results are interpreted
    correctly, and convincingly
    -Improve the quality of publication
    -Detect fabrication
    -Prevent duplicate publication
    -Pick the most interesting papers
    -Ensure quality
    -Ensure the article is right for the
    journal
    (*) AUTOMATICALLY
    

    View Slide

12. How did we get here?
    “We (Mr. Rosen and I) had sent you our manuscript for publication and had not
    authorised you to show it to specialists before it is printed. I see no reason to address
    the – in any case erroneous – comments of your anonymous expert.”
    1665: Publication of Journal des Sçavans and Philosophical Transactions of the Royal Society.
    1930s-1940s: Peer-review emerges, not very popular at first
    1960s: Peer-review becomes the institutional standard (Nature, 1967)
    Today: Enjoy overwhelming support despite issues*, alternatives emerge
    * Slow and costly, systemic biases, still misses fraudulent papers
    

    View Slide

13. Peer-Review Types
    -SINGLE BLIND: Most common
    -DOUBLE BLIND: Medical journals
    -OPEN: Atmospheric Chemistry and Physics
    -SIGNED: Non-anonymous referees, BMJ Open
    -TECHNICAL PEER-REVIEW ONLY: PLoS One, Scientific Reports, PeerJ,…
    -MIX OF THE ABOVE: Independent/Interactive, “Frontiers In”, EMBO…
    -NONE: Evaluation by community post-publication, arXiv, F1000,…
    -INDIPENDENT FROM JOURNAL: Rubiq, Peerage of Science
    -PORTABLE: Within the same publishing house, portfolio of journals
    -TRANSPARENT: Report published with accepted article
    

    View Slide

14. View Slide

15. The editorial office
    EXTERNAL EDITORS
    IN-HOUSE EDITORS
    …all ACS, T&F, OUP, most
    Springer-Nature, Elsevier, Wiley
    journals…
    …+ some APS, RSC, IOP titles,
    Cell Press, Science, PLOS, The
    Lancet…
    

    View Slide

16. The editorial office
    EXTERNAL EDITORS
    IN-HOUSE EDITORS
    Have PhDs, often PostDoc
    experience
    Work full time on journal – can
    dedicate more time and
    resources on new
    developments
    General view
    Have own research group
    Expert in specific field
    BOTH: peer-review, decision making, dealing with appeals,
    commissioning, conference participation and lab visits, writing news
    stories, contributing to “input” marketing …
    

    View Slide

17. What editors look for?
    MOST JOURNALS
    -Novelty
    -Importance (in specific field / in related disciplines)
    -Interest
    ALL JOURNALS
    -Scope
    -Format (Communication, full paper, review…)
    -Understandability
    -Compliance to guidelines, ethical behavior
    Editors are not always qualified to evaluate the technical merits of manuscripts.
    This is the job of the referees.
    

    View Slide

18. How are referees chosen?
    -Editors’ knowledge & experience
    -From related papers:
    - cited manuscripts
    - literature search
    -Additional research:
    - conference/lab visits
    - web search (good ‘ol Google)
    -Reviewer database:
    - keywords, interest, history…
    

    View Slide

19. Referee suggestions are welcome
    -Not just the big names, please
    -No collaborators, previous advisors, grant co-applicants, …
    -Tell us about circumstances that may prevent impartial review:
    - close competitors, who may “scoop” you
    - other conflicts
    …within reason…
    Follow the 3W rule when suggesting reviewers: Tell the editor WHO, WHERE, WHY
    

    View Slide

20. Accept, reject, or revise?
    -REJECTION
    - Without external referee reports (Editor)
    - Based on reports
    -REVISION
    - Reconsideration or resubmission
    possible after major revisions
    -ACCEPTANCE
    - Without changes (rare)
    - With minor changes
    The decision is the Editor’s job…the reviewer ‘s recommendation is not a vote -- it’s advice!
    

    View Slide

21. Revision
    -REVISION
    • Carefully consider referee
    comments
    • Not all changes have to be
    made…
    • …but need convincing
    arguments for changes not
    made
    • Prepare revision
    • Revise manuscript
    • Highlight changes in
    manuscript
    • Point-by-point response to
    all referee criticisms
    • Changes made
    • Why changes not made
    • Response may go back to
    referees!
    • Need to convince editor
    and referees
    The peer-review process is not a private conversation between authors and
    referees. Try to work your answers to the reviewers in the revised manuscript!
    

    View Slide

22. Rejection – not the end of the world!
    -REJECTION
    • Most scientists have been rejected– do not take it personally
    • Try to understand why the paper was rejected
    • Note that you have received the benefit of the Editors and reviewers’ time: take their advice seriously!
    • Re-evaluate your work
    • If you resubmit, begin as if you are going to write a new article
    • Consider offers to transfer for your manuscript to another related journal
    

    View Slide

23. Reject and Transfer
    -REJECTION
    -Sharing referee reports and communication
    among editors allow research to find the right
    audience
    -Fast publication, often without further peer-
    review (if rejected on reports)
    -40% of submissions/year (~1.3M manuscripts!)
    are rejected after peer-review (Rubriq, 2013).
    Often by the same reviewers.
    Impact Factor
    12.256
    Impact Factor
    3.075
    

    View Slide

24. Appeal
    USUALLY, NO
    -Risk of long time of publication
    -Good papers are found and cited
    -Editors & referees know journal well
    OCCASIONALLY, YES
    -Importance/novelty missed by editor/referees
    -Factual error in referee reports that lead to rejection
    -Need more clarification of decision
    Be calm, argumentative and give scientific justification for reassessment
    It’s false that Editors hate appeals. So should you appeal a rejection?
    Dear Dr. *****:
    Regarding your
    decision on my
    submission **-*****,
    I am afraid you and
    the referees are
    crazy. Please
    reconsider your
    decision.
    Best wishes
    -REJECTION
    

    View Slide

25. We are pleased to inform you…
    

    View Slide

26. Congratulations!
    -ACCEPTANCE
    • Production data checklist
    • Text format
    • Figure preparation
    • Proofs!
    • Return quickly!
    • Check CAREFULLY copyediting
    changes/queries
    • Ask a friend to check as well
    • NOW pop the prosecco bottle!
    Author
    Correction
    Early View
    Online
    Publication
    Issue Build
    and
    checking
    Issue
    Publishing
    and
    Distribution
    Typesetting
    Submission Peer review
    Copy-
    editing
    In the editorial office:
    

    View Slide

27. Welcome to the future…
    

    View Slide

28. Publishing is changing
    Science has changed a lot since the 17th century, but research dissemination not too much
    • Print is on the way out, how about the PDF?
    • Open Science & Open Access
    • Data sets and other forms of scientific output (codes!) as
    citable units
    • Referee’s reports are part of the discourse
    • Article metrics >> Journal metrics
    • Articles are discovered outside of issues
    • Demographic shift of authorship
    • Correct identification of authors is paramount!
    https://onlinelibrary.wiley.com/page/journal/1097461x/homepage/interactivearticle_vi.html
    

    View Slide

29. Search Engine Optimization (SEO) for your article
    • Search-Engine friendly Title/Abstract
    • Use keywords throughout the article
    • Be consistent with authors names
    • In-bound links rule Google. Link your article
    across social media, networking and
    institutional sites
    • Network, highlight/elevate your colleagues,
    they will do the same for you!
    • Share data, code. Open science leads to
    greater collaboration, increased confidence
    in results and goodwill between researchers
    • Most journals welcome preprints!
    HELP PEOPLE FIND YOU
    https://authorservices.wiley.com/author-resources/Journal-Authors/Prepare/writing-for-seo.html
    

    View Slide

30. Open Access
    A growing, global, movement
    800+ OA policies in place globally
    Grounded in shared Declarations Supported by policy changes
    0
    20
    40
    60
    80
    100
    120
    2011 2012 2013 2014 2015 2016 2017 2018 2019
    Substantiated by a growing OA portfolio
    

    View Slide

31. Be identified, be credited
    Persistent digital identifier that
    distinguishes researchers from each
    other. http://orcid.org
    High-level taxonomy that can be used to represent the roles
    typically played by contributors to scientific scholarly output.
    https://casrai.org/credit.html
    

    View Slide

32. Tl;dr THIS IS NOT HOW IT
    WORKS!
    Dr Matteo Cavalleri @physicsteo linkedin.com/teowaits
    mcavalleri@wiley.com
    

    View Slide