How to fix a broken system: Article-Level Metrics at the Public Library of Science

Presentation given at the Max Delbrück Center in Berlin on March 11, 2014.

1. Martin Fenner ! Technical Lead Article-Level Metrics http://orcid.org/0000-0003-1419-2405
2. 2 Scientific Publishing How to fix a broken system http://commons.wikimedia.org/wiki/File:Image-Wikimania--5_agosto--Broken_lock.png What do I think is broken?
3. 3 It has become more important where we publish than what we publish. https://en.wikipedia.org/wiki/File:Science_1883_Cover.png
4. 4 Do not use journal-based metrics, such as Journal Impact Factors, as a surrogate measure of the quality of individual research articles, to assess an individual scientist's contributions, or in hiring, promotion, or funding decisions. ! ! San Francisco Declaration on Research Assessment, May 2013 http://am.ascb.org/dora/
5. 5 Don't reduce individual research performance to a single number ! Don't use impact factors as measure of quality for individual researchers! Don't apply (hidden) bibliometric filters for selection, 
 e.g. minimum IF for inclusion in publication lists! Don't apply arbitrary weights to co-authorship. 
 Algorithms bases on author position might be problematic! Don't rank scientists according to one indicator. 
 Ranking should not be merely based on bibliometrics! ! Wolfgang Glänzel and Paul Wouters, July 2013 http://www.slideshare.net/paulwouters1/issi2013-wg-pw
6. 6 Funders and universities, too, have a role to play. They must tell the committees that decide on grants and positions not to judge papers by where they are published. It is the quality of the science, not the journal's brand, that matters. ! ! Randy Schekman, 9 Dec 2013 http://www.theguardian.com/commentisfree/2013/dec/09/how-journals- nature-science-cell-damage-science
7. 7 I think peer review is hindering science. In fact, I think it has become a completely corrupt system. It’s corrupt in many ways, in that scientists and academics have handed over to the editors of these journals the ability to make judgment on science and scientists. ! ! Sidney Brenner, 24 Feb 2014 http://kingsreview.co.uk/magazine/blog/2014/02/24/how-academia-and-publishing- are-destroying-scientific-innovation-a-conversation-with-sydney-brenner/
8. 8 Yet how do we actually know what excellence is and where it is worthwhile to foster a scientific elite? In reality, no one actually knows, least of all the politicians who enthusiastically launch such excellence initiatives. ! ! This is where the idea of artificially staged competition comes in. It is assumed that these competitions will automatically make the best rise to the top—without the need to care about neither content nor purpose of research. ! ! Mathias Binswanger, January 2014! ! Binswanger, M. (2014). Excellence by Nonsense: The Competition for Publications in Modern Science. In Opening Science (pp. 49–72). Springer-Verlag. 
 doi:10.1007/978-3-319-00026-8_3
9. 9 Unintended Consequences ! Cascading Journal Submissions http://commons.wikimedia.org/wiki/File:France_Lozere_Runes_Cascade.jpg
10. 10 Unintended Consequences II ! Salami Slicing https://en.wikipedia.org/wiki/Hungarian_cuisine
11. 11 Unintended Consequences III ! Guest and 
 Ghost Authors http://commons.wikimedia.org/wiki/ File:Winston_Churchill_receives_an_honorary_degree_from_Harvard_University_in_Massachusetts,_USA, _6_October_1943._H32723.jpg
12. !12 PLOS Biology
 October, 2003
 
 PLOS Medicine
 October, 2004 PLOS Community Journals
 June-September, 2005 October, 2007 PLOS ONE December, 2006
13. !13 2007 2008 2009 2010 2011 2012 2013 2014 0 500 1000 1500 2000 2500 3000 PLOS ONE Articles Published per Month Source: PLOS Search API at http://api.plos.org
14. Editorial criteria Scientifically rigorous Ethical Properly reported Conclusions supported by the data Editors and reviewers do not ask How important is the work? Which is the relevant audience? Use online tools to sort and filter scholarly content after publication, not before !14 PLOS ONE’s Key Innovation – 
 The editorial process Binfield, P. (2014). Novel Scholarly Journal Concepts. In Opening Science (pp. 155–163). Springer-Verlag. doi:10.1007/978-3-319-00026-8_10
15. 15 https://upload.wikimedia.org/wikipedia/commons/2/2b/Welcome%2C_Nature._Seriously_ %28from_PLoS%29_%285405189157%29.jpg
16. !16 −10 0 10 20 30 40 50 60 0.00 0.02 0.04 0.06 Citation Counts for 2010 PLOS ONE Papers Scopus citation counts for all 6,730 PLOS ONE papers published in 2010. Data collected May 20, 2013. Median 5, mean 9.7 citations, 10% of papers have at least 16 citations. Scopus Citations Probability http://blogs.plos.org/everyone/2013/06/14/plos-one-measuring-article-impact/
17. PLOS ONE is not a second-tier journal Combined Scopus citation counts for all PLOS Biology and PLOS Medicine articles published in 2009, as well as top 200 PLOS ONE articles in 2009 !17 Citation counts collected November 8, 2012 8849 3263 4341 264 199 200 PLOS Biology PLOS Medicine PLOS ONE Median 15.5 Median 11 Median 51.5
18. PLOS ONE was the 3th most cited journal in 2012 1% of the 2010 top 1% most cited papers were published in PLOS ONE !18 Are elite journals declining? http://arxiv.org/pdf/1304.6460v1 ! http://www.scimagojr.com/journalrank.php? area=0&category=0&country=all&year=2012&order=tc&min=0&min_type=cd
19. 19 http://www.flickr.com/photos/timetrax/376152628/ But how do we find the relevant content?
20. 20 Use online tools to sort and filter scholarly content after publication, not before http://www.flickr.com/photos/mecklenburg/5340182341
21. 7 Post-Publication Tools and Metrics Early Discovery Deep Discovery Assess Individual Assess Program Personal reference + + - - Peer review - - + ?? Journals + + ?? ?? Citations - + + + Usage Stats + + - ? Altmetrics + + ? ?
22. 22 20, 2013 and include all PLOS Biology articles published up to that day. Data for F1000Prime were collected on August 15, 2013. All charts were produced with R version 3.0.0. (ZIP) Figure 5. PLOS Biology articles: sites of recommendation and discussion. Number of PLOS Biology research articles published up to May 20, 2013 that have been recommended by F1000Prime (red) or mentioned in Wikipedia (blue). doi:10.1371/journal.pbio.1001687.g005 Different metrics measure different things: 
 F1000Prime vs. Wikipedia http://dx.doi.org/10.1371/journal.pbio.1001687
23. 23 0 20 40 60 80 100 120 0 5000 10000 15000 20000 25000 Usage and Citations for MDC PLOS Articles HTML views vs. PDF downloads from PLOS website for all 169 PLOS articles with affiliation Max Delbrück Center. Bubble size correlates with Scopus citations, color with journal. Data collected March 10, 2014. Months since Publication Views and Downloads
24. 24 Neural Differentiation of Embryonic Stem Cells In Vitro: A Road Map to Neurogenesis http://alm.plos.org/articles/info:doi/10.1371/journal.pone.0006286
25. 25 A Genetic Basis for Mechanosensory Traits in Humans in the Embryo http://alm.plos.org/articles/info:doi/10.1371/journal.pbio.1001318
26. 26 http://www.zeit.de/news/2012-05/02/wissenschaft-studie-schwerhoerige-haben-schlechteren-tastsinn-02064408
27. 27 Outlier-Based Identification of Copy Number Variations Using Targeted Resequencing in a Small Cohort of Patients with Tetralogy of Fallot in the Embryo http://alm.plos.org/articles/info:doi/10.1371/journal.pone.0085375
28. PLOS is working on improving article- level metrics as a discovery tool http://almreports.plos.org http://articlemetrics.github.io
29. 29 It should be more important what we publish than where we publish. ! Journals, libraries and others can build services that help us discover good science, but at the end only researchers themselves understanding the science can make that assessment. Slides available from https://speakerdeck.com/mfenner/how-to-fix-a-broken- system-article-level-metrics-at-the-public-library-of-science