Article-Level Metrics

Presentation given at a NISO Webinar on Alternative Metrics for Scholarship, more information at http://www.niso.org/news/events/2012/nisowebinars/alternative_metrics/

Transcript

1. http://www.flickr.com/photos/wallyg/156079118/ Martin Fenner Technical Lead ALM Project
2. Journal-based metrics correlate poorly with numbers for individual article 2 0 20 40 60 80 100 0.00 0.01 0.02 0.03 0.04 0.05 Many PLOS ONE Papers are Highly Cited Number of Scopus citations for all 4405 PLoS ONE papers published in 2009. Data collected November 8, 2012. Median is 9 citations, 10% of papers have at least 29 citations. Scopus Citations Probability
3. PLOS ONE is not a second-tier journal Combined Scopus citation counts for all PLOS Biology and PLOS Medicine articles published in 2009, as well as top 200 PLOS ONE articles in 2009 3 Citation counts collected November 8, 2012 8849 3263 4341 264 199 200 PLOS Biology PLOS Medicine PLOS ONE Median 15.5 Median 11 Median 51.5
4. The megajournal has changed how we use the journal as a filter 4 Megajournal Keyword/ Subject Area Journal Keyword/ Subject Area Metrics w ork in progress
5. 124,985,344 HTML pageviews 27,722,142 PDF downloads 374,958 CrossRef citations 5 0.3% 100% 22.2% Citations are only a small fraction of how a paper is reused and discussed Article-Level Metrics from November 8, 2012 for 63,771 PLOS Papers
6. 6 The PLOS Article-Level Metrics project started in 2009 and tracks usage, citations and social web activity for all PLOS articles Usage Citations Social Web PLOS Journals (HTML, PDF, XML) CrossRef Mendeley PLOS Journals (HTML, PDF, XML) Scopus CiteULike PubMed Central (abstract, fulltext, PDF, figures) Web of Science Facebook PubMed Central (abstract, fulltext, PDF, figures) PubMed Central Twitter PubMed Central (abstract, fulltext, PDF, figures) Research Blogging PLOS Comments Wikipedia
7. 7 PLoS HTML Views PMC HTML Views CrossRef Scopus PubMed Citations Mendeley CiteULike PLoS Comments Research Blogging Facebook Twitter Wikipedia 100% 98% 63% 62% 49% 71% 25% 12% 2% 27% 13% 6% Proportion of articles covered by source Metrics for 63,771 PLoS journal articles published until November 8, 2012.
8. 8 PLoS HTML Views PMC HTML Views CrossRef Scopus PubMed Citations Mendeley CiteULike PLoS Comments Research Blogging Facebook Twitter Wikipedia 100% 91% 17% 21% 3% 51% 48% 16% 4% 76% 92% 17% Proportion of articles covered by source Metrics for 98 PLoS Biology articles published May 8 − November 8, 2012.
9. Why use article-level metrics? 9 Data-driven stories about the post- publication reception of a paper Business intelligence for a journal, university or funder Evaluation of the impact of research and researchers Metrics as a discovery tool
10. Should article-level metrics be used to evaluate the impact of research? • Can numbers reflect the impact of research, across disciplines and over time? • Do the currently available metrics really measure impact or something else? • Does the use of metrics for evaluation create undesired incentives? • How easily can metrics be changed by self-promotion and gaming? 10
11. What do we measure? 11 Impact Attention Self- Promotion
12. 12 Gaming and Spam
13. Metrics as a discovery tool 13
14. Data-driven stories about the post- publication reception of a paper 14 0 20 40 60 80 0 10000 20000 30000 40000 50000 60000 PLOS Articles by Type: Review Total views are combined HTML views and PDF downloads from the PLOS website. Bubble size correlates with the number of CrossRef citations, and bubble color with the PLOS journal. Data collected August 23, 2012. Months Total Views
15. 15 http://blogs.bbsrc.ac.uk/index.php/2012/09/summertime-reprise/ http://dx.doi.org/10.1371/journal.pcbi.1000204
16. 16 0 20000 40000 60000 80000 58 182 251 66 241 265 17 51 90 69 18 97 138 62 44 42 33 29 68 173 15 10 29 Mendeley Ten Simple Rules for Getting Published Ten Simple Rules for Reviewers Ten Simple Rules for Getting Grants Ten Simple Rules for Graduate Students Ten Simple Rules for Doing Your Best Research, According to Hamming Ten Simple Rules for a Good Poster Presentation Ten Simple Rules for Making Good Oral Presentations Ten Simple Rules for a Successful Collaboration Ten Simple Rules for Organizing a Scientific Meeting Ten Simple Rules for Aspiring Scientists in a Low−Income Country Ten Simple Rules for Chairing a Scientific Session Ten Simple Rules for Choosing between Industry and Academia Ten Simple Rules To Combine Teaching and Research Ten Simple Rules for Editing Wikipedia Ten Simple Rules for Organizing a Virtual ConferenceÂAnywhere Ten Simple Rules for Developing a Short Bioinformatics Training Course Ten Simple Rules for Getting Involved in Your Scientific Community Ten Simple Rules for Teaching Bioinformatics at the High School Level Ten Simple Rules for Getting Help from Online Scientific Communities Ten Simple Rules for Building and Maintaining a Scientific Reputation Ten Simple Rules for Providing a Scientific Web\nResource Ten Simple Rules for Getting Ahead as a Computational Biologist in Academia Ten Simple Rules for Starting a Company Popularity of the Ten Simple Rules Collection PDF downloads (open circles), HTML pageviews (closed circles), and Mendeley readers (numbers on the right) for 23 editorials in PLoS Computational Biology. Articles are sorted by descending age. Total Article Views
17. Business intelligence for a journal, university of funder 17
18. 18 0 2 4 6 8 10 12 14 0 5000 10000 15000 20000 25000 30000 PLOS Medicine Articles August 2011 to August 2012 Combined HTML views and PDF downloads (PLOS website) for articles published between 08/16/11 and 08/15/12. Bubble size correlates with Scopus citations, bubble color with article type. Data collected 08/24/12. Months Total Views http://bit.ly/alm36
19. 19 0 50 100 150 200 250 0 2000 4000 6000 8000 10000 12000 14000 PLOS Articles from the Big Food collection Tweets vs. HTML views at PLOS for articles in the Big Food collection. Circle size correlates with Facebook activity. Data collected August 19, 2012. Tweets HTML Views
20. 20 S S M T W T F 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 1 2 3 S S M T W T F 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 1 2 S S M T W T F 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6 S S M T W T F 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 5 6 7 8 9 10 11 1 2 3 4 31 April S S M T W T F 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 2 3 4 5 6 7 8 1 28 29 30 May S S M T W T F 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 1 2 3 4 5 6 26 27 28 29 30 31 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 6 26 11 7 26 172 19 19 12 119 43 53 June S S M T W T F 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 4 5 6 7 8 9 10 1 2 3 30 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 16 2 2 23 3 2 3 4 1 7 5 5 8 4 2 7 17 9 105 28 20 July S S M T W T F 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 2 3 4 5 6 7 28 29 30 31 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 1 1 3 2 1 5 4 1 12 5 August S S M T W T F 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 1 2 3 4 5 25 26 27 28 29 30 31 September October November December 50 100 150 Tweets about Big Food Collection Data collected August 19, 2012.
21. 21 PLOS Search API 2562 Articles CORDIS 1166 Articles Identify Open Access articles published by PLOS, funded by the European Commission Seventh Framework Programme (FP7) Match articles with grant numbers Show the impact of these publications
22. Author locations from the ENGAGE project 22 First authors (blue) and last authors (green). European Network for Genetic and Genomic Epidemiology http://www.openaire.eu/en/component/openaire/project_info/default/530?grant=201413
23. 23 0 10 20 30 40 0 5000 10000 15000 PLOS Publications from the EU−funded ENGAGE Project Total views are combined HTML views and PDF downloads from the PLOS website. Bubble size correlates with Scopus citations, and bubble color with the PLOS journal. Data collected September 19, 2012. Months Total Views Scopus
24. 24 0 10 20 30 40 0 5000 10000 15000 PLOS Publications from the EU−funded ENGAGE Project Total views are combined HTML views and PDF downloads from the PLOS website. Bubble size correlates with Mendeley bookmarks, and bubble color with the PLOS journal. Data collected September 19, 2012. Months Total Views Mendeley
25. 25 Thank you Wordle of altmetrics manifesto by AJ Cann http://www.flickr.com/photos/ajc1/6795008004/