Where and how knowledge on digital library evaluation spreads: a case study on conference literature

Scholarly communication has not remained unaffected by the advance of the social networking culture. The traditional bibliometric paradigm is strongly questioned as a tool that accurately portrays the impact of research outcomes. New metrics, such as download or view rates and shares, have been proposed as alternative ways for measuring the impact of digital content published in the form of articles, datasets, etc. Mendeley's Readership Statistics are one of these metrics, based on the assumption that there is a linkage between a paper in a collection and the interests of collection owner. The current study explores the “altmetric” aspects of the literature of the Digital Libraries Evaluation domain, as it is expressed in two major conferences of the field, namely JCDL and ECDL. Our corpus consists of 224 papers, for which we extract readership data from Mendeley and examine in how many collections these papers belong to. Our goal is to investigate whether readership statistics can help us to understand where and to whom DL evaluation research has impact. Therefore the data are analyzed statistically to produce indicators of geographical, and topical distribution of Mendeley Readers as well as to explore and classify their their profession. Finally it derived that there is a loose correlation between the number of Google Scholar citations and the number of Mendeley readers.

Paper presented at LIDA2014, on Tuesday, July 17, 2014.
Full text paper available at: http://hdl.handle.net/10889/7587

[CC BY-NC-SA]

Transcript

1. W H E R E A N D H O W K N O W L E D G E O N
   D I G I TA L L I B R A RY E VA L U AT I O N S P R E A D S : A C A S E S T U D Y O N C O N F E R E N C E L I T E R AT U R E LEONIDAS PAPACHRISTOPOULOS, ANGELOS MITRELIS, CHRISTOS PAPATHEODOROU DEPARTMENT OF ARCHIVES, LIBRARY SCIENCE & MUSEOLOGY, IONIAN UNIVERSITY, GREECE ! GIANNIS TSAKONAS LIBRARY & INFORMATION CENTER, UNIVERSITY OF PATRAS, GREECE
2. A I M & S C O P E O F R E S E A R C H • Can ‘altmetrics’ - in the form of Mendeley Readership statistics - reveal knowledge diffusion patterns? • Can data from ‘altmetrics’ - in combination with traditional metrics - help us create quality profiles of conference papers?
3. A LT M E T R I C S • A new portfolio of metrics, based on automatically processed web interactions and transactions. • Downloads, Views, Shares, Likes, Tweets, etc. • Alternative: contradiction with the existing system of calculation and assessment. • Complementary: relation of ‘altmetrics’ with citations.
4. C O N F E R E N C E L I T E R AT U R E • An overlooked publication venue • Hard to index / calculate • differences in periodicity • unavailability of a commonly agreed quality system • superabundance of events
5. M E N D E L E Y : W H Y • A reference management system. • A social network for scholars. • Mendeley’s coverage, especially in cases of very specific venues, such as the conferences, has proven to be very broad. • www.mendeley.com
6. R E S E A R C H S E T T I N G • Corpus • number of papers: 224 • domain: digital library evaluation • sources: two conferences, JCDL and ECDL • period: 2001-2011 • Data • Mendeley Readership statistics • Google Scholar citations
7. R E A D E R S H I P N E T W O R K • nodes = countries • reader’s country (vai ) • paper’s country (vbi ) • edges (vai , vbi ) = state of readership • the inclusion of a paper b in the collection of a Mendeley reader a.
8. W H E R E I T F L O W S : J C D L
9. W H E R E I T F L O W S : E C D L
10. R E A D E R S 0 20 40 60 80 100 Practitioner PhD Student Researcher MSc Student Faculty 27 64 25 74 66 26 48 53 84 97 0 28 56 84 112 140 Computer Science Social Sciences Humanities Engineering Education Design Psychology 10 8 16 0 0 29 98 0 0 10 11 12 27 124 status discipline E C D L J C D L
11. R E A D E R S v s C I TAT I O N S 0 20 40 60 80 100 0 20 40 60 80 100 120 140 160 JCDL ECDL Google Scholar Citations Mendeley Readers Disambiguating Geographic Names in a Historical Digital Library [2001] Enhancing digital libraries with TechLens [2004]
12. A LT M E T R I C S A S ‘ C O N F M E T R I C S ’ ? • An altmetrics powered impact indicator • readers • citations • acceptance rate • year • The harmonic mean of two quality rates, the readers’ and the citations’. Conf. Cites Reads Year Acpt. Rate Indic. a ECDL 61 30 2003 29% 0.34 b JCDL 61 47 2007 36% 0.18
13. C L O S I N G R E M A R K S • Readership: a balanced kind of metric in the landscape of ‘altmetrics’ • More elaborate statistics are needed - at least in the case of Mendeley. • We were able to see which countries produce and which consume knowledge on digital library evaluation.
14. THANK YOU FOR YOUR ATTENTION ! full paper at: http://hdl.handle.net/10889/7587 addendum at: http://gtsak.info/blog/gallery/lida-addendum/ ! contact: john@lis.upatras.gr / twitter: @gtsakonas