An invited talk at the 2013 Society for Scholarly Publishing on Altmetrics for assessment of research.
Metrics for Assessing
Society for Scholarly
William Gunn, PhD
Head of Academic Outreach
Based in London, Mendeley is
researchers, graduates and software
• What’s the problem?
• What’s the opportunity?
• How do we serve researchers?
• How do we get the context
What is the problem?
• Is a researcher doing impactful research?
• Does an institution have an effective
• Is the research being effectively
• Is the work advancing the field?
How do you measure advancement?
What are the units?
People like numbers
“Not everything that can be counted
counts and not everything that
counts can be counted.” – William
Bruce Cameron (1967)
Let research be research
• Helping researchers make better decisions
• Some answers
– Find out what researchers really need & try to
meet that need.
• Some unanswered questions
– What do the different metrics tell you?
– Whose needs are best served by which info?
• Some unasked questions
What we do know
• Amgen: 47 of 53 “landmark” oncology
publications could not be reproduced.
• Bayer: 43 of 67 oncology & cardiovascular
projects were based on contradictory results
• Dr. John Ioannidis: Of 432 publications
purporting sex differences in hypertension,
multiple sclerosis, or lung cancer, only one data
set was reproducible.
“authors who make
data from their articles
available are cited twice as
frequently as articles with no
data… but otherwise equal
Gleditsch, N.P. & Strand, H. Posting your data: will you be
scooped or will you be famous? International Studies
Perspectives 4, 89-97 (2003).
“The state of knowledge
of the human race is
sitting in the scientists’
computers, and is
currently not shared […]
We need to get it
unlocked so we can
tackle those huge
Post-scarcity in data
be baked into the
tools that we use… I
would like for you to
think of yourselves as
people engaged in a
task that is important
to everyone and not
just people in a
...and aggregates research
data in the cloud
Mendeley makes science more collaborative and
• Let’s get better at understanding how to
meet researcher needs.
• Let’s get the context right for metrics.
• Practical tips
– Collect lots of data
– Build expertise in analysis
– Make that data available openly – it’s more
valuable in aggregate than siloed apart.