From opportune indicators to narratives: Conceptualizing altmetrics for social impact

Presentation at 4AM Altmetrics Conference in Toronto, Canada.

Altmetrics continue to grow in popularity in part because of their promise to measure ‘something’ beyond the academic impact captured by citations. Yet, despite over half a decade of research into the topic and their almost ubiquitous presence on journal websites, what that ‘something’ is remains elusive. Moreover, and most problematically for proponents of altmetrics as a measure of societal impact, is that there has been almost no theorizing about the role that social media plays towards the many ways in which research can have broader impacts in society. As we continue to collect metrics and create indicators based on data that is easy to gather, we propose taking a step back to consider how to move from these opportune indicators towards a indicators based on a solid theoretical and conceptual basis. In this presentation, we first propose bounding our definition of impact to three main types: epistemic (e.g., new knowledge is created, or someone’s understanding is changed), problem solving (e.g., progress is made towards a problem that needs to be solved), and accountability (e.g., an activity is legitimized). Doing so allows us to understand how different indicators are needed depending on the type of impact being sought, and how the same indicator may mean very different things depending on the impact of interest. Similarly, we present how various research assessment frameworks focus on measuring three different outcomes: behavioural changes, policy changes, and innovation. In doing so, this presentation tackles many of the most important questions facing the altmetrics community: what does it mean for a document to be seen, shared, or commented on? What makes a good indicators? And, how can we use indicators to construct narratives that speak to the societal impact of research?

Transcript

1. From opportune indicators to narratives: Conceptualizing altmetrics for social impact JUAN PABLO ALPERIN & KATHERINE REILLY SCHOLARLY COMMUNICATIONS LAB JUAN PABLO ALPERIN @juancommander #scholcommlab
2. JUAN PABLO ALPERIN @juancommander #scholcommlab opportune indicators 1. why do we count retweets but not shares? 2. why don’t we count 2nd order events? i. why don’t we count likes? ii. shares of news stories about research? 3. why don’t we have better coverage of non-English coverage?
3. JUAN PABLO ALPERIN @juancommander #scholcommlab traces metrics indicators understanding our terms
4. indicators for …
5. JUAN PABLO ALPERIN @juancommander #scholcommlab indicators for … stories of productivity JUAN PABLO ALPERIN @juancommander #scholcommlab story of impact. and
6. JUAN PABLO ALPERIN @juancommander #scholcommlab
7. JUAN PABLO ALPERIN @juancommander #scholcommlab 1. Many outputs can have no outcomes whatsoever, and even if they do result in identifiable outcomes, the ‘real world’ impacts are often unclear. 2. a direct causal link between an outcome and an impact is extremely difficult to demonstrate 3. more typical for impacts to emerge over a long period of time through a complex confluence of factors. 4. usually, it is an aggregation of different outputs that lead to a certain impact, so this makes it even harder to trace the link from specific outputs to outcomes and impacts.
8. JUAN PABLO ALPERIN @juancommander #scholcommlab Metrics really only have value once they become indicators of a particular type of impact.
9. JUAN PABLO ALPERIN @juancommander #scholcommlab epistemic : Has the research changed the way we think about a topic? problem solving : Has the research solved a problem? accountability : Has the research affected attitudes about research on a topic? bounding the problem
10. JUAN PABLO ALPERIN @juancommander #scholcommlab in practice RESEARCH EVALUATION FRAMEWORKS
11. JUAN PABLO ALPERIN @juancommander #scholcommlab behavioural change : growth in discussions of given topics policy changes : shares in policy circles innovation : productive interactions / network effects research assessment frameworks: outcomes
12. JUAN PABLO ALPERIN @juancommander #scholcommlab thank you @JUANCOMMANDER JUAN@ALPERIN.CA This work was supported by the Social Sciences and Humanities Research Council of Canada through an Insight Grant (435-2016-1029).