Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From opportune indicators to narratives: Conceptualizing altmetrics for social impact

From opportune indicators to narratives: Conceptualizing altmetrics for social impact

Presentation at 4AM Altmetrics Conference in Toronto, Canada.

Altmetrics continue to grow in popularity in part because of their promise to measure ‘something’ beyond the academic impact captured by citations. Yet, despite over half a decade of research into the topic and their almost ubiquitous presence on journal websites, what that ‘something’ is remains elusive. Moreover, and most problematically for proponents of altmetrics as a measure of societal impact, is that there has been almost no theorizing about the role that social media plays towards the many ways in which research can have broader impacts in society. As we continue to collect metrics and create indicators based on data that is easy to gather, we propose taking a step back to consider how to move from these opportune indicators towards a indicators based on a solid theoretical and conceptual basis. In this presentation, we first propose bounding our definition of impact to three main types: epistemic (e.g., new knowledge is created, or someone’s understanding is changed), problem solving (e.g., progress is made towards a problem that needs to be solved), and accountability (e.g., an activity is legitimized). Doing so allows us to understand how different indicators are needed depending on the type of impact being sought, and how the same indicator may mean very different things depending on the impact of interest. Similarly, we present how various research assessment frameworks focus on measuring three different outcomes: behavioural changes, policy changes, and innovation. In doing so, this presentation tackles many of the most important questions facing the altmetrics community: what does it mean for a document to be seen, shared, or commented on? What makes a good indicators? And, how can we use indicators to construct narratives that speak to the societal impact of research?

Juan Pablo Alperin

September 28, 2017
Tweet

More Decks by Juan Pablo Alperin

Other Decks in Education

Transcript

  1. From opportune indicators to narratives:
    Conceptualizing altmetrics for social impact
    JUAN PABLO ALPERIN & KATHERINE REILLY
    SCHOLARLY COMMUNICATIONS LAB
    JUAN PABLO ALPERIN @juancommander #scholcommlab

    View Slide

  2. JUAN PABLO ALPERIN @juancommander #scholcommlab
    opportune indicators
    1. why do we count retweets but not shares?
    2. why don’t we count 2nd order events?
    i. why don’t we count likes?
    ii. shares of news stories about research?
    3. why don’t we have better coverage of non-English
    coverage?

    View Slide

  3. JUAN PABLO ALPERIN @juancommander #scholcommlab
    traces
    metrics
    indicators
    understanding our terms

    View Slide

  4. indicators for …

    View Slide

  5. JUAN PABLO ALPERIN @juancommander #scholcommlab
    indicators for …
    stories of productivity
    JUAN PABLO ALPERIN @juancommander #scholcommlab
    story of impact.
    and

    View Slide

  6. JUAN PABLO ALPERIN @juancommander #scholcommlab

    View Slide

  7. JUAN PABLO ALPERIN @juancommander #scholcommlab
    1. Many outputs can have no outcomes whatsoever, and even if they do result in identifiable outcomes, the
    ‘real world’ impacts are often unclear.
    2. a direct causal link between an outcome and an impact is extremely difficult to demonstrate
    3. more typical for impacts to emerge over a long period of time through a complex confluence of factors.
    4. usually, it is an aggregation of different outputs that lead to a certain impact, so this makes it even harder
    to trace the link from specific outputs to outcomes and impacts.

    View Slide

  8. JUAN PABLO ALPERIN @juancommander #scholcommlab
    Metrics really only have
    value once they become
    indicators of a particular
    type of impact.

    View Slide

  9. JUAN PABLO ALPERIN @juancommander #scholcommlab
    epistemic
    : Has the research changed the way we think about a topic?
    problem solving
    : Has the research solved a problem?
    accountability
    : Has the research affected attitudes about research on a topic?
    bounding the problem

    View Slide

  10. JUAN PABLO ALPERIN @juancommander #scholcommlab
    in practice
    RESEARCH EVALUATION FRAMEWORKS

    View Slide

  11. JUAN PABLO ALPERIN @juancommander #scholcommlab
    behavioural change
    : growth in discussions of given topics
    policy changes
    : shares in policy circles
    innovation
    : productive interactions / network effects
    research assessment frameworks: outcomes

    View Slide

  12. JUAN PABLO ALPERIN @juancommander #scholcommlab
    thank you
    @JUANCOMMANDER
    [email protected]
    This work was supported by the Social Sciences and Humanities Research Council of Canada through an
    Insight Grant (435-2016-1029).

    View Slide