Étiquette : research impact

Catégories
EN

Diverse roles of twitter in research evaluation: original tweets and retweets capture different types of engagements with scholarly articles

Authors :  Ashraf Maleki, Kim Holmberg

Altmetrics need to be more critically assessed in terms of the extent to which they reflect impact and quality of research compared to popularity or mere attention. Twitter (now rebranded as X) is a popular platform to, among other things, discuss and share scientific articles.

Earlier altmetric studies have often focused on investigating whether the number of tweets mentioning scientific articles could be used as an indicator of scientific impact or attention, with results showing weak to moderate correlations with citation counts. But all tweets may not be equal, as original tweets and retweets may reflect different levels of engagement and impact. Using a dataset of over 330,000 PLOS publications, this study explores whether these two forms of Twitter activity correlate differently with traditional citation metrics and how these relationships vary across disciplines.

The findings showed the correlation between citations and original tweets was consistently higher than that between citations and retweets and significant weak or moderate, but higher in Social Science and Humanities than in Natural Science, Engineering and Medicine fields. Also, including zero citation counts improved the correlation coefficients for original tweets, but reduced that of retweets.

This indicates that original tweets may be more aligned with citation counts as an indicator of scholarly impact, whereas retweets might reflect broader dissemination and popularity. In conclusion, tweets and retweets are different altmetric indicators and should be considered as two different metrics and analysed separately.

URL : Diverse roles of twitter in research evaluation: original tweets and retweets capture different types of engagements with scholarly articles

DOI : https://doi.org/10.1093/reseval/rvag014

Catégories
EN

Evaluating Open Access Advantages for Citations and Altmetrics (2011-21): A Dynamic and Evolving Relationship

Author : Mike Taylor

Differences between the impacts of Open Access (OA) and non-OA research have been observed over a range of citation and altmetric indicators, usually finding an Open Access Advantage (OAA). However, science-wide analyses covering multiple years, indicators and disciplines are lacking. Using citations and six altmetrics for 33.3M articles published 2011-21, we compare OA and non-OA papers.

The results show that there is no universal OAA across all disciplines or impact indicators: the OAA for citations tends to be lower for recent papers, whereas the OAAs for news, blogs and Twitter are consistent across years and unrelated to volume of OA publications. Wikipedia OAAs are consistently pronounced for all subjects except Humanities (HU) and Social Sciences. Patent OAAs for are strongest for Medical & Health Sciences (MHS) and Life Sciences (LS).

Uniquely, the OAAs for Policy citations is stronger for recently published research. These results support different hypotheses for different subjects and indicators. The evidence is consistent with OA accelerating research impact in MHS, LS and HU; increased visibility/discoverability being a factor in promoting the socio-economic impact; and that OA is a factor in growing online engagement with research. OAAs are therefore complex, dynamic, multi-factorial and require considerable analysis to understand.

URL : Evaluating Open Access Advantages for Citations and Altmetrics (2011-21): A Dynamic and Evolving Relationship

DOI : Serendipity and Scientific Styles: An Ordinary

Catégories
EN

From ‘research impact’ to ‘research value’: a new approach to support research for societal benefit

Authors :  Ruth A O’Connor, Sejul Malde, A Wendy Russell, Maya Haviland, Kate Bellchambers, Kirsty Jones, Ginny M Sargent, Sara Bice

University research has a vital role to play in addressing complex societal challenges. The research impact (RI) agenda should enable this but is critiqued for creating an audit culture focused narrowly on economic returns on investment and university rankings. There is a need for alternative approaches that better support research for societal benefit. A current hiatus in research assessment processes in Australia provides an opportunity to explore alternatives.

In this study, we elicited responses from 53 university staff in academic and professional roles to explore what constitutes research impact in practice, and what helps to achieve it. The responses highlight a disconnect between the current institutional framing of research impact and both the practices and values of those seeking to create societal benefit through research.

We identify four tensions between the motivations and practice of research staff on one hand and the research impact agenda on the other. Tensions related to (1) narrow definitions of impact inadequately encompassing valuable work; (2) the premise of linear impact pathways inaccurately portraying the complexity of impact; (3) assessment rewarding individual endeavour over collaboration; and (4) assessment focusing on auditing rather than learning through evaluation.

We take these findings and apply current theories of public and cultural value to offer ‘research value’ as an alternative approach to address the four tensions and nurture research for societal benefit.

URL : From ‘research impact’ to ‘research value’: a new approach to support research for societal benefit

DOI : https://doi.org/10.1093/reseval/rvag002

Catégories
EN

And then a miracle occurs—a review of theory of change models for societal impact of research

Authors : Ole Henning Sørensen, Stine Dandanell Garn, Steffen Bohni Nielsen

Through an umbrella review, this article identified and surveyed 24 societal impact of research (SIR) models. Most of these models were developed within health domains and in Anglo-Saxon countries. The authors mapped the SIR models against constituent components of a robust theory of change.

The study found that logic models were predominantly used to conceive SIR models. Yet, only nine models had explicit causal links, and only two made explicit assumptions about why research contributes to societal change.

The old proverb among evaluators—when using theories of change to describe change—“and then a miracle occurs…,” rings uncomfortably true to the current state of SIR theorizing. Further theorizing and conceptual clarity are needed to advance the science of research impact.

URL : And then a miracle occurs—a review of theory of change models for societal impact of research

DOI : https://doi.org/10.1093/reseval/rvaf057

 

Catégories
EN

Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review

Authors : Paloma González, Martha Fors, Ariel Torres

Altmetrics have emerged as a complementary tool to traditional citation-based metrics in the assessment of scholarly impact. Unlike traditional metrics that primarily capture academic citations over long periods, altmetrics reflect immediate online attention across platforms such as Twitter, blogs, news outlets, and Mendeley.

This article critically examines whether altmetrics can serve as a substitute for traditional metrics by exploring their strengths, limitations, disciplinary variations, and correlation with conventional indicators.

Through a review of recent empirical studies and theoretical debates, the article argues that while altmetrics offer valuable insights into social impact and engagement, they are not yet mature or standardized enough to fully replace traditional metrics. Instead, a hybrid model that integrates both systems may offer a more holistic and inclusive measure of research influence.

URL : Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review

DOI : https://doi.org/10.3389/frma.2025.1693304

Catégories
EN

A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

Authors : Digant Gupta, Amandeep Kaur, Mansi Malik

Introduction

The primary objective of this study was to investigate the association between journal Impact Factor (IF) and study quality in real-world observational studies. The secondary objective was to explore whether the association changes as a function of different study factors (study design, funding type and geographic location).

Methods

Study quality was assessed using the Newcastle-Ottawa Scale (NOS). IFs were obtained from journal websites. The association between journal IF and NOS score was evaluated firstly using Spearman’s correlation coefficient, and secondly using one-way Analysis of Variance (ANOVA).

Results

We selected 457 studies published in 208 journals across 11 consecutive systematic literature reviews (SLRs) conducted at our organization over the last 5 years. Most studies were cross-sectional and from North America or Europe. Mean (SD) NOS score was 6.6 (1.03) and mean (SD) IF was 5.2 (4.5). Overall, there was a weak positive correlation between NOS score and IF (Spearman’s coefficient (ρ) = 0.23 [95% CI: 0.13–0.31]; p < 0.001). There was no correlation between NOS score and IF for prospective cohort studies (ρ = 0.07 [95% CI:−0.12–0.25]) and industry-funded studies (ρ = 0.06 [95% CI:−0.09–0.21]). Based on ANOVA, the effect size, eta squared (η2), was 0.04 (95% CI: 0.01–0.08), indicating a small effect.

Discussion

While there is some correlation between journal quality and study quality, our findings indicate that high-quality research can be found in journals with lower IF, and assessing study quality requires careful review of study design, methodology, analysis, interpretation, and significance of the findings. Notably, in industry-funded studies, no correlation was found between methodological quality and IF.

URL : A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

DOI : https://doi.org/10.3389/frma.2025.1679842

Catégories
EN

Assessing the Societal Impact of Academic Research With Artificial Intelligence (AI): A Scoping Review of Business School Scholarship as a ‘Force for Good’

Authors : David Steingard, Kathleen Rodenburg

This study addresses critical questions about how current evaluative frameworks for academic research can effectively translate scholarly findings into practical applications and policies to tackle societal ‘grand challenges’.

This scoping review analysis was conducted using bibliometric methods and AI tools. Articles were drawn from a wide range of disciplines, with particular emphasis on the business and management fields, focusing on the burgeoning scholarship area of ‘business as a force for good’.

The novel integration of generative AI research approaches underscores the transformative potential of AI-human collaboration in academic research. Metadata from 4051 articles were examined in the scoping review, with only 370 articles (9.1%) explicitly identified as relevant to societal impact.

This finding reveals a substantial and concerning gap in research addressing the urgent social and environmental issues of our time. To address this gap, the study identifies six meta-themes related to enhancing the societal impact of research: business applications; faculty publication pressure; societal impact focus; sustainable development; university and scholarly rankings; and reference to responsible research frameworks.

Key findings highlight critical misalignments between research outputs and the United Nations Sustainable Development Goals (SDGs) and a lack of practical business applications of research insights.

The results emphasise the urgent need for academic institutions to expand evaluation criteria beyond traditional metrics to prioritise real-world impacts. Recommendations include developing holistic evaluation frameworks and incentivising research that addresses pressing societal challenges—shifting academia from a ‘scholar-to-scholar’ to a ‘scholar-to-society’ paradigm.

The implications of this shift are applied to business-related scholarship and its potential to inspire meaningful societal impact through business practice.

URL : Assessing the Societal Impact of Academic Research With Artificial Intelligence (AI): A Scoping Review of Business School Scholarship as a ‘Force for Good’

DOI : https://doi.org/10.1002/leap.2010