Étiquette : research impact

Catégories
EN

Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

Authors : Kayvan Kousha, Mike Thelwall

Academics and departments are sometimes judged by how their research has benefited society. For example, the UK’s Research Excellence Framework (REF) assesses Impact Case Studies (ICSs), which are five-page evidence-based claims of societal impacts.

This article investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6220 public ICSs from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing ChatGPT’s predictions with published departmental average ICS scores.

The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text and to modify the original REF guidelines to encourage a stricter evaluation.

The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience).

At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.

URL : Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

DOI : https://doi.org/10.1002/asi.25021

Catégories
EN

Which kind of research papers influence policymaking

Author : Pablo Dorta-González

This study examines the use of evidence in policymaking by analysing a range of journal and article attributes, as well as online engagement metrics. It employs a large-scale citation analysis of nearly 150,000 articles covering diverse policy topics. The findings highlight that scholarly citations exert the strongest positive influence on policy citations.

Articles from journals with a higher citation impact and larger Mendeley readership are cited more frequently in policy documents. Other online engagements, such as news and blog mentions, also boost policy citations, while mentions on social media X have a negative effect.

The finding that highly cited and widely read papers are also frequently referenced in policy documents likely reflects the perception among policymakers that such research is more trustworthy. In contrast, papers that derive their influence primarily from social media tend to be cited less often in policy contexts.

DOI : https://doi.org/10.48550/arXiv.2504.18889

Catégories
EN

Unequal Access, Unequal Impact? The Role of Open Access Policies in Publishing and Citation Trends Across Three Countries

Authors : Shlomit Hadad, Daphne R. Raban, Noa Aharony

This bibliometric study investigates Open Access (OA) publication and citation trends in Austria, Israel, and Mexico from 2010 to 2020—three countries with comparable research output but differing OA infrastructures.

(1) Background: The study examines how national OA policies, funding mechanisms, and transformative agreements (TAs) shape publication and citation patterns across disciplines.

(2) Methods: Using Scopus data, the analysis focuses on four broad subject areas (health, physical, life, and social sciences), applying both three-way ANOVA and a Weighted OA Citation Impact index that adjusts citation shares based on the proportional representation of each subject area in national research output. An OA Engagement Score was also developed to assess each country’s policy and infrastructure support.

(3) Results: OA publications consistently receive more citations than closed-access ones, confirming a robust OA citation advantage. Austria leads in both OA publication volume and weighted impact, reflecting its strong policy frameworks and TA coverage. Israel, while publishing fewer OA articles, achieves high citation visibility in specific disciplines. Mexico demonstrates strengths in repositories and Diamond OA journals but lags in transformative agreements.

(4) Conclusions: National differences in OA policy maturity, infrastructure, and publishing models shape both visibility and citation impact. Structural limitations and indexing disparities may further affect how research from different regions and disciplines is represented globally, emphasizing the need for inclusive and context-sensitive frameworks for evaluating OA engagement.

URL : Unequal Access, Unequal Impact? The Role of Open Access Policies in Publishing and Citation Trends Across Three Countries

DOI : https://doi.org/10.3390/publications13020020

Catégories
EN

Does open access foster interdisciplinary citations? Decomposing open access citation advantage

Authors : Kai Nishikawa, Akiyoshi Murakami

The existence of an open access (OA) citation advantage—that is, whether OA increases citations—has been a topic of interest for many years. Although numerous studies have focused on whether OA increases citations, expectations for OA go beyond that. One such expectation is the promotion of knowledge transfer across various fields.

This study aimed to clarify what effects OA, particularly gold OA, has on knowledge transfer across fields. Specifically, we measure the effect of OA on interdisciplinary and within-discipline citation counts by decomposing an existing OA citation advantage metric.

OA increased both interdisciplinary and within-discipline citations in many fields studied, and only interdisciplinary citations in chemistry, computer science, and clinical medicine. In these three fields, clinical medicine showed a tendency toward interdisciplinary citations, independent of journal or paper.

These findings suggest that OA fosters knowledge transfer across disciplines.

URL : Does open access foster interdisciplinary citations? Decomposing open access citation advantage

DOI : https://doi.org/10.1007/s11192-025-05297-z

Catégories
EN

Evaluation of Open Science for co-creation of Social Innovations: Aconceptual framework

Author : Monika Maciuliene

Open Science is a rapidly expanding and diversifying field of social innovation with significant implications for and potential benefits to society, policy and various academic research areas. However, much is still unknown about the co-creation processes in Open Science and an overall conceptual framework which aids such understanding is missing.

The article aims to address these limitations and identify the key dimensions of an ecosystem allowing co-creation in Open Science to unfold its social and economic impact. The research presented integrates the literature analysis on co-creation in multi-stakeholder ecosystems and suggest that three important dimensions have to be considered in evaluation of Open Science ecosystems: framework conditions, system conditions and outcomes.

The proposed model was applied in qualitative analysis of thirty-three Open Science case studies. Based on the results of evaluation, it can be concluded that Open Science landscape is highly heterogenous, fragmented and not fully coordinated. The fragmentation appeared in all dimensions of evaluation.

The outcomes of the research provide a first exploratory step in proposing innovative measures to determine the elements of co-creation practices within Open Science context.

URL : Evaluation of Open Science for co-creation of Social Innovations: Aconceptual framework

DOI : https://doi.org/10.31637/epsir-2022-167

Catégories
EN

The academic impact of Open Science: a scoping review

Authors : Thomas Klebel, Vincent Traag, Ioanna Grypari, Lennart Stoy, Tony Ross-Hellauer

Open Science seeks to make research processes and outputs more accessible, transparent and inclusive, ensuring that scientific findings can be freely shared, scrutinized and built upon by researchers and others. To date, there has been no systematic synthesis of the extent to which Open Science (OS) reaches these aims.

We use the PRISMA scoping review methodology to partially address this gap, scoping evidence on the academic (but not societal or economic) impacts of OS. We identify 485 studies related to all aspects of OS, including Open Access (OA), Open/FAIR Data (OFD), Open Code/Software, Open Evaluation and Citizen Science (CS).

Analysing and synthesizing findings, we show that the majority of studies investigated effects of OA, CS and OFD. Key areas of impact studied are citations, quality, efficiency, equity, reuse, ethics and reproducibility, with most studies reporting positive or at least mixed impacts.

However, we also identified significant unintended negative impacts, especially those regarding equity, diversity and inclusion. Overall, the main barrier to academic impact of OS is lack of skills, resources and infrastructure to effectively re-use and build on existing research.

Building on this synthesis, we identify gaps within this literature and draw implications for future research and policy.

URL : The academic impact of Open Science: a scoping review

DOI : https://doi.org/10.1098/rsos.241248

Catégories
EN

Research on Research Visibility

Authors : Enrique Orduña-Malea, Cristina I. Font-Julián

This editorial explores the significance of research visibility within the evolving landscape of academic communication, mainly focusing on the role of search engines as online meta-markets shaping the impact of research. With the rapid expansion of scientific output and the increasing reliance on algorithm-driven platforms such as Google and Google Scholar, the online visibility of scholarly work has become an essential factor in determining its reach and influence.

The need for more rigorous research into academic search engine optimization (A-SEO), a field still in its infancy despite its growing relevance, is also discussed, highlighting key challenges in the field, including the lack of robust research methodologies, the skepticism within the academic community regarding the commercialization of science, and the need for standardization in reporting and measurement techniques.

This editorial thus invites a multidisciplinary dialogue on the future of research visibility, with significant implications for academic publishing, science communication, research evaluation, and the global scientific ecosystem.

URL : Research on Research Visibility

DOI : https://doi.org/10.1344/bid2024.53.01