Étiquette : research evaluation

Catégories
EN

Responsible research evaluation: integrating quality, leadership, and integrity in national systems. The case of Peru

Authors : Pablo Alejandro Millones-Gomez, Jean Paul Simon Castillo-Nunez, Kenyie Jossuet Paucca-Calla, Carlos Alberto Minchon-Medina, Vanessa Lizet Castro-Delgado, Samantha Sotelo-Llancari, Cecilia Ignacio-Punin, David Yeret Rodríguez-Salazar, Víctor Hugo Urrutia-Baca

Introduction:

National research evaluation systems often rely on publication-based metrics that equate productivity with performance while overlooking scientific leadership and research integrity. This study examines the Peruvian National Registry of Science, Technology, and Innovation (RENACYT) to inform a more multidimensional framework for research evaluation.

Materials and methods:

An observational, non-experimental, and analytical study was conducted using data from RENACYT, Scopus, and SciVal for 9,651 researchers during 2019–2024. Four dimensions were assessed across hierarchical levels: scientific production, journal-based impact (Q1–Q4), corresponding authorship as a proxy of leadership, and retractions as indicators of research integrity. Descriptive statistics, ANOVA, repeated-measures tests, and count regression models (Poisson, negative binomial, and zero-inflated specifications) were applied.

Results:

A total of 92,284 publications were identified. Productivity increased across RENACYT levels (3.9 publications in Level VII vs. 62.5 in Distinguished; F = 1,162.572, p < 0.001), although with substantial within-level dispersion and differentiated temporal trajectories (Time × Level: F = 44.662, p < 0.001). Higher levels concentrated Q1 output (30.8 vs. 0.7 articles per author; F = 1,090.183, p < 0.001), while differences became less pronounced in Q3–Q4 journals. Corresponding authorship increased with level (β = 1.624 for Level I, p < 0.001) but remained heterogeneous even among top categories. Retractions were positively associated with productivity (coef. = 0.013, p < 0.001) reflecting differential exposure to integrity-related risks rather than uniform patterns across levels.

Conclusion:

RENACYT captures gradients in productivity and quality but insufficiently differentiates leadership and integrity. These findings support the proposal of a hybrid evaluation framework integrating productivity with explicit recognition of intellectual leadership and research integrity.

URL : Responsible research evaluation: integrating quality, leadership, and integrity in national systems. The case of Peru

Original location : https://www.frontiersin.org/journals/research-metrics-and-analytics/articles/10.3389/frma.2026.1842222/full

Catégories
EN

Studying ‘predatory publishing’ in the context of research evaluation: conceptual and methodological challenges

Authors :  Dimity Stephen, Meta Cramer, Emanuel Kulczycki, Martin Reinhart, Federico Vasen, Jakub Krzeski, Rita Fari, Moumita Koley, Marilena Drymioti

Over the past years, the phenomenon of ‘predatory publishing’ has undergone fundamental changes raising pressing methodological and conceptual challenges for its study, particularly in the context of research evaluation.

The complex constellation of commercial, evaluative, and scholarly actors and logics now involved necessitates an interdisciplinary, geographically-diverse, and collaborative approach to studying scholarly – and especially ‘predatory’ – publishing. In this piece, we outline four key conceptual and methodological dimensions that, we argue, scholars must account for when studying this phenomenon.

Firstly, the constantly changing dynamics of who and what constitutes predatory publishers and practices. Secondly, disentangling the complex relationships between evaluation and practice, accounting methodologically for the myriad factors that influence these ties, and recognising that scholarly practices are not a unidirectional effect of evaluations.

Thirdly, scholars must recognise that evaluation regimes are embedded in distinct political economies of academia and that the notion of predatoriness is not universal but culturally, methodologically, and institutionally contingent.

Finally, the common practice of using quantitative analyses alone to study questionable publishing practices risks reproducing existing biases and overlooking structural dynamics, and thus mixed approaches incorporating qualitative methods are necessary to ensure a nuanced understanding of the topic.

We argue that scholars’ approach to ‘predatory publishing’ crucially shapes what empirical dynamics are observed, and consequently call for scholars to take a holistic approach to studying this phenomenon.

URL : Studying ‘predatory publishing’ in the context of research evaluation: conceptual and methodological challenges

DOI : https://doi.org/10.1093/reseval/rvag025

Catégories
EN

Generative AI can and should accelerate research evaluation reform to better recognize ‘distinctly human contributions’

Authors :  Mohammad Hosseini, Brian D Earp, Sebastian Porsdam Mann, Kristi Holmes

As generative artificial intelligence (GenAI) revolutionizes how research is conducted, it also challenges traditional methods of scholarly evaluation. Productivity metrics such as publication and citation counts are widely understood to be poor proxies for gauging meaningful impact. These metrics are becoming even less reliable as GenAI accelerates text-based and computational work while leaving other forms of research labor (e.g. community engagement, in-person mentorship and team development) largely unaffected. This uneven effect risks exacerbating existing evaluative biases.

We argue that evaluation reforms should be organized around two categories of ‘distinctly human contributions’ that are indispensable to research, but which are inadequately captured by metrics: (1) the epistemic-ethical category, encompassing situated judgment under accountability (e.g. deciding what to trust, justifying that decision, and standing behind it); and (2) the socio-relational category, encompassing sustained forms of valuable human engagement (e.g. mentoring, teaching, community partnership and trust-building).

We suggest practical mechanisms for supporting evaluation reform including modified CRediT (Contributor Role Taxonomy) statements, recognition of a broader array of outputs, and strengthened narrative CVs and third-person testimonies.

However, we acknowledge that these suggestions, particularly those relying on narrative self-presentation, are themselves vulnerable to GenAI manipulation and are insufficient on their own. If distinctly human contributions to research require judgment and relationships that resist automation, then evaluation cannot be reduced to instruments designed to minimize human evaluative effort.

GenAI, therefore, does not require entirely new systems of evaluation. Rather, it increases the cost of avoiding what good and ethically sound performance evaluation has always required.

URL : Generative AI can and should accelerate research evaluation reform to better recognize ‘distinctly human contributions’

DOI : https://doi.org/10.1093/reseval/rvag020

Catégories
EN

Back to the roots: reimagining scientific evaluation of research without peer review

Author : Malik Sallam

The peer review system, once a noble aspiration, now lags behind the accelerating demands of modern science.

This opinion piece calls for a decisive departure from that peer review system and advocates for a return to a more accountable, editorially driven model of scholarly evaluation. Scientific editors – already vested with decision‑making authority – should no longer outsource their judgement to external referees. Instead, they must reclaim their rightful role as the primary arbiters of scientific merit.

Too often, editorial judgement is diluted by ritualized consultation, where peer review delays innovation, rewards consensus and obscures responsibility.

I argue for a future in which academic editors decide independently, sign their decisions and are recognized – publicly and professionally – for the intellectual stewardship they provide. By linking editorial work to scientific databases such as Web of Science, Scopus and Google Scholar, we can incentivize rigour, transparency and accountability.

This model would not erode scientific integrity but elevate it, replacing bureaucracy with responsibility. It is time to shed the cloak of anonymity and return authority – and credit – to those best positioned to shape the scientific record; the academic editors themselves.

URL : Back to the roots: reimagining scientific evaluation of research without peer review

DOI : https://doi.org/10.1629/uksg.714

Catégories
EN

From ‘research impact’ to ‘research value’: a new approach to support research for societal benefit

Authors :  Ruth A O’Connor, Sejul Malde, A Wendy Russell, Maya Haviland, Kate Bellchambers, Kirsty Jones, Ginny M Sargent, Sara Bice

University research has a vital role to play in addressing complex societal challenges. The research impact (RI) agenda should enable this but is critiqued for creating an audit culture focused narrowly on economic returns on investment and university rankings. There is a need for alternative approaches that better support research for societal benefit. A current hiatus in research assessment processes in Australia provides an opportunity to explore alternatives.

In this study, we elicited responses from 53 university staff in academic and professional roles to explore what constitutes research impact in practice, and what helps to achieve it. The responses highlight a disconnect between the current institutional framing of research impact and both the practices and values of those seeking to create societal benefit through research.

We identify four tensions between the motivations and practice of research staff on one hand and the research impact agenda on the other. Tensions related to (1) narrow definitions of impact inadequately encompassing valuable work; (2) the premise of linear impact pathways inaccurately portraying the complexity of impact; (3) assessment rewarding individual endeavour over collaboration; and (4) assessment focusing on auditing rather than learning through evaluation.

We take these findings and apply current theories of public and cultural value to offer ‘research value’ as an alternative approach to address the four tensions and nurture research for societal benefit.

URL : From ‘research impact’ to ‘research value’: a new approach to support research for societal benefit

DOI : https://doi.org/10.1093/reseval/rvag002

Catégories
FR

Pour une éthique de l’intelligence artificielle dans le domaine de l’évaluation de la recherche

Authors : Otmane Azeroual, Joachim Schöpfel

L’intelligence artificielle (IA) s’impose aujourd’hui dans de multiples secteurs, de la médecine à la logistique, en passant par la finance et l’éducation. Son intégration croissante dans les systèmes d’information sur la recherche (SI recherche) ouvre de nouvelles perspectives, mais soulève aussi des enjeux éthiques majeurs.

Cet article propose une réflexion sur le rôle de l’IA dans l’évaluation de la recherche, en mettant l’accent sur ses bénéfices, ses limites et la nécessité d’un cadre éthique rigoureux.

URL : Pour une éthique de l’intelligence artificielle dans le domaine de l’évaluation de la recherche

DOI : https://doi.org/10.4000/15gp8

Catégories
EN

Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review

Authors : Paloma González, Martha Fors, Ariel Torres

Altmetrics have emerged as a complementary tool to traditional citation-based metrics in the assessment of scholarly impact. Unlike traditional metrics that primarily capture academic citations over long periods, altmetrics reflect immediate online attention across platforms such as Twitter, blogs, news outlets, and Mendeley.

This article critically examines whether altmetrics can serve as a substitute for traditional metrics by exploring their strengths, limitations, disciplinary variations, and correlation with conventional indicators.

Through a review of recent empirical studies and theoretical debates, the article argues that while altmetrics offer valuable insights into social impact and engagement, they are not yet mature or standardized enough to fully replace traditional metrics. Instead, a hybrid model that integrates both systems may offer a more holistic and inclusive measure of research influence.

URL : Altmetrics in the evaluation of scholarly impact: a systematic and critical literature review

DOI : https://doi.org/10.3389/frma.2025.1693304