Archives des mots-clés : research evaluation

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Posté le par

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345

On The Peer Review Reports: Does Size Matter?

Posté le par

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

Evaluative altmetrics: is there evidence for its application to research evaluation?

Posté le par

Authors : Wenceslao Arroyo-Machado, Daniel Torres-Salinas

Introduction

Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions.

Methods

This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics.

Results

Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level.

Discussion

The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.

URL : Evaluative altmetrics: is there evidence for its application to research evaluation?

DOI : https://doi.org/10.3389/frma.2023.1188131

Metrics and peer review agreement at the institutional level

Posté le par

Authors : Vincent A Traag, Marco Malgarini, Sarlo Scipione

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review.

A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011–2014).

In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation.

We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context.

This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.

URL : https://arxiv.org/abs/2006.14830

Influence of research on open science in the public policy sphere

Posté le par

Authors : Daniela De Filippo, Pablo Sastrón‑Toledo

This paper analyses the scientific activity related to open science in Spain and its influence on public policy from a bibliometric perspective. For this purpose, Spanish centres’ projects and publications on open science from 2010 to 2020 are studied. Subsequently, policy documents using papers related to open science are analysed to study their influence on policymaking.

A total of 142 projects and 1491 publications are analysed, 15% of which are mentioned in policy documents.

The publications cited in policy documents display high proportions of international collaboration, open access publication and publication in first-quartile journals. The findings underline governments’ leading role in the implementation of open science policies and the funding of open science research.

The same government agencies that promote and fund open science research are shown to use that research in their institutional reports, a process known as knowledge flow feedback.

Other non-academic actors are also observed to make use of the knowledge produced by open science research, showing how the open science movement has crossed the boundaries of academia.

URL : Influence of research on open science in the public policy sphere

DOI : https://doi.org/10.1007/s11192-023-04645-1

Les systèmes d’information recherche : un nouvel objet du questionnement éthique

Posté le par

Auteur/Authors : Joachim Schöpfel, Otmane Azeroual

La politique en faveur de la science ouverte interroge les critères et les procédures de l’évaluation de la recherche, tout en mettant en avant les principes fondamentaux de l’éthique scientifique, comme la transparence, l’ouverture et l’intégrité.

Dans ce contexte, nous menons depuis 2020 une analyse de la dimension éthique des systèmes d’information consacrés à l’évaluation de la recherche (SI recherche).

Cet article présente les résultats d’une enquête internationale conduite en 2021 avec un petit échantillon de professionnels et de chercheurs sur deux aspects : l’éthique comme objet du modèle de données de ces systèmes (métriques), et l’aspect éthique de la mise en place et de l’utilisation de ces systèmes.

La discussion fait le lien avec la qualité de ces systèmes, insiste sur la distinction entre l’évaluation des institutions et des personnes et propose l’analyse de ces systèmes à partir du concept d’une responsabilité morale répartie des infrastructures éthiques (infraéthique).

DOI : https://doi.org/10.4000/rfsic.13254

Open Editors: A dataset of scholarly journals’ editorial board positions

Posté le par

Authors : Andreas Nishikawa-Pacher, Tamara Heck, Kerstin Schoch

Editormetrics analyses the role of editors of academic journals and their impact on the scientific publication system. Such analyses would best rely on open, structured, and machine-readable data about editors and editorial boards, which still remains rare.

To address this shortcoming, the project Open Editors collects data about academic journal editors on a large scale and structures them into a single dataset. It does so by scraping the websites of 7,352 journals from 26 publishers (including predatory ones), thereby structuring publicly available information (names, affiliations, editorial roles, ORCID etc.) about 594,580 researchers.

The dataset shows that journals and publishers are immensely heterogeneous in terms of editorial board sizes, regional diversity, and editorial role labels. All codes and data are made available at Zenodo, while the result is browsable at a dedicated website (https://openeditors.ooir.org).

This dataset carries implications for both practical purposes of research evaluation and for meta-scientific investigations into the landscape of scholarly publications, and allows for critical inquiries regarding the representation of diversity and inclusivity across academia.

URL : Open Editors: A dataset of scholarly journals’ editorial board positions

DOI : https://doi.org/10.1093/reseval/rvac037