Catégories
EN

Science of science — Citation models and research evaluation

Author : Vincent Traag

Citations in science are being studied from several perspectives, among which approaches such as scientometrics and science of science. In this chapter I briefly review some of the literature on citations, citation distributions and models of citations. These citations feature prominently in another part of the literature which is dealing with research evaluation and the role of metrics and indicators in that process.

Here I briefly review part of the discussion in research evaluation. This also touches on the subject of how citations relate to peer review. Finally, I conclude by trying to integrate the two literatures. The fundamental problem in research evaluation is that research quality is unobservable. This has consequences for conclusions that we can draw from quantitative studies of citations and citation models.

The term « indicators » is a relevant concept in this context, which I try to clarify. Causality is important for properly understanding indicators, especially when indicators are used in practice: when we act on indicators, we enter causal territory. Even when an indicator might have been valid, through its very use, the consequences of its use may invalidate it. By combining citation models with proper causal reasoning and acknowledging the fundamental problem about unobservable research quality, we may hope to make progress.

DOI : https://doi.org/10.48550/arXiv.2207.11116

Catégories
EN

Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platforms

Authors : Mike Thelwall, Abdallah Yaghi

Academic peer review is at the heart of scientific quality control, yet the process is slow and time-consuming. Technology that can predict peer review outcomes may help with this, for example by fast-tracking desk rejection decisions. While previous studies have demonstrated that Large Language Models (LLMs) can predict peer review outcomes to some extent, this paper introduces two new contexts and employs a more robust method—averaging multiple ChatGPT scores.

Averaging 30 ChatGPT predictions, based on reviewer guidelines and using only the submitted titles and abstracts failed to predict peer review outcomes for F1000Research (Spearman’s rho = 0.00). However, it produced mostly weak positive correlations with the quality dimensions of SciPost Physics (rho = 0.25 for validity, rho = 0.25 for originality, rho = 0.20 for significance, and rho = 0.08 for clarity) and a moderate positive correlation for papers from the International Conference on Learning Representations (ICLR) (rho = 0.38). Including article full texts increased the correlation for ICLR (rho = 0.46) and slightly improved it for F1000Research (rho = 0.09), with variable effects on the four quality dimension correlations for SciPost LaTeX files.

The use of simple chain-of-thought system prompts slightly increased the correlation for F1000Research (rho = 0.10), marginally reduced it for ICLR (rho = 0.37), and further decreased it for SciPost Physics (rho = 0.16 for validity, rho = 0.18 for originality, rho = 0.18 for significance, and rho = 0.05 for clarity). Overall, the results suggest that in some contexts, ChatGPT can produce weak pre-publication quality predictions.

However, their effectiveness and the optimal strategies for employing them vary considerably between platforms, journals, and conferences. Finally, the most suitable inputs for ChatGPT appear to differ depending on the platform.

URL : Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platforms

DOI : https://doi.org/10.1007/s11192-025-05287-1

Catégories
EN

Gaps between Open Science activities and actual recognition systems: Insights from an international survey

Authors : Florencia Grattarola, Hanna Shmagun, Christopher Erdmann, Anne Cambon-Thomsen, Mogens Thomsen, Jaesoo Kim, Laurence Mabile

There are global movements aiming to promote reform of the traditional research evaluation and reward systems. However, a comprehensive picture of the existing best practices and efforts across various institutions to integrate Open Science into these frameworks remains underdeveloped and not fully known. The aim of this study was to identify perceptions and expectations of various research communities worldwide regarding how Open Science activities are (or should be) formally recognised and rewarded.

To achieve this, a global survey was conducted in the framework of the Research Data Alliance, recruiting 230 participants from five continents and 37 countries. Despite most participants reporting that their organisation had one form or another of formal Open Science policies, the majority indicated that their organisation lacks any initiative or tool that provides specific credits or rewards for Open Science activities. However, researchers from France, the United States, the Netherlands and Finland affirmed having such mechanisms in place. T

he study found that, among various Open Science activities, Open or FAIR data management and sharing stood out as especially deserving of explicit recognition and credit. Open Science indicators in research evaluation and/or career progression processes emerged as the most preferred type of reward.

URL : Gaps between Open Science activities and actual recognition systems: Insights from an international survey

DOI : https://doi.org/10.1371/journal.pone.0315632

Catégories
FR

Stratégies et comportements de communication scientifique de chercheurs en sciences humaines et sociales face à l’évaluation de la recherche publique

Auteur/Author : Cheikh Ndiaye

Cette thèse se penche sur les enjeux de la communication et de l’évaluation dans le domaine de la recherche en Sciences de l’Information et de la Communication. Partant du principe que la science contribue au progrès d’une société en lui permettant de répondre à ses préoccupations (santé, économie, etc.), divers pays industrialisés comme la France investissent des sommes importantes pour son essor. Ces investissements suscitant, à leur tour, des prescrits afin d’en assurer une gestion efficiente et un impact en adéquation avec les objectifs de développement. Ainsi ont été institutionnalisées l’évaluation de la recherche publique et sa valorisation.

Leur montée en puissance a eu des effets bouleversants sur le paysage scientifique, notamment la gouvernance des universités, leurs missions et rapports au monde économique ou encore la production des connaissances. Au cœur de ses changements, l’évaluation impacte les orientations politiques autour de la recherche, en particulier le choix des priorités thématiques et la distribution des financements. Elle induit un besoin de valorisation individuelle qui semble obliger les chercheurs à intégrer davantage dans leurs activités la quête construite d’autorité, donc de reconnaissance scientifique.Ayant toujours été un régulateur de la compétition entre savants et un vecteur de diffusion de leurs connaissances produites, la communication scientifique est-elle devenue un outil de stratégie ?

Le but de cette recherche est de répondre à cette interrogation. En se fondant sur un modèle psychosociologique, la théorie du comportement planifié, il s’agit d’examiner la construction de ce changement comportemental éventuel, à partir de la motivation, de l’attitude, des perceptions des normes sociales et de la capacité à conduire le changement et l’intention comportementale. Il s’agit également de voir les liens existant entre ces variables et l’intention de s’adapter à l’évaluation, voire le comportement final adopté.

La première partie de l’étude pose les bases conceptuelles de la thèse en examinant les relations entre les Sciences Humaines et Sociales, l’espace scientifique, l’économie et les politiques publiques. Elle explore la dualité entre les SHS et les sciences exactes, ainsi que l’impact du numérique sur la recherche.

La deuxième partie se concentre sur le chercheur en tant que communicateur et objet d’évaluation. Elle aborde les différents moyens de communication scientifique, y compris les publications, les médias sociaux, et les médias alternatifs, ainsi que les enjeux sociaux de la communication dans le domaine académique.

La troisième et dernière partie examine le chercheur en tant qu’acteur central de la recherche. D’une part, elle se penche sur les aspects de l’évaluation de la recherche, notamment la bibliométrie et les classements universitaires. De l’autre, elle expose les théories du comportement humain, le modèle théorique et la méthodologie de la recherche. Ce qui permet de présenter les résultats de l’enquête par un questionnaire auprès des chercheurs.

En conclusion, l’étude met en évidence une perception contraignante de l’évaluation scientifique au-delà d’une incitation à publier. Elle apparaît, en effet, comme une obligation professionnelle dont l’inexécution peut avoir des effets néfastes sur la carrière.

Son attitude est plutôt favorable au changement, et les normes sociales perçues l’y poussent alors qu’il a confiance en ses capacités à intégrer les prescriptions dans sa stratégie de communication. De ce fait, le chercheur en SHS à l’intention de s’adapter en construisant une communication adaptative, en sélectionnant les prescrits lui convenant.

Au final, son comportement adopté consiste à élaborer une stratégie au cas par cas, basée sur la publication évaluée par les pairs afin d’obtenir divers bénéfices : visibilité et reconnaissance scientifiques, intégration sociale, accès aux médias et autorités politiques, responsabilités administratives ou scientifiques.

URL : https://theses.fr/2023MON30038

Catégories
EN

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345

Catégories
EN

On The Peer Review Reports: Does Size Matter?

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

Catégories
EN

Evaluative altmetrics: is there evidence for its application to research evaluation?

Authors : Wenceslao Arroyo-Machado, Daniel Torres-Salinas

Introduction

Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions.

Methods

This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics.

Results

Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level.

Discussion

The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.

URL : Evaluative altmetrics: is there evidence for its application to research evaluation?

DOI : https://doi.org/10.3389/frma.2023.1188131