Gaps between Open Science activities and actual recognition systems: Insights from an international survey

Authors : Florencia Grattarola, Hanna Shmagun, Christopher Erdmann, Anne Cambon-Thomsen, Mogens Thomsen, Jaesoo Kim, Laurence Mabile

There are global movements aiming to promote reform of the traditional research evaluation and reward systems. However, a comprehensive picture of the existing best practices and efforts across various institutions to integrate Open Science into these frameworks remains underdeveloped and not fully known. The aim of this study was to identify perceptions and expectations of various research communities worldwide regarding how Open Science activities are (or should be) formally recognised and rewarded.

To achieve this, a global survey was conducted in the framework of the Research Data Alliance, recruiting 230 participants from five continents and 37 countries. Despite most participants reporting that their organisation had one form or another of formal Open Science policies, the majority indicated that their organisation lacks any initiative or tool that provides specific credits or rewards for Open Science activities. However, researchers from France, the United States, the Netherlands and Finland affirmed having such mechanisms in place. T

he study found that, among various Open Science activities, Open or FAIR data management and sharing stood out as especially deserving of explicit recognition and credit. Open Science indicators in research evaluation and/or career progression processes emerged as the most preferred type of reward.

URL : Gaps between Open Science activities and actual recognition systems: Insights from an international survey

DOI : https://doi.org/10.1371/journal.pone.0315632

Stratégies et comportements de communication scientifique de chercheurs en sciences humaines et sociales face à l’évaluation de la recherche publique

Auteur/Author : Cheikh Ndiaye

Cette thèse se penche sur les enjeux de la communication et de l’évaluation dans le domaine de la recherche en Sciences de l’Information et de la Communication. Partant du principe que la science contribue au progrès d’une société en lui permettant de répondre à ses préoccupations (santé, économie, etc.), divers pays industrialisés comme la France investissent des sommes importantes pour son essor. Ces investissements suscitant, à leur tour, des prescrits afin d’en assurer une gestion efficiente et un impact en adéquation avec les objectifs de développement. Ainsi ont été institutionnalisées l’évaluation de la recherche publique et sa valorisation.

Leur montée en puissance a eu des effets bouleversants sur le paysage scientifique, notamment la gouvernance des universités, leurs missions et rapports au monde économique ou encore la production des connaissances. Au cœur de ses changements, l’évaluation impacte les orientations politiques autour de la recherche, en particulier le choix des priorités thématiques et la distribution des financements. Elle induit un besoin de valorisation individuelle qui semble obliger les chercheurs à intégrer davantage dans leurs activités la quête construite d’autorité, donc de reconnaissance scientifique.Ayant toujours été un régulateur de la compétition entre savants et un vecteur de diffusion de leurs connaissances produites, la communication scientifique est-elle devenue un outil de stratégie ?

Le but de cette recherche est de répondre à cette interrogation. En se fondant sur un modèle psychosociologique, la théorie du comportement planifié, il s’agit d’examiner la construction de ce changement comportemental éventuel, à partir de la motivation, de l’attitude, des perceptions des normes sociales et de la capacité à conduire le changement et l’intention comportementale. Il s’agit également de voir les liens existant entre ces variables et l’intention de s’adapter à l’évaluation, voire le comportement final adopté.

La première partie de l’étude pose les bases conceptuelles de la thèse en examinant les relations entre les Sciences Humaines et Sociales, l’espace scientifique, l’économie et les politiques publiques. Elle explore la dualité entre les SHS et les sciences exactes, ainsi que l’impact du numérique sur la recherche.

La deuxième partie se concentre sur le chercheur en tant que communicateur et objet d’évaluation. Elle aborde les différents moyens de communication scientifique, y compris les publications, les médias sociaux, et les médias alternatifs, ainsi que les enjeux sociaux de la communication dans le domaine académique.

La troisième et dernière partie examine le chercheur en tant qu’acteur central de la recherche. D’une part, elle se penche sur les aspects de l’évaluation de la recherche, notamment la bibliométrie et les classements universitaires. De l’autre, elle expose les théories du comportement humain, le modèle théorique et la méthodologie de la recherche. Ce qui permet de présenter les résultats de l’enquête par un questionnaire auprès des chercheurs.

En conclusion, l’étude met en évidence une perception contraignante de l’évaluation scientifique au-delà d’une incitation à publier. Elle apparaît, en effet, comme une obligation professionnelle dont l’inexécution peut avoir des effets néfastes sur la carrière.

Son attitude est plutôt favorable au changement, et les normes sociales perçues l’y poussent alors qu’il a confiance en ses capacités à intégrer les prescriptions dans sa stratégie de communication. De ce fait, le chercheur en SHS à l’intention de s’adapter en construisant une communication adaptative, en sélectionnant les prescrits lui convenant.

Au final, son comportement adopté consiste à élaborer une stratégie au cas par cas, basée sur la publication évaluée par les pairs afin d’obtenir divers bénéfices : visibilité et reconnaissance scientifiques, intégration sociale, accès aux médias et autorités politiques, responsabilités administratives ou scientifiques.

URL : https://theses.fr/2023MON30038

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345

On The Peer Review Reports: Does Size Matter?

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

Evaluative altmetrics: is there evidence for its application to research evaluation?

Authors : Wenceslao Arroyo-Machado, Daniel Torres-Salinas

Introduction

Altmetrics have been demonstrated as a promising tool for analyzing scientific communication on social media. Nevertheless, its application for research evaluation remains underdeveloped, despite the advancement of research in the study of diverse scientific interactions.

Methods

This paper develops a method for applying altmetrics in the evaluation of researchers, focusing on a case study of the Environment/Ecology ESI field publications by researchers at the University of Granada. We considered Twitter as a mirror of social attention, news outlets as media, and Wikipedia as educational, exploring mentions from these three sources and the associated actors in their respective media, contextualizing them using various metrics.

Results

Our analysis evaluated different dimensions such as the type of audience, local attention, engagement generated around the mention, and the profile of the actor. Our methodology effectively provided dashboards that gave a comprehensive view of the different instances of social attention at the author level.

Discussion

The use of altmetrics for research evaluation presents significant potential, as shown by our case study. While this is a novel method, our results suggest that altmetrics could provide valuable insights into the social attention that researchers garner. This can be an important tool for research evaluation, expanding our understanding beyond traditional metrics.

URL : Evaluative altmetrics: is there evidence for its application to research evaluation?

DOI : https://doi.org/10.3389/frma.2023.1188131

Metrics and peer review agreement at the institutional level

Authors : Vincent A Traag, Marco Malgarini, Sarlo Scipione

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review.

A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011–2014).

In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation.

We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context.

This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.

URL : https://arxiv.org/abs/2006.14830

Influence of research on open science in the public policy sphere

Authors : Daniela De Filippo, Pablo Sastrón‑Toledo

This paper analyses the scientific activity related to open science in Spain and its influence on public policy from a bibliometric perspective. For this purpose, Spanish centres’ projects and publications on open science from 2010 to 2020 are studied. Subsequently, policy documents using papers related to open science are analysed to study their influence on policymaking.

A total of 142 projects and 1491 publications are analysed, 15% of which are mentioned in policy documents.

The publications cited in policy documents display high proportions of international collaboration, open access publication and publication in first-quartile journals. The findings underline governments’ leading role in the implementation of open science policies and the funding of open science research.

The same government agencies that promote and fund open science research are shown to use that research in their institutional reports, a process known as knowledge flow feedback.

Other non-academic actors are also observed to make use of the knowledge produced by open science research, showing how the open science movement has crossed the boundaries of academia.

URL : Influence of research on open science in the public policy sphere

DOI : https://doi.org/10.1007/s11192-023-04645-1