Archives des mots-clés : Bibliometrics

The use of ChatGPT for identifying disruptive papers in science: a first exploration

Posté le par

Authors : Lutz Bornmann, Lingfei Wu, Christoph Ettl

ChatGPT has arrived in quantitative research evaluation. With the exploration in this Letter to the Editor, we would like to widen the spectrum of the possible use of ChatGPT in bibliometrics by applying it to identify disruptive papers.

The identification of disruptive papers using publication and citation counts has become a popular topic in scientometrics. The disadvantage of the quantitative approach is its complexity in the computation. The use of ChatGPT might be an easy to use alternative.

URL : The use of ChatGPT for identifying disruptive papers in science: a first exploration

DOI : https://doi.org/10.1007/s11192-024-05176-z

On The Peer Review Reports: Does Size Matter?

Posté le par

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

How open are hybrid journals included in transformative agreements?

Posté le par

Author : Najko Jahn

The ongoing controversy surrounding transformative agreements, which aim to transition journal publishing to full open access, highlight the need for large-scale studies assessing the uptake of open access in hybrid journals. This includes evaluating the extent to which transformative agreements enabled open access.

By combining publicly available data from various sources, including cOAlition S Journal Checker, Crossref, and OpenAlex, this study presents a novel approach that analyses over 700 agreements and nine million journal articles published in more than 11.000 hybrid journals. Estimates suggest a strong growth in open access between 2018 and 2022 from 4.3% to 15%. In 2022, 58% of hybrid open access was enabled by transformative agreements.

This trend was largely driven by the three commercial publishers Elsevier, Springer Nature, and Wiley, but the open access uptake varied substantially across journals, publishers, disciplines, and country affiliations. In particular, comparing the developments in the OECD and BRICS areas revealed different publication trends relative to hybrid open access.

In conclusion, estimates suggest that current levels of implementation of transformative agreements is insufficient to bring about a large-scale transition to full open access.

URL : How open are hybrid journals included in transformative agreements?

Arxiv : https://arxiv.org/abs/2402.18255

Scholar Metrics Scraper (SMS): automated retrieval of citation and author data

Posté le par

Authors : Yutong Cao, Nicole A. Cheung, Dean Giustini, Jeffrey LeDue, Timothy H. Murphy

Academic departments, research clusters and evaluators analyze author and citation data to measure research impact and to support strategic planning. We created Scholar Metrics Scraper (SMS) to automate the retrieval of bibliometric data for a group of researchers.

The project contains Jupyter notebooks that take a list of researchers as an input and exports a CSV file of citation metrics from Google Scholar (GS) to visualize the group’s impact and collaboration. A series of graph outputs are also available. SMS is an open solution for automating the retrieval and visualization of citation data.

URL : Scholar Metrics Scraper (SMS): automated retrieval of citation and author data

DOI : https://doi.org/10.3389/frma.2024.1335454

How can revivals of scientific publications be explained using bibliometric methods? A case study discovering booster papers for the 1985 Physics Nobel Prize paper

Posté le par

Authors : Robin Haunschild, Werner Marx, Jürgen Weis

The unusual citation profile of the 1985 Physics Nobel Prize paper has been analyzed. The number of citing papers per year exhibits a maximum of 123 citations in the mid-1980s and increases to more than 200 citations about two decades later.

The publication set of the citing papers was analyzed in terms of co-authorships and research topics. The USA and (more recently) the People’s Republic of China appear prominently among the countries of the citing authors. A keyword analysis of the citing papers revealed research dealing with topological insulators as one of the major newly evolving research topics. An analysis of the co-cited papers has been performed via reference publication year spectroscopy (RPYS).

The most-frequently co-cited papers (the peak papers of the RPYS spectrogram) were identified and discussed. As a result, we found two primary booster papers and three secondary booster papers that renewed the interest in the 1985 Physics Nobel Prize paper.

URL : How can revivals of scientific publications be explained using bibliometric methods? A case study discovering booster papers for the 1985 Physics Nobel Prize paper

DOI : https://doi.org/10.1007/s11192-023-04906-z

Metrics and peer review agreement at the institutional level

Posté le par

Authors : Vincent A Traag, Marco Malgarini, Sarlo Scipione

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review.

A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011–2014).

In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation.

We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context.

This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.

URL : https://arxiv.org/abs/2006.14830

How do journals deal with problematic articles. Editorial response of journals to articles commented in PubPeer

Posté le par

Authors : José-Luis Ortega, Lorena Delgado-Quirós

The aim of this article is to explore the editorial response of journals to research articles that may contain methodological errors or misconduct. A total of 17,244 articles commented on in PubPeer, a post-publication peer review site, were processed and classified according to several error and fraud categories.

Then, the editorial response (i.e., editorial notices) to these papers were retrieved from PubPeer, Retraction Watch, and PubMed to obtain the most comprehensive picture. The results show that only 21.5% of the articles that deserve an editorial notice (i.e., honest errors, methodological flaws, publishing fraud, manipulation) were corrected by the journal. This percentage would climb to 34% for 2019 publications.

This response is different between journals, but cross-sectional across all disciplines. Another interesting result is that high-impact journals suffer more from image manipulations, while plagiarism is more frequent in low-impact journals.

The study concludes with the observation that the journals have to improve their response to problematic articles.

URL : How do journals deal with problematic articles. Editorial response of journals to articles commented in PubPeer

DOI : https://doi.org/10.3145/epi.2023.ene.18