Étiquette : Open peer review

Publié le

The Journal of Open Source Software (JOSS): Bringing Open-Source Software Practices to the Scholarly Publishing Community for Authors, Reviewers, Editors, and Publishers

Authors : Patrick Diehl, Charlotte Soneson, Rachel C. Kurchin, Ross Mounce, Daniel S. Katz

Introduction

Open-source software (OSS) is a critical component of open science, but contributions to the OSS ecosystem are systematically undervalued in the current academic system. The Journal of Open Source Software (JOSS) contributes to addressing this by providing a venue (that is itself free, diamond open access, and all open-source, built in a layered structure using widely available elements/services of the scholarly publishing ecosystem) for publishing OSS, run in the style of OSS itself.

A particularly distinctive element of JOSS is that it uses open peer review in a collaborative, iterative format, unlike most publishers. Additionally, all the components of the process—from the reviews to the papers to the software that is the subject of the papers to the software that the journal runs—are open.

Background

We describe JOSS’s history and its peer review process using an editorial bot, and we present statistics gathered from JOSS’s public review history on GitHub showing an increasing number of peer reviewed papers each year. We discuss the new JOSSCast and use it as a data source to understand reasons why interviewed authors decided to publish in JOSS.

Discussion and Outlook

JOSS’s process differs significantly from traditional journals, which has impeded JOSS’s inclusion in indexing services such as Web of Science. In turn, this discourages researchers within certain academic systems, such as Italy’s, which emphasize the importance of Web of Science and/or Scopus indexing for grant applications and promotions. JOSS is a fully diamond open-access journal with a cost of around US$5 per paper for the 401 papers published in 2023. The scalability of running JOSS with volunteers and financing JOSS with grants and donations is discussed.

URL : The Journal of Open Source Software (JOSS): Bringing Open-Source Software Practices to the Scholarly Publishing Community for Authors, Reviewers, Editors, and Publishers

DOI : https://doi.org/10.31274/jlsc.18285

Publié le

On The Peer Review Reports: Does Size Matter?

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

Publié le

Additional experiments required: A scoping review of recent evidence on key aspects of Open Peer Review

Authors : Tony Ross-Hellauer, Serge P.J.M. Horbach

Diverse efforts are underway to reform the journal peer review system. Combined with growing interest in Open Science practices, Open Peer Review (OPR) has become of central concern to the scholarly community. However, what OPR is understood to encompass and how effective some of its elements are in meeting the expectations of diverse communities, are uncertain.

This scoping review updates previous efforts to summarize research on OPR to May 2022. Following the PRISMA methodological framework, it addresses the question: “What evidence has been reported in the scientific literature from 2017 to May 2022 regarding uptake, attitudes, and efficacy of two key aspects of OPR (Open Identities and Open Reports)?”

The review identifies, analyses and synthesizes 52 studies matching inclusion criteria, finding that OPR is growing, but still far from common practice. Our findings indicate positive attitudes towards Open Reports and more sceptical approaches to Open Identities.

Changes in reviewer behaviour seem limited and no evidence for lower acceptance rates of review invitations or slower turnaround times is reported in those studies examining those issues. Concerns about power dynamics and potential backfiring on critical reviews are in need of further experimentation.

We conclude with an overview of evidence gaps and suggestions for future research. Also, we discuss implications for policy and practice, both in the scholarly communications community and the research evaluation community more broadly.

URL : Additional experiments required: A scoping review of recent evidence on key aspects of Open Peer Review

DOI : https://doi.org/10.1093/reseval/rvae004

Publié le

Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

Authors : Leonardo Ferreira Fontenelle, Thiago Dias Sarti

Scholarly journals should consider the attitudes of their communities before adopting any of the seven traits of open peer review. Unfortunately, surveys from the Global North might not apply to the Global South, where double-blind peer review is commonplace even among natural sciences and medicine journals.

This paper reports the findings of a survey on attitudes toward open peer review among four stakeholder groups of a scholar-led medical journal in Brazil: society members, journal readers, authors, and reviewers.

Compared to a previous survey, which mostly recruited natural sciences researchers from Europe, this survey found similar support for open peer review in general and for most of its traits.

One important exception was open identities, which were considered detrimental by most participants, even more in this survey than in the previous one. Interestingly, participants were more dismissive of open identities as a whole than of statements about its specific consequences.

Because preprints are increasingly popular but incompatible with double-blind review, future research should examine the effects of transitioning from double-blind to open identities, especially on gender bias.

Meanwhile, scholarly journals with double-blind review might prefer to begin by adopting other traits of open review or to make open identities optional at first.

URL : Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

DOI : https://doi.org/10.1590/2318-0889202133e200072

Publié le

Modes d’évaluation ouverte par les pairs : de la revue à la plateforme

Auteurs/Authors : Evelyne Broudoux, Madjid Ihadjadene

Cet article a pour but de proposer un état de l’art des différentes formes de l’évaluation d’articles ou de communications par les pairs. De l’évaluation « aveugle» à l’évaluation « ouverte », de multiples possibilités existent et sont expérimentées.

C’est dans le champ des sciences que l’on trouve le plus d’innovations sociotechniques s’appuyant sur des plateformes de publication modélisant des workflows éditoriaux originaux.

L’ouverture de l’évaluation peut se produire entre pairs, en rendant publiques les identités et/ou les rapports des évaluateurs, à différents stades de l’article scientifique : préprint, en cours de rédaction, ou encore après publication.

Cet état de l’art est basé sur un ensemble de publications essentiellement produites par les acteurs de l’évaluation ouverte, issus principalement des disciplines STM.

URL : Modes d’évaluation ouverte par les pairs : de la revue à la plateforme

URL : https://revue-cossi.numerev.com/articles/revue-9/2496-modes-d-evaluation-ouverte-par-les-pairs-de-la-revue-a-la-plateforme

Publié le

Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece

Authors : Eirini Delikoura, Dimitrios Kouis

Recently significant initiatives have been launched for the dissemination of Open Access as part of the Open Science movement. Nevertheless, two other major pillars of Open Science such as Open Research Data (ORD) and Open Peer Review (OPR) are still in an early stage of development among the communities of researchers and stakeholders.

The present study sought to unveil the perceptions of a medical and health sciences community about these issues. Through the investigation of researchers‘ attitudes, valuable conclusions can be drawn, especially in the field of medicine and health sciences, where an explosive growth of scientific publishing exists.

A quantitative survey was conducted based on a structured questionnaire, with 179 valid responses. The participants in the survey agreed with the Open Peer Review principles. However, they ignored basic terms like FAIR (Findable, Accessible, Interoperable, and Reusable) and appeared incentivized to permit the exploitation of their data.

Regarding Open Peer Review (OPR), participants expressed their agreement, implying their support for a trustworthy evaluation system.

Conclusively, researchers need to receive proper training for both Open Research Data principles and Open Peer Review processes which combined with a reformed evaluation system will enable them to take full advantage of the opportunities that arise from the new scholarly publishing and communication landscape.

URL : Open Research Data and Open Peer Review: Perceptions of a Medical and Health Sciences Community in Greece

DOI : https://doi.org/10.3390/publications9020014

Publié le

What Have We Learned from OpenReview?

Authors : Gang Wang, Qi Peng, Yanfeng Zhang, Mingyang Zhang

Anonymous peer review is used by the great majority of computer science conferences. OpenReview is such a platform that aims to promote openness in peer review process. The paper, (meta) reviews, rebuttals, and final decisions are all released to public. We collect 5,527 submissions and their 16,853 reviews from the OpenReview platform.

We also collect these submissions’ citation data from Google Scholar and their non-peer-reviewed versions from arXiv.org. By acquiring deep insights into these data, we have several interesting findings that could help understand the effectiveness of the public-accessible double-blind peer review process.

Our results can potentially help writing a paper, reviewing it, and deciding on its acceptance.

URL : https://arxiv.org/abs/2103.05885v4