Archives des mots-clés : Peer Review

Accelerated Peer Review and Paper Processing Models in Academic Publishing

Posté le par

Authors : Jaime A. Teixeira da Silva, Yuki Yamada

Some journals and publishers offer a free or paid rapid peer review service. In the latter case, such a service is offered at a premium, i.e., for an additional fee, and authors receive, in return, a privileged service, namely faster peer review.

In the cut-throat world of survival in academia, the difference of a few weeks or months in terms of speed of peer review and publication may bring untold benefits to authors that manage to benefit from accelerated peer review.

We examine the deontological aspects behind this two-tier peer review system, including some positive, but mainly negative, aspects. Some paid accelerated peer review services thrive.

We examine the paid accelerated peer review services by Taylor & Francis, Future Medicine Ltd., Elsevier, and two stand-alone journals that are OASPA members. This suggests that there is a demand, and thus market, for faster peer review.

However, this privilege risks creating a two-tiered system that may divide academics between those who can pay versus those who cannot.

We recommend that those papers that have benefited from accelerated peer review clearly indicate this in the published papers, as either a disclaimer or within the acknowledgements, for maximum transparency of the peer review and publication process.

DOI : https://doi.org/10.1007/s12109-022-09891-4

Transparency versus anonymity: which is better to eliminate bias in peer review?

Posté le par

Authors: Faye Holst, Kim Eggleton, Simon Harris

Peer review is a critical component of the scientific process. When conducted properly by dedicated and competent reviewers, it helps to safeguard the quality, validity, authority and rigour of academic work. However, bias in peer review is well documented and can skew objectivity of the review and hinder fair assessment of research.

To mitigate against bias and enhance accountability, IOP Publishing has introduced two different, but complementary, approaches to all their peer-reviewed, open access (OA) journals: double-anonymous peer review and transparent peer review.

Double-anonymous peer review, where the reviewer and author identities are concealed, is designed to tackle inequality in the scholarly publishing process as it reduces bias with respect to gender, race, country of origin or affiliation.

Transparent peer review shows readers the full peer review history, including reviewer reports, editor decision letters and the authors’ responses alongside the published article. Making this process visible to the community increases accountability, allows reviewers to be recognized more for their work and can aid the training of aspiring reviewers.

IOP Publishing is the first physics publisher to adopt both of these approaches portfolio wide. In this article we discuss how applying these methods has altered different elements of the publishing process. Early indicators show that there may be a marked difference in acceptance rates across regions.

URL : Transparency versus anonymity: which is better to eliminate bias in peer review?

DOI : http://doi.org/10.1629/uksg.584

The Role of Publons in the Context of Open Peer Review

Posté le par

Authors : Jaime A. Teixeira da Silva, Serhii Nazarovets

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers.

Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes.

In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform.

Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services.

We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

DOI : https://doi.org/10.1007/s12109-022-09914-0

Gender diversity of research consortia contributes to funding decisions in a multi-stage grant peer-review process

Posté le par

Authors : Stefano Bianchini, Patrick Llerena, Sıla Öcalan-Öze, Emre Özel

This study seeks to draw connections between the grant proposal peer-review and the gender representation in research consortia.

We examined the implementation of a multi-disciplinary, pan-European funding scheme—EUROpean COllaborative RESearch Scheme (2003–2015)—and the reviewers’ materials that this generated. EUROCORES promoted investigator-driven, multinational collaborative research in multiple scientific areas and brought together 9158 Principal Investigators (PI) who teamed up in 1347 international consortia that were sequentially evaluated by 467 expert panel members and 1862 external reviewers.

We found systematically unfavourable evaluations for consortia with a higher proportion of female PIs. This gender effect was evident in the evaluation outcomes of both panel members and reviewers: applications from consortia with a higher share of female scientists were less successful in panel selection and received lower scores from external reviewers.

Interestingly, we found a systematic discrepancy between the evaluative language of written review reports and the scores assigned by reviewers that works against consortia with a higher share of female participants.

Reviewers did not perceive female scientists as being less competent in their comments, but they were negatively sensitive to a high female ratio within a consortium when scoring the proposed research project.

URL : Gender diversity of research consortia contributes to funding decisions in a multi-stage grant peer-review process

DOI : https://doi.org/10.1057/s41599-022-01204-6

Characteristics of ‘mega’ peer-reviewers

Posté le par

Authors : Danielle B. Rice, Ba’ Pham, Justin Presseau, Andrea C. Tricco, David Moher

Background

The demand for peer reviewers is often perceived as disproportionate to the supply and availability of reviewers. Considering characteristics associated with peer review behaviour can allow for the development of solutions to manage the growing demand for peer reviewers.

The objective of this research was to compare characteristics among two groups of reviewers registered in Publons.

Methods

A descriptive cross-sectional study design was used to compare characteristics between (1) individuals completing at least 100 peer reviews (‘mega peer reviewers’) from January 2018 to December 2018 as and (2) a control group of peer reviewers completing between 1 and 18 peer reviews over the same time period.

Data was provided by Publons, which offers a repository of peer reviewer activities in addition to tracking peer reviewer publications and research metrics. Mann Whitney tests and chi-square tests were conducted comparing characteristics (e.g., number of publications, number of citations, word count of peer review) of mega peer reviewers to the control group of reviewers.

Results

A total of 1596 peer reviewers had data provided by Publons. A total of 396 M peer reviewers and a random sample of 1200 control group reviewers were included. A greater proportion of mega peer reviews were male (92%) as compared to the control reviewers (70% male).

Mega peer reviewers demonstrated a significantly greater average number of total publications, citations, receipt of Publons awards, and a higher average h index as compared to the control group of reviewers (all p < .001). We found no statistically significant differences in the number of words between the groups (p > .428).

Conclusions

Mega peer reviewers registered in the Publons database also had a higher number of publications and citations as compared to a control group of reviewers. Additional research that considers motivations associated with peer review behaviour should be conducted to help inform peer reviewing activity.

URL : Characteristics of ‘mega’ peer-reviewers

DOI : https://doi.org/10.1186/s41073-022-00121-1

Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

Posté le par

Author : Dimity Stephen

The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function.

Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings.

These studies also found the effect of peer review on manuscripts’ content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers’ comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers’ comments.

I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts.

I also examined changes in references used between versions and linked changes to reviewers’ comments. Reviewers’ comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%).

Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints.

Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

URL : Peer reviewers equally critique theory, method, and writing, with limited effect on the final content of accepted manuscripts

DOI : https://doi.org/10.1007/s11192-022-04357-y

How to improve scientific peer review: Four schools of thought

Posté le par

Authors : Ludo Waltman, Wolfgang Kaltenbrunner, Stephen Pinfield, Helen Buckley Woods

Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review.

Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved.

To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school.

Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between the schools.

We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.

URL : How to improve scientific peer review: Four schools of thought

DOI : https://doi.org/10.31235/osf.io/v8ghj