Étiquette : Peer Review

Catégories
EN

Paying for Publication with Peer Review Design and Simulation of a Fee-Free, Credit-Based Open Access Journal for Information Technology

Author : Murat Ozer

Open access publishing has moved much of the cost of scholarly communication from readers to authors, while the labor of peer review remains unpaid and unevenly shared.

This paper describes the design of Quorum: An Open Journal of Information Technology (this http URL), a journal that charges neither readers nor authors and uses peer review as its currency instead. Every first complete review a scholar writes on a manuscript earns one credit, and one credit pays for one submission.

Each manuscript receives three double-blind reviews from reviewers matched automatically by expertise, and editorial decisions follow published rules rather than editor discretion.

The paper documents the credit ledger, a bootstrapping mechanism called the genesis block, a reviewer search that widens after seven days and opens the paper to volunteers, the decision rules, and the integrity safeguards. An agent-based simulation, run for two simulated years with 40 replications per scenario, evaluates these choices.

Without the genesis block, no paper was submitted in any replication. With it, a community growing from 40 to about 330 members submitted about 603 papers, and papers waited a median of 7 days for a full review panel.

The day-7 escalation had a modest effect in a large community but a decisive one in a small community of about 60 members, where removing it left 48.8% of papers without a panel after 60 days, compared with 4.6% with it. Unspent credits also accumulated to about 3.7 per member after two years, which points to a policy choice the journal will face as it grows.

DOI : https://doi.org/10.48550/arXiv.2609.33864

Catégories
EN

If You Really Want to Know How AI Is Changing Peer Review Talk to Early Career Researchers

Authors : David Nicholas, Blanca Rodríguez-Bravo, David Clark, Abdullah Abrizah, Jorge Revez, Marzena Swigon, John Akeroyd, Jie Xu, Tatyana Polezhaeva, Eti Herman

The Harbingers study of early career researchers (ECRs), their work life and scholarly communications, began by studying generational—Millennial—change (H1 c.2016), then pandemic change (H2 c2020) and is now investigating yet another: artificial intelligence (H3 2024–).

We report here on a scoping pilot study, which looks at the impact of AI on peer review from over 150 international ECRs from all disciplines. The data were collected by open-ended, in-depth interviews, but being a convenience sample the findings need to be treated with caution.

We find that ECRs are displaying a complex, shifting attitude towards the integration of AI into peer review, characterised by cautious optimism on the one hand and scepticism on the other. ECRs are already reporting specific instances of AI increasingly influencing reviewing practices: both potentially beneficial applications and actual misuse.

AI is seen as a new systemic threat, potentially worsening the alleged exploitative nature of publishing if adopted without ethical guidelines.

URL : If You Really Want to Know How AI Is Changing Peer Review Talk to Early Career Researchers

DOI : https://doi.org/10.1002/leap.2082

Catégories
EN

Effectiveness of the researcher-led “Peerspectives” peer review training course on review quality, knowledge, and skills among doctoral students in the biomedical sciences: a pre-post study

Authors : Jessica L. Rohmann, Nadja Wülk, Kerstin Rubarth, Hannah Grillmaier, Iman Abdikarim, Mariana Lopes Simões, Sara Schroter, Marco Piccininni, Tobias Kurth, Toivo Glatz

Background

Peer review remains a cornerstone of scientific knowledge dissemination, yet comprehensive, practically relevant training is limited. This inspired us to develop Peerspectives, a peer review training course for doctoral students in the biomedical sciences in Berlin, Germany. We aimed to assess the effectiveness of the Peerspectives course on editor-judged quality of peer review reports.

Methods

Doctoral students in health research fields who enrolled in the Peerspectives course between October 2020 and August 2022 were invited to participate in the study, and 80 consented. The ~18 week-long course provided training on the structure, purpose, and conduct of peer review and editorial processes in biomedical journals. It included 12 h of lectures, homework assignments, and 12 h of hands-on, small-group workshops, during which students reviewed original research manuscripts currently under consideration at The BMJ under the guidance of experienced mentors.

The primary outcome was the overall quality of the peer review reports as judged by two independent BMJ editors using the global score of the Review Quality Instrument (RQI) pre- and post-intervention. Additionally, we compared participants’ post-course scores with those of actual BMJ reviewers. We also compared participants’ self-assessed knowledge and skills related to scholarly peer review (1–5 Likert scale) before and after the course.

Results

After course completion, the editor-assessed overall quality of the participants’ peer review reports was higher than before the course (median increase of 0.5 points, p < 0.001; mean increase of 0.36 points, p < 0.001). The RQI scores of participants’ post-course reports were not non-inferior to those of actual BMJ reviewers for the same manuscripts. Self-assessed peer review-related knowledge skills increased across all questionnaire items after course completion. Greatest improvements were seen in understanding reviewer expectations (increase in means from 2.9 to 4.5), confidence in reviewing (2.5 to 3.9), and knowing what to look for while reviewing (2.8 to 4.2).

Conclusions

Providing doctoral students with comprehensive training resulted in an editorially significant increase in review report quality and improved understanding of the role and expectations of peer reviewers in the scholarly publishing processes and confidence in giving constructive feedback.

URL : Effectiveness of the researcher-led “Peerspectives” peer review training course on review quality, knowledge, and skills among doctoral students in the biomedical sciences: a pre-post study

DOI : https://doi.org/10.1186/s41073-026-00220-3

Catégories
EN

Un/Sustainable Peer Review and Generative AI: Ethical Gaps, Editorial Acceleration, and the Whitewashing of Technological Solutionism

Authors : Angel Gord, Chris H. Gray, Ana Rodrígue, Elías Said-Hung, Raúl Tabaré

Generative AI in peer review raises ethical and environmental concerns and risks deepening existing inequities in scholarly publishing. Celebrated gains in speed often mask declines in quality and accountability.

Training and deploying large models impose environmental costs. In editorial workflows, AI can privilege technical fixes over structural reform, and evidence shows it reproduces human biases while being cast as neutral. We call for a renewed commitment to open-science principles anchored in human oversight, deep sustainability, and broader justice.

The paper concludes by interrogating sustainability’s absence from green-economy debates and mapping the values likely to shape the future of peer review.

URL : Un/Sustainable Peer Review and Generative AI: Ethical Gaps, Editorial Acceleration, and the Whitewashing of Technological Solutionism

DOI : https://doi.org/10.17742/IMAGE29731

Catégories
EN

Determining quality dimensions for peer review reports using a Delphi approach

Authors : Amanda Sizo, Adriano Lino, Álvaro Rocha, Luis Paulo Reis

The quality of peer review reports is essential to the integrity and effectiveness of scholarly communication. Yet review reports are often criticized for being vague, biased, or unconstructive, which limits their usefulness for both authors and editors. Existing frameworks for assessing review quality remain fragmented and are rarely validated through expert consensus.

This study aims to define and validate a comprehensive set of quality dimensions for peer review reports, encompassing comments addressed to both authors and editors. We employed a two-phase design combining a thematic analysis of the literature with a Delphi study involving 43 scientific editors, primarily from journals in Computer Science and Engineering.

Consensus was reached after two Delphi rounds, resulting in 62 validated statements organized into eight quality dimensions: Helpfulness, Specificity, Fairness, Thoroughness, Courteousness, Readability, Consistency, and Relevance. These findings provide an empirically grounded framework to inform the development of clearer standards for peer review practice.

URL : Determining quality dimensions for peer review reports using a Delphi approach

DOI : https://doi.org/10.1007/s11192-026-05603-3

Catégories
EN

Gaming the peer review system: Evidence for a review mill in medicine highlights the need to ensure reviewer integrity

Authors : M. Ángeles Oviedo-García, René Aquarius, Dorothy V. M. Bishop

Background

Review mills are recognized when individuals generate numerous generic review reports, typically containing suggestions for citations to their own work. Here, we report a network with characteristics of a review mill in the field of gynecological oncology.

Methods

Our search started with a review that contained “boilerplate” comments as well as suggestions that specific PubMed IDs be cited. We searched the internet using Google for review reports using the same boilerplate comments. We coded text to quantify similarities between reviews and compiled citations suggested by reviewers. For comparison, we analyzed 59 reviews of the same articles by other peer reviewers.

Results

We identified a network of 195 review reports that shared boilerplate text from 170 articles. One hundred and eighty-six reports suggested citing articles coauthored by a member of the network. Five members of the network had editorial roles. Authors of 142 articles complied with suggestions for citation. Boilerplate text and citation recommendations were rare in the comparison reports.

Conclusions

Review mills lead to articles being published without proper peer review. This is of particular concern in medical research. Open peer review and transparent reporting of the editors responsible for handling papers will make it easier to detect review mills.

URL : Gaming the peer review system: Evidence for a review mill in medicine highlights the need to ensure reviewer integrity

DOI : https://doi.org/10.1080/08989621.2026.2640012

Catégories
EN

Back to the roots: reimagining scientific evaluation of research without peer review

Author : Malik Sallam

The peer review system, once a noble aspiration, now lags behind the accelerating demands of modern science.

This opinion piece calls for a decisive departure from that peer review system and advocates for a return to a more accountable, editorially driven model of scholarly evaluation. Scientific editors – already vested with decision‑making authority – should no longer outsource their judgement to external referees. Instead, they must reclaim their rightful role as the primary arbiters of scientific merit.

Too often, editorial judgement is diluted by ritualized consultation, where peer review delays innovation, rewards consensus and obscures responsibility.

I argue for a future in which academic editors decide independently, sign their decisions and are recognized – publicly and professionally – for the intellectual stewardship they provide. By linking editorial work to scientific databases such as Web of Science, Scopus and Google Scholar, we can incentivize rigour, transparency and accountability.

This model would not erode scientific integrity but elevate it, replacing bureaucracy with responsibility. It is time to shed the cloak of anonymity and return authority – and credit – to those best positioned to shape the scientific record; the academic editors themselves.

URL : Back to the roots: reimagining scientific evaluation of research without peer review

DOI : https://doi.org/10.1629/uksg.714