Étiquette : Peer Review

Catégories
EN

Opening Pandora’s box: Developing reviewer rhetorical sensitivity through retracted articles

Author : Baraa Khuder

Retractions issued for misconduct offer a unique window into how questionable research is rhetorically constructed and made to appear credible. This study investigates how engaging with retracted articles can serve as a pedagogical tool for reviewer training, with particular attention to the rhetorical mechanisms through which unreliability is performed.

Twenty STEM doctoral researchers analyzed self-selected retracted papers using guided critical-reading questions to identify problematic rhetorical features. Across the analyses, five recurring issues emerged: intertextual falsification, methodological opacity, rhetorical inconsistency, rhetorical overstatement, and terminological distortion.

The findings indicate that this approach has the potential to raise doctoral students’ rhetorical sensitivity by enabling them to detect subtle markers of unreliability and to adopt a more evaluative rhetorical stance toward scholarly texts.

Retracted articles thus can provide an authentic pedagogical resource for developing reviewer rhetorical sensitivity within doctoral education.

URL : Opening Pandora’s box: Developing reviewer rhetorical sensitivity through retracted articles

DOI : https://doi.org/10.1080/08989621.2025.2607681

 

Catégories
EN

A Cross-Disciplinary Analysis of AI Policies in Academic Peer Review

Authors : Zhongshi Wang, Mengyue Gong

Rapid advances of artificial intelligence (AI) have substantially impacted the field of academic publishing. This study examines AI integration in peer review by analysing policies from 439 high- and 363 middle-impact factor (IF) journals across disciplines. Using grounded theory, we identify patterns in AI policy adoption.

Results show 83% of high-IF journals have AI guidelines, with varying stringency across disciplines. Meanwhile, only 75% of middle-IF journals have AI guidelines. Science, technology, and medicine (STM) disciplines exhibit stricter regulations, while humanities and social sciences adopt more lenient approaches.

Key ethical concerns focus on confidentiality risks, accountability gaps, and AI’s inability to replicate critical human judgement. Publisher policies emphasise transparency, human oversight, and restricted AI usage for auxiliary tasks only, such as grammar checks or reviewer finding.

Disciplinary differences highlight the need for tailored guidelines that balance efficiency gains with research integrity. This study proposes collaborative frameworks for responsible AI integration. It focuses on accountability, transparency, and interdisciplinary policy development to address peer review challenges.

URL : A Cross-Disciplinary Analysis of AI Policies in Academic Peer Review

DOI : https://doi.org/10.1002/leap.2035

Catégories
EN

Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement

Authors : Maher M. Alarfaj

This paper investigates the inconsistencies present in peer review by analysing the evaluation patterns of reviewers involved in an educational award in the Arab Gulf Country States. A statistical approach was used to assess the degree of variation in scores assigned to 270 manuscripts reviewed by three different groups of reviewers.

The study revealed significant differences in the evaluations, suggesting that at least two reviewers often showed discrepancies in their assessments despite using the standardised evaluation form. The observed discrepancies appear to reflect underlying complexities related to reviewer perspectives and evaluation standards, highlighting challenges in achieving uniformity across assessments.

Additionally, it proposes a model to enhance peer review consistency, including methods for score adjustment and calibration to mitigate reviewer differences. The goal is to offer practical recommendations for improving the fairness, transparency and reliability of peer review systems, contributing to the ongoing development of academic publishing practices.

For audiences beyond the academic community including publishers, editors and academic librarians, these findings show how practical statistical tools can strengthen peer review and build greater trust in academic publishing.

URL : Enhancing Consistency in Peer Review: A Statistical Analysis of Discrepancies and Proposals for Improvement

DOI : https://doi.org/10.1002%2Fleap.2033

Catégories
EN

Reforming research funding: Combining editorial preregistration with grant peer review

Authors : Lutz Bornmann, Gerald Schweiger

Competitive grant funding is associated with high costs and a potential bias to favor conservative research. This comment proposes integrating editorial preregistration, in the form of registered reports, into grant peer review processes as a reform strategy.

Linking funding decisions to in principle accepted study protocols would reduce reviewer burden, strengthen methodological rigor, and provide an institutional foundation for (more) replication, theory driven research, and high risk research. Our proposal also minimizes strategic proposal writing and ensures scholarly output through the publication of preregistered protocols, regardless of funding outcomes.

Possible implementation models include direct coupling of journal acceptance with funding, co review mechanisms, voucher systems, and lotteries. While challenges remain in aligning journal and funding agency procedures, the integration of preregistration and funding offers a promising pathway toward a more transparent and efficient research ecosystem.

DOI : https://doi.org/10.48550/arXiv.2511.01439

Catégories
EN

Scientific publishing without gatekeeping: an empirical investigation of eLife’s new peer review process

Authors : Rüdiger Mutz, Lutz Bornmann, Hans‑Dieter Daniel

At the end of January 2023, eLife introduced a new publishing model (alongside the old-traditional-publishing model): all manuscripts submitted as preprints are peer-reviewed and published if they are deemed worthy of review by the editorial team (“editorial triage”). The model abandons the gatekeeping function and retains the previous “consultative approach to peer review”.

Even under the changed conditions, the question of the quality of judgements in the peer review process remains. In this study, the reviewers’ ratings of manuscripts submitted to eLife were examined in terms of both descriptive comparisons of peer review models, and the following selected quality criteria of peer review: interrater agreement and interrater reliability. eLife provided us with the data on all manuscripts submitted in 2023 according to the new publishing model (group 3, N = 3,846), as well as manuscripts submitted according to the old publishing model (group 1: N = 6,592 submissions from 2019; group 2: N = 364 submissions from 2023).

The interrater agreement and interrater reliability for the criteria “significance of findings” and “strength of support” were similarly low, as previous empirical studies for gatekeeping journals have shown.

The fairness of peer review is not or only slightly compromised. We used the empirical results of our study to recommend several improvements to the new publishing model introduced by eLife as for example, increasing transparency, masking author identity or increasing the number of expert reviewers.

URL : Scientific publishing without gatekeeping: an empirical investigation of eLife’s new peer review process

DOI : https://doi.org/10.1007/s11192-025-05422-y

Catégories
EN

Peer review across borders: benefits and challenges of international review panels in research funding organizations

Authors : Helen Peterson, Liisa Husu

Peer review by external experts is widely recognized as a legitimate and trustworthy academic practice, essential for ensuring the quality and rigor of research, providing more objective and less impartial assessments, and promoting transparent decision-making in science and academia. Research Funding Organisations (RFOs) usually rely on some form of peer review to evaluate the scientific quality of research proposals to allocate their limited resources.

The peer review system is, however, also associated with several weaknesses, such as risks for bias and conflict of interest. This article explores the implications of replacing National Review Panels (NRPs) with International Review Panels (IRPs) in a national RFO, examining how this shift may impact the peer review process.

Drawing on semi-structured interviews with staff from a national RFO in a European country and members of its IRPs, the article provides a nuanced analysis of both the potential benefits and challenges with substituting NRPs with IRPs.

The results highlight how IRPs increase the distance between applicants and reviewers, which benefits the impartiality of the process. Nevertheless, this distance needs to be balanced by domestic panel members, chairs or research officers possessing appropriate knowledge of the local academic context, culture and structure.

IRPs also introduce a greater diversity of perspectives into the assessments of applicants, which may promote objective and balanced assessments. The diversity may however also lower inter-reviewer reliability, and, in turn, complicate calibration practices and hinder the development of informal deliberative norms during the process of reaching decisions and consensus.

URL : Peer review across borders: benefits and challenges of international review panels in research funding organizations

DOI : https://doi.org/10.1093/reseval/rvaf030

Catégories
EN

Peer Review as Structured Commentary: Immutable Identity, Public Dialogue, and Reproducible Scholarship

Author : Craig Steven Wright

This paper reconceptualises peer review as structured public commentary. Traditional academic validation is hindered by anonymity, latency, and gatekeeping. We propose a transparent, identity-linked, and reproducible system of scholarly evaluation anchored in open commentary.

Leveraging blockchain for immutable audit trails and AI for iterative synthesis, we design a framework that incentivises intellectual contribution, captures epistemic evolution, and enables traceable reputational dynamics.

This model empowers fields from computational science to the humanities, reframing academic knowledge as a living process rather than a static credential.

DOI : https://doi.org/10.48550/arXiv.2506.22497