Authors : Rüdiger Mutz, Lutz Bornmann, Hans‑Dieter Daniel
At the end of January 2023, eLife introduced a new publishing model (alongside the old-traditional-publishing model): all manuscripts submitted as preprints are peer-reviewed and published if they are deemed worthy of review by the editorial team (“editorial triage”). The model abandons the gatekeeping function and retains the previous “consultative approach to peer review”.
Even under the changed conditions, the question of the quality of judgements in the peer review process remains. In this study, the reviewers’ ratings of manuscripts submitted to eLife were examined in terms of both descriptive comparisons of peer review models, and the following selected quality criteria of peer review: interrater agreement and interrater reliability. eLife provided us with the data on all manuscripts submitted in 2023 according to the new publishing model (group 3, N = 3,846), as well as manuscripts submitted according to the old publishing model (group 1: N = 6,592 submissions from 2019; group 2: N = 364 submissions from 2023).
The interrater agreement and interrater reliability for the criteria “significance of findings” and “strength of support” were similarly low, as previous empirical studies for gatekeeping journals have shown.
The fairness of peer review is not or only slightly compromised. We used the empirical results of our study to recommend several improvements to the new publishing model introduced by eLife as for example, increasing transparency, masking author identity or increasing the number of expert reviewers.