Reading Peer Review : PLOS ONE and Institutional Change in Academia

Authors : Martin Paul Eve, Cameron Neylon, Daniel Paul O’Donnell, Samuel Moore, Robert Gadie, Victoria Odeniyi, Shahina Parvin

This Element describes for the first time the database of peer review reports at PLOS ONE, the largest scientific journal in the world, to which the authors had unique access.

Specifically, this Element presents the background contexts and histories of peer review, the data-handling sensitivities of this type of research, the typical properties of reports in the journal to which the authors had access, a taxonomy of the reports, and their sentiment arcs.

This unique work thereby yields a compelling and unprecedented set of insights into the evolving state of peer review in the twenty-first century, at a crucial political moment for the transformation of science.

It also, though, presents a study in radicalism and the ways in which PLOS’s vision for science can be said to have effected change in the ultra-conservative contemporary university.

URL : Reading Peer Review : PLOS ONE and Institutional Change in Academia

Original location : https://www.cambridge.org/core/elements/reading-peer-review/42F027E4C67D246DD8C3AC440A68C7A7

Attitudes, behaviours and experiences of authors of COVID-19 preprints

Authors : Narmin Rzayeva, Susana Oliveira Henriques, Stephen Pinfield, Ludo Waltman

The COVID-19 pandemic caused a rise in preprinting, apparently triggered by the need for open and rapid dissemination of research outputs. We surveyed authors of COVID-19 preprints to learn about their experience of preprinting as well as publishing in a peer-reviewed journal.

A key aim was to consider preprints in terms of their effectiveness for authors to receive feedback on their work. We also aimed to compare the impact of feedback on preprints with the impact of comments of editors and reviewers on papers submitted to journals. We observed a high rate of new adopters of preprinting who reported positive intentions regarding preprinting their future work.

This allows us to posit that the boost in preprinting may have a structural effect that will last after the pandemic. We also saw a high rate of feedback on preprints but mainly through “closed” channels – directly to the authors.

This means that preprinting was a useful way to receive feedback on research, but the value of feedback could be increased further by facilitating and promoting “open” channels for preprint feedback. At the same time, almost a quarter of the preprints that received feedback received comments resembling journal peer review.

This shows the potential of preprint feedback to provide valuable detailed comments on research. However, journal peer review resulted in a higher rate of major changes in the papers surveyed, suggesting that the journal peer review process has significant added value compared to preprint feedback.

URL : Attitudes, behaviours and experiences of authors of COVID-19 preprints

DOI : https://doi.org/10.31235/osf.io/d96yj

Accelerated Peer Review and Paper Processing Models in Academic Publishing

Authors : Jaime A. Teixeira da Silva, Yuki Yamada

Some journals and publishers offer a free or paid rapid peer review service. In the latter case, such a service is offered at a premium, i.e., for an additional fee, and authors receive, in return, a privileged service, namely faster peer review.

In the cut-throat world of survival in academia, the difference of a few weeks or months in terms of speed of peer review and publication may bring untold benefits to authors that manage to benefit from accelerated peer review.

We examine the deontological aspects behind this two-tier peer review system, including some positive, but mainly negative, aspects. Some paid accelerated peer review services thrive.

We examine the paid accelerated peer review services by Taylor & Francis, Future Medicine Ltd., Elsevier, and two stand-alone journals that are OASPA members. This suggests that there is a demand, and thus market, for faster peer review.

However, this privilege risks creating a two-tiered system that may divide academics between those who can pay versus those who cannot.

We recommend that those papers that have benefited from accelerated peer review clearly indicate this in the published papers, as either a disclaimer or within the acknowledgements, for maximum transparency of the peer review and publication process.

DOI : https://doi.org/10.1007/s12109-022-09891-4

Transparency versus anonymity: which is better to eliminate bias in peer review?

Authors: Faye Holst, Kim Eggleton, Simon Harris

Peer review is a critical component of the scientific process. When conducted properly by dedicated and competent reviewers, it helps to safeguard the quality, validity, authority and rigour of academic work. However, bias in peer review is well documented and can skew objectivity of the review and hinder fair assessment of research.

To mitigate against bias and enhance accountability, IOP Publishing has introduced two different, but complementary, approaches to all their peer-reviewed, open access (OA) journals: double-anonymous peer review and transparent peer review.

Double-anonymous peer review, where the reviewer and author identities are concealed, is designed to tackle inequality in the scholarly publishing process as it reduces bias with respect to gender, race, country of origin or affiliation.

Transparent peer review shows readers the full peer review history, including reviewer reports, editor decision letters and the authors’ responses alongside the published article. Making this process visible to the community increases accountability, allows reviewers to be recognized more for their work and can aid the training of aspiring reviewers.

IOP Publishing is the first physics publisher to adopt both of these approaches portfolio wide. In this article we discuss how applying these methods has altered different elements of the publishing process. Early indicators show that there may be a marked difference in acceptance rates across regions.

URL : Transparency versus anonymity: which is better to eliminate bias in peer review?

DOI : http://doi.org/10.1629/uksg.584

The Role of Publons in the Context of Open Peer Review

Authors : Jaime A. Teixeira da Silva, Serhii Nazarovets

Publons was a peer reviewer rewards platform that aimed to recognize the contribution that academics made during peer review to a journal. For about 10 years of its existence, Publons became the most popular service among peer reviewers.

Having gained traction and popularity, Publons was purchased in 2017 by Clarivate Analytics (now Clarivate), and many academics, journals and publishers invested time and effort to participate in Publons. Using Publons, various peer review-related experiments or pilot programs were initiated by some academic publishers regarding the introduction of open peer review into their journals’ editorial processes.

In this paper, we examine pertinent literature related to Publons, and reflect on its benefits and flaws during its short-lived history. In mid-August 2022, Clarivate fused Publons into the Web of Science platform.

Publons, as a brand peer review service, has now ceased to exist but some of the functionality remains in Web of Science while other aspects that used to be open and free at Publons are now paid-for services.

We reflect on the effect of such experiments, which initially had bold and ambitious academic objectives to fortify peer review, on academics’ trust, especially when such projects become commercialized.

DOI : https://doi.org/10.1007/s12109-022-09914-0

Gender diversity of research consortia contributes to funding decisions in a multi-stage grant peer-review process

Authors : Stefano Bianchini, Patrick Llerena, Sıla Öcalan-Öze, Emre Özel

This study seeks to draw connections between the grant proposal peer-review and the gender representation in research consortia.

We examined the implementation of a multi-disciplinary, pan-European funding scheme—EUROpean COllaborative RESearch Scheme (2003–2015)—and the reviewers’ materials that this generated. EUROCORES promoted investigator-driven, multinational collaborative research in multiple scientific areas and brought together 9158 Principal Investigators (PI) who teamed up in 1347 international consortia that were sequentially evaluated by 467 expert panel members and 1862 external reviewers.

We found systematically unfavourable evaluations for consortia with a higher proportion of female PIs. This gender effect was evident in the evaluation outcomes of both panel members and reviewers: applications from consortia with a higher share of female scientists were less successful in panel selection and received lower scores from external reviewers.

Interestingly, we found a systematic discrepancy between the evaluative language of written review reports and the scores assigned by reviewers that works against consortia with a higher share of female participants.

Reviewers did not perceive female scientists as being less competent in their comments, but they were negatively sensitive to a high female ratio within a consortium when scoring the proposed research project.

URL : Gender diversity of research consortia contributes to funding decisions in a multi-stage grant peer-review process

DOI : https://doi.org/10.1057/s41599-022-01204-6

Characteristics of ‘mega’ peer-reviewers

Authors : Danielle B. Rice, Ba’ Pham, Justin Presseau, Andrea C. Tricco, David Moher

Background

The demand for peer reviewers is often perceived as disproportionate to the supply and availability of reviewers. Considering characteristics associated with peer review behaviour can allow for the development of solutions to manage the growing demand for peer reviewers.

The objective of this research was to compare characteristics among two groups of reviewers registered in Publons.

Methods

A descriptive cross-sectional study design was used to compare characteristics between (1) individuals completing at least 100 peer reviews (‘mega peer reviewers’) from January 2018 to December 2018 as and (2) a control group of peer reviewers completing between 1 and 18 peer reviews over the same time period.

Data was provided by Publons, which offers a repository of peer reviewer activities in addition to tracking peer reviewer publications and research metrics. Mann Whitney tests and chi-square tests were conducted comparing characteristics (e.g., number of publications, number of citations, word count of peer review) of mega peer reviewers to the control group of reviewers.

Results

A total of 1596 peer reviewers had data provided by Publons. A total of 396 M peer reviewers and a random sample of 1200 control group reviewers were included. A greater proportion of mega peer reviews were male (92%) as compared to the control reviewers (70% male).

Mega peer reviewers demonstrated a significantly greater average number of total publications, citations, receipt of Publons awards, and a higher average h index as compared to the control group of reviewers (all p < .001). We found no statistically significant differences in the number of words between the groups (p > .428).

Conclusions

Mega peer reviewers registered in the Publons database also had a higher number of publications and citations as compared to a control group of reviewers. Additional research that considers motivations associated with peer review behaviour should be conducted to help inform peer reviewing activity.

URL : Characteristics of ‘mega’ peer-reviewers

DOI : https://doi.org/10.1186/s41073-022-00121-1