Étiquette : Peer Review

The limitations to our understanding of peer review

Authors : Jonathan P. Tennant, Tony Ross-Hellauer

Peer review is embedded in the core of our scholarly knowledge generation systems, conferring legitimacy on research while distributing academic capital and prestige on individuals.

Despite its critical importance, it curiously remains poorly understood in a number of dimensions. In order to address this, we have programmatically analysed peer review to assess where the major gaps in our theoretical and empirical understanding of it lie.

We distill this into core themes around editorial accountability, the subjectivity and bias of reviewers, the function and quality of peer review, the role of the reviewer, the social and epistemic implications of peer review, and regarding innovations in open peer review platforms and services.

We use this to present a guide for the future of peer review, and the development of a new research discipline based on the study of peer review. Such a field requires sustained funding and commitment from publishers and research funders, who both have a commitment to uphold the integrity of the published scholarly record.

This will require the design of a consensus for a minimal set of standards for what constitutes peer review, and the development of a shared data infrastructure to support this.

We recognise that many of the criticisms attributed to peer review might reflect wider issues within academia and wider society, and future care will be required in order to carefully demarcate and address these.

DOI : https://doi.org/10.31235/osf.io/jq623

Open Up: A Survey on Open and Non-anonymized Peer Reviewing

Authors : Lonni Besançon, Niklas Rönnberg, Jonas Löwgren, Jonathan P. Tennant, Matthew Cooper

We present a discussion and analysis regarding the benefits and limitations of open and non-anonymized peer review based on literature results and responses to a survey on the reviewing process of alt.chi, a more or less open-review track within the CHI conference, the predominant conference in the field of human-computer interaction (HCI).

This track currently is the only implementation of an open-peer-review process in the field of HCI while, with the recent increase in interest in open science practices, open review is now being considered and used in other fields.

We collected 30 responses from alt.chi authors and reviewers and found that, while the benefits are quite clear and the system is generally well liked by alt.chi participants, they are reluctant to see it used in other venues.

This concurs with a number of recent studies that suggest a divergence between support for a more open review process and its practical implementation. The data and scripts are available on https://osf.io/vuw7h/, and the figures and follow-up work on http://tiny.cc/OpenReviews.

URL : Open Up: A Survey on Open and Non-anonymized Peer Reviewing

Alternative location : https://www.preprints.org/manuscript/201905.0098/v2

 

Acceptance rates of scholarly peer-reviewed journals: A literature survey

Author : Bo-Christer Bjórk

The acceptance rate of scholarly journals is an important selection criterion for authors choosing where to submit their manuscripts. Unfortunately, information about the acceptance (or rejection rates) of individual journals is seldom available.

This article surveys available systematic information and studies of acceptance rates. The overall global average is around 35-40%. There are significant differences between fields of science, with biomedicine having higher acceptance rates compared to for instance the social sciences.

Open access journals usually have higher acceptance rates than subscription journals, and this is particularly true for so-called OA mega-journals, which have peer review criteria focusing on sound science only.

URL : https://recyt.fecyt.es/index.php/EPI/article/view/epi.2019.jul.07

The F3-index. Valuing reviewers for scholarly journals

Authors : Federico Bianchi, Francisco Grimaldo, Flaminio Squazzoni

This paper presents an index that measures reviewer contribution to editorial processes of scholarly journals. Following a metaphor of ranking algorithms in sports tournaments, we created an index that considers reviewers on different context-specific dimensions, i.e., report delivery time, the length of the report and the alignment of recommendations to editorial decisions.

To test the index, we used a dataset of peer review in a multi-disciplinary journal, including 544 reviewers on 606 submissions in six years. Although limited by sample size, the test showed that the index identifies outstanding contributors and weak performing reviewers efficiently.

Our index is flexible, contemplates extensions and could be incorporated into available scholarly journal management tools. It can assist editors in rewarding high performing reviewers and managing editorial turnover.

URL : The F3-index. Valuing reviewers for scholarly journals

DOI : https://doi.org/10.1016/j.joi.2018.11.007

What does better peer review look like? Underlying principles and recommendations for better practice

Authors : Heidi Allen, Alexandra Cury, Thomas Gaston, Chris Graf, Hannah Wakley, Michael Willis

We conducted a literature review of best practice in peer review. Following this research, we identified five principles for better peer review: Content Integrity, Content Ethics, Fairness, Usefulness, and Timeliness. For each of these principles, we have developed a set of recommendations to improve peer review standards.

In this article, we describe the role of peer review and how our five principles support that goal. This article is intended to continue the conversation about improving peer review standards and provide guidance to journal teams looking to improve their standards. It is accompanied by a detailed checklist, which could be used by journal teams to assess their current peer review standards.

URL : What does better peer review look like? Underlying principles and recommendations for better practice

DOI : https://doi.org/10.1002/leap.1222

Using AI to solve business problems in scholarly publishing

Author: Michael Upshall

Artificial intelligence (AI) tools are widely used today in many areas, and are now being introduced into scholarly publishing. This article provides a brief overview of present-day AI and machine learning as used for text-based resources such as journal articles and book chapters, and provides an example of its application to identify suitable peer reviewers for manuscript submissions.

It describes how one company, UNSILO, has created a tool for this purpose, and the underlying technology used to deliver it. The article also offers a glimpse into a future where AI will profoundly change the way that academic publishing will work.

URL : Using AI to solve business problems in scholarly publishing

DOI : http://doi.org/10.1629/uksg.460

Who Is (Likely) Peer-Reviewing Your Papers? A Partial Insight into the World’s Top Reviewers

Authors : Francesco Pomponi, Bernardino D’Amico, Tom Rye

Scientific publishing is experiencing unprecedented growth in terms of outputs across all fields. Inevitably this creates pressure throughout the system on a number of entities.

One key element is represented by peer-reviewers, whose demand increases at an even higher pace than that of publications, since more than one reviewer per paper is needed and not all papers that get reviewed get published.

The relatively recent Publons platform allows for unprecedented insight into the usual ‘blindness’ of the peer-review system. At a time where the world’s top peer-reviewers are announced and celebrated, we have taken a step back in order to attempt a partial mapping of their profiles to identify trends and key dimensions of this community of ‘super-reviewers’.

This commentary focuses necessarily on a limited sample due to manual processing of data, which needs to be done within a single day for the type of information we seek. In investigating the numbers of performed reviews vs. academic citations, our analysis suggests that most reviews are carried out by relatively inexperienced academics.

For some of these early career academics, peer-reviewing seems to be the only activity they engage with, given the high number of reviews performed (e.g., three manuscripts per day) and the lack of outputs (zero academic papers and citations in some cases). Additionally, the world’s top researchers (i.e., highly-cited researchers) are understandably busy with research activities and therefore far less active in peer-reviewing.

Lastly, there seems to be an uneven distribution at a national level between scientific outputs (e.g., publications) and reviews performed. Our analysis contributes to the ongoing global discourse on the health of scientific peer-review, and it raises some important questions for further discussion.

URL : Who Is (Likely) Peer-Reviewing Your Papers? A Partial Insight into the World’s Top Reviewers

URL : https://www.mdpi.com/2304-6775/7/1/15