Étiquette : Peer Review

Catégories
EN

Does double-blind peer review reduce bias? Evidence from a top computer science conference

Authors : Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy

Peer review is essential for advancing scientific research, but there are long-standing concerns that authors’ prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed.

Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018.

First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly.

Second, the inter-reviewer disagreement increased significantly in the double-blind format.

Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers.

Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers’ acceptance was affected.

These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.

URL : Does double-blind peer review reduce bias? Evidence from a top computer science conference

DOI : https://doi.org/10.1002/asi.24582

Catégories
EN

Science rules! A qualitative study of scientists’ approaches to grant lottery

Author : Axel Philipps

Using peer review to assess the validity of research proposals has always had its fair share of critics, including a more-than-fair-share of scholars. The debate about this method of assessing these proposals now seems trivial when compared with assessing the validity for granting funding by lottery.

Some of the same scholars have suggested that the way grant lottery was being assessed has made random allocation seem even-handed, less biased and more supportive of innovative research.

But we know little of what researchers actually think about grant lottery and even less about the thoughts of those scientists who rely on funding. This paper examines scientists’ perspectives on selecting grants by ‘lots’ and how they justify their support or opposition.

How do they approach something scientifically that is, in itself, not scientific? These approaches were investigated with problem-centered interviews conducted with natural scientists in Germany.

The qualitative interviews for this paper reveal that scientists in dominated and dominating field positions are, more or less, open to the idea of giving a selection process by lots a try. Nonetheless, they are against pure randomization because from their point of view it is incompatible with scientific principles.

They rather favor a combination of grant lottery and peer review processes, assuming that only under these conditions could randomly allocated funding be an integral and legitimate part of science.

URL : Science rules! A qualitative study of scientists’ approaches to grant lottery

DOI : https://doi.org/10.1093/reseval/rvaa027

Catégories
EN

The role of metrics in peer assessments

Authors :  Liv Langfeldt, Ingvild Reymert, Dag W Aksnes

Metrics on scientific publications and their citations are easily accessible and are often referred to in assessments of research and researchers. This paper addresses whether metrics are considered a legitimate and integral part of such assessments. Based on an extensive questionnaire survey in three countries, the opinions of researchers are analysed.

We provide comparisons across academic fields (cardiology, economics, and physics) and contexts for assessing research (identifying the best research in their field, assessing grant proposals and assessing candidates for positions).

A minority of the researchers responding to the survey reported that metrics were reasons for considering something to be the best research. Still, a large majority in all the studied fields indicated that metrics were important or partly important in their review of grant proposals and assessments of candidates for academic positions.

In these contexts, the citation impact of the publications and, particularly, the number of publications were emphasized. These findings hold across all fields analysed, still the economists relied more on productivity measures than the cardiologists and the physicists. Moreover, reviewers with high scores on bibliometric indicators seemed more frequently (than other reviewers) to adhere to metrics in their assessments.

Hence, when planning and using peer review, one should be aware that reviewers—in particular reviewers who score high on metrics—find metrics to be a good proxy for the future success of projects and candidates, and rely on metrics in their evaluation procedures despite the concerns in scientific communities on the use and misuse of publication metrics.

URL : The role of metrics in peer assessments

DOI : https://doi.org/10.1093/reseval/rvaa032

Catégories
EN

Overburdening of peer reviewers: A multi-stakeholder perspective on causes and effects

Authors : Anna Severin, Joanna Chataway

Peer review of manuscripts is labour-intensive and time-consuming. Individual reviewers might feel themselves overburdened with the amount of reviewing they are requested to do. Aiming to explore how stakeholder groups perceive reviewing burden and what they believe to be the causes of a potential overburdening of reviewers, we conducted focus groups with early-, mid-, and senior career scholars, editors, and publishers.

By means of a thematic analysis, we aimed to identify the causes of overburdening of reviewers. First, we show that, across disciplines and roles, stakeholders believed that the reviewing burden is distributed unequally across members of the academic community, resulting in the overburdening of small groups of reviewers. Second, stakeholders believed this to be caused by (i) an increase in manuscript submissions; (ii) inefficient manuscript handling; (iii) lack of institutionalization of peer review; (iv) lack of reviewing instructions and (v) inadequate reviewer recruiting strategies.

These themes were assumed to relate to an inadequate incentive structure in academia that favours publications over peer review. In order to alleviate reviewing burden, a holistic approach is required that addresses both the increased demand for and the insufficient supply of reviewing resources.

URL : Overburdening of peer reviewers: A multi-stakeholder perspective on causes and effects

DOI : https://doi.org/10.1002/leap.1392

Catégories
EN

Problematizing Peer Review: Academic Librarians’ Pedagogical Approaches to Peer Review

Authors : Lana Mariko Wood, Gr Keer

INTRODUCTION

This study is the first to consider how academic librarians’ understanding of and participation in the peer review process influences their information literacy pedagogy and practice.

METHODS

This mixed-methods study uses a modified sequential explanatory design, beginning with a survey of academic librarians in the United States and Canada, followed by interviews with interested study participants.

RESULTS & DISCUSSION

The researchers find that academic librarians frequently teach about peer review, but approaches vary widely, and though some have adapted the Framework to fit their instruction about peer review, there are no best practices.

Instructor demands, the length of instructional sessions, and student level influence whether and how academic librarians contextualize the peer review process. While some academic librarians draw from their personal experience in the peer review process as authors, reviewers, and/or editors in their teaching, academic librarians do not consistently report their personal experience as an influence on their teaching of the peer review process to students.

CONCLUSION

This article argues that academic librarians should consider the place of peer review in information literacy instruction, including interrogating how scaffolding instruction about peer review may provide a disservice to students from an equity perspective.

The authors urge academic librarians who have it to draw on personal experience to contextualize their instruction about peer review.

URL : Problematizing Peer Review: Academic Librarians’ Pedagogical Approaches to Peer Review

DOI : https://doi.org/10.7710/2162-3309.2399

Catégories
FR

Modes d’évaluation ouverte par les pairs : de la revue à la plateforme

Auteurs/Authors : Evelyne Broudoux, Madjid Ihadjadene

Cet article a pour but de proposer un état de l’art des différentes formes de l’évaluation d’articles ou de communications par les pairs. De l’évaluation « aveugle» à l’évaluation « ouverte », de multiples possibilités existent et sont expérimentées.

C’est dans le champ des sciences que l’on trouve le plus d’innovations sociotechniques s’appuyant sur des plateformes de publication modélisant des workflows éditoriaux originaux.

L’ouverture de l’évaluation peut se produire entre pairs, en rendant publiques les identités et/ou les rapports des évaluateurs, à différents stades de l’article scientifique : préprint, en cours de rédaction, ou encore après publication.

Cet état de l’art est basé sur un ensemble de publications essentiellement produites par les acteurs de l’évaluation ouverte, issus principalement des disciplines STM.

URL : Modes d’évaluation ouverte par les pairs : de la revue à la plateforme

URL : https://revue-cossi.numerev.com/articles/revue-9/2496-modes-d-evaluation-ouverte-par-les-pairs-de-la-revue-a-la-plateforme

Catégories
EN

What Have We Learned from OpenReview?

Authors : Gang Wang, Qi Peng, Yanfeng Zhang, Mingyang Zhang

Anonymous peer review is used by the great majority of computer science conferences. OpenReview is such a platform that aims to promote openness in peer review process. The paper, (meta) reviews, rebuttals, and final decisions are all released to public. We collect 5,527 submissions and their 16,853 reviews from the OpenReview platform.

We also collect these submissions’ citation data from Google Scholar and their non-peer-reviewed versions from arXiv.org. By acquiring deep insights into these data, we have several interesting findings that could help understand the effectiveness of the public-accessible double-blind peer review process.

Our results can potentially help writing a paper, reviewing it, and deciding on its acceptance.

URL : https://arxiv.org/abs/2103.05885v4