Does double-blind peer review reduce bias? Evidence from a top computer science conference

Authors : Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy

Peer review is essential for advancing scientific research, but there are long-standing concerns that authors’ prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed.

Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018.

First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly.

Second, the inter-reviewer disagreement increased significantly in the double-blind format.

Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers.

Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers’ acceptance was affected.

These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.

URL : Does double-blind peer review reduce bias? Evidence from a top computer science conference

DOI : https://doi.org/10.1002/asi.24582