Authors : Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy
Peer review is essential for advancing scientific research, but there are long-standing concerns that authors’ prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed.
Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018.
First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly.
Second, the inter-reviewer disagreement increased significantly in the double-blind format.
Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers.
Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers’ acceptance was affected.
These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.
URL : Does double-blind peer review reduce bias? Evidence from a top computer science conference