Archives des mots-clés : Peer Review

The Uptake and Impact of a Label for Peer-Reviewed Books

Posté le par

Authors : Eline Vandewalle, Raf Guns, Tim C. E. Engels

This article presents an analysis of the uptake of the GPRC label (Guaranteed Peer Reviewed Content label) since its introduction in 2010 until 2019. GPRC is a label for books that have been peer reviewed introduced by the Flemish publishers association.

The GPRC label allows locally published scholarly books to be included in the regional database for the Social Sciences and Humanities which is used in the Flemish performance-based research funding system. Ten years after the start of the GPRC label, this is the first systematic analysis of the uptake of the label.

We use a mix of qualitative and quantitative methods. Our two main data sources are the Flemish regional database for the Social Sciences and Humanities, which currently includes 2,580 GPRC-labeled publications, and three interviews with experts on the GPRC label. Firstly, we study the importance of the label in the Flemish performance-based research funding system.

Secondly, we analyse the label in terms of its possible effect on multilingualism and the local or international orientation of publications. Thirdly, we analyse to what extent the label has been used by the different disciplines.

Lastly, we discuss the potential implications of the label for the peer review process among book publishers. We find that the GPRC label is of limited importance to the Flemish performance-based research funding system.

However, we also conclude that the label has a specific use for locally oriented book publications and in particular for the discipline Law. Furthermore, by requiring publishers to adhere to a formalized peer review procedure, the label affects the peer review practices of local publishers because not all book publishers were using a formal system of peer review before the introduction of the label and even at those publishers who already practiced peer review, the label may have required the publishers to make these procedures more uniform.

URL : The Uptake and Impact of a Label for Peer-Reviewed Books

DOI : https://doi.org/10.3389/frma.2021.746452

Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

Posté le par

Authors : Leonardo Ferreira Fontenelle, Thiago Dias Sarti

Scholarly journals should consider the attitudes of their communities before adopting any of the seven traits of open peer review. Unfortunately, surveys from the Global North might not apply to the Global South, where double-blind peer review is commonplace even among natural sciences and medicine journals.

This paper reports the findings of a survey on attitudes toward open peer review among four stakeholder groups of a scholar-led medical journal in Brazil: society members, journal readers, authors, and reviewers.

Compared to a previous survey, which mostly recruited natural sciences researchers from Europe, this survey found similar support for open peer review in general and for most of its traits.

One important exception was open identities, which were considered detrimental by most participants, even more in this survey than in the previous one. Interestingly, participants were more dismissive of open identities as a whole than of statements about its specific consequences.

Because preprints are increasingly popular but incompatible with double-blind review, future research should examine the effects of transitioning from double-blind to open identities, especially on gender bias.

Meanwhile, scholarly journals with double-blind review might prefer to begin by adopting other traits of open review or to make open identities optional at first.

URL : Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

DOI : https://doi.org/10.1590/2318-0889202133e200072

Peer Review in Law Journals

Posté le par

Authors : Jadranka Stojanovski, Elías Sanz-Casado, Tommaso Agnoloni, Ginevra Peruginelli

The field of law has retained its distinctiveness regarding peer review to this day, and reviews are often conducted without following standardized rules and principles. External and independent evaluation of submissions has recently become adopted by European law journals, and peer review procedures are still poorly defined, investigated, and attuned to the legal science publishing landscape.

The aim of our study was to gain a better insight into current editorial policies on peer review in law journals by exploring editorial documents (instructions, guidelines, policies) issued by 119 Croatian, Italian, and Spanish law journals.

We relied on automatic content analysis of 135 publicly available documents collected from the journal websites to analyze the basic features of the peer review processes, manuscript evaluation criteria, and related ethical issues using WordStat.

Differences in covered topics between the countries were compared using the chi-square test. Our findings reveal that most law journals have adopted a traditional approach, in which the editorial board manages mostly anonymized peer review (104, 77%) engaging independent/external reviewers (65, 48%).

Submissions are evaluated according to their originality and relevance (113, 84%), quality of writing and presentation (94, 70%), comprehensiveness of literature references (93, 69%), and adequacy of methods (57, 42%).

The main ethical issues related to peer review addressed by these journals are reviewer’s competing interests (42, 31%), plagiarism (35, 26%), and biases (30, 22%). We observed statistically significant differences between countries in mentioning key concepts such as “Peer review ethics”, “Reviewer”, “Transparency of identities”, “Publication type”, and “Research misconduct”.

Spanish journals favor reviewers’ “Independence” and “Competence” and “Anonymized” peer review process. Also, some manuscript types popular in one country are rarely mentioned in other countries.

Even though peer review is equally conventional in all three countries, high transparency in Croatian law journals, respect for research integrity in Spanish ones, and diversity and inclusion in Italian are promising indicators of future development.

URL : Peer Review in Law Journals

DOI : https://doi.org/10.3389/frma.2021.787768

Editors between Support and Control by the Digital Infrastructure — Tracing the Peer Review Process with Data from an Editorial Management System

Posté le par

Authors : Judith Hartstein, Clemens Blüme

Many journals now rely on editorial management systems, which are supposed to support the administration and decision making of editors, while aiming at making the process of communication faster and more transparent to both reviewers and authors. Yet, little is known about how these infrastructures support, stabilize, transform or change existing editorial practices.

Research suggests that editorial management systems as digital infrastructures are adapted to the local needs at scholarly journals and reflect main realms of activities. Recently, it has been established that in a minimal case, the peer review process is comprised of postulation, consultation, decision and administration.

By exploring process generated data from a publisher’s editorial management system, we investigate the ways by which the digital infrastructure is used and how it represents the different realms of the process of peer review. How does the infrastructure support, strengthen or restrain editorial agency for administrating the process?

In our study, we investigate editorial processes and practices with their data traces captured by an editorial management system. We do so by making use of the internal representation of manuscript life cycles from submission to decision for 14,000 manuscripts submitted to a biomedical publisher.

Reconstructing the processes applying social network analysis, we found that the individual steps in the process have no strict order, other than could be expected with regard to the software patent. However, patterns can be observed, as to which stages manuscripts are most likely to go through in an ordered fashion.

We also found the different realms of the peer review process represented in the system, some events, however, indicate that the infrastructure offers more control and observation of the peer review process, thereby strengthening the editorial role in the governance of peer review while at the same time the infrastructure oversees the editors’ performance.

URL : Editors between Support and Control by the Digital Infrastructure — Tracing the Peer Review Process with Data from an Editorial Management System

DOI : https://doi.org/10.3389/frma.2021.747562

Ensuring Quality and Status: Peer Review Practices in Kriterium, A Portal for Quality-Marked Monographs and Edited Volumes in Swedish SSH

Posté le par

Authors : Björn Hammarfelt, Isak Hammar, Helena Francke

Although established forms of peer review are often criticized for being slow, secretive, and even unfair, they are repeatedly mentioned by academics as the most important indicator of quality in scholarly publishing.

In many countries, the peer review of books is a less codified practice than that of journal articles or conference papers, and the processes and actors involved are far from uniform. In Sweden, the review process of books has seldom been formalized.

However, more formal peer review of books has been identified as a response to the increasing importance placed on streamlined peer-reviewed publishing of journal articles in English, which has been described as a direct challenge to more pluralistic publication patterns found particularly in the humanities.

In this study, we focus on a novel approach to book review, Kriterium, where an independent portal maintained by academic institutions oversees the reviewing of academic books. The portal administers peer reviews, providing a mark of quality through a process which involves reviewers, an academic coordinator, and an editorial board.

The paper studies how this process functions in practice by exploring materials concerning 24 scholarly books reviewed within Kriterium. Our analysis specifically targets tensions identified in the process of reviewing books with a focus on three main themes, namely the intended audience, the edited volume, and the novel role of the academic coordinator.

Moreover, we find that the two main aims of the portal–quality enhancement (making research better) and certification (displaying that research is of high quality)–are recurrent in deliberations made in the peer review process.

Consequently, we argue that reviewing procedures and criteria of quality are negotiated within a broader discussion where more traditional forms of publishing are challenged by new standards and evaluation practices.

URL : Ensuring Quality and Status: Peer Review Practices in Kriterium, A Portal for Quality-Marked Monographs and Edited Volumes in Swedish SSH

DOI : https://doi.org/10.3389/frma.2021.740297

Does double-blind peer review reduce bias? Evidence from a top computer science conference

Posté le par

Authors : Mengyi Sun, Jainabou Barry Danfa, Misha Teplitskiy

Peer review is essential for advancing scientific research, but there are long-standing concerns that authors’ prestige or other characteristics can bias reviewers. Double-blind peer review has been proposed as a way to reduce reviewer bias, but the evidence for its effectiveness is limited and mixed.

Here, we examine the effects of double-blind peer review by analyzing the review files of 5,027 papers submitted to a top computer science conference that changed its reviewing format from single- to double-blind in 2018.

First, we find that the scores given to the most prestigious authors significantly decreased after switching to double-blind review. However, because many of these papers were above the threshold for acceptance, the change did not affect paper acceptance significantly.

Second, the inter-reviewer disagreement increased significantly in the double-blind format.

Third, papers rejected in the single-blind format are cited more than those rejected under double-blind, suggesting that double-blind review better excludes poorer quality papers.

Lastly, an apparently unrelated change in the rating scale from 10 to 4 points likely reduced prestige bias significantly such that papers’ acceptance was affected.

These results support the effectiveness of double-blind review in reducing biases, while opening new research directions on the impact of peer-review formats.

URL : Does double-blind peer review reduce bias? Evidence from a top computer science conference

DOI : https://doi.org/10.1002/asi.24582

Science rules! A qualitative study of scientists’ approaches to grant lottery

Posté le par

Author : Axel Philipps

Using peer review to assess the validity of research proposals has always had its fair share of critics, including a more-than-fair-share of scholars. The debate about this method of assessing these proposals now seems trivial when compared with assessing the validity for granting funding by lottery.

Some of the same scholars have suggested that the way grant lottery was being assessed has made random allocation seem even-handed, less biased and more supportive of innovative research.

But we know little of what researchers actually think about grant lottery and even less about the thoughts of those scientists who rely on funding. This paper examines scientists’ perspectives on selecting grants by ‘lots’ and how they justify their support or opposition.

How do they approach something scientifically that is, in itself, not scientific? These approaches were investigated with problem-centered interviews conducted with natural scientists in Germany.

The qualitative interviews for this paper reveal that scientists in dominated and dominating field positions are, more or less, open to the idea of giving a selection process by lots a try. Nonetheless, they are against pure randomization because from their point of view it is incompatible with scientific principles.

They rather favor a combination of grant lottery and peer review processes, assuming that only under these conditions could randomly allocated funding be an integral and legitimate part of science.

URL : Science rules! A qualitative study of scientists’ approaches to grant lottery

DOI : https://doi.org/10.1093/reseval/rvaa027