Étiquette : Peer Review

Catégories
EN

PRINCIPIA: a Decentralized Peer-Review Ecosystem

Authors : Andrea Mambrini, Andrea Baronchelli, Michele Starnini, Daniele Marinazzo, Manlio De Domenico

Peer review is a cornerstone of modern scientific endeavor. However, there is growing consensus that several limitations of the current peer review system, from lack of incentives to reviewers to lack of transparency, risks to undermine its benefits.

Here, we introduce the PRINCIPIA (http://www.principia.network/) framework for peer-review of scientific outputs (e.g., papers, grant proposals or patents).

The framework allows key players of the scientific ecosystem — including existing publishing groups — to create and manage peer-reviewed journals, by building a free market for reviews and publications. PRINCIPIA’s referees are transparently rewarded according to their efforts and the quality of their reviews.

PRINCIPIA also naturally allows to recognize the prestige of users and journals, with an intrinsic reputation system that does not depend on third-parties. PRINCIPIA re-balances the power between researchers and publishers, stimulates valuable assessments from referees, favors a fair competition between journals, and reduces the costs to access research output and to publish.

URL : https://arxiv.org/abs/2008.09011

Catégories
EN

Quantifying professionalism in peer review

Authors : Travis G. Gerwing, Alyssa M. Allen Gerwing, Stephanie Avery-Gomm, Chi-Yeung Choi, Jeff C. Clements, Joshua A. Rash

Background

The process of peer-review in academia has attracted criticism surrounding issues of bias, fairness, and professionalism; however, frequency of occurrence of such comments is unknown.

Methods

We evaluated 1491 sets of reviewer comments from the fields of “Ecology and Evolution” and “Behavioural Medicine,” of which 920 were retrieved from the online review repository Publons and 571 were obtained from six early career investigators.

Comment sets were coded for the occurrence of “unprofessional comments” and “incomplete, inaccurate or unsubstantiated critiques” using an a-prior rubric based on our published research. Results are presented as absolute numbers and percentages.

Results

Overall, 12% (179) of comment sets included at least one unprofessional comment towards the author or their work, and 41% (611) contained incomplete, inaccurate of unsubstantiated critiques (IIUC).

Conclusions

The large number of unprofessional comments, and IIUCs observed could heighten psychological distress among investigators, particularly those at an early stage in their career. We suggest that development and adherence to a universally agreed upon reviewer code of conduct is necessary to improve the quality and professional experience of peer review.

URL : Quantifying professionalism in peer review

DOI : https://doi.org/10.1186/s41073-020-00096-x

Catégories
EN

Open peer review: promoting transparency in open science

Authors : Dietmar Wolfram, Peiling Wang, Adam Hembree, Hyoungjoo Park

Open peer review (OPR), where review reports and reviewers’ identities are published alongside the articles, represents one of the last aspects of the open science movement to be widely embraced, although its adoption has been growing since the turn of the century.

This study provides the first comprehensive investigation of OPR adoption, its early adopters and the implementation approaches used. Current bibliographic databases do not systematically index OPR journals, nor do the OPR journals clearly state their policies on open identities and open reports.

Using various methods, we identified 617 OPR journals that published at least one article with open identities or open reports as of 2019 and analyzed their wide-ranging implementations to derive emerging OPR practices.

The findings suggest that: (1) there has been a steady growth in OPR adoption since 2001, when 38 journals initially adopted OPR, with more rapid growth since 2017; (2) OPR adoption is most prevalent in medical and scientific disciplines (79.9%); (3) five publishers are responsible for 81% of the identified OPR journals; (4) early adopter publishers have implemented OPR in different ways, resulting in different levels of transparency.

Across the variations in OPR implementations, two important factors define the degree of transparency: open identities and open reports. Open identities may include reviewer names and affiliation as well as credentials; open reports may include timestamped review histories consisting of referee reports and author rebuttals or a letter from the editor integrating reviewers’ comments.

When and where open reports can be accessed are also important factors indicating the OPR transparency level. Publishers of optional OPR journals should add metric data in their annual status reports.

URL : Open peer review: promoting transparency in open science

DOI : https://doi.org/10.1007/s11192-020-03488-4

Catégories
EN

Innovative Strategies for Peer Review

Author : Edward Barroga

Peer review is a crucial part of research and publishing. However, it remains imperfect and suffers from bias, lack of transparency, and professional jealousy. It is also overburdened by an increasing quantity of complex papers against the stagnant pool of reviewers, causing delays in peer review.

Additionally, many medical, nursing, and healthcare educators, peer reviewers, and authors may not be completely familiar with the current changes in peer review. Moreover, reviewer education and training have unfortunately remained lacking.

This is especially crucial since current initiatives to improve the review process are now influenced by factors other than academic needs. Thus, increasing attention has recently focused on ways of streamlining the peer review process and implementing alternative peer-review methods using new technologies and open access models.

This article aims to give an overview of the innovative strategies for peer review and to consider perspectives that may be helpful in introducing changes to peer review. Critical assessments of peer review innovations and incentives based on past and present experiences are indispensable.

A theoretical appraisal must be balanced by a realistic appraisal of the ethical roles of all stakeholders in enhancing the peer review process.

As the peer review system is far from being perfect, identifying and developing core competencies among reviewers, continuing education of researchers, reviewer education and training, and professional engagement of the scientific community in various disciplines may help bridge gaps in an imperfect but indispensable peer review system.

URL : Innovative Strategies for Peer Review

DOI : https://doi.org/10.3346/jkms.2020.35.e138

Catégories
EN

Dear Reviewer 2: Go F’ Yourself

Author : David A. M. Peterson

Objectives

The objective of this study was to empirically test the wide belief that Reviewer #2 is a uniquely poor reviewer.

Methods

The test involved analyzing the reviewer database from Political Behavior . There are two main tests. First, the reviewer’s categorical evaluation of the manuscript was compared by reviewer number. Second, the data were analyzed to test if Reviewer #2 was disproportionately likely to be more than one category below the mean of the other reviewers of the manuscript.

Results

There is no evidence that Reviewer #2 is either more negative about the manuscript or out of line with the other reviewers. There is, however, evidence that Reviewer #3 is more likely to be more than one category below the other reviewers.

Conclusions

Reviewer #2 is not the problem. Reviewer #3 is. In fact, he is such a bad actor that he even gets the unwitting Reviewer #2 blamed for his bad behavior.

DOI : https://doi.org/10.1111/ssqu.12824

Catégories
Non classé

Towards a typology of edited books and conference proceedings according to the applied peer-review procedures

Author : Iva Zlodi

In the last years here is an increasing need to ensure a more objective and transparent evaluation of scientific research in the Humanities and Social Sciences. This short paper explores some of the underlying issues and suggests a study using the suvey method based on a sample of 146 publications.

The results of this study could contribute to the identification and describing distinctive types of edited books and conference proceedings according to their peer-review procedures, and thus to facilitate the recognition of their scholarly value and reliability.

URL : https://hal.archives-ouvertes.fr/hal-02544293

Catégories
EN

Who reviews for predatory journals? A study on reviewer characteristics

Authors : Anna Severin, Michaela Strinzel, Matthias Egger, Marc Domingo, Tiago Barros

Background

While the characteristics of scholars who publish in predatory journals are relatively well-understood, nothing is known about the scholars who review for these journals.

We aimed to answer the following questions: Can we observe patterns of reviewer characteristics for scholars who review for predatory journals and for legitimate journals? Second, how are reviews for potentially predatory journals distributed globally?

Methods

We matched random samples of 1,000 predatory journals and 1,000 legitimate journals of the Cabells Scholarly Analytics’ journal lists with the Publons database of review reports, using the Jaro-Winkler string metric.

For reviewers of matched reviews, we descriptively analysed meta-data on reviewing and publishing behaviour.

Results

We matched 183,743 unique Publons reviews that were claimed by 19,598 reviewers. 6,077 reviews were conducted for 1160 unique predatory journals (3.31% of all reviews). 177,666 were claimed for 6,403 legitimate journals (96.69% of all reviews).

The vast majority of scholars either never or only occasionally submitted reviews for predatory journals to Publons (89.96% and 7.55% of all reviewers, respectively). Smaller numbers of scholars claimed reviews predominantly or exclusively for predatory journals (0.26% and 0.35% of all reviewers, respectively).

The two latter groups of scholars are of younger academic age and have fewer publications and fewer reviews than the first two groups of scholars.Developing regions feature larger shares of reviews for predatory reviews than developed regions.

Conclusion

The characteristics of scholars who review for potentially predatory journals resemble those of authors who publish their work in these outlets. In order to combat potentially predatory journals, stakeholders will need to adopt a holistic approach that takes into account the entire research workflow.

DOI : https://doi.org/10.1101/2020.03.09.983155