Étiquette : Peer Review

Publié le

Status of peer review guidelines in international surgical journals: A cross-sectional survey

Authors : Min Dong, Wenjing Wang, Xuemei Liu, Fang Lei, Yunmei Luo

Aim

To gain insight into the current status of peer review guidelines in international surgical journals and to offer guidance for the development of peer review guidelines for surgical journals.

Methods

We selected the top 100 journals in the category of ‘Surgery’ according to the Journal Citation Report 2021. We conducted a search of the websites of these journals, and Web of Science, PubMed, other databases, in order to gather the peer review guidelines published by these top 100 journals up until June 30, 2022. Additionally, we analysed the contents of these peer review guidelines.

Results

Only 52% (52/100) of journals provided guidelines for reviewers. Sixteen peer review guidelines which were published by these 52 surgical journals were included in this study. The contents of these peer review guidelines were classified into 33 items. The most common item was research methodology, which was mentioned by 13 journals (25%, 13/52). Other important items include statistical methodology, mentioned by 11 journals (21.2%), the rationality of figures, tables, and data, mentioned by 11 journals (21.2%), innovation of research, mentioned by nine journals (17.3%), and language expression, readability of papers, ethical review, references, and so forth, mentioned by eight journals (15.4%).

Two journals described items for quality assessment of peer review. Forty-three journals offered a checklist to guide reviewers on how to write a review report. Some surgical journals developed peer review guidelines for reviewers with different academic levels, such as professional reviewers and patient/public reviewers. Additionally, some surgical journals provided specific items for different types of papers, such as original articles, reviews, surgical videos, surgical database research, surgery-related outcome measurements, and case reports in their peer review guidelines.

Conclusions

Key contents of peer review guidelines for the reviewers of surgical journals not only include items relating to reviewing research methodology, statistical methods, figures, tables and data, research innovation, ethical review, but also cover items concerning reviewing surgical videos, surgical database research, surgery-related outcome measurements, instructions on how to write a review report, and guidelines on how to assess quality of peer review.

URL : Status of peer review guidelines in international surgical journals: A cross-sectional survey

DOI : https://doi.org/10.1002/leap.1624

Publié le

Evolution of Peer Review in Scientific Communication

Author : Dmitry Kochetkov

It is traditionally believed that peer review is the backbone of an academic journal and scientific communication, ensuring high quality and trust in the published materials. However, peer review only became an institutionalized practice in the second half of the 20th century, although the first scientific journals appeared three centuries earlier. By the beginning of the 21st century, there emerged an opinion that the traditional model of peer review is in deep crisis.

The aim of this article is to formulate a perspective model of peer review for scientific communication. The article discusses the evolution of the institution of scientific peer review and the formation of the current crisis. The author analyzed the modern landscape of innovations in peer review and scientific communication. Based on this analysis, three main peer review models in relation to editorial workflow were identified: pre-publication peer review (traditional model),  registered reports, and post-publication (peer) review (including preprints (peer) review).

The author argues that the third model offers the best way to implement the main functions of scientific communication.

URL : Evolution of Peer Review in Scientific Communication

DOI : https://doi.org/10.31235/osf.io/b2ra3

Publié le

On The Peer Review Reports: Does Size Matter?

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274

Publié le

Peer review’s irremediable flaws: Scientists’ perspectives on grant evaluation in Germany

Authors : Eva Barlösius, Laura Paruschke, Axel Philipps

Peer review has developed over time to become the established procedure for assessing and assuring the scientific quality of research. Nevertheless, the procedure has also been variously criticized as conservative, biased, and unfair, among other things. Do scientists regard all these flaws as equally problematic?

Do they have the same opinions on which problems are so serious that other selection procedures ought to be considered? The answers to these questions hints at what should be modified in peer review processes as a priority objective. The authors of this paper use survey data to examine how members of the scientific community weight different shortcomings of peer review processes.

Which of those processes’ problems do they consider less relevant? Which problems, on the other hand, do they judge to be beyond remedy? Our investigation shows that certain defects of peer review processes are indeed deemed irreparable: (1) legitimate quandaries in the process of fine-tuning the choice between equally eligible research proposals and in the selection of daring ideas; and (2) illegitimate problems due to networks. Science-policy measures to improve peer review processes should therefore clarify the distinction between field-specific remediable and irremediable flaws than is currently the case.

URL : Peer review’s irremediable flaws: Scientists’ perspectives on grant evaluation in Germany

DOI : https://doi.org/10.1093/reseval/rvad032

Publié le

Beyond journals and peer review: towards a more flexible ecosystem for scholarly communication

Author : Michael Wood

This article challenges the assumption that journals and peer review are essential for developing,evaluating and disseminating scientific and other academic knowledge. It suggests a more flexible ecosystem, and examines some of the possibilities this might facilitate. The market for academic outputs should be opened up by encouraging the separation of the dissemination service from the evaluation service.

Publishing research in subject-specific journals encourages compartmentalising research into rigid categories. The dissemination of knowledge would be better served by an open access, web-based repository system encompassing all disciplines. There would then be a role for organisations to assess the items in this repository to help users find relevant, high-quality work.

There could be a variety of such organisations which could enable reviews from peers to be supplemented with evaluation by non-peers from a variety of different perspectives: user reviews, statistical reviews, reviews from the perspective of different disciplines, and so on. This should reduce the inevitably conservative influence of relying on two or three peers, and make the evaluation system more critical, multi-dimensional and responsive to the requirements of different audience groups, changing circumstances, and new ideas.

Non-peer review might make it easier to challenge dominant paradigms, and expanding the potential audience beyond a narrow group of peers might encourage the criterion of simplicity to be taken more seriously – which is essential if human knowledge is to continue to progress.

Arxiv : https://arxiv.org/abs/1311.4566

Publié le

Additional experiments required: A scoping review of recent evidence on key aspects of Open Peer Review

Authors : Tony Ross-Hellauer, Serge P.J.M. Horbach

Diverse efforts are underway to reform the journal peer review system. Combined with growing interest in Open Science practices, Open Peer Review (OPR) has become of central concern to the scholarly community. However, what OPR is understood to encompass and how effective some of its elements are in meeting the expectations of diverse communities, are uncertain.

This scoping review updates previous efforts to summarize research on OPR to May 2022. Following the PRISMA methodological framework, it addresses the question: “What evidence has been reported in the scientific literature from 2017 to May 2022 regarding uptake, attitudes, and efficacy of two key aspects of OPR (Open Identities and Open Reports)?”

The review identifies, analyses and synthesizes 52 studies matching inclusion criteria, finding that OPR is growing, but still far from common practice. Our findings indicate positive attitudes towards Open Reports and more sceptical approaches to Open Identities.

Changes in reviewer behaviour seem limited and no evidence for lower acceptance rates of review invitations or slower turnaround times is reported in those studies examining those issues. Concerns about power dynamics and potential backfiring on critical reviews are in need of further experimentation.

We conclude with an overview of evidence gaps and suggestions for future research. Also, we discuss implications for policy and practice, both in the scholarly communications community and the research evaluation community more broadly.

URL : Additional experiments required: A scoping review of recent evidence on key aspects of Open Peer Review

DOI : https://doi.org/10.1093/reseval/rvae004

Publié le

Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials

Authors : Mauricia Davidson, Theodoros Evrenoglou, Carolina Graña, Anna Chaimani, Isabelle Boutron

Background

Preprints are increasingly used to disseminate research results, providing multiple sources of information for the same study. We assessed the consistency in effect estimates between preprint and subsequent journal article of COVID-19 randomized controlled trials.

Methods

The study utilized data from the COVID-NMA living systematic review of pharmacological treatments for COVID-19 (covid-nma.com) up to July 20, 2022. We identified randomized controlled trials (RCTs) evaluating pharmacological treatments vs. standard of care/placebo for patients with COVID-19 that were originally posted as preprints and subsequently published as journal articles.

Trials that did not report the same analysis in both documents were excluded. Data were extracted independently by pairs of researchers with consensus to resolve disagreements. Effect estimates extracted from the first preprint were compared to effect estimates from the journal article.

Results

The search identified 135 RCTs originally posted as a preprint and subsequently published as a journal article. We excluded 26 RCTs that did not meet the eligibility criteria, of which 13 RCTs reported an interim analysis in the preprint and a final analysis in the journal article. Overall, 109 preprint–article RCTs were included in the analysis.

The median (interquartile range) delay between preprint and journal article was 121 (73–187) days, the median sample size was 150 (71–464) participants, 76% of RCTs had been prospectively registered, 60% received industry or mixed funding, 72% were multicentric trials. The overall risk of bias was rated as ‘some concern’ for 80% of RCTs.

We found that 81 preprint–article pairs of RCTs were consistent for all outcomes reported. There were nine RCTs with at least one outcome with a discrepancy in the number of participants with outcome events or the number of participants analyzed, which yielded a minor change in the estimate of the effect. Furthermore, six RCTs had at least one outcome missing in the journal article and 14 RCTs had at least one outcome added in the journal article compared to the preprint. There was a change in the direction of effect in one RCT. No changes in statistical significance or conclusions were found.

Conclusions

Effect estimates were generally consistent between COVID-19 preprints and subsequent journal articles. The main results and interpretation did not change in any trial. Nevertheless, some outcomes were added and deleted in some journal articles.

URL : Comparison of effect estimates between preprints and peer-reviewed journal articles of COVID-19 trials

DOI : https://doi.org/10.1186/s12874-023-02136-8