Archives des mots-clés : Peer Review

Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

Posté le par

Authors : Gaëlle Vallée-Tourangeau, Ana Wheelock, Tushna Vandrevala, Priscilla Harris

Independent evaluations of grant applications by subject experts are an important part of the peer-review system. However, little is known about the real-time experiences of peer reviewers or experts who perform reviews of a grant application independently.

This study sought to gain insight into this stage of the grant evaluation process by observing how experts conduct an independent review in near real time. Using the think aloud approach and Critical Decision Method of interviewing, in-depth interviews were conducted with 16 peer reviewers from a range of roles and disciplines within the medical humanities and social sciences.

Participants were asked to think aloud while reviewing applications to different grant schemes from a single prestigious funder. The analysis shows reviewers encountered five dilemmas during the evaluation process.

These dilemmas were related to whether or not one should (1) accept an invitation to review, (2) rely exclusively on the information presented in the application, (3) pay attention to institutional prestige, (4) offer comments about aspects that are not directly related to academics’ area of expertise, and (5) to take risks and overlook shortcomings rather than err on the side of caution.

In order to decide on the appropriate course of action, reviewers often engaged in a series of deliberations and trade-offs—varying in length and complexity.

However, their interpretation of what was ‘right’ was influenced by their values, preferences and experiences, but also by relevant norms and their understanding of the funder’s guidelines and priorities.

As a result, the way reviewers approached the identified dilemmas was idiosyncratic and sometimes diametrically opposed to other reviewers’ views, which could lead to variation in peer-review outcomes.

The dilemmas we have uncovered suggest that peer reviewers engage in thoughtful considerations during the peer-review process.

We should, therefore, be wary of reducing the absence of consensus as resulting from biased, instinctive thinking. Rather, these findings highlight the diversity of values, priorities and habits and ways of working each reviewer brings to the fore when reviewing the applicants and their project proposals and call for further reflection on, and study of, this “invisible work” to better understand and continue to improve the peer-reviewing process.

URL : Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

DOI : https://doi.org/10.1057/s41599-022-01050-6

Reshaping How Universities Can Evaluate the Research Impact of Open Humanities for Societal Benefit

Posté le par

Authors : Paul Longley Arthur, Lydia Hearn

During the twenty-first century, for the first time, the volume of digital data has surpassed the amount of analog data. As academic practices increasingly become digital, opportunities arise to reshape the future of scholarly communication through more accessible, interactive, open, and transparent methods that engage a far broader and more diverse public.

Yet despite these advances, the research performance of universities and public research institutes remains largely evaluated through publication and citation analysis rather than by public engagement and societal impact.

This article reviews how changes to bibliometric evaluations toward greater use of altmetrics, including social media mentions, could enhance uptake of open scholarship in the humanities.

In addition, the article highlights current challenges faced by the open scholarship movement, given the complexity of the humanities in terms of its sources and outputs that include monographs, book chapters, and journals in languages other than English; the use of popular media not considered as scholarly papers; the lack of time and energy to develop digital skills among research staff; problems of authority and trust regarding the scholarly or non-academic nature of social media platforms; the prestige of large academic publishing houses; and limited awareness of and familiarity with advanced digital applications.

While peer review will continue to be a primary method for evaluating research in the humanities, a combination of altmetrics and other assessment of research impact through different data sources may provide a way forward to ensure the increased use, sustainability, and effectiveness of open scholarship in the humanities.

DOI : https://doi.org/10.3998/jep.788

Innovations in peer review in scholarly publishing: a meta-summary

Posté le par

Authors : Helen Buckley Woods, Johanna Brumberg, Wolfgang Kaltenbrunner, Stephen Pinfield, Ludo Waltman

Background

There are currently numerous innovations in peer review and quality assurance in scholarly publishing. The Research on Research Institute conducted a programme of co-produced projects investigating these innovations.

This literature review was part of one such project ‘Experiments in peer review’ (Kaltenbrunner et al, 2022), which created an inventory and framework of peer review innovations. The aim of this literature review was to aid the development of the inventory by identifying innovations in peer review reported in the scholarly literature and by providing a summary of the different approaches.

Methods

This meta-summary is based on data identified from Web of Science and Scopus limited from 2010 to 2021. A total of 247 papers were screened, with 6 review articles chosen for the focus of the literature review. Items were selected that described approaches to innovating peer review or illustrated examples.

Results

The summary of innovations are drawn from 6 review articles: Bruce et al (2016); Tennant et al (2017); Burley (2017); Horbach and Halffman (2018); Tennant (2018); Barroga (2020). The innovations are divided into three high-level categories: approaches to peer review, reviewer focussed initiatives and technology to support peer review with sub-categories of results presented in tabular form and summarised. A summary of all innovations found is also presented.

Conclusions

From a simple synthesis of the review authors’ conclusions, three key messages are presented: observations on current practice; authors’ views on the implications of innovations in peer review; and calls for action in peer review research and practice.

URL : https://osf.io/preprints/socarxiv/qaksd/

Examining linguistic shifts between preprints and publications

Posté le par

Authors : David N. Nicholson, Vincent Rubinetti, Dongbo Hu, Marvin Thielk, Lawrence E. Hunter, Casey S. Greene

Preprints allow researchers to make their findings available to the scientific community before they have undergone peer review. Studies on preprints within bioRxiv have been largely focused on article metadata and how often these preprints are downloaded, cited, published, and discussed online.

A missing element that has yet to be examined is the language contained within the bioRxiv preprint repository. We sought to compare and contrast linguistic features within bioRxiv preprints to published biomedical text as a whole as this is an excellent opportunity to examine how peer review changes these documents.

The most prevalent features that changed appear to be associated with typesetting and mentions of supporting information sections or additional files. In addition to text comparison, we created document embeddings derived from a preprint-trained word2vec model.

We found that these embeddings are able to parse out different scientific approaches and concepts, link unannotated preprint–peer-reviewed article pairs, and identify journals that publish linguistically similar papers to a given preprint.

We also used these embeddings to examine factors associated with the time elapsed between the posting of a first preprint and the appearance of a peer-reviewed publication. We found that preprints with more versions posted and more textual changes took longer to publish.

Lastly, we constructed a web application (https://greenelab.github.io/preprint-similarity-search/) that allows users to identify which journals and articles that are most linguistically similar to a bioRxiv or medRxiv preprint as well as observe where the preprint would be positioned within a published article landscape.

URL : Examining linguistic shifts between preprints and publications

DOI : https://doi.org/10.1371/journal.pbio.3001470

The Uptake and Impact of a Label for Peer-Reviewed Books

Posté le par

Authors : Eline Vandewalle, Raf Guns, Tim C. E. Engels

This article presents an analysis of the uptake of the GPRC label (Guaranteed Peer Reviewed Content label) since its introduction in 2010 until 2019. GPRC is a label for books that have been peer reviewed introduced by the Flemish publishers association.

The GPRC label allows locally published scholarly books to be included in the regional database for the Social Sciences and Humanities which is used in the Flemish performance-based research funding system. Ten years after the start of the GPRC label, this is the first systematic analysis of the uptake of the label.

We use a mix of qualitative and quantitative methods. Our two main data sources are the Flemish regional database for the Social Sciences and Humanities, which currently includes 2,580 GPRC-labeled publications, and three interviews with experts on the GPRC label. Firstly, we study the importance of the label in the Flemish performance-based research funding system.

Secondly, we analyse the label in terms of its possible effect on multilingualism and the local or international orientation of publications. Thirdly, we analyse to what extent the label has been used by the different disciplines.

Lastly, we discuss the potential implications of the label for the peer review process among book publishers. We find that the GPRC label is of limited importance to the Flemish performance-based research funding system.

However, we also conclude that the label has a specific use for locally oriented book publications and in particular for the discipline Law. Furthermore, by requiring publishers to adhere to a formalized peer review procedure, the label affects the peer review practices of local publishers because not all book publishers were using a formal system of peer review before the introduction of the label and even at those publishers who already practiced peer review, the label may have required the publishers to make these procedures more uniform.

URL : The Uptake and Impact of a Label for Peer-Reviewed Books

DOI : https://doi.org/10.3389/frma.2021.746452

Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

Posté le par

Authors : Leonardo Ferreira Fontenelle, Thiago Dias Sarti

Scholarly journals should consider the attitudes of their communities before adopting any of the seven traits of open peer review. Unfortunately, surveys from the Global North might not apply to the Global South, where double-blind peer review is commonplace even among natural sciences and medicine journals.

This paper reports the findings of a survey on attitudes toward open peer review among four stakeholder groups of a scholar-led medical journal in Brazil: society members, journal readers, authors, and reviewers.

Compared to a previous survey, which mostly recruited natural sciences researchers from Europe, this survey found similar support for open peer review in general and for most of its traits.

One important exception was open identities, which were considered detrimental by most participants, even more in this survey than in the previous one. Interestingly, participants were more dismissive of open identities as a whole than of statements about its specific consequences.

Because preprints are increasingly popular but incompatible with double-blind review, future research should examine the effects of transitioning from double-blind to open identities, especially on gender bias.

Meanwhile, scholarly journals with double-blind review might prefer to begin by adopting other traits of open review or to make open identities optional at first.

URL : Attitudes toward open peer review among stakeholders of a scholar-led journal in Brazil

DOI : https://doi.org/10.1590/2318-0889202133e200072

Peer Review in Law Journals

Posté le par

Authors : Jadranka Stojanovski, Elías Sanz-Casado, Tommaso Agnoloni, Ginevra Peruginelli

The field of law has retained its distinctiveness regarding peer review to this day, and reviews are often conducted without following standardized rules and principles. External and independent evaluation of submissions has recently become adopted by European law journals, and peer review procedures are still poorly defined, investigated, and attuned to the legal science publishing landscape.

The aim of our study was to gain a better insight into current editorial policies on peer review in law journals by exploring editorial documents (instructions, guidelines, policies) issued by 119 Croatian, Italian, and Spanish law journals.

We relied on automatic content analysis of 135 publicly available documents collected from the journal websites to analyze the basic features of the peer review processes, manuscript evaluation criteria, and related ethical issues using WordStat.

Differences in covered topics between the countries were compared using the chi-square test. Our findings reveal that most law journals have adopted a traditional approach, in which the editorial board manages mostly anonymized peer review (104, 77%) engaging independent/external reviewers (65, 48%).

Submissions are evaluated according to their originality and relevance (113, 84%), quality of writing and presentation (94, 70%), comprehensiveness of literature references (93, 69%), and adequacy of methods (57, 42%).

The main ethical issues related to peer review addressed by these journals are reviewer’s competing interests (42, 31%), plagiarism (35, 26%), and biases (30, 22%). We observed statistically significant differences between countries in mentioning key concepts such as “Peer review ethics”, “Reviewer”, “Transparency of identities”, “Publication type”, and “Research misconduct”.

Spanish journals favor reviewers’ “Independence” and “Competence” and “Anonymized” peer review process. Also, some manuscript types popular in one country are rarely mentioned in other countries.

Even though peer review is equally conventional in all three countries, high transparency in Croatian law journals, respect for research integrity in Spanish ones, and diversity and inclusion in Italian are promising indicators of future development.

URL : Peer Review in Law Journals

DOI : https://doi.org/10.3389/frma.2021.787768