How to improve scientific peer review: Four schools of thought

Authors : Ludo Waltman, Wolfgang Kaltenbrunner, Stephen Pinfield, Helen Buckley Woods

Peer review plays an essential role as one of the cornerstones of the scholarly publishing system. There are many initiatives that aim to improve the way in which peer review is organized, resulting in a highly complex landscape of innovation in peer review.

Different initiatives are based on different views on the most urgent challenges faced by the peer review system, leading to a diversity of perspectives on how the system can be improved.

To provide a more systematic understanding of the landscape of innovation in peer review, we suggest that the landscape is shaped by four schools of thought: The Quality & Reproducibility school, the Democracy & Transparency school, the Equity & Inclusion school, and the Efficiency & Incentives school.

Each school has a different view on the key problems of the peer review system and the innovations necessary to address these problems. The schools partly complement each other, but we argue that there are also important tensions between the schools.

We hope that the four schools of thought offer a useful framework to facilitate conversations about the future development of the peer review system.

URL : How to improve scientific peer review: Four schools of thought

DOI : https://doi.org/10.31235/osf.io/v8ghj

How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation

Authors : Cinzia Daraio, Alessio Vaccari

In this paper we answer the question of how evaluation should be by proposing a good evaluation of research practices. A good evaluation of research practices, intended as social practices à la MacIntyre, should take into account the stable motivations and the traits of the characters (i.e. the virtues) of researchers.

We also show that a good evaluation is also just, beyond the sense of fairness, as working on good research practices implies keep into account a broader sense of justice. After that, we propose the development of a knowledge base for the assessment of “good” evaluations of research practices to implement a questionnaire for the assessment of researchers’ virtues.

Although the latter is a challenging task, the use of ontologies and taxonomic knowledge, and the reasoning algorithms that can draw inferences on the basis of such knowledge represents a way for testing the consistency of the information reported in the questionnaire and to analyse correctly and coherently how the data is gathered through it.

Finally, we describe the potential application usefulness of our proposal for the reform of current research assessment systems.

URL : How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation

DOI : https://doi.org/10.1007/s11192-022-04329-2

Are female scientists less inclined to publish alone? The gender solo research gap

Authors : Marek Kwiek, Wojciech Roszka

In solo research, scientists compete individually for prestige, sending clear signals about their research ability, avoiding problems in credit allocation, and reducing conflicts about authorship. We examine to what extent male and female scientists differ in their use of solo publishing across various dimensions.

This research is the first to comprehensively study the “gender solo research gap” among all internationally visible scientists within a whole national higher education system. We examine the gap through mean “individual solo publishing rates” found in “individual publication portfolios” constructed for each Polish university professor.

We use the practical significance/statistical significance difference (based on the effect-size r coefficient) and our analyses indicate that while some gender differences are statistically significant, they have no practical significance.

Using a partial effects of fractional logistic regression approach, we estimate the probability of conducting solo research. In none of the models does gender explain the variability of the individual solo publishing rate.

The strongest predictor of individual solo publishing rate is the average team size, publishing in STEM fields negatively affects the rate, publishing in male-dominated disciplines positively affects it, and the influence of international collaboration is negative.

The gender solo research gap in Poland is much weaker than expected: within a more general trend toward team research and international research, gender differences in solo research are much weaker and less relevant than initially assumed.

We use our unique biographical, administrative, publication, and citation database (“Polish Science Observatory”) with metadata on all Polish scientists present in Scopus (N = 25,463) and their 158,743 Scopus-indexed articles published in 2009–2018, including 18,900 solo articles.

URL : Are female scientists less inclined to publish alone? The gender solo research gap

DOI : https://doi.org/10.1007/s11192-022-04308-7

Free access to scientific literature and its influence on the publishing activity in developing countries: The effect of Sci-Hub in the field of mathematics

Authors : Kilian Buehling, Matthias Geissler, Dorothea Strecker

This paper investigates whether free access to scientific literature increases the participation of under-represented groups in scientific discourse. To this end, we aggregate and match data tracing access to Sci-Hub, a widely used black open access (OA) repository or shadow library, and publication data from the Web of Science (WoS).

We treat the emergence of Sci-Hub as an exogenous event granting relatively unrestricted access to publications, which are otherwise hidden behind a paywall. We analyze changes in the publication count of researchers from developing countries in a given journal as a proxy for general participation in scientific discourse.

Our results indicate that in the exemplary field of mathematics, free access to academic knowledge is likely to improve the representation of authors from developing countries in international journals.

Assuming the desirability of greater international diversity in science (e.g., to generate more original work, reproduce empirical findings in different settings, or shift the research focus toward topics that are overlooked by researchers from more developed countries), our findings lend evidence to the claim of the OA movement that scientific knowledge should be free and widely distributed.

URL : Free access to scientific literature and its influence on the publishing activity in developing countries: The effect of Sci-Hub in the field of mathematics

DOI : https://doi.org/10.1002/asi.24636

Research data management and services: Resources for different data practitioners

Author : Gilbert Mushi

The emergence of data-driven research and demands for the establishment of Research Data Management (RDM) has created interest in academic institutions and research organizations globally.

Some of the libraries especially in developed countries have started offering RDM services to their communities. Although lagging behind, some academic libraries in developing countries are at the stage of planning or implementing the service. However, the level of RDM awareness is very low among researchers, librarians and other data practitioners.

The objective of this paper is to present available open resources for different data practitioners particularly researchers and librarians.

It includes training resources for both researchers and librarians, Data Management Plan (DMP) tool for researchers; data repositories available for researchers to freely archive and share their research data to the local and international communities.

A case study with a survey was conducted at the University of Dodoma to identify relevant RDM services so that librarians could assist researchers to make their data accessible to the local and international community.

The study findings revealed a low level of RDM awareness among researchers and librarians. Over 50% of the respondent indicated their perceived knowledge as poor in the following RDM knowledge areas; DMP, data repository, long term digital preservation, funders RDM mandates, metadata standards describing data and general awareness of RDM.

Therefore, this paper presents available open resources for different data practitioners to improve RDM knowledge and boost the confidence of academic and research libraries in establishing the service.

URL : Research data management and services: Resources for different data practitioners

DOI : https://doi.org/10.29173/iq995

Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

Authors : Gaëlle Vallée-Tourangeau, Ana Wheelock, Tushna Vandrevala, Priscilla Harris

Independent evaluations of grant applications by subject experts are an important part of the peer-review system. However, little is known about the real-time experiences of peer reviewers or experts who perform reviews of a grant application independently.

This study sought to gain insight into this stage of the grant evaluation process by observing how experts conduct an independent review in near real time. Using the think aloud approach and Critical Decision Method of interviewing, in-depth interviews were conducted with 16 peer reviewers from a range of roles and disciplines within the medical humanities and social sciences.

Participants were asked to think aloud while reviewing applications to different grant schemes from a single prestigious funder. The analysis shows reviewers encountered five dilemmas during the evaluation process.

These dilemmas were related to whether or not one should (1) accept an invitation to review, (2) rely exclusively on the information presented in the application, (3) pay attention to institutional prestige, (4) offer comments about aspects that are not directly related to academics’ area of expertise, and (5) to take risks and overlook shortcomings rather than err on the side of caution.

In order to decide on the appropriate course of action, reviewers often engaged in a series of deliberations and trade-offs—varying in length and complexity.

However, their interpretation of what was ‘right’ was influenced by their values, preferences and experiences, but also by relevant norms and their understanding of the funder’s guidelines and priorities.

As a result, the way reviewers approached the identified dilemmas was idiosyncratic and sometimes diametrically opposed to other reviewers’ views, which could lead to variation in peer-review outcomes.

The dilemmas we have uncovered suggest that peer reviewers engage in thoughtful considerations during the peer-review process.

We should, therefore, be wary of reducing the absence of consensus as resulting from biased, instinctive thinking. Rather, these findings highlight the diversity of values, priorities and habits and ways of working each reviewer brings to the fore when reviewing the applicants and their project proposals and call for further reflection on, and study of, this “invisible work” to better understand and continue to improve the peer-reviewing process.

URL : Peer reviewers’ dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

DOI : https://doi.org/10.1057/s41599-022-01050-6

Scientific research in news media: a case study of misrepresentation, sensationalism and harmful recommendations

Authors : Georgia Dempster, Georgina Sutherland, Louise Keogh

Accurate news media reporting of scientific research is important as most people receive their health information from the media and inaccuracies in media reporting can have adverse health outcomes.

We completed a quantitative and qualitative analysis of a journal article, the corresponding press release and the online news reporting of a scientific study.

Four themes were identified in the press release that were directly translated to the news reports that contributed to inaccuracies: sensationalism, misrepresentation, clinical recommendations and subjectivity.

The pressures on journalists, scientists and their institutions has led to a mutually beneficial relationship between these actors that can prioritise newsworthiness ahead of scientific integrity to the detriment of public health.

URL : Scientific research in news media: a case study of misrepresentation, sensationalism and harmful recommendations

DOI : https://doi.org/10.22323/2.21010206