Étiquette : research quality

Catégories
EN

A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

Authors : Digant Gupta, Amandeep Kaur, Mansi Malik

Introduction

The primary objective of this study was to investigate the association between journal Impact Factor (IF) and study quality in real-world observational studies. The secondary objective was to explore whether the association changes as a function of different study factors (study design, funding type and geographic location).

Methods

Study quality was assessed using the Newcastle-Ottawa Scale (NOS). IFs were obtained from journal websites. The association between journal IF and NOS score was evaluated firstly using Spearman’s correlation coefficient, and secondly using one-way Analysis of Variance (ANOVA).

Results

We selected 457 studies published in 208 journals across 11 consecutive systematic literature reviews (SLRs) conducted at our organization over the last 5 years. Most studies were cross-sectional and from North America or Europe. Mean (SD) NOS score was 6.6 (1.03) and mean (SD) IF was 5.2 (4.5). Overall, there was a weak positive correlation between NOS score and IF (Spearman’s coefficient (ρ) = 0.23 [95% CI: 0.13–0.31]; p < 0.001). There was no correlation between NOS score and IF for prospective cohort studies (ρ = 0.07 [95% CI:−0.12–0.25]) and industry-funded studies (ρ = 0.06 [95% CI:−0.09–0.21]). Based on ANOVA, the effect size, eta squared (η2), was 0.04 (95% CI: 0.01–0.08), indicating a small effect.

Discussion

While there is some correlation between journal quality and study quality, our findings indicate that high-quality research can be found in journals with lower IF, and assessing study quality requires careful review of study design, methodology, analysis, interpretation, and significance of the findings. Notably, in industry-funded studies, no correlation was found between methodological quality and IF.

URL : A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

DOI : https://doi.org/10.3389/frma.2025.1679842

Catégories
EN

Peer review across borders: benefits and challenges of international review panels in research funding organizations

Authors : Helen Peterson, Liisa Husu

Peer review by external experts is widely recognized as a legitimate and trustworthy academic practice, essential for ensuring the quality and rigor of research, providing more objective and less impartial assessments, and promoting transparent decision-making in science and academia. Research Funding Organisations (RFOs) usually rely on some form of peer review to evaluate the scientific quality of research proposals to allocate their limited resources.

The peer review system is, however, also associated with several weaknesses, such as risks for bias and conflict of interest. This article explores the implications of replacing National Review Panels (NRPs) with International Review Panels (IRPs) in a national RFO, examining how this shift may impact the peer review process.

Drawing on semi-structured interviews with staff from a national RFO in a European country and members of its IRPs, the article provides a nuanced analysis of both the potential benefits and challenges with substituting NRPs with IRPs.

The results highlight how IRPs increase the distance between applicants and reviewers, which benefits the impartiality of the process. Nevertheless, this distance needs to be balanced by domestic panel members, chairs or research officers possessing appropriate knowledge of the local academic context, culture and structure.

IRPs also introduce a greater diversity of perspectives into the assessments of applicants, which may promote objective and balanced assessments. The diversity may however also lower inter-reviewer reliability, and, in turn, complicate calibration practices and hinder the development of informal deliberative norms during the process of reaching decisions and consensus.

URL : Peer review across borders: benefits and challenges of international review panels in research funding organizations

DOI : https://doi.org/10.1093/reseval/rvaf030

Catégories
EN

Academic Quality or Commercial Concern? The Role of APCs in Open-Access Communication Studies Journals

Author : Burak Ili

Despite the positive effects of the open access (OA) movement on academic publishing, commercial publishers’ profit-driven policies continue to prevail, making the publishing process increasingly difficult for many researchers, particularly those from developing countries. T

his study critically examines open-access Q1 and Q2 journals listed in the Scimago Journal & Country Rank (SJR) within the field of Media and Communication Studies.

Despite the OA movement’s goal of increasing access to information, the capitalist academic publishing model transforms knowledge production into a commercial activity through article processing charges (APCs). The research reveals that high APCs demanded by high-impact journals represent a significant barrier, especially for researchers with limited financial and institutional support.

This situation underscores the urgent need for institutional reform in the structure of academic publishing, particularly within the field of Media and Communication Studies.

The proposed reforms should focus on critical areas such as increased support for OA models, freeing journals and editorial boards from Western monopolies, fairly compensating the labour of reviewers and editors, and offering greater language support.

Steps taken in this direction will contribute to the creation of a more transparent, fair, and inclusive structure for academic production and sharing processes.

URL : Academic Quality or Commercial Concern? The Role of APCs in Open-Access Communication Studies Journals

DOI : https://doi.org/10.31269/triplec.v23i1.1547

Catégories
EN

Enhancing Research Methodology and Academic Publishing: A Structured Framework for Quality and Integrity

Authors : Md. Jalil Piran, Nguyen H. Tran

Following a brief introduction to research, research processes, research types, papers, reviews, and evaluations, this paper presents a structured framework for addressing inconsistencies in research methodology, technical writing, quality assessment, and publication standards across academic disciplines. Using a four-dimensional evaluation model that focuses on 1) technical content, 2) structural coherence, 3) writing precision, and 4) ethical integrity, this framework not only standardizes review and publication processes but also serves as a practical guide for authors in preparing high-quality manuscripts. Each of these four dimensions cannot be compromised for the sake of another.

Following that, we discuss the components of a research paper adhering to the four-dimensional evaluation model in detail by providing guidelines and principles. By aligning manuscripts with journal standards, reducing review bias, and enhancing transparency, the framework contributes to more reliable and reproducible research results. Moreover, by strengthening cross-disciplinary credibility, improving publication consistency, and fostering public trust in academic literature, this initiative is expected to positively influence both research quality and scholarly publishing’s reputation.

Arxiv : https://arxiv.org/abs/2412.05683

Catégories
EN

Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with quality

Author : Dimity Stephen

This study investigates the viability of distinguishing articles in questionable journals (QJs) from those in non-QJs on the basis of quantitative indicators typically associated with quality. Subsequently, I examine what can be deduced about the quality of articles in QJs based on the differences observed.

I contrast the length of abstracts and full-texts, prevalence of spelling errors, text readability, number of references and citations, the size and internationality of the author team, the documentation of ethics and informed consent statements, and the presence erroneous decisions based on statistical errors in 1,714 articles from 31 QJs, 1,691 articles from 16 journals indexed in Web of Science (WoS), and 1,900 articles from 45 mid-tier journals, all in the field of psychology.

The results suggest that QJ articles do diverge from the disciplinary standards set by peer-reviewed journals in psychology on quantitative indicators of quality that tend to reflect the effect of peer review and editorial processes. However, mid-tier and WoS journals are also affected by potential quality concerns, such as under-reporting of ethics and informed consent processes and the presence of errors in interpreting statistics. Further research is required to develop a comprehensive understanding of the quality of articles in QJs.

Arxiv : https://arxiv.org/abs/2405.06308

Catégories
EN

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345

Catégories
EN

On The Peer Review Reports: Does Size Matter?

Authors : Abdelghani Maddi, Luis Miotti

Amidst the ever-expanding realm of scientific production and the proliferation of predatory journals, the focus on peer review remains paramount for scientometricians and sociologists of science. Despite this attention, there is a notable scarcity of empirical investigations into the tangible impact of peer review on publication quality.

This study aims to address this gap by conducting a comprehensive analysis of how peer review contributes to the quality of scholarly publications, as measured by the citations they receive. Utilizing an adjusted dataset comprising 57,482 publications from Publons to Web of Science and employing the Raking Ratio method, our study reveals intriguing insights. Specifically, our findings shed light on a nuanced relationship between the length of reviewer reports and the subsequent citations received by publications.

Through a robust regression analysis, we establish that, beginning from 947 words, the length of reviewer reports is significantly associated with an increase in citations. These results not only confirm the initial hypothesis that longer reports indicate requested improvements, thereby enhancing the quality and visibility of articles, but also underscore the importance of timely and comprehensive reviewer reports.

Furthermore, insights from Publons’ data suggest that open access to reports can influence reviewer behavior, encouraging more detailed reports. Beyond the scholarly landscape, our findings prompt a reevaluation of the role of reviewers, emphasizing the need to recognize and value this resource-intensive yet underappreciated activity in institutional evaluations.

Additionally, the study sounds a cautionary note regarding the challenges faced by peer review in the context of an increasing volume of submissions, potentially compromising the vigilance of peers in swiftly assessing numerous articles.

HAL : https://cnrs.hal.science/hal-04492274