Étiquette : research quality

Catégories
EN

Scientific rigor beyond open science: Faculty perspectives on research quality and evaluation

Authors :  Holly Cordner, Gabrielle Y Garza, Isabelle Freiling

In recent years, policy reforms and funding mandates have accelerated the adoption of open science practices. These practices are increasingly being promoted and mandated to enhance transparency and replicability.

As such, they may serve as signals of the quality of a paper. However, transparency and replicability are only parts of what makes a study rigorous, and far less attention has been paid to how this emphasis on open science practices relates to the criteria researchers use to evaluate scientific rigor.

Drawing on in-depth interviews with tenure-line faculty across disciplines, we examine how researchers understand scientific rigor and how they interpret the growing use of open science practices as signals of quality. Faculty consistently valued open science for improving transparency and accessibility but emphasized a broader set of criteria they consider equally important for rigorous research, including theoretical grounding, appropriate methods, accurate claims, and ethical considerations.

Many noted that these dimensions of rigor are not always fully captured by open science frameworks or by journal badges that indicate adherence to open science practices. Faculty also expressed mixed views about the extent to which open science practices function as meaningful indicators of research quality across disciplines and methodologies.

Our findings suggest that while open science can strengthen scientific rigor, evaluation systems that treat transparency and replicability as prominent evaluative signals may affect perceptions of what constitutes rigorous research.

These findings have implications for the design and implementation of research evaluation frameworks that reflect the multidimensional and context-dependent nature of scientific rigor.

URL : Scientific rigor beyond open science: Faculty perspectives on research quality and evaluation

DOI : https://doi.org/10.1093/reseval/rvag061

Catégories
EN

A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

Authors : Digant Gupta, Amandeep Kaur, Mansi Malik

Introduction

The primary objective of this study was to investigate the association between journal Impact Factor (IF) and study quality in real-world observational studies. The secondary objective was to explore whether the association changes as a function of different study factors (study design, funding type and geographic location).

Methods

Study quality was assessed using the Newcastle-Ottawa Scale (NOS). IFs were obtained from journal websites. The association between journal IF and NOS score was evaluated firstly using Spearman’s correlation coefficient, and secondly using one-way Analysis of Variance (ANOVA).

Results

We selected 457 studies published in 208 journals across 11 consecutive systematic literature reviews (SLRs) conducted at our organization over the last 5 years. Most studies were cross-sectional and from North America or Europe. Mean (SD) NOS score was 6.6 (1.03) and mean (SD) IF was 5.2 (4.5). Overall, there was a weak positive correlation between NOS score and IF (Spearman’s coefficient (ρ) = 0.23 [95% CI: 0.13–0.31]; p < 0.001). There was no correlation between NOS score and IF for prospective cohort studies (ρ = 0.07 [95% CI:−0.12–0.25]) and industry-funded studies (ρ = 0.06 [95% CI:−0.09–0.21]). Based on ANOVA, the effect size, eta squared (η2), was 0.04 (95% CI: 0.01–0.08), indicating a small effect.

Discussion

While there is some correlation between journal quality and study quality, our findings indicate that high-quality research can be found in journals with lower IF, and assessing study quality requires careful review of study design, methodology, analysis, interpretation, and significance of the findings. Notably, in industry-funded studies, no correlation was found between methodological quality and IF.

URL : A pilot study investigating the relationship between journal impact factor and methodological quality of real-world observational studies

DOI : https://doi.org/10.3389/frma.2025.1679842

Catégories
EN

Peer review across borders: benefits and challenges of international review panels in research funding organizations

Authors : Helen Peterson, Liisa Husu

Peer review by external experts is widely recognized as a legitimate and trustworthy academic practice, essential for ensuring the quality and rigor of research, providing more objective and less impartial assessments, and promoting transparent decision-making in science and academia. Research Funding Organisations (RFOs) usually rely on some form of peer review to evaluate the scientific quality of research proposals to allocate their limited resources.

The peer review system is, however, also associated with several weaknesses, such as risks for bias and conflict of interest. This article explores the implications of replacing National Review Panels (NRPs) with International Review Panels (IRPs) in a national RFO, examining how this shift may impact the peer review process.

Drawing on semi-structured interviews with staff from a national RFO in a European country and members of its IRPs, the article provides a nuanced analysis of both the potential benefits and challenges with substituting NRPs with IRPs.

The results highlight how IRPs increase the distance between applicants and reviewers, which benefits the impartiality of the process. Nevertheless, this distance needs to be balanced by domestic panel members, chairs or research officers possessing appropriate knowledge of the local academic context, culture and structure.

IRPs also introduce a greater diversity of perspectives into the assessments of applicants, which may promote objective and balanced assessments. The diversity may however also lower inter-reviewer reliability, and, in turn, complicate calibration practices and hinder the development of informal deliberative norms during the process of reaching decisions and consensus.

URL : Peer review across borders: benefits and challenges of international review panels in research funding organizations

DOI : https://doi.org/10.1093/reseval/rvaf030

Catégories
EN

Academic Quality or Commercial Concern? The Role of APCs in Open-Access Communication Studies Journals

Author : Burak Ili

Despite the positive effects of the open access (OA) movement on academic publishing, commercial publishers’ profit-driven policies continue to prevail, making the publishing process increasingly difficult for many researchers, particularly those from developing countries. T

his study critically examines open-access Q1 and Q2 journals listed in the Scimago Journal & Country Rank (SJR) within the field of Media and Communication Studies.

Despite the OA movement’s goal of increasing access to information, the capitalist academic publishing model transforms knowledge production into a commercial activity through article processing charges (APCs). The research reveals that high APCs demanded by high-impact journals represent a significant barrier, especially for researchers with limited financial and institutional support.

This situation underscores the urgent need for institutional reform in the structure of academic publishing, particularly within the field of Media and Communication Studies.

The proposed reforms should focus on critical areas such as increased support for OA models, freeing journals and editorial boards from Western monopolies, fairly compensating the labour of reviewers and editors, and offering greater language support.

Steps taken in this direction will contribute to the creation of a more transparent, fair, and inclusive structure for academic production and sharing processes.

URL : Academic Quality or Commercial Concern? The Role of APCs in Open-Access Communication Studies Journals

DOI : https://doi.org/10.31269/triplec.v23i1.1547

Catégories
EN

Enhancing Research Methodology and Academic Publishing: A Structured Framework for Quality and Integrity

Authors : Md. Jalil Piran, Nguyen H. Tran

Following a brief introduction to research, research processes, research types, papers, reviews, and evaluations, this paper presents a structured framework for addressing inconsistencies in research methodology, technical writing, quality assessment, and publication standards across academic disciplines. Using a four-dimensional evaluation model that focuses on 1) technical content, 2) structural coherence, 3) writing precision, and 4) ethical integrity, this framework not only standardizes review and publication processes but also serves as a practical guide for authors in preparing high-quality manuscripts. Each of these four dimensions cannot be compromised for the sake of another.

Following that, we discuss the components of a research paper adhering to the four-dimensional evaluation model in detail by providing guidelines and principles. By aligning manuscripts with journal standards, reducing review bias, and enhancing transparency, the framework contributes to more reliable and reproducible research results. Moreover, by strengthening cross-disciplinary credibility, improving publication consistency, and fostering public trust in academic literature, this initiative is expected to positively influence both research quality and scholarly publishing’s reputation.

Arxiv : https://arxiv.org/abs/2412.05683

Catégories
EN

Distinguishing articles in questionable and non-questionable journals using quantitative indicators associated with quality

Author : Dimity Stephen

This study investigates the viability of distinguishing articles in questionable journals (QJs) from those in non-QJs on the basis of quantitative indicators typically associated with quality. Subsequently, I examine what can be deduced about the quality of articles in QJs based on the differences observed.

I contrast the length of abstracts and full-texts, prevalence of spelling errors, text readability, number of references and citations, the size and internationality of the author team, the documentation of ethics and informed consent statements, and the presence erroneous decisions based on statistical errors in 1,714 articles from 31 QJs, 1,691 articles from 16 journals indexed in Web of Science (WoS), and 1,900 articles from 45 mid-tier journals, all in the field of psychology.

The results suggest that QJ articles do diverge from the disciplinary standards set by peer-reviewed journals in psychology on quantitative indicators of quality that tend to reflect the effect of peer review and editorial processes. However, mid-tier and WoS journals are also affected by potential quality concerns, such as under-reporting of ethics and informed consent processes and the presence of errors in interpreting statistics. Further research is required to develop a comprehensive understanding of the quality of articles in QJs.

Arxiv : https://arxiv.org/abs/2405.06308

Catégories
EN

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345