Authors : Holly Cordner, Gabrielle Y Garza, Isabelle Freiling
In recent years, policy reforms and funding mandates have accelerated the adoption of open science practices. These practices are increasingly being promoted and mandated to enhance transparency and replicability.
As such, they may serve as signals of the quality of a paper. However, transparency and replicability are only parts of what makes a study rigorous, and far less attention has been paid to how this emphasis on open science practices relates to the criteria researchers use to evaluate scientific rigor.
Drawing on in-depth interviews with tenure-line faculty across disciplines, we examine how researchers understand scientific rigor and how they interpret the growing use of open science practices as signals of quality. Faculty consistently valued open science for improving transparency and accessibility but emphasized a broader set of criteria they consider equally important for rigorous research, including theoretical grounding, appropriate methods, accurate claims, and ethical considerations.
Many noted that these dimensions of rigor are not always fully captured by open science frameworks or by journal badges that indicate adherence to open science practices. Faculty also expressed mixed views about the extent to which open science practices function as meaningful indicators of research quality across disciplines and methodologies.
Our findings suggest that while open science can strengthen scientific rigor, evaluation systems that treat transparency and replicability as prominent evaluative signals may affect perceptions of what constitutes rigorous research.
These findings have implications for the design and implementation of research evaluation frameworks that reflect the multidimensional and context-dependent nature of scientific rigor.
URL : Scientific rigor beyond open science: Faculty perspectives on research quality and evaluation