Étiquette : reproducibility of research

Catégories
EN

Modeling the impact of research data unavailability on science

Authors : Jorge Chamorro-Padial, Francisco-Javier Rodrigo-Ginés, Rosa Rodrí­guez-Sánchez, R.M. Gil, Roberto García

Scientific progress depends on the accessibility and reproducibility of research outputs. Unfortunately, datasets and other referenced resources in academic publications frequently become unavailable over time, limiting reproducibility and reuse.

In this work, we quantitatively analyze the potential impact of research data unavailability by applying economic, probabilistic, and network based models to scientific citation networks. Rather than measuring knowledge directly, we use citation based network metrics as proxies for the dissemination and potential reuse of scientific results, and study how the absence of data-linked resources affects impact propagation and productivity-related indicators.

We further examine the resilience of citation networks under different modeling assumptions and analyze the role of highly influential nodes, or superpropagators, in amplifying the effects of dataset loss.

Our results reveal structural dependencies on vulnerable data sources and show that the magnitude of the impact depends strongly on network position and model assumptions.

These findings provide quantitative evidence of the systemic consequences of data unavailability and underline the importance of long-term data preservation and accessibility policies in scientific research.

URL : Modeling the impact of research data unavailability on science

DOI : https://doi.org/10.1016/j.joi.2026.101813

Catégories
EN

“Well, Parts of Linguistics Is Open…”: Insights into Linguists’ Diverse Understandings of Open Science

Author : Elen Le Foll

Broadly defined as the study of language, linguistics is a diverse field spanning many disciplines. Recent studies on the prevalence of Questionable Research Practices (QRPs) in linguistics (e.g. Isbell et al. 2022) suggest that it suffers from many of the same issues that triggered the replication crisis in psychology (see e.g. Sönning and Werner 2021). While surveys have indicated that linguists are generally in favour of Open Science/Scholarship (OS), there appears to be a “a misalignment between the attitude to and the adoption of OS practices” (Liu and de Cat 2024, 64).

The present study aims to gain insights into this misalignment by exploring linguists’ understanding of what constitutes OS and of the specificities of linguistic research that (can) affect its applicability to (subdisciplines of) linguistics. To this end, the study draws on the results of an anonymous, small-scale survey and the qualitative analysis of semi-structured interviews conducted with 26 linguists based in Northern Europe, representing all career stages, and a wide range of subdisciplines within linguistics.

The results reveal diverse understandings of OS among linguists. While some focus on the accessibility of research (for both academics and the wider public), others prioritise the sharing of data, materials, and code to promote transparency, reproducibility, and replicability. The latter group also emphasises the importance of OS principles and values like rigour, fairness, and collaboration. Linguists report learning about OS through conferences, workshops, library services, and social media but, most importantly, in personal interactions with other researchers, thus making much of this knowledge network-dependent.

The interviewees highlight several challenges and considerations that they believe need to be addressed when applying OS to linguistics. These include ethical and legal issues concerning data sharing, the high inter-person variability inherent to many linguistic studies, the need for (more) funding for open-access monographs, and for training in data management and statistical methods.

URL : “Well, Parts of Linguistics Is Open…”: Insights into Linguists’ Diverse Understandings of Open Science

DOI : https://doi.org/10.3998/jep.7974

Catégories
EN

A retrospective analysis of 400 publications reveals patterns of irreproducibility across an entire life sciences research field

Authors : Joseph Lemaitre, Désirée Popelka, Blandine Ribotta, Hannah Westlake, Sveta Chakrabarti, Li Xiaoxue, Mark A. Hanson, Haobo Jiang, Francesca Di Cara, Estee Kurant, Fabrice David, Bruno Lemaitre

The ReproSci project retrospectively analyzed the reproducibility of 1006 claims from 400 papers published between 1959 and 2011 in the field of Drosophila immunity. This project attempts to provide a comprehensive assessment, 14 years later, of the replicability of nearly all publications across an entire scientific community in experimental life sciences.

We found that 61% of claims were verified, while only 7% were directly challenged (not reproducible), a replicability rate higher than previous assessments. Notably, 24% of claims had never been independently tested and remain unchallenged.

We performed experimental validations of a selection of 45 unchallenged claim, that revealed that a significant fraction (38/45) of them is in fact non-reproducible. We also found that high-impact journals and top-ranked institutions are more likely to publish challenged claims.

In line with the reproducibility crisis narrative, the rates of both challenged and unchallenged claims increased over time, especially as the field gained popularity. We characterized the uneven distribution of irreproducibility among first and last authors.

Surprisingly, irreproducibility rates were similar between PhD students and postdocs, and did not decrease with experience or publication count. However, group leaders, who had prior experience as first authors in another Drosophila immunity team, had lower irreproducibility rates, underscoring the importance of early-career training.

Finally, authors with a more exploratory, short-term engagement with the field exhibited slightly higher rates of challenged claims and a markedly higher proportion of unchallenged ones. This systematic, field-wide retrospective study offers meaningful insights into the ongoing discussion on reproducibility in experimental life sciences.

DOI : https://doi.org/10.1101/2025.07.07.663460

Catégories
EN

Making Reproducibility a Reality by 2035? Enabling Publisher Collaboration for Enhanced Data Policy Enforcement

Authors : Rebecca Taylor-Grant, Matthew Cannon, Allyson Lister, Susanna-Assunta Sansone

This paper describes a project which identified practical and pragmatic ways to increase the FAIRness and reproducibility of published research. Academic journals have supported Open Science through the implementation of data sharing policies for over ten years; some evidence has since emerged on the additional time, resources and expertise that policy enforcement requires as part of an editorial workflow.

A series of publisher workshops facilitated by the EC-funded TIER2 project aimed to identify the key checks needed to enforce strengthened journal data sharing policies and to understand which editorial roles have the capacity to undertake such enforcement. The intended outcome of this work was to establish the workflows and resourcing which can support academic journals to enforce stronger data sharing policies in future.

URL : Making Reproducibility a Reality by 2035? Enabling Publisher Collaboration for Enhanced Data Policy Enforcement

DOI : https://doi.org/10.2218/ijdc.v19i1.1064

Catégories
EN

The State of Reproducibility Stamps for Visualization Research Papers

Author : Tobias Isenberg

I analyze the evolution of papers certified by the Graphics Replicability Stamp Initiative (GRSI) to be reproducible, with a specific focus on the subset of publications that address visualization-related topics. With this analysis I show that, while the number of papers is increasing overall and within the visualization field, we still have to improve quite a bit to escape the replication crisis.

I base my analysis on the data published by the GRSI as well as publication data for the different venues in visualization and lists of journal papers that have been presented at visualization-focused conferences. I also analyze the differences between the involved journals as well as the percentage of reproducible papers in the different presentation venues.

Furthermore, I look at the authors of the publications and, in particular, their affiliation countries to see where most reproducible papers come from. Finally, I discuss potential reasons for the low reproducibility numbers and suggest possible ways to overcome these obstacles.

This paper is reproducible itself, with source code and data available from this http URL as well as a free paper copy and all supplemental materials at this http URL.

Arxiv : https://arxiv.org/abs/2408.03889

Catégories
EN

Reproducible and Attributable Materials Science Curation Practices: A Case Study

Authors : Ye Li, Sarah Laura Wilson, Micah Altman

While small labs produce much of the fundamental experimental research in Material Science and Engineering (MSE), little is known about their data management and sharing practices and the extent to which they promote trust in, and transparency of, the published research.

In this research, we conduct a case study of a leading MSE research lab to characterize the limits of current data management and sharing practices concerning reproducibility and attribution. We systematically reconstruct the workflows, underpinning four research projects by combining interviews, document review, and digital forensics. We then apply information graph analysis and computer-assisted retrospective auditing to identify where critical research information is unavailable or at risk.

We find that while data management and sharing practices in this leading lab protect against computer and disk failure, they are insufficient to ensure reproducibility or correct attribution of work — especially when a group member withdraws before project completion.

We conclude with recommendations for adjustments to MSE data management and sharing practices to promote trustworthiness and transparency by adding lightweight automated file-level auditing and automated data transfer processes.

URL : Reproducible and Attributable Materials Science Curation Practices: A Case Study

DOI : https://doi.org/10.2218/ijdc.v18i1.940

Catégories
EN

Data Science at the Singularity

Author : David Donoho

Something fundamental to computation-based research has really changed in the last ten years. In certain fields, progress is simply dramatically more rapid than previously. Researchers in affected fields are living through a period of profound transformation, as the fields undergo a transition to frictionless reproducibility (FR).

This transition markedly changes the rate of spread of ideas and practices, affects scientific mindsets and the goals of science, and erases memories of much that came before. The emergence of FR flows from 3 data science principles that matured together after decades of work by many technologists and numerous research communities.

The mature principles involve data sharing, code sharing, and competitive challenges, however implemented in the particularly strong form of frictionless open services. Empirical Machine Learning is today’s leading adherent field; its hidden superpower is adherence to frictionless reproducibility practices; these practices are responsible for the striking and surprising progress in AI that we see everywhere; they can be learned and adhered to by researchers in whatever research field, automatically increasing the rate of progress in each adherent field.

URL : Data Science at the Singularity

DOI : https://doi.org/10.1162/99608f92.b91339ef