Étiquette : replication

Catégories
EN

“Well, Parts of Linguistics Is Open…”: Insights into Linguists’ Diverse Understandings of Open Science

Author : Elen Le Foll

Broadly defined as the study of language, linguistics is a diverse field spanning many disciplines. Recent studies on the prevalence of Questionable Research Practices (QRPs) in linguistics (e.g. Isbell et al. 2022) suggest that it suffers from many of the same issues that triggered the replication crisis in psychology (see e.g. Sönning and Werner 2021). While surveys have indicated that linguists are generally in favour of Open Science/Scholarship (OS), there appears to be a “a misalignment between the attitude to and the adoption of OS practices” (Liu and de Cat 2024, 64).

The present study aims to gain insights into this misalignment by exploring linguists’ understanding of what constitutes OS and of the specificities of linguistic research that (can) affect its applicability to (subdisciplines of) linguistics. To this end, the study draws on the results of an anonymous, small-scale survey and the qualitative analysis of semi-structured interviews conducted with 26 linguists based in Northern Europe, representing all career stages, and a wide range of subdisciplines within linguistics.

The results reveal diverse understandings of OS among linguists. While some focus on the accessibility of research (for both academics and the wider public), others prioritise the sharing of data, materials, and code to promote transparency, reproducibility, and replicability. The latter group also emphasises the importance of OS principles and values like rigour, fairness, and collaboration. Linguists report learning about OS through conferences, workshops, library services, and social media but, most importantly, in personal interactions with other researchers, thus making much of this knowledge network-dependent.

The interviewees highlight several challenges and considerations that they believe need to be addressed when applying OS to linguistics. These include ethical and legal issues concerning data sharing, the high inter-person variability inherent to many linguistic studies, the need for (more) funding for open-access monographs, and for training in data management and statistical methods.

URL : “Well, Parts of Linguistics Is Open…”: Insights into Linguists’ Diverse Understandings of Open Science

DOI : https://doi.org/10.3998/jep.7974

Catégories
EN

Reproducibility in Management Science

Authors : Miloš Fišar, Ben Greiner, Christoph Huber, Elena Katok, Ali I. Ozkes

With the help of more than 700 reviewers, we assess the reproducibility of nearly 500 articles published in the journal Management Science before and after the introduction of a new Data and Code Disclosure policy in 2019.

When considering only articles for which data accessibility and hardware and software requirements were not an obstacle for reviewers, the results of more than 95% of articles under the new disclosure policy could be fully or largely computationally reproduced. However, for 29% of articles, at least part of the data set was not accessible to the reviewer. Considering all articles in our sample reduces the share of reproduced articles to 68%.

These figures represent a significant increase compared with the period before the introduction of the disclosure policy, where only 12% of articles voluntarily provided replication materials, of which 55% could be (largely) reproduced. Substantial heterogeneity in reproducibility rates across different fields is mainly driven by differences in data set accessibility.

Other reasons for unsuccessful reproduction attempts include missing code, unresolvable code errors, weak or missing documentation, and software and hardware requirements and code complexity. Our findings highlight the importance of journal code and data disclosure policies and suggest potential avenues for enhancing their effectiveness.

DOI : https://doi.org/10.1287/mnsc.2023.03556

Catégories
EN

Open science, the replication crisis, and environmental public health

Author : Daniel J. Hicks

Concerns about a crisis of mass irreplicability across scientific fields (“the replication crisis”) have stimulated a movement for open science, encouraging or even requiring researchers to publish their raw data and analysis code.

Recently, a rule at the US Environmental Protection Agency (US EPA) would have imposed a strong open data requirement. The rule prompted significant public discussion about whether open science practices are appropriate for fields of environmental public health.

The aims of this paper are to assess (1) whether the replication crisis extends to fields of environmental public health; and (2) in general whether open science requirements can address the replication crisis.

There is little empirical evidence for or against mass irreplicability in environmental public health specifically. Without such evidence, strong claims about whether the replication crisis extends to environmental public health – or not – seem premature.

By distinguishing three concepts – reproducibility, replicability, and robustness – it is clear that open data initiatives can promote reproducibility and robustness but do little to promote replicability.

I conclude by reviewing some of the other benefits of open science, and offer some suggestions for funding streams to mitigate the costs of adoption of open science practices in environmental public health.

URL : Open science, the replication crisis, and environmental public health

DOI : https://doi.org/10.1080/08989621.2021.1962713

Catégories
EN

Replication and trustworthiness

Authors : Rik Peels, Lex Bouter

This paper explores various relations that exist between replication and trustworthiness. After defining “trust”, “trustworthiness”, “replicability”, “replication study”, and “successful replication”, we consider, respectively, how trustworthiness relates to each of the three main kinds of replication: reproductions, direct replications, and conceptual replications.

Subsequently, we explore how trustworthiness relates to the intentionality of a replication. After that, we discuss whether the trustworthiness of research findings depends merely on evidential considerations or also on what is at stake.

We conclude by adding replication to the other issues that should be considered in assessing the trustworthiness of research findings: (1) the likelihood of the findings before the primary study was done (that is, the prior probability of the findings), (2) the study size and the methodological quality of the primary study, (3) the number of replications that were performed and the quality and consistency of their aggregated findings, and (4) what is at stake.

URL : Replication and trustworthiness

DOI : https://doi.org/10.1080/08989621.2021.1963708

Catégories
EN

Reading the fine print: A review and analysis of business journals’ data sharing policies

Authors : Brianne Dosch, Tyler Martindale

Business librarians offer many data services to their researchers. These services are often focused more on discovery, visualization, and analysis than general data management. But, with the replication crisis facing many business disciplines, there is a need for business librarians to offer more data sharing and general data management support to their researchers.

To find evidence of this data need, 146 business journal’s data sharing policies were reviewed and analyzed to uncover meaningful trends in business research. Results of the study indicate data sharing is not mandated by business journals.

However, data sharing is often encouraged and recommended. This journal policy content analysis provides evidence that business researchers have opportunities to share their research data, and with the right data management support, business librarians can play a significant role in improving the data sharing behaviors of business researchers.

DOI : https://doi.org/10.1080/08963568.2020.1847549

Catégories
EN

Two years into the Brazilian Reproducibility Initiative: reflections on conducting a large-scale replication of Brazilian biomedical science

Author : Kleber Neves, Clarissa FD Carneiro, Ana Paula Wasilewska-Sampaio, Mariana Abreu, Bruna Valério-Gomes, Pedro B Tan, Olavo B Amaral

Scientists have increasingly recognised that low methodological and analytical rigour combined with publish-or-perish incentives can make the published scientific literature unreliable.

As a response to this, large-scale systematic replications of the literature have emerged as a way to assess the problem empirically. The Brazilian Reproducibility Initiative is one such effort, aimed at estimating the reproducibility of Brazilian biomedical research.

Its goal is to perform multicentre replications of a quasi-random sample of at least 60 experiments from Brazilian articles published over a 20-year period, using a set of common laboratory methods.

In this article, we describe the challenges of managing a multicentre project with collaborating teams across the country, as well as its successes and failures over the first two years.

We end with a brief discussion of the Initiative’s current status and its possible future contributions after the project is concluded in 2021.

URL : Two years into the Brazilian Reproducibility Initiative: reflections on conducting a large-scale replication of Brazilian biomedical science

DOI : https://doi.org/10.1590/0074-02760200328

Catégories
EN

What is replication?

Authors : Brian A. Nosek, Timothy M. Errington

Credibility of scientific claims is established with evidence for their replicability using new data. According to common understanding, replication is repeating a study’s procedure and observing whether the prior finding recurs. This definition is intuitive, easy to apply, and incorrect.

We propose that replication is a study for which any outcome would be considered diagnostic evidence about a claim from prior research. This definition reduces emphasis on operational characteristics of the study and increases emphasis on the interpretation of possible outcomes.

The purpose of replication is to advance theory by confronting existing understanding with new evidence. Ironically, the value of replication may be strongest when existing understanding is weakest.

Successful replication provides evidence of generalizability across the conditions that inevitably differ from the original study; Unsuccessful replication indicates that the reliability of the finding may be more constrained than recognized previously.

Defining replication as a confrontation of current theoretical expectations clarifies its important, exciting, and generative role in scientific progress.

URL : What is replication?

DOI : https://doi.org/10.1371/journal.pbio.3000691