Archives des mots-clés : research impact

The use of ChatGPT for identifying disruptive papers in science: a first exploration

Posté le par

Authors : Lutz Bornmann, Lingfei Wu, Christoph Ettl

ChatGPT has arrived in quantitative research evaluation. With the exploration in this Letter to the Editor, we would like to widen the spectrum of the possible use of ChatGPT in bibliometrics by applying it to identify disruptive papers.

The identification of disruptive papers using publication and citation counts has become a popular topic in scientometrics. The disadvantage of the quantitative approach is its complexity in the computation. The use of ChatGPT might be an easy to use alternative.

URL : The use of ChatGPT for identifying disruptive papers in science: a first exploration

DOI : https://doi.org/10.1007/s11192-024-05176-z

Open access improves the dissemination of science: insights from Wikipedia

Posté le par

Authors : Puyu Yang, Ahad Shoaib, Robert West, Giovanni Colavizza

Wikipedia is a well-known platform for disseminating knowledge, and scientific sources, such as journal articles, play a critical role in supporting its mission. The open access movement aims to make scientific knowledge openly available, and we might intuitively expect open access to help further Wikipedia’s mission. However, the extent of this relationship remains largely unknown.

To fill this gap, we analyse a large dataset of citations from the English Wikipedia and model the role of open access in Wikipedia’s citation patterns. Our findings reveal that Wikipedia relies on open access articles at a higher overall rate (44.1%) compared to their availability in the Web of Science (23.6%) and OpenAlex (22.6%). Furthermore, both the accessibility (open access status) and academic impact (citation count) significantly increase the probability of an article being cited on Wikipedia.

Specifically, open access articles are extensively and increasingly more cited in Wikipedia, as they show an approximately 64.7% higher likelihood of being cited in Wikipedia when compared to paywalled articles, after controlling for confounding factors. This open access citation effect is particularly strong for articles with high citation counts or published in recent years.

Our findings highlight the pivotal role of open access in facilitating the dissemination of scientific knowledge, thereby increasing the likelihood of open access articles reaching a more diverse audience through platforms such as Wikipedia. Simultaneously, open access articles contribute to the reliability of Wikipedia as a source by affording editors timely access to novel results.

URL : Open access improves the dissemination of science: insights from Wikipedia

DOI : https://doi.org/10.1007/s11192-024-05163-4

Economic valuation of open research data: A conceptual framework and methodological approach

Posté le par

Authors : Zhifang Tu, Jiashu Shen

The economic significance of open research data is widely acknowledged, yet its quantification remains challenging. This paper presents an effective valuation instrument to help stakeholders understand and evaluate the economic benefits of open research data. By conducting a scoping review and prioritizing user engagement, this study introduces a comprehensive conceptual framework for the economic valuation of open research data. The valuation is based on economic value and willingness to pay, employing the Contingent Valuation Method (CVM).

It incorporates per-use models (view, download, and request) and periodic subscription models (monthly and yearly). An empirical survey was conducted at the National Basic Science Data Center (NBSDC) in China to verify this framework. Both pricing models, comprising five distinct tactics, was supported by surveyed users. Measuring economic value by views and by year was preferred, while willingness to pay by downloads and by year was considered more reasonable.

Overall, the most applicable valuation approach is on a yearly basis. Through this case study at NBSDC, specific pricing tactics were identified, and the total economic value and users’ willingness to pay were assessed. This study is arguably the first to establish a conceptual framework with pricing tactics from a user perspective. This methodological approach for economic valuation of open research data provides evidence and tools for future research, policy formulation, and resource allocation in the context of open science and innovation.

URL : Economic valuation of open research data: A conceptual framework and methodological approach

DOI : https://doi.org/10.1093/reseval/rvae033

The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact

Posté le par

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact.

This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact. Leveraging data from the first Italian Research Assessment Exercise (VTR 2001–2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

URL : The role of non-scientific factors vis-à-vis the quality of publications in determining their scholarly impact

DOI : https://doi.org/10.1007/s11192-024-05106-z

Effects of Open Access. Literature study on empirical research 2010–2021

Posté le par

Authors : David Hopf, Sarah Dellmann, Christian Hauschke, Marco Tullney

Open access — the free availability of scholarly publications — intuitively offers many benefits. At the same time, some academics, university administrators, publishers, and political decision-makers express reservations. Many empirical studies on the effects of open access have been published in the last decade. This report provides an overview of the state of research from 2010 to 2021.

The empirical results on the effects of open access help to determine the advantages and disadvantages of open access and serve as a knowledge base for academics, publishers, research funding and research performing institutions, and policy makers.

This overview of current findings can inform decisions about open access and publishing strategies. In addition, this report identifies aspects of the impact of open access that are potentially highly relevant but have not yet been sufficiently studied.

URL : Effects of Open Access. Literature study on empirical research 2010–2021

DOI : https://doi.org/10.34657/13648

An analysis of the effects of sharing research data, code, and preprints on citations

Posté le par

Authors : Giovanni Colavizza, Lauren Cadwallader, Marcel LaFlamme, Grégory Dozot, Stéphane Lecorney, Daniel Rappo, Iain Hrynaszkiewicz

Calls to make scientific research more open have gained traction with a range of societal stakeholders. Open Science practices include but are not limited to the early sharing of results via preprints and openly sharing outputs such as data and code to make research more reproducible and extensible. Existing evidence shows that adopting Open Science practices has effects in several domains.

In this study, we investigate whether adopting one or more Open Science practices leads to significantly higher citations for an associated publication, which is one form of academic impact. We use a novel dataset known as Open Science Indicators, produced by PLOS and DataSeer, which includes all PLOS publications from 2018 to 2023 as well as a comparison group sampled from the PMC Open Access Subset. In total, we analyze circa 122’000 publications. We calculate publication and author-level citation indicators and use a broad set of control variables to isolate the effect of Open Science Indicators on received citations.

We show that Open Science practices are adopted to different degrees across scientific disciplines. We find that the early release of a publication as a preprint correlates with a significant positive citation advantage of about 20.2% on average. We also find that sharing data in an online repository correlates with a smaller yet still positive citation advantage of 4.3% on average.

However, we do not find a significant citation advantage for sharing code. Further research is needed on additional or alternative measures of impact beyond citations. Our results are likely to be of interest to researchers, as well as publishers, research funders, and policymakers.

Arxiv : https://arxiv.org/abs/2404.16171

The role of non-scientific factors vis-a-vis the quality of publications in determining their scholarly impact

Posté le par

Authors : Giovanni Abramo, Ciriaco Andrea D’Angelo, Leonardo Grilli

In the evaluation of scientific publications’ impact, the interplay between intrinsic quality and non-scientific factors remains a subject of debate. While peer review traditionally assesses quality, bibliometric techniques gauge scholarly impact. This study investigates the role of non-scientific attributes alongside quality scores from peer review in determining scholarly impact.

Leveraging data from the first Italian Research Assessment Exercise (VTR 2001-2003) and Web of Science citations, we analyse the relationship between quality scores, non-scientific factors, and publication short- and long-term impact.

Our findings shed light on the significance of non-scientific elements overlooked in peer review, offering policymakers and research management insights in choosing evaluation methodologies. Sections delve into the debate, identify non-scientific influences, detail methodologies, present results, and discuss implications.

Arxiv : https://arxiv.org/abs/2404.05345