EN

Metrics and peer review agreement at the institutional level

Posted on by Hans Dillaerts

Authors : Vincent A Traag, Marco Malgarini, Sarlo Scipione

In the past decades, many countries have started to fund academic institutions based on the evaluation of their scientific performance. In this context, post-publication peer review is often used to assess scientific performance. Bibliometric indicators have been suggested as an alternative to peer review.

A recurrent question in this context is whether peer review and metrics tend to yield similar outcomes. In this paper, we study the agreement between bibliometric indicators and peer review based on a sample of publications submitted for evaluation to the national Italian research assessment exercise (2011–2014).

In particular, we study the agreement between bibliometric indicators and peer review at a higher aggregation level, namely the institutional level. Additionally, we also quantify the internal agreement of peer review at the institutional level. We base our analysis on a hierarchical Bayesian model using cross-validation.

We find that the level of agreement is generally higher at the institutional level than at the publication level. Overall, the agreement between metrics and peer review is on par with the internal agreement among two reviewers for certain fields of science in this particular context.

This suggests that for some fields, bibliometric indicators may possibly be considered as an alternative to peer review for the Italian national research assessment exercise. Although results do not necessarily generalise to other contexts, it does raise the question whether similar findings would obtain for other research assessment exercises, such as in the United Kingdom.

URL : https://arxiv.org/abs/2006.14830

EN

Hybrid Gold Open Access Citation Advantage in Clinical Medicine: Analysis of Hybrid Journals in the Web of Science

Posted on by Hans Dillaerts

Authors : Chompunuch Saravudecha, Duangruthai Na Thungfai, Chananthida Phasom, Sodsri Gunta-in, Aorrakanya Metha, Peangkobfah Punyaphet, Tippawan Sookruay, Wannachai Sakuludomkan, Nut Koonrungsesomboon

Biomedical fields have seen a remarkable increase in hybrid Gold open access articles. However, it is uncertain whether the hybrid Gold open access option contributes to a citation advantage, an increase in the citations of articles made immediately available as open access regardless of the article’s quality or whether it involves a trending topic of discussion.

This study aimed to compare the citation counts of hybrid Gold open access articles to subscription articles published in hybrid journals. The study aimed to ascertain if hybrid Gold open access publications yield an advantage in terms of citations.

This cross-sectional study included the list of hybrid journals under 59 categories in the ‘Clinical Medicine’ group from Clarivate’s Journal Citation Reports (JCR) during 2018–2021. The number of citable items with ‘Gold Open Access’ and ‘Subscription and Free to Read’ in each journal, as well as the number of citations of those citable items, were extracted from JCR.

A hybrid Gold open access citation advantage was computed by dividing the number of citations per citable item with hybrid Gold open access by the number of citations per citable item with a subscription.

A total of 498, 636, 1009, and 1328 hybrid journals in the 2018 JCR, 2019 JCR, 2020 JCR, and 2021 JCR, respectively, were included in this study. The citation advantage of hybrid Gold open access articles over subscription articles in 2018 was 1.45 (95% confidence interval (CI), 1.24–1.65); in 2019, it was 1.31 (95% CI, 1.20–1.41); in 2020, it was 1.30 (95% CI, 1.20–1.39); and in 2021, it was 1.31 (95% CI, 1.20–1.42).

In the ‘Clinical Medicine’ discipline, the articles published in the hybrid journal as hybrid Gold open access had a greater number of citations when compared to those published as a subscription, self-archived, or otherwise openly accessible option.

URL : Hybrid Gold Open Access Citation Advantage in Clinical Medicine: Analysis of Hybrid Journals in the Web of Science

DOI : https://doi.org/10.3390/publications11020021

EN

Biases in scholarly recommender systems: impact, prevalence, and mitigation

Posted on by Hans Dillaerts

Authors : Michael Färber, Melissa Coutinho, Shuzhou Yuan

With the remarkable increase in the number of scientific entities such as publications, researchers, and scientific topics, and the associated information overload in science, academic recommender systems have become increasingly important for millions of researchers and science enthusiasts.

However, it is often overlooked that these systems are subject to various biases. In this article, we first break down the biases of academic recommender systems and characterize them according to their impact and prevalence. In doing so, we distinguish between biases originally caused by humans and biases induced by the recommender system.

Second, we provide an overview of methods that have been used to mitigate these biases in the scholarly domain.

Based on this, third, we present a framework that can be used by researchers and developers to mitigate biases in scholarly recommender systems and to evaluate recommender systems fairly.

Finally, we discuss open challenges and possible research directions related to scholarly biases.

URL : Biases in scholarly recommender systems: impact, prevalence, and mitigation

DOI : https://doi.org/10.1007/s11192-023-04636-2

EN

Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021

Posted on by Hans Dillaerts

Authors : Mike Thelwall, Kayvan Kousha, Mahshid Abdoli, Emma Stuart, Meiko Makita, Paul Wilson, Jonathan Levitt

Altmetrics are web-based quantitative impact or attention indicators for academic articles that have been proposed to supplement citation counts. This article reports the first assessment of the extent to which mature altmetrics from Altmetric.com and Mendeley associate with individual article quality scores.

It exploits expert norm-referenced peer review scores from the UK Research Excellence Framework 2021 for 67,030+ journal articles in all fields 2014–2017/2018, split into 34 broadly field-based Units of Assessment (UoAs). Altmetrics correlated more strongly with research quality than previously found, although less strongly than raw and field normalized Scopus citation counts.

Surprisingly, field normalizing citation counts can reduce their strength as a quality indicator for articles in a single field. For most UoAs, Mendeley reader counts are the best altmetric (e.g., three Spearman correlations with quality scores above 0.5), tweet counts are also a moderate strength indicator in eight UoAs (Spearman correlations with quality scores above 0.3), ahead of news (eight correlations above 0.3, but generally weaker), blogs (five correlations above 0.3), and Facebook (three correlations above 0.3) citations, at least in the United Kingdom.

In general, altmetrics are the strongest indicators of research quality in the health and physical sciences and weakest in the arts and humanities.

URL : Do altmetric scores reflect article quality? Evidence from the UK Research Excellence Framework 2021

Original location : https://asistdl.onlinelibrary.wiley.com/doi/full/10.1002/asi.24751

EN

The Transformation of the Green Road to Open Access

Posted on by Hans Dillaerts

Authors : Joachim Schöpfel, Stéphane Chaudiron, Bernard Jacquemin, Eric Kergosien, Hélène Prost, Florence Thiault

(1) Background: The 2002 Budapest Open Access Initiative recommended on self-archiving of scientific articles in open repositories as the “green road” to open access. Twenty years later, only one part of the researchers deposits their publications in open repositories; moreover, one part of the repositories’ content is not based on self-archived deposits but on mediated nonfaculty contributions. The purpose of the paper is to provide more empirical evidence on this situation and to assess the impact on the future of the green road.

(2) Methods: We analyzed the contributions on the French national HAL repository from more than 1,000 laboratories affiliated to the ten most important French research universities, with a focus on 2020, representing 14,023 contributor accounts and 166,939 deposits.

(3) Results: We identified seven different types of contributor accounts, including deposits from nonfaculty staff and import flows from other platforms. Mediated nonfaculty contribution accounts for at least 48% of the deposits. We also identified difference between institutions and disciplines.

(4) Conclusions: Our empirical results reveal a transformation of open repositories from self-archiving and direct scientific communication towards research information management. Repositories like HAL are somewhere in the middle of the process. The paper describes data quality as the main issue and major challenge of this transformation.

URL : The Transformation of the Green Road to Open Access

DOI : https://doi.org/10.20944/preprints202302.0268.v1

EN

Publishers, funders and institutions: who is supporting UKRI-funded researchers to share data?

Posted on by Hans Dillaerts

Authors : Beth Montague-Hellen, Kate Montague-Hellen

Researchers are increasingly being asked by funders, publishers and their institutions to share research data alongside written publications, and to include data availability statements to support their readers in finding this data.

In the UK, UKRI (UK Research and Innovation) is one of the largest funding bodies and has had data-sharing policies for several years. This article investigates the reasons why a researcher may or may not share their data and assesses whether funders, publishers and institutions are supporting data-sharing behaviour through their policies and actions.

A survey with 166 responses gave an indicative assessment of researcher opinions around data sharing, and a corpus of 3,277 journal articles retrieved from four UK institutions was analysed using multivariate logistic regression models to provide empirical evidence as to researcher behaviour around data sharing.

The regression models provide insight into how this is affected by the funder, institution and publisher of the research. This study identifies that those publishers and funders who give clear guidance in their policies as to which data should be shared, and where this data should be shared, are most likely to encourage good practice in researchers.

URL : Publishers, funders and institutions: who is supporting UKRI-funded researchers to share data?

DOI : https://doi.org/10.1629/uksg.602

EN

The Landscape of Scholarly Book Publishing in Croatia: Finding Pathways for Viable Open Access Models

Posted on by Hans Dillaerts

Author : Iva Melinščak Zlodi

1) Background: Open access to scholarly works is globally recognized as a goal to be achieved as soon as possible; however, there is not yet a general understanding of how to achieve open access for books. In considering the most appropriate models of transition, an accurate and detailed insight into national and regional specifics can be of great importance.

The aim of this research is to show the current state of scholarly book publishing in Croatia: recognising the key stakeholders, their characteristics, and the current level of open access to scholarly books.

(2) Methods: The existing data from two different sources were used: the data about the public subsidies for book publishers by the Ministry of Science and Education and the data on published books from the Croatian Scientific Bibliography CROSBI, both for the period from 2018 to 2021.

(3) Results: In the four-year period, 224 Croatian publishers were awarded subsidies to publish 2359 book titles. The majority of the publishers received support for only a small number of titles and relatively low amounts of subsidies. More than half of the titles are published by small private commercial publishers.

However, the uptake of digital publishing among commercial publishers is very modest. Open access to scholarly books is almost entirely in the domain of non-commercial publishers. Most open access titles are available on the websites of their publishers.

(4) Conclusions: The analysis of the data from these two sources have resulted in an overview of the current state of book publishing in Croatia. Such an overview provides a good basis for designing future measures and creating a national open science plan and can also be a useful contribution to international discussions.

URL : The Landscape of Scholarly Book Publishing in Croatia: Finding Pathways for Viable Open Access Models

DOI : https://doi.org/10.3390/publications11010017