EN

Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial

Posted on by Hans Dillaerts

Authors : Melissa L Rethlefsen, Sara Schroter, Lex M Bouter, Jamie J Kirkham,  David Moher, Ana Patricia Ayala, David Blanco, Tara J Brigham, Holly K Grossetta Nardini,  Shona Kirtley, Kate Nyhan, Whitney Townsend, Maurice Zeegers

Objective

To evaluate the impact of adding librarians and information specialists (LIS) as methodological peer reviewers to the formal journal peer review process on the quality of search reporting and risk of bias in systematic review searches in the medical literature.

Design

Pragmatic two-group parallel randomised controlled trial.

Setting

Three biomedical journals.

Participants

Systematic reviews and related evidence synthesis manuscripts submitted to The BMJ, BMJ Open and BMJ Medicine and sent out for peer review from 3 January 2023 to 1 September 2023. Randomisation (allocation ratio, 1:1) was stratified by journal and used permuted blocks (block size=4). Of 2670 manuscripts sent to peer review during study enrollment, 400 met inclusion criteria and were randomised (62 The BMJ, 334 BMJ Open, 4 BMJ Medicine). 76 manuscripts were revised and resubmitted in the intervention group and 90 in the control group by 2 January 2024.

Interventions

All manuscripts followed usual journal practice for peer review, but those in the intervention group had an additional (LIS) peer reviewer invited.

Main outcome measures

The primary outcomes are the differences in first revision manuscripts between intervention and control groups in the quality of reporting and risk of bias. Quality of reporting was measured using four prespecified PRISMA-S items. Risk of bias was measured using ROBIS Domain 2. Assessments were done in duplicate and assessors were blinded to group allocation. Secondary outcomes included differences between groups for each individual PRISMA-S and ROBIS Domain 2 item. The difference in the proportion of manuscripts rejected as the first decision post-peer review between the intervention and control groups was an additional outcome.

Results

Differences in the proportion of adequately reported searches (4.4% difference, 95% CI: −2.0% to 10.7%) and risk of bias in searches (0.5% difference, 95% CI: −13.7% to 14.6%) showed no statistically significant differences between groups. By 4 months post-study, 98 intervention and 70 control group manuscripts had been rejected after peer review (13.8% difference, 95% CI: 3.9% to 23.8%).

Conclusions

Inviting LIS peer reviewers did not impact adequate reporting or risk of bias of searches in first revision manuscripts of biomedical systematic reviews and related review types, though LIS peer reviewers may have contributed to a higher rate of rejection after peer review.

URL : Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial

DOI : https://doi.org/10.1136/bmjebm-2024-113527

EN

The academic impact of Open Science: a scoping review

Posted on by Hans Dillaerts

Authors : Thomas Klebel, Vincent Traag, Ioanna Grypari, Lennart Stoy, Tony Ross-Hellauer

Open Science seeks to make research processes and outputs more accessible, transparent and inclusive, ensuring that scientific findings can be freely shared, scrutinized and built upon by researchers and others. To date, there has been no systematic synthesis of the extent to which Open Science (OS) reaches these aims.

We use the PRISMA scoping review methodology to partially address this gap, scoping evidence on the academic (but not societal or economic) impacts of OS. We identify 485 studies related to all aspects of OS, including Open Access (OA), Open/FAIR Data (OFD), Open Code/Software, Open Evaluation and Citizen Science (CS).

Analysing and synthesizing findings, we show that the majority of studies investigated effects of OA, CS and OFD. Key areas of impact studied are citations, quality, efficiency, equity, reuse, ethics and reproducibility, with most studies reporting positive or at least mixed impacts.

However, we also identified significant unintended negative impacts, especially those regarding equity, diversity and inclusion. Overall, the main barrier to academic impact of OS is lack of skills, resources and infrastructure to effectively re-use and build on existing research.

Building on this synthesis, we identify gaps within this literature and draw implications for future research and policy.

URL : The academic impact of Open Science: a scoping review

DOI : https://doi.org/10.1098/rsos.241248

EN

The Origins and Veracity of References ‘Cited’ by Generative Artificial Intelligence Applications: Implications for the Quality of Responses

Posted on by Hans Dillaerts

Author :  Dirk H. R. Spennemann

The public release of ChatGPT in late 2022 has resulted in considerable publicity and has led to widespread discussion of the usefulness and capabilities of generative Artificial intelligence (Ai) language models. Its ability to extract and summarise data from textual sources and present them as human-like contextual responses makes it an eminently suitable tool to answer questions users might ask.

Expanding on a previous analysis of the capabilities of ChatGPT3.5, this paper tested what archaeological literature appears to have been included in the training phase of three recent generative Ai language models: ChatGPT4o, ScholarGPT, and DeepSeek R1. While ChatGPT3.5 offered seemingly pertinent references, a large percentage proved to be fictitious. While the more recent model ScholarGPT, which is purportedly tailored towards academic needs, performed much better, it still offered a high rate of fictitious references compared to the general models ChatGPT4o and DeepSeek.

Using ‘cloze’ analysis to make inferences on the sources ‘memorized’ by a generative Ai model, this paper was unable to prove that any of the four genAi models had perused the full texts of the genuine references. It can be shown that all references provided by ChatGPT and other OpenAi models, as well as DeepSeek, that were found to be genuine, have also been cited on Wikipedia pages.

This strongly indicates that the source base for at least some, if not most, of the data is found in those pages and thus represents, at best, third-hand source material. This has significant implications in relation to the quality of the data available to generative Ai models to shape their answers. The implications of this are discussed.

URL : The Origins and Veracity of References ‘Cited’ by Generative Artificial Intelligence Applications: Implications for the Quality of Responses

DOI : https://doi.org/10.3390/publications13010012

EN

Diamond Open Access Landscape in Croatia: DIAMAS Survey Results

Posted on by Hans Dillaerts

Authors :  Jadranka Stojanovski, Danijel Mofardin

As open science initiatives address the crisis in scholarly communication driven by commercialisation, diamond open access publishing—promoting equity for authors and readers—has emerged as a focal point in open access scholarly publishing. This study examines the landscape of institutional publishing in Croatia, focusing on the community-owned diamond open access model.

Through the DIAMAS project survey, which targeted 251 institutional publishers and achieved a response rate of 77, the research identifies the distinct features of Croatian institutional publishing. Institutional publishers are characterised by governance structures, funding challenges, voluntary staffing, and alignment with open science principles. Notable traits include reliance on public funding, use of the national open access journal platform, and a strong diamond open access publishing tradition.

Key findings emphasise the critical role of national infrastructure, services, and multilingual publishing. Persistent challenges include meeting indexing criteria, advancing open science practices, and ensuring metadata quality. This study provides a comprehensive mapping of Croatian institutional publishers, offering insights into their strengths and weaknesses while proposing strategies for improvement.

The findings underscore the importance of national policy frameworks, capacity building, and international collaboration to ensure the sustainability and visibility of Croatian institutional publishing.

URL : Diamond Open Access Landscape in Croatia: DIAMAS Survey Results

DOI : https://doi.org/10.3390/publications13010013

FR

A la recherche d’une maîtrise des données : une étude des discours scientifiques et professionnels sur la data literacy

Posted on by Hans Dillaerts

Auteur : Ugo Verdi

Depuis deux décennies, l’évolution et la démocratisation des logiciels et des technologies numériques, à l’instar des applications des smartphones, ont généré de nouveaux usages liés aux données. Ces dernières, considérées comme un « nouvel or noir » nécessaire aux prises de décisions, ont été investies d’enjeux centraux au sein de discours qui véhiculent de nombreux mythes essentialisant.

Leur maîtrise est de plus en plus sollicitée par tout un ensemble d’acteurs et la data literacy, également nommée « culture des données » ou « littératie des données », est en ce sens évoquée comme une, si ce n’est « la » solution miracle qui permettrait une acculturation efficace des individus.

L’objectif de cette thèse est de présenter un état de l’art des caractéristiques de la data literacy issues de représentations véhiculées par les discours scientifiques et professionnels. Elle doit ainsi permettre d’une part de répondre à un ensemble de questionnements, à savoir quelles sont les caractéristiques de la data literacy et ont-elles évolué dans le temps ? quelles sont les visions portées sur les données et quel lien est tissé avec l’information ? quels sont les enchevêtrements conceptuels et littéraciques ?

Depuis quand pouvons-nous véritablement parler de data literacy et a-t-il existé des proto data literacies aux objectifs similaires ? que recouvre une « acculturation » aux données et quelles sont ses modalités de déploiement ? quels sont les acteurs qui se sont exprimés sur cette thématique ? dans quels contextes et selon quelle temporalité ? pouvons-nous parler d’une vision purement française de la data literacy ?

D’autre part, de déconstruire les mythes qui entourent la data literacy et in extenso les usages des données. Pour ce faire, trois grand corpus seront exploités : (1) plusieurs centaines de publications scientifiques et professionnelles, (2) 44544 tweets contenant le terme « data literacy » produits entre septembre 2021 et septembre 2022 et (3) 32 entretiens semi-directifs réalisés auprès d’acteurs français.

URL : A la recherche d’une maîtrise des données : une étude des discours scientifiques et professionnels sur la data literacy

TEL : https://theses.hal.science/tel-04905349

 

EN

Motivations and barriers to publishing open access book chapters and monographs: An institutional perspective

Posted on by Hans Dillaerts

Authors : Wm. Joseph Thomas, Allison Kaefring, Jeanne K. Hoover

Introduction

Recent years have seen an increase in publishers exploring open access for monographs and book chapters. Programs like the Direct to Open from MIT Press and JSTOR’s Path to Open have provided avenues for libraries and authors to support open access monographs generally, but not campus authors specifically. On our campus, we have seen an increase in requests for and questions about publishing monographs and book chapters open access.

Description of Program

We offer several options for support for open access article publishing, including transformational agreements, institutional memberships, and an open access fund, but have limited resources and strategies for supporting book and chapter authors to make their publications open access.

To gauge our authors’ awareness and interest, we surveyed faculty who recently published a book or chapter about their publishing experiences with a focus on open access publishing. In addition to our survey, we conducted interviews with faculty to gain a better understanding of open access publishing from their perspective as recent authors.

Next Steps

In response to this research, the library has explored new methods of supporting open monograph publishing and plans to develop open education resources and webinars about the open monograph publishing process.

URL : Motivations and barriers to publishing open access book chapters and monographs: An institutional perspective

DOI : https://doi.org/10.31274/jlsc.18280

 

EN

Automation Applied to the Collection and Generation of Scientific Literature

Posted on by Hans Dillaerts

Authors : Nadia Paola Valadez-de la Paz , Jose Antonio Vazquez-Lopez , Aidee Hernandez-Lopez , Jaime Francisco Aviles-Viñas, Jose Luis Navarro-Gonzalez,  Alfredo Valentin Reyes-Acosta, Ismael Lopez-Juarez

Preliminary activities of searching and selecting relevant articles are crucial in scientific research to determine the state of the art (SOTA) and enhance overall outcomes.

While there are automatic tools for keyword extraction, these algorithms are often computationally expensive, storage-intensive, and reliant on institutional subscriptions for metadata retrieval. Most importantly, they still require manual selection of literature.

This paper introduces a framework that automates keyword searching in article abstracts to help select relevant literature for the SOTA by identifying key terms matching that we, hereafter, call source words. A case study in the food and beverage industry is provided to demonstrate the algorithm’s application.

In the study, five relevant knowledge areas were defined to guide literature selection. The database from scientific repositories was categorized using six classification rules based on impact factor (IF), Open Access (OA) status, and JCR journal ranking.

This classification revealed the knowledge area with the highest presence and highlighted the effectiveness of the selection rules in identifying articles for the SOTA. The approach included a panel of experts who confirmed the algorithm’s effectiveness in identifying source words in high-quality articles. The algorithm’s performance was evaluated using the 𝐹1 Score, which reached 0.83 after filtering out non-relevant articles.

This result validates the algorithm’s ability to extract significant source words and demonstrates its usefulness in building the SOTA by focusing on the most scientifically impactful articles.

URL : Automation Applied to the Collection and Generation of Scientific Literature

DOI : https://doi.org/10.3390/publications13010011