Publié le

Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platforms

Authors : Mike Thelwall, Abdallah Yaghi

Academic peer review is at the heart of scientific quality control, yet the process is slow and time-consuming. Technology that can predict peer review outcomes may help with this, for example by fast-tracking desk rejection decisions. While previous studies have demonstrated that Large Language Models (LLMs) can predict peer review outcomes to some extent, this paper introduces two new contexts and employs a more robust method—averaging multiple ChatGPT scores.

Averaging 30 ChatGPT predictions, based on reviewer guidelines and using only the submitted titles and abstracts failed to predict peer review outcomes for F1000Research (Spearman’s rho = 0.00). However, it produced mostly weak positive correlations with the quality dimensions of SciPost Physics (rho = 0.25 for validity, rho = 0.25 for originality, rho = 0.20 for significance, and rho = 0.08 for clarity) and a moderate positive correlation for papers from the International Conference on Learning Representations (ICLR) (rho = 0.38). Including article full texts increased the correlation for ICLR (rho = 0.46) and slightly improved it for F1000Research (rho = 0.09), with variable effects on the four quality dimension correlations for SciPost LaTeX files.

The use of simple chain-of-thought system prompts slightly increased the correlation for F1000Research (rho = 0.10), marginally reduced it for ICLR (rho = 0.37), and further decreased it for SciPost Physics (rho = 0.16 for validity, rho = 0.18 for originality, rho = 0.18 for significance, and rho = 0.05 for clarity). Overall, the results suggest that in some contexts, ChatGPT can produce weak pre-publication quality predictions.

However, their effectiveness and the optimal strategies for employing them vary considerably between platforms, journals, and conferences. Finally, the most suitable inputs for ChatGPT appear to differ depending on the platform.

URL : Evaluating the predictive capacity of ChatGPT for academic peer review outcomes across multiple platforms

DOI : https://doi.org/10.1007/s11192-025-05287-1

Publié le

Open access, open infrastructures, and their funding: Learning from histories to more effectively enhance diamond OA ecologies for books

Authors  : Kira Hopkins, Kevin Sanders

The decade since the “Bottlenecks in the Open Access System” special issue of JLSC in 2014 has been an expansive one for open access (OA) and OA books in particular. The creation of a scholarly publishing ecosystem that enables works to be freely accessible for readers has been successful in many ways.

However, the underlying politics and economics of OA scholarly publishing often remain opaque or under-interrogated (Lawson et al., 2015). The problems with journal OA funding, specifically regarding inequality of access to publishing, discussed by Bonaccorso et al. (2014) in their contribution to that issue, have also increased and become entrenched as we discuss below.

This entrenchment has been largely via the growth and consolidation of gold OA, “transformative” agreements, and read-and-publish journal deals, which have effectively, and unnecessarily, commodified OA publications. We would argue that this is in direct tension with some of the foundations of contemporary OA.

OA was explicitly described from early principles as not a business model and as aiming to reduce financial barriers from authors, libraries, and other groups (Suber, 2024). We would like to note that, while the main focus of this paper is books, we begin with a discussion of journals. This is because we are focusing on the history, development, and critiques of OA fundings in the intervening ten years following the “Bottlenecks” special issue.

OA journal publishing has been at the forefront of discussions of OA funding, and it has dominated the last decade, and more, of this discussion; it would therefore be remiss of us not to discuss this history, the resulting current landscape of inequity, and the potential ramifications if this were to be transferred to OA books, a more nascent field in general.

URL : Open access, open infrastructures, and their funding: Learning from histories to more effectively enhance diamond OA ecologies for books

DOI : https://doi.org/10.31274/jlsc.18284

 

Publié le

Identifying trust cues: how trust in science is mediated in content about science

Authors :  Justin T. Schröder, Janise Brück , Lars Guenther

Most public audiences in Germany receive scientific information via a variety of (digital) media; in these contexts, media act as intermediaries of trust in science by providing information that present reasons for public audiences to place their trust in science. T

o describe this process, the study introduces the term “trust cues”. To identify such content-related trust cues, an explorative qualitative content analysis has been applied to German journalistic, populist, social, and other (non-journalistic) online media (“n” = 158). In total, “n” = 1,329 trust cues were coded. The findings emphasize the diversity of mediated trust, with trust cues being connected to dimensions of trust in science (established: expertise, integrity, benevolence; recently introduced: transparency, dialogue).

Through this analysis, the study aims for a better understanding of mediated trust in science. Deriving this finding is crucial since public trust in science is important for individual and collective informed decision-making and crises management.

URL : Identifying trust cues: how trust in science is mediated in content about science

DOI : https://doi.org/10.22323/2.24010206

Publié le

Knowledge Production and Intellectual Property: A Perspective on Scientific Publications in the Capitalist System

Author : Sofia Guilhem Basilio

The digital revolution has reshaped the production, dissemination, and accessibility of scientific knowledge. However, capitalist logic persists, commodifying intellectual labour and concentrating market power within a few mega-publishers.

This article critically examines scientific publishing through the lens of Marx’s theory of value, focusing on intellectual property rent as a mechanism of capital accumulation.

By highlighting the Brazilian higher education system – where public resources are redirected to private publishers via paywalls and Article Processing Charges (APCs) – the paper exposes the contradictions of contemporary academic publishing.

It critiques the dual exploitation of researchers as producers and consumers of knowledge and argues for alternative, equitable models like Open Access. Situating the analysis within global and local contexts, the article advocates for the democratisation of scientific knowledge as a resistance to commodification and privatisation.

URL : Knowledge Production and Intellectual Property: A Perspective on Scientific Publications in the Capitalist System

DOI : https://doi.org/10.31269/triplec.v23i1.1520

Publié le

Why is it important to implement meta-research in universities and institutes with medical research activities

Authors : Ivan David Lozada-Martinez, Dionicio Neira-Rodado, Darly Martinez-Guevara, Hary Salome Cruz-Soto, Maria Paula Sanchez-Echeverry, Yamil Liscano

In recent years, there has been a growing concern over questionable practices and a lack of rigor in scientific activities, particularly in health and medical sciences.

Universities and research institutes are key players in the development of science, technology, and innovation. Academic institutions, whose primary mission is to generate and disseminate knowledge, bear the responsibility in many parts of the world to act as consultants and guardians of scientific integrity in health research.

Then, universities and research institutes must act as guardians of the research and technological development process, utilizing methodological and operational evaluation tools to validate the rigor and quality of medical research.

Meta-research is defined as the research of research itself. Some of the most important specific objectives of meta-research include the assessment of research relevance, the evaluation of evidence validity, and the exploration of scientific integrity.

A significant portion of evidence in the medical and health sciences literature has been found to be redundant, misleading, or inconsistent. Although this issue is of great importance in global health, discussions about practical and tangible solutions remain fragmented and limited.

The aim of this manuscript is to highlight the significance of employing meta-research within universities and research institutes as a tool to monitor scientific rigor and promote responsible practices in medical research.

URL : Why is it important to implement meta-research in universities and institutes with medical research activities

DOI : https://doi.org/10.3389/frma.2025.1497280

Publié le

Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial

Authors : Melissa L Rethlefsen, Sara Schroter, Lex M Bouter, Jamie J Kirkham,  David Moher, Ana Patricia Ayala, David Blanco, Tara J Brigham, Holly K Grossetta Nardini,  Shona Kirtley, Kate Nyhan, Whitney Townsend, Maurice Zeegers

Objective

To evaluate the impact of adding librarians and information specialists (LIS) as methodological peer reviewers to the formal journal peer review process on the quality of search reporting and risk of bias in systematic review searches in the medical literature.

Design

Pragmatic two-group parallel randomised controlled trial.

Setting

Three biomedical journals.

Participants

Systematic reviews and related evidence synthesis manuscripts submitted to The BMJ, BMJ Open and BMJ Medicine and sent out for peer review from 3 January 2023 to 1 September 2023. Randomisation (allocation ratio, 1:1) was stratified by journal and used permuted blocks (block size=4). Of 2670 manuscripts sent to peer review during study enrollment, 400 met inclusion criteria and were randomised (62 The BMJ, 334 BMJ Open, 4 BMJ Medicine). 76 manuscripts were revised and resubmitted in the intervention group and 90 in the control group by 2 January 2024.

Interventions

All manuscripts followed usual journal practice for peer review, but those in the intervention group had an additional (LIS) peer reviewer invited.

Main outcome measures

The primary outcomes are the differences in first revision manuscripts between intervention and control groups in the quality of reporting and risk of bias. Quality of reporting was measured using four prespecified PRISMA-S items. Risk of bias was measured using ROBIS Domain 2. Assessments were done in duplicate and assessors were blinded to group allocation. Secondary outcomes included differences between groups for each individual PRISMA-S and ROBIS Domain 2 item. The difference in the proportion of manuscripts rejected as the first decision post-peer review between the intervention and control groups was an additional outcome.

Results

Differences in the proportion of adequately reported searches (4.4% difference, 95% CI: −2.0% to 10.7%) and risk of bias in searches (0.5% difference, 95% CI: −13.7% to 14.6%) showed no statistically significant differences between groups. By 4 months post-study, 98 intervention and 70 control group manuscripts had been rejected after peer review (13.8% difference, 95% CI: 3.9% to 23.8%).

Conclusions

Inviting LIS peer reviewers did not impact adequate reporting or risk of bias of searches in first revision manuscripts of biomedical systematic reviews and related review types, though LIS peer reviewers may have contributed to a higher rate of rejection after peer review.

URL : Improving peer review of systematic reviews and related review types by involving librarians and information specialists as methodological peer reviewers: a randomised controlled trial

DOI : https://doi.org/10.1136/bmjebm-2024-113527

Publié le

The academic impact of Open Science: a scoping review

Authors : Thomas Klebel, Vincent Traag, Ioanna Grypari, Lennart Stoy, Tony Ross-Hellauer

Open Science seeks to make research processes and outputs more accessible, transparent and inclusive, ensuring that scientific findings can be freely shared, scrutinized and built upon by researchers and others. To date, there has been no systematic synthesis of the extent to which Open Science (OS) reaches these aims.

We use the PRISMA scoping review methodology to partially address this gap, scoping evidence on the academic (but not societal or economic) impacts of OS. We identify 485 studies related to all aspects of OS, including Open Access (OA), Open/FAIR Data (OFD), Open Code/Software, Open Evaluation and Citizen Science (CS).

Analysing and synthesizing findings, we show that the majority of studies investigated effects of OA, CS and OFD. Key areas of impact studied are citations, quality, efficiency, equity, reuse, ethics and reproducibility, with most studies reporting positive or at least mixed impacts.

However, we also identified significant unintended negative impacts, especially those regarding equity, diversity and inclusion. Overall, the main barrier to academic impact of OS is lack of skills, resources and infrastructure to effectively re-use and build on existing research.

Building on this synthesis, we identify gaps within this literature and draw implications for future research and policy.

URL : The academic impact of Open Science: a scoping review

DOI : https://doi.org/10.1098/rsos.241248