Evaluating Research Quality with Large Language Models: An Analysis of ChatGPT’s Effectiveness with Different Settings and Inputs

Posté le par

Author : Mike Thelwall

Evaluating the quality of academic journal articles is a time consuming but critical task for national research evaluation exercises, appointments and promotion. It is therefore important to investigate whether Large Language Models (LLMs) can play a role in this process.

This article assesses which ChatGPT inputs (full text without tables, figures and references; title and abstract; title only) produce better quality score estimates, and the extent to which scores are affected by ChatGPT models and system prompts.

The results show that the optimal input is the article title and abstract, with average ChatGPT scores based on these (30 iterations on a dataset of 51 papers) correlating at 0.67 with human scores, the highest ever reported. ChatGPT 4o is slightly better than 3.5-turbo (0.66), and 4o-mini (0.66).

The results suggest that article full texts might confuse LLM research quality evaluations, even though complex system instructions for the task are more effective than simple ones.

Thus, whilst abstracts contain insufficient information for a thorough assessment of rigour, they may contain strong pointers about originality and significance. Finally, linear regression can be used to convert the model scores into the human scale scores, which is 31% more accurate than guessing.

Arxiv : https://arxiv.org/abs/2408.06752

Does Open Access Foster Interdisciplinary Citation? Decomposing Open Access Citation Advantage

Posté le par

Authors : Kai Nishikawa, Akiyoshi Murakami

The existence of an open access (OA) citation advantage, that is, whether OA increases citations, has been a topic of interest for many years. Although numerous previous studies have focused on whether OA increases citations, expectations for OA go beyond that. One such expectation is the promotion of knowledge transfer across various fields.

This study aimed to clarify whether OA, especially gold OA, increases interdisciplinary citations in various natural science fields. Specifically, we measured the effect of OA on interdisciplinary and within-discipline citation counts by decomposing an existing metric of the OA citation advantage.

The results revealed that OA increases both interdisciplinary and within-discipline citations in many fields and increases only interdisciplinary citations in chemistry, computer science, and clinical medicine. Among these fields, clinical medicine tends to obtain more interdisciplinary citations without being influenced by specific journals or papers.

The findings indicate that OA fosters knowledge transfer to different fields, which extends our understanding of its effects.

Arxiv : https://arxiv.org/abs/2411.14653

Open scholarship and bibliodiversity

Posté le par

Authors : Maureen P. Walsh, Nataliia Kaliuzhna, Nokuthula Mchunu, Mohamad Mostafa, Katherine Witzig, Tony Alves

This paper is based on the Open Scholarship and Bibliodiversity panel presented at the 2024 NISO Plus conference in Baltimore, Maryland on February 13, 2024, and brings together five perspectives on the interdependency of open scholarship and bibliodiversity. Bibliodiversity in the context of open scholarship refers to the diversity of publishing models, platforms, and formats that are available for scholarly communication.

It emphasizes the importance of a varied and inclusive ecosystem for acquiring academic knowledge and for the dissemination of research. An important part of bibliodiversity is the inclusion and the promotion of a diversity of scholarly voices.

The authors explore how to ensure that a scholarly infrastructure includes a multitude of voices, is accessible to everyone, and can be expressed in a variety of ways.

URL : Open scholarship and bibliodiversity

DOI : https://doi.org/10.1177/18758789241296760

Repository (R)evolution: Metadata, Interoperability, and Sustainability

Posté le par

Authors : Linda Eells, Julia Kelly, Shannon Farrell

Introduction

Successfully managing an open-access repository requires constant attention to user community priorities in order to inform the development or selection of a platform that fulfills constantly evolving functional demands in an increasingly complex operational environment.

This paper uses AgEcon Search (AES) as an example of the way that varying platforms address the metadata and other platform needs of a repository. AES is a successful subject repository with an international scope that has resided on several different platforms in its 25-year lifespan.

Elements and Considerations

Critical among the technical requirements of a repository is interoperability with other information sources and the ability to accommodate and describe different types of objects, including data. Experienced in the use of easy and widely used Dublin Core (DC), as well as Machine-Readable Cataloging 21 (MARC 21)-based repository platforms, we discuss both metadata schemas from administrative and user perspectives.

Reconsidering underlying metadata issues might positively impact both technical and administrative issues that are currently restricting the development of robust, interoperable systems. As managers of AES, we are uniquely placed to discuss both technical and sustainability issues.

Conclusions

Although many institutional and subject repositories are on platforms that use DC for their metadata, other options are available. MARC, the well-established library standard, can provide the wide range of fields needed to fully and accurately describe the variety of document and data types that are included in repositories.

URL : Repository (R)evolution: Metadata, Interoperability, and Sustainability

DOI : https://doi.org/10.31274/ jlsc.16890 

Open Research Data in Spanish University Repositories

Posté le par

Authors : Pablo Monteagudo-Haro, Juan Jose Prieto-Gutierrez

The current situation of open research data in Spanish university repositories is analyzed by means of twelve indicators that allow us to compare them with each other. The twelve self-developed indicators deal with research datasets and institutional policies linked to open access, as well as some of the key characteristics of the repositories.

The methodology used consists of comparing the repositories of the different universities linked to REBIUN. The result has been that datasets in institutional repositories are scarce, and the situation is heterogeneous across the territory. This raises questions about future open access policies for research data in the country’s main scientific institutions.

Arxiv : https://arxiv.org/abs/2411.10470

Fundamental problems in the peer-review process and stakeholders’ perceptions of potential suggestions for improvement

Posté le par

Authors : Cigdem Kadaifci, Erkan Isikli, Y. Ilker Topcu

Academic papers are essential for researchers to communicate their work to their peers and industry experts. Quality research is published in prestigious scientific journals, and is considered as part of the hiring and promotion criteria at leading universities. Scientific journals conduct impartial and anonymous peer reviews of submitted manuscripts; however, individuals involved in this process may encounter issues related to the duration, impartiality, and transparency of these reviews.

To explore these concerns, we created a questionnaire based on a comprehensive review of related literature and expert opinions, which was distributed to all stakeholders (authors, reviewers, and editors) who participated in the peer-review process from a variety of countries and disciplines. Their opinions on the primary issues during the process and suggestions for improvement were collected. The data were then analysed based on various groups, such as gender, country of residence, and contribution type, using appropriate multivariate statistical techniques to determine the perceptions and experiences of participants in the peer-review process.

The results showed that unethical behaviour was not uncommon and that editors and experienced reviewers encountered it more frequently. Women and academics from Türkiye were more likely to experience ethical violations and perceived them as more ethically severe. Incentives and stakeholder involvement were seen as ways to enhance the quality and impartiality of peer review. The scale developed can serve as a useful tool for addressing difficulties in the peer-review process and improving its effectiveness and performance.

URL : Fundamental problems in the peer-review process and stakeholders’ perceptions of potential suggestions for improvement

DOI : https://doi.org/10.1002/leap.1637

Adoption and use of author identifier services: A French national survey

Posté le par

Authors : Christophe Boudry, Aline Bouchard

This paper studies awareness and use of author identifier services (AIDs) in the French academic community and explores needs and forms of support required for these tools, using a national questionnaire survey. ArXivID, IdHAL, ORCID, ResearcherID and Scopus Author ID were investigated. A total of 6125 people completed the questionnaire in full. The results of this survey show that discipline and age play an important role in French researchers’ familiarity with AIDs.

IdHAL and ORCID were by far the two best known AIDs, probably because they have been promoted by institutions in France for several years. French researchers use AIDs mainly to respond to external requests (e.g., to submit an article or a research project), while, surprisingly, few use them to ‘facilitate their work’.

When French researchers were asked about their needs and the form of support required for AIDs, more than 30% of them said they either required an introduction to or practical training in these tools. The results of this national survey should help stakeholders to adapt their policies and to guide and support researchers more efficiently in the use of these tools.

URL : Adoption and use of author identifier services: A French national survey

DOI : https://doi.org/10.1002/leap.1640