Catégorie : EN

Catégories
EN

Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

Authors : Kayvan Kousha, Mike Thelwall

Academics and departments are sometimes judged by how their research has benefited society. For example, the UK’s Research Excellence Framework (REF) assesses Impact Case Studies (ICSs), which are five-page evidence-based claims of societal impacts.

This article investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6220 public ICSs from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing ChatGPT’s predictions with published departmental average ICS scores.

The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text and to modify the original REF guidelines to encourage a stricter evaluation.

The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience).

At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.

URL : Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

DOI : https://doi.org/10.1002/asi.25021

Catégories
EN

Science of science — Citation models and research evaluation

Author : Vincent Traag

Citations in science are being studied from several perspectives, among which approaches such as scientometrics and science of science. In this chapter I briefly review some of the literature on citations, citation distributions and models of citations. These citations feature prominently in another part of the literature which is dealing with research evaluation and the role of metrics and indicators in that process.

Here I briefly review part of the discussion in research evaluation. This also touches on the subject of how citations relate to peer review. Finally, I conclude by trying to integrate the two literatures. The fundamental problem in research evaluation is that research quality is unobservable. This has consequences for conclusions that we can draw from quantitative studies of citations and citation models.

The term « indicators » is a relevant concept in this context, which I try to clarify. Causality is important for properly understanding indicators, especially when indicators are used in practice: when we act on indicators, we enter causal territory. Even when an indicator might have been valid, through its very use, the consequences of its use may invalidate it. By combining citation models with proper causal reasoning and acknowledging the fundamental problem about unobservable research quality, we may hope to make progress.

DOI : https://doi.org/10.48550/arXiv.2207.11116

Catégories
EN

Evolving library practice toward the sustainability of supporting open access

Authors : Maureen P Walsh, Miranda Bennett, Matthew W Goddard, Joshua Shelly

This paper is based on the Evolving Library Practice toward the Sustainability of Supporting Open Access panel presented at the 2024 NISO Plus Global/Online conference on September 17, 2024, and brings together four perspectives on how academic research library practices are evolving in response to developments in the global open access landscape.

The authors discuss current pain points in library support of open access publishing and explore how we might collectively work toward scalable and sustainable open access workflows.

URL : Evolving library practice toward the sustainability of supporting open access

DOI : https://doi.org/10.1177/18758789251339911

 

Catégories
EN

The African Platform for Open Scholarship Advancing Diamond Open Access and Inclusivity

Authors : Reggie Raju, Saiansha Maharaj

Geographic, language, peer review, and editorial biases have to be navigated by Global South authors to get published. Initially, the open access movement was praised for bridging the information access divide.

However, commercial publishers have hijacked the philanthropic ethos, turning it into a business model. Publishing charges add to the aforementioned biases, consolidating the exclusion of Global South scholarship. The African Platform for Open Scholarship developed by the University of Cape Town (UCT) counters these biases by offering free publishing infrastructure to advance the publishing of African scholarship without compromising academic rigor.

The platform adopts the diamond open access model to demarginalize Global South scholarship. Further, there is a discussion on the challenges and opportunities associated with creating an inclusive and equitable scholarly communication ecosystem. This paper focuses on UCT’s use of the platform to transition its commercial publishing arm (UCT Press) and to grow UCT Libraries Press. The paper will use exemplars to demonstrate the positive impact of these initiatives on the growth of diamond open access.

URL : The African Platform for Open Scholarship Advancing Diamond Open Access and Inclusivity

DOI : https://doi.org/10.31274/jlsc.18278

Catégories
EN

Implementation of Transformative Agreements at the University of Chicago Library: A Case Study

Authors : Jessica Harris, Greg Fleming, Jennifer Hart, Adrian K. Ho, Barbara Kern,
Catherine Mardikes, Debra A. Werner

The University of Chicago Library created a working group, composed of librarians across the library, to engage in dedicated and focused work around transformative agreements, including understanding how they work and how the library should be engaging with them in a complex open access (OA) landscape.

The working group was charged with specific tasks, including determining challenges and opportunities around transformative agreements, developing criteria for determining when to use the library’s OA fund to pursue an agreement, conducting ongoing assessments of the agreements, and developing a set of recommendations to communicate this out to our wider campus community.

The group’s work included piloting several transformative agreements and establishing a rubric to evaluate these agreements. The creation of the group allowed the library to gain valuable knowledge and expertise, engage actively in new models for supporting OA, and start critical conversations on campus.

The group continues the work, with the ultimate goal of affordable OA publishing and communicating the value of OA with researchers and campus partners, as well as positioning the library as a campus leader in OA.

URL : Implementation of Transformative Agreements at the University of Chicago Library: A Case Study

DOI : https://doi.org/10.31274/jlsc.18265

Catégories
EN

Understanding ORCID adoption among academic researchers

Author : Stephen R. Porter

Just over a decade ago, the ORCID (Open Researcher and Contributor Identifier) was created to provide a unique digital identifier for researchers around the world. The ORCID has proven essential in identifying individual researchers and their publications, both for bibliometric research analyses and for universities and other organizations tracking the research productivity and impact of their personnel.

Yet widespread adoption of the ORCID by individual researchers has proved elusive, with previous studies finding adoption rates ranging from 3% to 42%. Using a national survey of U.S. academic researchers at 31 research universities, we investigate why some researchers adopt an ORCID and some do not. We found an overall adoption rate of 72%, with adoptions rates ranging between academic disciplines from a low of 17% in the visual and performing arts to a high of 93% in biological and biomedical sciences.

Many academic journals require an ORCID to submit a manuscript, and this is the main reason why researchers adopt an ORCID. The top three reasons for not having an ORCID are not seeing the benefits, being far enough in the academic career to not need it, and working in an academic discipline where it is not needed.

URL : Understanding ORCID adoption among academic researchers

DOI : https://doi.org/10.1007/s11192-025-05300-7

Catégories
EN

Which kind of research papers influence policymaking

Author : Pablo Dorta-González

This study examines the use of evidence in policymaking by analysing a range of journal and article attributes, as well as online engagement metrics. It employs a large-scale citation analysis of nearly 150,000 articles covering diverse policy topics. The findings highlight that scholarly citations exert the strongest positive influence on policy citations.

Articles from journals with a higher citation impact and larger Mendeley readership are cited more frequently in policy documents. Other online engagements, such as news and blog mentions, also boost policy citations, while mentions on social media X have a negative effect.

The finding that highly cited and widely read papers are also frequently referenced in policy documents likely reflects the perception among policymakers that such research is more trustworthy. In contrast, papers that derive their influence primarily from social media tend to be cited less often in policy contexts.

DOI : https://doi.org/10.48550/arXiv.2504.18889