Étiquette : Kim Holmberg

Catégories
EN

Diverse roles of twitter in research evaluation: original tweets and retweets capture different types of engagements with scholarly articles

Authors :  Ashraf Maleki, Kim Holmberg

Altmetrics need to be more critically assessed in terms of the extent to which they reflect impact and quality of research compared to popularity or mere attention. Twitter (now rebranded as X) is a popular platform to, among other things, discuss and share scientific articles.

Earlier altmetric studies have often focused on investigating whether the number of tweets mentioning scientific articles could be used as an indicator of scientific impact or attention, with results showing weak to moderate correlations with citation counts. But all tweets may not be equal, as original tweets and retweets may reflect different levels of engagement and impact. Using a dataset of over 330,000 PLOS publications, this study explores whether these two forms of Twitter activity correlate differently with traditional citation metrics and how these relationships vary across disciplines.

The findings showed the correlation between citations and original tweets was consistently higher than that between citations and retweets and significant weak or moderate, but higher in Social Science and Humanities than in Natural Science, Engineering and Medicine fields. Also, including zero citation counts improved the correlation coefficients for original tweets, but reduced that of retweets.

This indicates that original tweets may be more aligned with citation counts as an indicator of scholarly impact, whereas retweets might reflect broader dissemination and popularity. In conclusion, tweets and retweets are different altmetric indicators and should be considered as two different metrics and analysed separately.

URL : Diverse roles of twitter in research evaluation: original tweets and retweets capture different types of engagements with scholarly articles

DOI : https://doi.org/10.1093/reseval/rvag014

Catégories
EN

Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?

Authors : Mike Thelwall, Marianna Lehtisaari, Irini Katsirea, Kim Holmberg, Er-Te Zheng

Large language models (LLMs) like ChatGPT seem to be increasingly used for information seeking and analysis, including to support academic literature reviews. To test whether the results might sometimes include retracted research, we identified 217 retracted or otherwise concerning academic studies with high altmetric scores and asked ChatGPT 4o-mini to evaluate their quality 30 times each.

Surprisingly, none of its 6510 reports mentioned that the articles were retracted or had relevant errors, and it gave 190 relatively high scores (world leading, internationally excellent, or close). The 27 articles with the lowest scores were mostly accused of being weak, although the topic (but not the article) was described as controversial in five cases (e.g., about hydroxychloroquine for COVID-19).

In a follow-up investigation, 61 claims were extracted from retracted articles from the set, and ChatGPT 4o-mini was asked 10 times whether each was true. It gave a definitive yes or a positive response two-thirds of the time, including for at least one statement that had been shown to be false over a decade ago.

The results therefore emphasise, from an academic knowledge perspective, the importance of verifying information from LLMs when using them for information seeking or analysis.

URL : Does ChatGPT Ignore Article Retractions and Other Reliability Concerns?

DOI : https://doi.org/10.1002/leap.2018

Catégories
EN

Who Are Tweeting About Academic Publications? A Cochrane Systematic Review and Meta-Analysis of Altmetric Studies

Authors : Ashraf Maleki, Kim Holmberg

Previous studies have developed different categorizations of Twitter users who interact with scientific publications online, reflecting the difficulty in creating a unified approach. Using Cochrane Review meta-analysis to analyse earlier research (including 79,014 Twitter users, over twenty million tweets, and over five million tweeted publications from 23 studies), we created a consolidated robust categorization consisting of 11 user categories, at different dimensions, covering most of any future needs for user categorizations on Twitter and possibly also other social media platforms.

Our findings showed, with moderate certainty, covering all the earlier different approaches employed, that the predominant Twitter group was individual users (66%), responsible for the majority of tweets (55%) and tweeted publications (50%), while organizations (22%, 27%, and 28%, respectively) and science communicators (16%, 13%, and 30%) clearly contributed smaller proportions.

The cumulative findings from prior investigations indicated a statistically equal extent of academic individuals (33%) and other individuals (28%). While academic individuals shared more academic publications than other individuals (42% vs. 31%), they posted fewer tweets overall (22% vs. 30%), but these differences do not reach statistical significance.

Despite significant heterogeneity arising from variations in categorization methods, the findings consistently indicate the importance of academics in disseminating academic publications.

URL : https://arxiv.org/abs/2312.06399