Archives des mots-clés : metrics

Deep Impact: A Study on the Impact of Data Papers and Datasets in the Humanities and Social Sciences

Posté le par

Authors : Barbara McGillivray, Paola Marongiu, Nilo Pedrazzini, Marton Ribary, Mandy Wigdorowitz, Eleonora Zordan

The humanities and social sciences (HSS) have recently witnessed an exponential growth in data-driven research. In response, attention has been afforded to datasets and accompanying data papers as outputs of the research and dissemination ecosystem.

In 2015, two data journals dedicated to HSS disciplines appeared in this landscape: Journal of Open Humanities Data (JOHD) and Research Data Journal for the Humanities and Social Sciences (RDJ).

In this paper, we analyse the state of the art in the landscape of data journals in HSS using JOHD and RDJ as exemplars by measuring performance and the deep impact of data-driven projects, including metrics (citation count; Altmetrics, views, downloads, tweets) of data papers in relation to associated research papers and the reuse of associated datasets.

Our findings indicate: that data papers are published following the deposit of datasets in a repository and usually following research articles; that data papers have a positive impact on both the metrics of research papers associated with them and on data reuse; and that Twitter hashtags targeted at specific research campaigns can lead to increases in data papers’ views and downloads.

HSS data papers improve the visibility of datasets they describe, support accompanying research articles, and add to transparency and the open research agenda.

URL : Deep Impact: A Study on the Impact of Data Papers and Datasets in the Humanities and Social Sciences

DOI : https://doi.org/10.3390/publications10040039

Implementing the Declaration on Research Assessment: a publisher case study

Posté le par

Authors : Victoria Gardner, Mark Robinson, Elisabetta O’Connell

There has been much debate around the role of metrics in scholarly communication, with particular focus on the misapplication of journal metrics, such as the impact factor in the assessment of research and researchers.

Various initiatives have advocated for a change in this culture, including the Declaration on Research Assessment (DORA), which invites stakeholders throughout the scholarly communication ecosystem to sign up and show their support for practices designed to address the misuse of metrics.

This case study provides an overview of the process undertaken by a large academic publisher (Taylor & Francis Group) in signing up to DORA and implementing some of its key practices in the hope that it will provide some guidance to others considering becoming a signatory.

Our experience suggests that research, consultation and flexibility are crucial components of the process. Additionally, approaching signing with a project mindset versus a ‘sign and forget’ mentality can help organizations to understand the practical implications of signing, to anticipate and mitigate potential obstacles and to support cultural change.

URL : Implementing the Declaration on Research Assessment: a publisher case study

DOI : http://doi.org/10.1629/uksg.573

Toward More Inclusive Metrics and Open Science to Measure Research Assessment in Earth and Natural Sciences

Posté le par

Authors : Olivier Pourret, Dasapta Erwin Irawan, Najmeh Shaghaei, Elenora M. van Rijsingen, Lonni Besançon

The conventional assessment of scientists relies on a set of metrics which are mostly based on the production of scientific articles and their citations. These metrics are primarily established at the journal level (e.g., the Journal Impact Factor), the article-level (e.g., times cited), and the author level (e.g., h-index; Figure 1).

These metrics form the basis of criteria that have been widely used to measure institutional reputation, as well as that of authors and research groups. By relying mostly on citations (Langfeldt et al., 2021), however, they are inherently flawed in that they provide only a limited picture of scholarly production. Indeed, citations only count document use within scholarly works and thus provide a very limited view of the use and impact of an article.

Those reveal only the superficial dimensions of a research’s impact on society. Even within academia, citations are limited since the link they express does not hold any value (Tennant et al., 2019). As an example, one could be cited for the robustness of the presented work while the other could be cited for its main limitation (Aksnes et al., 2019).

As such, two articles could be cited the same number of times for very different reasons, and relying on citations to evaluate scientific work therefore displays obvious limitations (Tahamtan et al., 2016). Beyond this issue, however, the conventional assessment of scientists is clearly beneficial to some scientists more than others and does not reflect or encourage the dissemination of knowledge back to the public that is ultimately paying scientists.

This is visible in the Earth and natural sciences which has been organized to solve local community problems in dealing with the Earth system like groundwater hazards (Irawan et al., 2021; Dwivedi et al., 2022).

URL: Toward More Inclusive Metrics and Open Science to Measure Research Assessment in Earth and Natural Sciences

DOI : https://doi.org/10.3389/frma.2022.850333

RipetaScore: Measuring the Quality, Transparency, and Trustworthiness of a Scientific Work

Posté le par

Authors : Josh Q. Sumner, Cynthia Hudson Vitale, Leslie D. McIntosh

A wide array of existing metrics quantifies a scientific paper’s prominence or the author’s prestige. Many who use these metrics make assumptions that higher citation counts or more public attention must indicate more reliable, better quality science.

While current metrics offer valuable insight into scientific publications, they are an inadequate proxy for measuring the quality, transparency, and trustworthiness of published research.

Three essential elements to establishing trust in a work include: trust in the paper, trust in the author, and trust in the data. To address these elements in a systematic and automated way, we propose the ripetaScore as a direct measurement of a paper’s research practices, professionalism, and reproducibility.

Using a sample of our current corpus of academic papers, we demonstrate the ripetaScore’s efficacy in determining the quality, transparency, and trustworthiness of an academic work.

In this paper, we aim to provide a metric to evaluate scientific reporting quality in terms of transparency and trustworthiness of the research, professionalism, and reproducibility.

URL : RipetaScore: Measuring the Quality, Transparency, and Trustworthiness of a Scientific Work

DOI : https://doi.org/10.3389/frma.2021.751734

Perspectives on Open Science and The Future of Scholarly Communication: Internet Trackers and Algorithmic Persuasion

Posté le par

Authors : Tiberius Ignat, Paul Ayris, Beatrice Gini, Olga Stepankova, Deniz Özdemir, Damla Bal, Yordanka Deyanov

The current digital content industry is heavily oriented towards building platforms that track users’ behaviour and seek to convince them to stay longer and come back sooner onto the platform. Similarly, authors are incentivised to publish more and to become champions of dissemination.

Arguably, these incentive systems are built around public reputation supported by a system of metrics, hard to be assessed. Generally, the digital content industry is permeable to non-human contributors (algorithms that are able to generate content and reactions), anonymity and identity fraud. It is pertinent to present a perspective paper about early signs of track and persuasion in scholarly communication.

Building our views, we have run a pilot study to determine the opportunity for conducting research about the use of “track and persuade” technologies in scholarly communication. We collected observations on a sample of 148 relevant websites and we interviewed 15 that are experts related to the field.

Through this work, we tried to identify 1) the essential questions that could inspire proper research, 2) good practices to be recommended for future research, and 3) whether citizen science is a suitable approach to further research in this field.

The findings could contribute to determining a broader solution for building trust and infrastructure in scholarly communication. The principles of Open Science will be used as a framework to see if they offer insights into this work going forward.

URL : Perspectives on Open Science and The Future of Scholarly Communication: Internet Trackers and Algorithmic Persuasion

DOI : https://doi.org/10.3389/frma.2021.748095

Why does library holding format really matter for book impact assessment?: Modelling the relationship between citations and altmetrics with print and electronic holdings

Posté le par

Author : Ashraf Maleki

Scholarly books are important outputs in some fields and their many publishing formats seem to introduce opportunities to scrutinize their impact. As there is a growing interest in the publisher-enforced massive collection of ebooks in libraries in the past decade, this study examined how this influences the relationship that library print holdings (LPH), library electronic holdings (LEH) and total library holdings (TLH) have with other metrics.

As a follow up study to a previous research on OCLC library holdings, the relationship between library holdings and twelve other metrics including Scopus Citations, Google Books (GB) Citations, Goodreads engagements, and Altmetric indicators were examined for 119,794 Scopus-indexed book titles across 26 fields.

Present study confirms the weak correlation levels observed between TLH and other indicators in previous studies and contributes additional evidence that print holdings can moderately reflect research, educational and online impact of books consistently more efficient than eholdings and total holdings across fields and over time, except for Mendeley for which eholdings slightly prevailed.

Regression models indicated that along with other dimensions, Google Books Citations frequently best explained LPH (in 14 out of 26 fields), whereas Goodreads User counts were weak, but the best predictor of both LEH and TLH (in 15 fields out of 26), suggesting significant association of eholdings with online uptake of books.

Overall, findings suggest that inclusion of eholdings overrides the more impactful counts of print holdings in Total Library Holdings metric and therefore undermines the statistical results, whilst print holdings has both statistically and theoretically promising underlying assumptions for prediction of impact of books and shows greater promise than the general Library Holding metric for book impact assessment.

Thus, there is a need for a distinction between print and electronic holding counts to be made, otherwise total library holding data need to be interpreted with caution.

URL : Why does library holding format really matter for book impact assessment?: Modelling the relationship between citations and altmetrics with print and electronic holdings

DOI : https://doi.org/10.1007/s11192-021-04239-9