A “basket of metrics”—the best support for understanding journal merit

Authors : Lisa Colledge, Chris James

Aim

To survey opinion of the assertion that useful metricbased input requires a “basket of metrics” to allow more varied and nuanced insights into merit than is possible by using one metric alone.

Methods

A poll was conducted to survey opinions (N=204; average response rate=61%) within the international research community on using usage metrics in merit systems.

Results

“Research is best quantified using multiple criteria” was selected by most (40%) respondents as the reason that usage metrics are valuable, and 95% of respondents indicated that they would be likely or very likely to use usage metrics in their assessments of research merit, if they had access to them.

There was a similar degree of preference for simple and sophisticated usage metrics confirming that one size does not fit all, and that a one-metric approach to merit is insufficient.

Conclusion

This survey demonstrates a clear willingness and a real appetite to use a “basket of metrics” to broaden the ways in which research merit can be detected and demonstrated.

URL : http://europeanscienceediting.eu/articles/a-basket-of-metrics-the-best-support-for-understanding-journal-merit/

How significant are the public dimensions of faculty work in review, promotion, and tenure documents?

Authors : Juan Pablo Alperin, Gustavo E. Fischman, Erin C. McKiernan, Carol Muñoz Nieves, Meredith T. Niles, Lesley Schimanski

Much of the work of universities, even private institutions, has significant public dimensions. Faculty work in particular is often funded by public funds, is aimed at serving the public good, and is subject to public evaluation.

To understand how the public dimensions of faculty work are valued, we analyzed review, tenure and promotion documents from a representative sample of 129 Canadian and American universities.

We found terms and concepts related to public and community are mentioned in a large portion of documents, but mostly in ways that relate to service—an undervalued aspect of academic careers.

Moreover, we find significant mentions of traditional research outputs and citation-based metrics. Such outputs and metrics reward faculty work targeted to academics, and mostly disregard the public dimensions.

We conclude that institutions that want to live up to their public mission need to work towards systemic change in how faculty work is assessed and incentivized.

URL : How significant are the public dimensions of faculty work in review, promotion, and tenure documents?

DOI : https://hcommons.org/deposits/item/hc:21015

Do all citations value the same? Valuing citations by the value of the citing items

Authors : Cristiano Giuffrida, Giovanni Abramo, Ciriaco Andrea D’Angelo

Bibliometricians have long recurred to citation counts to measure the impact of publications on the advancement of science. However, since the earliest days of the field, some scholars have questioned whether all citations should value the same, and have gone on to weight them by a variety of factors.

However sophisticated the operationalization of the measures, the methodologies used in weighting citations still present limits in their underlying assumptions. This work takes an alternate approach to resolving the underlying problem: the proposal is to value citations by the impact of the citing articles.

As well as conceptualizing a new indicator of impact, the work illustrates its application to the 2004-2012 Italian scientific production indexed in the WoS.

The new indicator appears highly correlated to traditional field normalized citations, however the shifts observed between the two measures are frequent and the number of outliers not at all negligible. Moreover, the new indicator seems to show greater “sensitivity” when used in identification of the top-cited papers.

URL : https://arxiv.org/abs/1809.06088

Mesurer la science

Auteurs/Authors : Vincent Larivière, Cassidy R. Sugimoto

L’ensemble de la communauté scientifique réclame depuis plusieurs années des indicateurs fiables permettant de mesurer les répercussions de la recherche. La ferveur inégalée autour de la mesure de l’influence de la recherche, combinée avec les nouveaux modes de diffusion des connaissances à l’ère numérique, a révolutionné le domaine de la scientométrie.

Il s’agit là d’une discipline qui comprend toutes les façons dont nous collectons les documents savants et analysons quantitativement leur production ainsi que leurs usages, des citations aux tweets. Les données et les indicateurs ainsi recueillis sont utilisés pour comprendre la science, stimuler la recherche ou distribuer les ressources.

Curieusement, il n’existe aucun ouvrage qui explique les fondements historiques, les concepts et les sources de la scientométrie, ou qui en fournirait une critique éclairée ou même qui formulerait des recommandations pour un usage optimal. D’où l’importance de celui-ci.

À sa façon, chacun est un acteur de la société du savoir et devrait se soucier des outils qui aident à guider son évolution : c’est pourquoi ce livre s’adresse à tous, savants comme profanes.

URL : https://pum.umontreal.ca/catalogue/mesurer-la-science/

Systematic analysis of agreement between metrics and peer review in the UK REF

Authors : Vincent Traag, Ludo Waltman

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found.

However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review.

We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review.

In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF.

The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review.

To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

URL : https://arxiv.org/abs/1808.03491

Universalizing science: alternative indices to direct research

Authors : Ari Melo Mariano, Maíra Rocha Santos

Measurement is a complicated but very necessary task. Many indices have been created in an effort to define the quality of knowledge produced but they have attracted strong criticism, having become synonymous with individualism, competition and mere productivity and, furthermore, they fail to head science towards addressing local demands or towards producing international knowledge by means of collaboration.

Institutions, countries, publishers, governments and authors have a latent need to create quality and productivity indices because they can serve as filters that influence far-reaching decision making and even decisions on the professional promotion of university teachers.

Even so, in the present-day context, the very creators of those indices admit that they were not designed for that purpose, given that different research areas, the age of the researcher, the country and the language spoken all have an influence on the index calculations.

Accordingly, this research sets out three indices designed to head science towards its universal objective by valuing collaboration and the dissemination of knowledge.

It is hoped that the proposed indices may provoke new discussions and the proposal of new, more assertive indicators for the analysis of scientific research quality.

URL : https://arxiv.org/abs/1807.07595

Collecting Inclusive Usage Metrics for Open Access Publications: the HIRMEOS Project

Author : Javier Arias

Open Access has matured for journals, but its uptake in the book market is still delayed, despite the fact that books continue to be the leading publishing format for social sciences and humanities.

The 30-months EU-funded project HIRMEOS (High Integration of Research Monographs in the European Open Science infrastructure) tackles the main obstacles of the full integration of five important digital platforms supporting open access monographs.

The content of participating platforms will be enriched with tools that enable identification, authentication and interoperability (via DOI, ORCID, Fundref), and tools that enrich information and entity extraction (INRIA (N)ERD), the ability to annotate monographs (Hypothes.is), and gather usage and alternative metric data.

This paper focuses on the development and implementation of Open Source Metrics Services that enable the collection of OA Metrics and Altmetrics from third-party platforms, and how the architecture of these tools will allow implementation in any external platform, particularly in start-up Open Access publishers.

URL : https://hal.archives-ouvertes.fr/hal-01816811