Étiquette : research evaluation

Catégories
EN

Academic social networks metrics: an effective indicator for university performance?

Authors : Łukasz Wiechetek, Zbigniew Pastuszak

Social networks have become popular among researchers and scientists. Specialized platforms for researchers offer many metrics and indicators which are used to evaluate various scientists and assess the strength of their impact. In this article the authors perform systematic comparison between the main university level ResearchGate (RG) metrics: total RG Score, number of publications, number of affiliated profiles and ARWU.

A tool for acquiring the RG metrics of research units and a framework for calculating alternative university ranks was implemented and tested. As a point of reference the ranking system of the Academic Ranking of World Universities (ARWU, 2019) was used. The authors used a web scraping technique to acquire data. Data analysis was based on Spearman’s rho and multiple linear regression (MLR).

Ten additional ranks were developed and compared with the benchmark ranking. The k-means clustering method was used to identify the groups of ARWU universities. The research results show that the metrics provided by specialized social networks can be used for the assessment of universities, however, an in-depth evaluation requires a more advanced procedure and indicators to measure many areas of scholarly activity like research, integration, application, teaching, and co-creation.

Clustering method showed also that the distance between the ARWU universities measured in values of RG metrics are bigger for the top of the ranking. The university authorities should encourage researchers to use specialized social networks, and train them how to do it, to promote not only their own achievements, but also to increase the impact and recognition of their respective research units.

At the end of the article some limitations of the method used and some practical recommendations for the university authorities were formulated.

URL : Academic social networks metrics: an effective indicator for university performance?

DOI : https://doi.org/10.1007/s11192-021-04258-6

Catégories
EN

What Is Wrong With the Current Evaluative Bibliometrics?

Author : Endel Põder

Bibliometric data are relatively simple and describe objective processes of publishing articles and citing others. It seems quite straightforward to define reasonable measures of a researcher’s productivity, research quality, or overall performance based on these data. Why do we still have no acceptable bibliometric measures of scientific performance?

Instead, there are hundreds of indicators with nobody knowing how to use them. At the same time, an increasing number of researchers and some research fields have been excluded from the standard bibliometric analysis to avoid manifestly contradictive conclusions.

I argue that the current biggest problem is the inadequate rule of credit allocation for multiple authored articles in mainstream bibliometrics. Clinging to this historical choice excludes any systematic and logically consistent bibliometrics-based evaluation of researchers, research groups, and institutions.

During the last 50 years, several authors have called for a change. Apparently, there are no serious methodologically justified or evidence-based arguments in the favor of the present system.

However, there are intractable social, psychological, and economical issues that make adoption of a logically sound counting system almost impossible.

URL : What Is Wrong With the Current Evaluative Bibliometrics?

DOI : https://doi.org/10.3389/frma.2021.824518

Catégories
EN

Evaluation and Merit-Based Increase in Academia: A Case Study in the First Person

Author : Christine Musselin

This article provides a reflexive account of the process of defining and implementing a mechanism to evaluate a group of academics in a French higher education institution. The situation is a rather unusual case for France, as the assessed academics are not civil servants but are employed by their university and this evaluation leads to merit-based salary increases.

To improve and implement this strategy was one of the author’s tasks, when she was vice-president for research at the institution in this case.

The article looks at this experience retrospectively, emphasizing three issues of particular relevance in the context of discussions about valuation studies and management proposed in this symposium: (1) the decision to distinguish between different types of profiles and thus categorize, or to apply the same criteria to all; (2) the concrete forms of commensuration to be developed in order to be able to evaluate and rank individuals from different disciplines; (3) the quantification of qualitative appreciation, i.e. their transformation into merit-based salary increases.

URL : Evaluation and Merit-Based Increase in Academia: A Case Study in the First Person

DOI : https://doi.org/10.3384/VS.2001-5992.2021.8.2.73-88

Catégories
EN

The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

Authors : Jorrit P Smit, Laurens K Hessels

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak.

This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective.

The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value.

While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process.

We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research.

Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

URL : The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

DOI : https://doi.org/10.1093/reseval/rvab002

Catégories
EN

The impact of geographical bias when judging scientific studies

Authors : Marta Kowal, Piotr Sorokowski, Emanuel Kulczycki, Agnieszka Żelaźniewicz

The beauty of science lies within its core assumption that it seeks to identify the truth, and as such, the truth stands alone and does not depend on the person who proclaims it. However, people’s proclivity to succumb to various stereotypes is well known, and the scientific world may not be exceptionally immune to the tendency to judge a book by its cover.

An interesting example is geographical bias, which includes distorted judgments based on the geographical origin of, inter alia, the given work and not its actual quality or value. Here, we tested whether both laypersons (N = 1532) and scientists (N = 480) are prone to geographical bias when rating scientific projects in one of three scientific fields (i.e., biology, philosophy, or psychology).

We found that all participants favored more biological projects from the USA than China; in particular, expert biologists were more willing to grant further funding to Americans. In philosophy, however, laypersons rated Chinese projects as better than projects from the USA. Our findings indicate that geographical biases affect public perception of research and influence the results of grant competitions.

URL : The impact of geographical bias when judging scientific studies

DOI : https://doi.org/10.1007/s11192-021-04176-7

Catégories
FR

Comment sauver l’ouverture de la science ? l’évaluation

Auteur/Author : Denis Jerome

Les mondes de la recherche et celui des éditeurs encouragent une disponibilité des résultats de la recherche à tous et gratuitement. La transition vers une science ouverte se développe rapidement mais elle n’est pas sans poser de sérieux problèmes qui ne sont pas uniquement d’ordre budgétaire mais peuvent aussi porter atteinte à l’éthique et au bon fonctionnement de la recherche.

Les acteurs incontournables que sont les chercheurs individuellement ou via les sociétés savantes et les académies doivent reprendre le contrôle de cette transition en reconsidérant le rôle de l’évaluation qui est le nœud du problème. C’est la pratique de l’évaluation qu’il faut revoir.

URL : https://hal.archives-ouvertes.fr/hal-03291013

Catégories
EN

Researchers’ attitudes towards the h-index on Twitter 2007–2020: criticism and acceptance

Authors : Mike Thelwall, Kayvan Kousha

The h-index is an indicator of the scientific impact of an academic publishing career. Its hybrid publishing/citation nature and inherent bias against younger researchers, women, people in low resourced countries, and those not prioritizing publishing arguably give it little value for most formal and informal research evaluations.

Nevertheless, it is well-known by academics, used in some promotion decisions, and is prominent in bibliometric databases, such as Google Scholar. In the context of this apparent conflict, it is important to understand researchers’ attitudes towards the h-index.

This article used public tweets in English to analyse how scholars discuss the h-index in public: is it mentioned, are tweets about it positive or negative, and has interest decreased since its shortcomings were exposed?

The January 2021 Twitter Academic Research initiative was harnessed to download all English tweets mentioning the h-index from the 2006 start of Twitter until the end of 2020. The results showed a constantly increasing number of tweets.

Whilst the most popular tweets unapologetically used the h-index as an indicator of research performance, 28.5% of tweets were critical of its simplistic nature and others joked about it (8%). The results suggest that interest in the h-index is still increasing online despite scientists willing to evaluate the h-index in public tending to be critical.

Nevertheless, in limited situations it may be effective at succinctly conveying the message that a researcher has had a successful publishing career.

DOI : https://doi.org/10.1007/s11192-021-03961-8