Étiquette : research evaluation

Catégories
FR

Le partage des données vu par les chercheurs : une approche par la valeur

Auteur/Author : Violaine Rebouillat

Le propos de cet article porte sur la compréhension des logiques qui interviennent dans la définition de la valeur des données de la recherche, celles-ci pouvant avoir une influence sur les critères déterminant leur motivation au partage.

L’approche méthodologique repose sur une enquête qualitative, menée dans le cadre d’une recherche doctorale, qui a déployé 57 entretiens semi-directifs. Alors que les travaux menés autour des données sont focalisés sur les freins et motivations du partage, l’originalité de cette recherche consiste à identifier les différents prismes par lesquels la question de la valeur des données impacte la motivation et la décision de leur partage.

L’analyse des résultats montre que, tous domaines confondus, la valeur des données reste encore cristallisée autour de la publication et de la reconnaissance symbolique du travail du chercheur.

Les résultats permettent de comprendre que la question du partage est confrontée à un impensé : celui du cadre actuel de l’évaluation de la recherche, qui met l’article scientifique au cœur de son dispositif.

Ce travail contribue donc à montrer que l’avenir du partage des données dépend des systèmes alternatifs futurs d’évaluation de la recherche, associés à la science ouverte.

URL : https://lesenjeux.univ-grenoble-alpes.fr/2021/varia/03-le-partage-des-donnees-vu-par-les-chercheurs-une-approche-par-la-valeur/

Catégories
EN

How faculty define quality, prestige, and impact in research

Authors : Esteban Morales, Erin McKiernan, Meredith T. Niles, Lesley Schimanski, Juan Pablo Alperin

Despite the calls for change, there is significant consensus that when it comes to evaluating publications, review, promotion, and tenure processes should aim to reward research that is of high “quality,” has an “impact,” and is published in “prestigious” journals.

Nevertheless, such terms are highly subjective and present challenges to ascertain precisely what such research looks like. Accordingly, this article responds to the question: how do faculty from universities in the United States and Canada define the terms quality, prestige, and impact?

We address this question by surveying 338 faculty members from 55 different institutions. This study’s findings highlight that, despite their highly varied definitions, faculty often describe these terms in overlapping ways. Additionally, results shown that marked variance in definitions across faculty does not correspond to demographic characteristics.

This study’s results highlight the need to more clearly implement evaluation regimes that do not rely on ill-defined concepts.

DOI : https://doi.org/10.1101/2021.04.14.439880

Catégories
EN

Research impact evaluation and academic discourse

Author : Marta Natalia Wróblewska

The introduction of ‘impact’ as an element of assessment constitutes a major change in the construction of research evaluation systems. While various protocols of impact evaluation exist, the most articulated one was implemented as part of the British Research Excellence Framework (REF).

This paper investigates the nature and consequences of the rise of ‘research impact’ as an element of academic evaluation from the perspective of discourse. Drawing from linguistic pragmatics and Foucauldian discourse analysis, the study discusses shifts related to the so-called Impact Agenda on four stages, in chronological order: (1) the ‘problematization’ of the notion of ‘impact’, (2) the establishment of an ‘impact infrastructure’, (3) the consolidation of a new genre of writing–impact case study, and (4) academics’ positioning practices towards the notion of ‘impact’, theorized here as the triggering of new practices of ‘subjectivation’ of the academic self.

The description of the basic functioning of the ‘discourse of impact’ is based on the analysis of two corpora: case studies submitted by a selected group of academics (linguists) to REF2014 (no = 78) and interviews (n = 25) with their authors.

Linguistic pragmatics is particularly useful in analyzing linguistic aspects of the data, while Foucault’s theory helps draw together findings from two datasets in a broader analysis based on a governmentality framework. This approach allows for more general conclusions on the practices of governing (academic) subjects within evaluation contexts.

URL : Research impact evaluation and academic discourse

DOI : https://doi.org/10.1057/s41599-021-00727-8

Catégories
EN

Rethinking success, integrity, and culture in research (part 2) — a multi-actor qualitative study on problems of science

Authors : Noémie Aubert Bonn, Wim Pinxten

Background

Research misconduct and questionable research practices have been the subject of increasing attention in the past few years. But despite the rich body of research available, few empirical works also include the perspectives of non-researcher stakeholders.

Methods

We conducted semi-structured interviews and focus groups with policy makers, funders, institution leaders, editors or publishers, research integrity office members, research integrity community members, laboratory technicians, researchers, research students, and former-researchers who changed career to inquire on the topics of success, integrity, and responsibilities in science.

We used the Flemish biomedical landscape as a baseline to be able to grasp the views of interacting and complementary actors in a system setting.

Results

Given the breadth of our results, we divided our findings in a two-paper series with the current paper focusing on the problems that affect the integrity and research culture. We first found that different actors have different perspectives on the problems that affect the integrity and culture of research.

Problems were either linked to personalities and attitudes, or to the climates in which researchers operate. Elements that were described as essential for success (in the associate paper) were often thought to accentuate the problems of research climates by disrupting research culture and research integrity.

Even though all participants agreed that current research climates need to be addressed, participants generally did not feel responsible nor capable of initiating change. Instead, respondents revealed a circle of blame and mistrust between actor groups.

Conclusions

Our findings resonate with recent debates, and extrapolate a few action points which might help advance the discussion.

First, the research integrity debate must revisit and tackle the way in which researchers are assessed.

Second, approaches to promote better science need to address the impact that research climates have on research integrity and research culture rather than to capitalize on individual researchers’ compliance.

Finally, inter-actor dialogues and shared decision making must be given priority to ensure that the perspectives of the full research system are captured. Understanding the relations and interdependency between these perspectives is key to be able to address the problems of science.

URL : Rethinking success, integrity, and culture in research (part 2) — a multi-actor qualitative study on problems of science

DOI : https://doi.org/10.1186/s41073-020-00105-z

Catégories
EN

Rethinking success, integrity, and culture in research (part 1) — a multi-actor qualitative study on success in science

Authors : Noémie Aubert Bonn, Wim Pinxten

Background

Success shapes the lives and careers of scientists. But success in science is difficult to define, let alone to translate in indicators that can be used for assessment. In the past few years, several groups expressed their dissatisfaction with the indicators currently used for assessing researchers.

But given the lack of agreement on what should constitute success in science, most propositions remain unanswered. This paper aims to complement our understanding of success in science and to document areas of tension and conflict in research assessments.

Methods

We conducted semi-structured interviews and focus groups with policy makers, funders, institution leaders, editors or publishers, research integrity office members, research integrity community members, laboratory technicians, researchers, research students, and former-researchers who changed career to inquire on the topics of success, integrity, and responsibilities in science.

We used the Flemish biomedical landscape as a baseline to be able to grasp the views of interacting and complementary actors in a system setting.

Results

Given the breadth of our results, we divided our findings in a two-paper series, with the current paper focusing on what defines and determines success in science. Respondents depicted success as a multi-factorial, context-dependent, and mutable construct.

Success appeared to be an interaction between characteristics from the researcher (Who), research outputs (What), processes (How), and luck. Interviewees noted that current research assessments overvalued outputs but largely ignored the processes deemed essential for research quality and integrity.

Interviewees suggested that science needs a diversity of indicators that are transparent, robust, and valid, and that also allow a balanced and diverse view of success; that assessment of scientists should not blindly depend on metrics but also value human input; and that quality should be valued over quantity.

Conclusions

The objective of research assessments may be to encourage good researchers, to benefit society, or simply to advance science. Yet we show that current assessments fall short on each of these objectives. Open and transparent inter-actor dialogue is needed to understand what research assessments aim for and how they can best achieve their objective.

URL : Rethinking success, integrity, and culture in research (part 1) — a multi-actor qualitative study on success in science

DOI : https://doi.org/10.1186/s41073-020-00104-0

Catégories
FR

L’effet SIGAPS : La recherche médicale française sous l’emprise de l’évaluation comptable

Auteurs/Authors : Yves Gingras, Mahdi Khelfaoui

Cette recherche a pour but de mettre en évidence les effets pervers générés par l’introduction du système SIGAPS (Système d’interrogation, de gestion, et d’analyse des publications scientifiques) sur la production scientifique française en médecine et en sciences biomédicales.

Cet outil biblio-métrique de gestion et de financement de la recherche présente un exemple emblématique des dé-rives que peuvent générer les méthodes d’évaluation de la recherche reposant sur des critères pu-rement comptables.

Dans cette note, nous présentons d’abord le fonctionnement de SIGAPS, pour ensuite expliquer précisément en quoi les méthodes de calcul des « points SIGAPS », fondés sur les facteurs d’impact des revues et l’ordre des noms des co-auteurs, posent de nombreux problèmes.

Nous identifions notamment les effets du système SIGAPS sur les dynamiques de publications, les choix des lieux de publications, la langue de publication et les critères de recrutement et de promotion des chercheurs.

Finalement, nous montrons que l’utilisation du système SIGAPS ne répond pas bien à tous les critères de ce que l’on pourrait appeler une « éthique de l’évaluation » qui devrait respecter certaines règles, comme la transparence, l’équité et la validité des indicateurs.

URL : https://cirst2.openum.ca/files/sites/179/2020/10/Note_2020-05vf.pdf

Catégories
EN

Changing how we evaluate research is difficult, but not impossible

Authors : Anna Hatch, Stephen Curry

The San Francisco Declaration on Research Assessment (DORA) was published in 2013 and described how funding agencies, institutions, publishers, organizations that supply metrics, and individual researchers could better evaluate the outputs of scientific research.

Since then DORA has evolved into an active initiative that gives practical advice to institutions on new ways to assess and evaluate research. This article outlines a framework for driving institutional change that was developed at a meeting convened by DORA and the Howard Hughes Medical Institute.

The framework has four broad goals: understanding the obstacles to changes in the way research is assessed; experimenting with different approaches; creating a shared vision when revising existing policies and practices; and communicating that vision on campus and beyond.

URL : Changing how we evaluate research is difficult, but not impossible

DOI : https://doi.org/10.7554/eLife.58654