How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation

Authors : Cinzia Daraio, Alessio Vaccari

In this paper we answer the question of how evaluation should be by proposing a good evaluation of research practices. A good evaluation of research practices, intended as social practices à la MacIntyre, should take into account the stable motivations and the traits of the characters (i.e. the virtues) of researchers.

We also show that a good evaluation is also just, beyond the sense of fairness, as working on good research practices implies keep into account a broader sense of justice. After that, we propose the development of a knowledge base for the assessment of “good” evaluations of research practices to implement a questionnaire for the assessment of researchers’ virtues.

Although the latter is a challenging task, the use of ontologies and taxonomic knowledge, and the reasoning algorithms that can draw inferences on the basis of such knowledge represents a way for testing the consistency of the information reported in the questionnaire and to analyse correctly and coherently how the data is gathered through it.

Finally, we describe the potential application usefulness of our proposal for the reform of current research assessment systems.

URL : How should evaluation be? Is a good evaluation of research also just? Towards the implementation of good evaluation

DOI : https://doi.org/10.1007/s11192-022-04329-2

A Framework for the Assessment of Research and Its Impacts

Author : Cinzia Daraio

This paper proposes a holistic framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions, including in an original way, data as a main dimension, together with theory and methodology.

Each dimension of the framework is further characterized by three main building blocks: education, research, and innovation (theory); efficiency, effectiveness, and impact (methodology); and availability, interoperability, and “unit-free” property (data).

The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as “quality.” Three additional quality attributes are identified as implementation factors (tailorability, transparency, and openness) and three “enabling” conditions (convergence, mixed methods, and knowledge infrastructures) complete the framework.

A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation.

It can act as a common denominator for different analytical levels and relevant aspects and is able to embrace many different and heterogeneous streams of literature. Directions for future research are provided.

URL : A Framework for the Assessment of Research and Its Impacts

DOI : https://doi.org/10.1515/jdis-2017-0018