Étiquette : societal impact

Catégories
EN

And then a miracle occurs—a review of theory of change models for societal impact of research

Authors : Ole Henning Sørensen, Stine Dandanell Garn, Steffen Bohni Nielsen

Through an umbrella review, this article identified and surveyed 24 societal impact of research (SIR) models. Most of these models were developed within health domains and in Anglo-Saxon countries. The authors mapped the SIR models against constituent components of a robust theory of change.

The study found that logic models were predominantly used to conceive SIR models. Yet, only nine models had explicit causal links, and only two made explicit assumptions about why research contributes to societal change.

The old proverb among evaluators—when using theories of change to describe change—“and then a miracle occurs…,” rings uncomfortably true to the current state of SIR theorizing. Further theorizing and conceptual clarity are needed to advance the science of research impact.

URL : And then a miracle occurs—a review of theory of change models for societal impact of research

DOI : https://doi.org/10.1093/reseval/rvaf057

 

Catégories
EN

Assessing the Societal Impact of Academic Research With Artificial Intelligence (AI): A Scoping Review of Business School Scholarship as a ‘Force for Good’

Authors : David Steingard, Kathleen Rodenburg

This study addresses critical questions about how current evaluative frameworks for academic research can effectively translate scholarly findings into practical applications and policies to tackle societal ‘grand challenges’.

This scoping review analysis was conducted using bibliometric methods and AI tools. Articles were drawn from a wide range of disciplines, with particular emphasis on the business and management fields, focusing on the burgeoning scholarship area of ‘business as a force for good’.

The novel integration of generative AI research approaches underscores the transformative potential of AI-human collaboration in academic research. Metadata from 4051 articles were examined in the scoping review, with only 370 articles (9.1%) explicitly identified as relevant to societal impact.

This finding reveals a substantial and concerning gap in research addressing the urgent social and environmental issues of our time. To address this gap, the study identifies six meta-themes related to enhancing the societal impact of research: business applications; faculty publication pressure; societal impact focus; sustainable development; university and scholarly rankings; and reference to responsible research frameworks.

Key findings highlight critical misalignments between research outputs and the United Nations Sustainable Development Goals (SDGs) and a lack of practical business applications of research insights.

The results emphasise the urgent need for academic institutions to expand evaluation criteria beyond traditional metrics to prioritise real-world impacts. Recommendations include developing holistic evaluation frameworks and incentivising research that addresses pressing societal challenges—shifting academia from a ‘scholar-to-scholar’ to a ‘scholar-to-society’ paradigm.

The implications of this shift are applied to business-related scholarship and its potential to inspire meaningful societal impact through business practice.

URL : Assessing the Societal Impact of Academic Research With Artificial Intelligence (AI): A Scoping Review of Business School Scholarship as a ‘Force for Good’

DOI : https://doi.org/10.1002/leap.2010

Catégories
EN

Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

Authors : Kayvan Kousha, Mike Thelwall

Academics and departments are sometimes judged by how their research has benefited society. For example, the UK’s Research Excellence Framework (REF) assesses Impact Case Studies (ICSs), which are five-page evidence-based claims of societal impacts.

This article investigates whether ChatGPT can evaluate societal impact claims and therefore potentially support expert human assessors. For this, various parts of 6220 public ICSs from REF2021 were fed to ChatGPT 4o-mini along with the REF2021 evaluation guidelines, comparing ChatGPT’s predictions with published departmental average ICS scores.

The results suggest that the optimal strategy for high correlations with expert scores is to input the title and summary of an ICS but not the remaining text and to modify the original REF guidelines to encourage a stricter evaluation.

The scores generated by this approach correlated positively with departmental average scores in all 34 Units of Assessment (UoAs), with values between 0.18 (Economics and Econometrics) and 0.56 (Psychology, Psychiatry and Neuroscience).

At the departmental level, the corresponding correlations were higher, reaching 0.71 for Sport and Exercise Sciences, Leisure and Tourism. Thus, ChatGPT-based ICS evaluations are simple and viable to support or cross-check expert judgments, although their value varies substantially between fields.

URL : Assessing the societal influence of academic research with ChatGPT: Impact case study evaluations

DOI : https://doi.org/10.1002/asi.25021

Catégories
EN

The societal impact of Open Science: a scoping review

Authors : Nicki Lisa Cole, Eva Kormann, Thomas Klebel, Simon Apartis, Tony Ross-Hellauer

Open Science (OS) aims, in part, to drive greater societal impact of academic research. Government, funder and institutional policies state that it should further democratize research and increase learning and awareness, evidence-based policy-making, the relevance of research to society’s problems, and public trust in research. Yet, measuring the societal impact of OS has proven challenging and synthesized evidence of it is lacking.

This study fills this gap by systematically scoping the existing evidence of societal impact driven by OS and its various aspects, including Citizen Science (CS), Open Access (OA), Open/FAIR Data (OFD), Open Code/Software and others. Using the PRISMA Extension for Scoping Reviews and searches conducted in Web of Science, Scopus and relevant grey literature, we identified 196 studies that contain evidence of societal impact. The majority concern CS, with some focused on OA, and only a few addressing other aspects.

Key areas of impact found are education and awareness, climate and environment, and social engagement. We found no literature documenting evidence of the societal impact of OFD and limited evidence of societal impact in terms of policy, health, and trust in academic research. Our findings demonstrate a critical need for additional evidence and suggest practical and policy implications.

URL : https://doi.org/10.1098/rsos.240286

Catégories
EN

The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

Authors : Jorrit P Smit, Laurens K Hessels

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak.

This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective.

The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value.

While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process.

We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research.

Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

URL : The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

DOI : https://doi.org/10.1093/reseval/rvab002