Enriching research quality: A proposition for stakeholder heterogeneity

Authors : Thomas Franssen

Dominant approaches to research quality rest on the assumption that academic peers are the only relevant stakeholders in its assessment. In contrast, impact assessment frameworks recognize a large and heterogeneous set of actors as stakeholders. In transdisciplinary research non-academic stakeholders are actively involved in all phases of the research process and actor-network theorists recognize a broad and heterogeneous set of actors as stakeholders in all types of research as they are assigned roles in the socio-material networks, also termed ‘problematizations’, that researchers reconfigure.

Actor-network theorists consider research as a performative act that changes the reality of the stakeholders it, knowingly or unknowingly, involves. Established approaches to, and notions of, research quality do not recognize the heterogeneity of relevant stakeholders nor allow for reflection on the performative effects of research.

To enrich the assessment of research quality this article explores the problematization as a potential new object of evaluation. Problematizations are proposals for how the future might look. Hence, their acceptance does not only concern fellow academics but also all other human and other-than-human actors that figure in them.

To enrich evaluative approaches, this article argues for the inclusion of stakeholder involvement and stakeholder representation as dimensions of research quality.

It considers a number of challenges to doing so including the identification of stakeholders, developing quality criteria for stakeholder involvement and stakeholder representation, and the possibility of participatory research evaluation. It can alternatively be summarized as raising the question: for whose benefit do we conduct evaluations of research quality?

URL : Enriching research quality: A proposition for stakeholder heterogeneity

DOI : https://doi.org/10.1093/reseval/rvac012

Open access and its potential impact on public health – A South African perspective

Authors : Adéle Strydom, Juanita Mellet, Jeanne Van Rensburg, Ignatius Viljoen, Anastasios Athanasiadis, Michael S. Pepper

Traditionally, access to research information has been restricted through journal subscriptions. This means that research entities and individuals who were unable to afford subscription costs did not have access to journal articles. There has however been a progressive shift toward electronic access to journal publications and subsequently growth in the number of journals available globally.

In the context of electronic journals, both open access and restricted access options exist. While the latter option is comparable to traditional, subscription-based paper journals, open access journal publications follow an “open science” publishing model allowing scholarly communications and outputs to be publicly available online at no cost to the reader.

However, for readers to enjoy open access, publication costs are shifted elsewhere, typically onto academic institutions and authors. SARS-CoV-2, and the resulting COVID-19 pandemic have highlighted the benefits of open science through accelerated research and unprecedented levels of collaboration and data sharing. South Africa is one of the leading open access countries on the African continent.

This paper focuses on open access in the South African higher education research context with an emphasis on our Institution and our own experiences. It also addresses the financial implications of open access and provides possible solutions for reducing the cost of publication for researchers and their institutions.

Privacy in open access and the role of the Protection of Personal Information Act (POPIA) in medical research and secondary use of data in South Africa will also be discussed.

URL : Open access and its potential impact on public health – A South African perspective

DOI : https://doi.org/10.3389/frma.2022.975109

The impact of open and reproducible scholarship on students’ scientific literacy, engagement, and attitudes towards science: A review and synthesis of the evidence

Authors : Madeleine Pownall, Flávio Azevedo, Laura M. König, et al. 

In recent years, the scientific community has called for improvements in the credibility, robustness, and reproducibility of research, characterized by higher standards of scientific evidence, increased interest in open practices, and promotion of transparency. While progress has been positive, there is a lack of consideration about how this approach can be embedded into undergraduate and postgraduate research training.

Currently, the impact of integrating an open and reproducible approach into the curriculum on student outcomes is not well articulated in the literature.

Therefore, in this paper, we provide the first comprehensive review of how integrating open and reproducible scholarship into teaching and learning may impact students, using a large-scale, collaborative, team-science approach. Our review highlighted how embedding open and reproducible scholarship may impact: (1) students’ scientific literacies (i.e., students’ understanding of open research, consumption of science, and the development of transferable skills); (2) student engagement (i.e., motivation and engagement with learning, collaboration, and engagement in open research), and (3) students’ attitudes towards science (i.e., trust in science and confidence in research findings).

Our review also identified a need for more robust and rigorous methods within evaluations of teaching practice. We discuss implications for teaching and learning scholarship in this area.

URL : The impact of open and reproducible scholarship on students’ scientific literacy, engagement, and attitudes towards science: A review and synthesis of the evidence

DOI : https://doi.org/10.31222/osf.io/9e526

Analysing Elsevier Journal Metadata with a New Specialized Workbench inside ICSR Lab

Authors : Ramadurai Petchiappan, Kristy James, Andrew Plume, Efthymios Tsakonas, Ana Marušić, Mario Malicki, Francisco Grimaldo, Bahar Mehmani

In this white paper we introduce Elsevier’s Peer Review Workbench which will be available via the computational platform ICSR Lab. The workbench offers a unique dataset to interested researchers who want to run research on journal evaluation and peer review processes.

We describe its properties, advantages, and limitations as well as the process of proposal application. This is a living document and will be updated on a regular basis.

DOI : https://dx.doi.org/10.2139/ssrn.4211833

Reading Peer Review : PLOS ONE and Institutional Change in Academia

Authors : Martin Paul Eve, Cameron Neylon, Daniel Paul O’Donnell, Samuel Moore, Robert Gadie, Victoria Odeniyi, Shahina Parvin

This Element describes for the first time the database of peer review reports at PLOS ONE, the largest scientific journal in the world, to which the authors had unique access.

Specifically, this Element presents the background contexts and histories of peer review, the data-handling sensitivities of this type of research, the typical properties of reports in the journal to which the authors had access, a taxonomy of the reports, and their sentiment arcs.

This unique work thereby yields a compelling and unprecedented set of insights into the evolving state of peer review in the twenty-first century, at a crucial political moment for the transformation of science.

It also, though, presents a study in radicalism and the ways in which PLOS’s vision for science can be said to have effected change in the ultra-conservative contemporary university.

URL : Reading Peer Review : PLOS ONE and Institutional Change in Academia

Original location : https://www.cambridge.org/core/elements/reading-peer-review/42F027E4C67D246DD8C3AC440A68C7A7

Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals

Authors : Inga Patarčić, Jadranka Stojanovski

Journal policies continuously evolve to enable knowledge sharing and support reproducible science. However, that change happens within a certain framework. Eight modular standards with three levels of increasing stringency make Transparency and Openness Promotion (TOP) guidelines which can be used to evaluate to what extent and with which stringency journals promote open science.

Guidelines define standards for data citation, transparency of data, material, code and design and analysis, replication, plan and study pre-registration, and two effective interventions: “Registered reports” and “Open science badges”, and levels of adoption summed up across standards define journal’s TOP Factor. In this paper, we analysed the status of adoption of TOP guidelines across two thousand journals reported in the TOP Factor metrics.

We show that the majority of the journals’ policies align with at least one of the TOP’s standards, most likely “Data citation” (70%) followed by “Data transparency” (19%). Two-thirds of adoptions of TOP standard are of the stringency Level 1 (less stringent), whereas only 9% is of the stringency Level 3.

Adoption of TOP standards differs across science disciplines and multidisciplinary journals (N = 1505) and journals from social sciences (N = 1077) show the greatest number of adoptions. Improvement of the measures that journals take to implement open science practices could be done: (1) discipline-specific, (2) journals that have not yet adopted TOP guidelines could do so, (3) the stringency of adoptions could be increased.

URL : Adoption of Transparency and Openness Promotion (TOP) Guidelines across Journals

DOI : https://doi.org/10.3390/publications10040046

Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study

Authors : Antoni D. Gardener, Ellen J. Hick, Chloe Jacklin, Gifford Tan, Aidan G. Cashin, Hopin Lee, David Nunan, Elaine C. Toomey, Georgia C. Richards

Objectives

To audit the transparent and open science standards of health and medical sciences journal policies and explore the impact of the COVID-19 pandemic.

Design

Repeat cross-sectional study.

Setting

19 journals listed in Google Scholar’s Top Publications for health and medical sciences.

Participants

Blood, Cell, Circulation, European Heart Journal, Gastroenterology, Journal of Clinical Oncology, Journal of the American College of Cardiology, Nature Genetics, Nature Medicine, Nature Neuroscience, Neuron, PLoS ONE, Proceedings of the National Academy of Sciences, Science Translational Medicine, The British Medical Journal, The Journal of the American Medical Association, The Lancet, The Lancet Oncology, and The New England Journal of Medicine.

Main outcome measures

We used the Transparency and Openness Promotion (TOP) guideline and the International Committee of Medical Journal Editors (ICMJE) requirements for disclosing conflicts
of interest (COIs) to evaluate journals standards.

Results

TOP scores slightly improved during the COVID-19 pandemic, from a median of 5 (IQR: 212.5) out of a possible 24 points in February 2020 to 7 (IQR: 4–12) in May 2021, but overall, scores were very low at both time points. Journal policies scored highest for their adherence to data transparency and scored lowest for preregistration of study protocols and analysis plans and the submission of replication studies. Most journals fulfilled all ICMJE provisions for reporting COIs before (84%; n = 16) and during (95%; n = 18) the COVID-19 pandemic.

Conclusions

The COVID-19 pandemic has highlighted the importance of practising open science. However, requirements for open science practices in audited policies were overall low, which may impede progress in health and medical research. As key stakeholders in disseminating research, journals should promote a research culture of greater transparency and more robust open science practices.

URL : Open science and conflict of interest policies of medical and health sciences journals before and during the COVID-19 pandemic: A repeat cross-sectional study

DOI : https://doi.org/10.1177/20542704221132139