Societal and scientific impact of policy research: A large-scale empirical study of some explanatory factors using Altmetric and Overton

Authors: Pablo Dorta-González, Alejandro Rodríguez-Caro, María Isabel Dorta-González

This study investigates how scientific research influences policymaking by analyzing citations of research articles in policy documents (policy impact) for nearly 125,000 articles across 434 public policy journals. We reveal distinct citation patterns between policymakers and other stakeholders like researchers, journalists, and the public.

News and blog mentions, social media engagement, and open access publications (excluding fully open access) significantly increase the likelihood of a research article being cited in policy documents. Conversely, articles locked behind paywalls and those published under the full open access model (based on Altmetric data) have a lower chance of being policy-cited. Publication year and policy type show no significant influence. Our findings emphasize the crucial role of science communication channels like news media and social media in bridging the gap between research and policy.

Interestingly, academic citations hold a weaker influence on policy citations compared to news mentions, suggesting a potential disconnect between how researchers reference research and how policymakers utilize it. This highlights the need for improved communication strategies to ensure research informs policy decisions more effectively.

This study provides valuable insights for researchers, policymakers, and science communicators. Researchers can tailor their dissemination efforts to reach policymakers through media channels. Policymakers can leverage these findings to identify research with higher policy relevance. Science communicators can play a critical role in translating research for policymakers and fostering dialogue between the scientific and policymaking communities.

Arxiv : https://arxiv.org/abs/2403.06714

Citation differences across research funding and access modalities

Authors : Pablo Dorta-González, María Isabel Dorta-González

This research provides insight into the complex relationship between open access, funding, and citation advantage. It presents an analysis of research articles and their citations in the Scopus database across 40 subject categories.

The sample includes 12 categories from Health Sciences, 7 from Life Sciences, 10 from Physical Sciences & Engineering, and 11 from Social Sciences & Humanities. Specifically, the analysis focuses on articles published in 2016 and the citations they received from 2016 to 2020.

Our findings show that open access articles published in hybrid journals receive considerably more citations than those published in gold open access journals. Articles under the hybrid gold modality are cited on average twice as much as those in the gold modality, regardless of funding.

Furthermore, we found that funded articles generally obtain 50 % more citations than unfunded ones within the same publication modality. Open access repositories significantly increase citations, particularly for articles without funding. Thus, articles in open access repositories receive 50 % more citations than paywalled ones.

URL : Citation differences across research funding and access modalities

DOI : https://doi.org/10.1016/j.acalib.2023.102734

The influence of funding on the Open Access citation advantage

Authors : Pablo Dorta-González, María Isabel Dorta-González

Some of the citation advantage in open access is likely due to more access allows more people to read and hence cite articles they otherwise would not. However, causation is difficult to establish and there are many possible bias. Several factors can affect the observed differences in citation rates.

Funder mandates can be one of them. Funders are likely to have OA requirement, and well-funded studies are more likely to receive more citations than poorly funded studies. In this paper this hypothesis is tested. Thus, we studied the effect of funding on the publication modality and the citations received in more than 128 thousand research articles, of which 31% were funded.

These research articles come from 40 randomly selected subject categories in the year 2016, and the citations received from the period 2016-2020 in the Scopus database. We found open articles published in hybrid journals were considerably more cited than those in open access journals.

Thus, articles under the hybrid gold modality are cite on average twice as those in the gold modality. This is the case regardless of funding, so this evidence is strong. Moreover, within the same publication modality, we found that funded articles generally obtain 50% more citations than unfunded ones.

The most cited modality is the hybrid gold and the least cited is the gold, well below even the paywalled. Furthermore, the use of open access repositories considerably increases the citations received, especially for those articles without funding. Thus, the articles in open access repositories (green) are 50% more cited than the paywalled ones.

This evidence is remarkable and does not depend on funding. Excluding the gold modality, there is a citation advantage in more than 75% of the cases and it is considerably greater among unfunded articles. This result is strong both across fields and over time

URL : https://arxiv.org/abs/2202.02082v1

Contribution of the Open Access modality to the impact of hybrid journals controlling by field and time effects

Authors : Pablo Dorta-González, María Isabel Dorta-González

Purpose

Researchers are more likely to read and cite papers to which they have access than those that they cannot obtain. Thus, the objective of this work is to analyze the contribution of the Open Access (OA) modality to the impact of hybrid journals.

Design / methodology /approach

The “research articles” in the year 2017 from 200 hybrid journals in four subject areas, and the citations received by such articles in the period 2017-2020 in the Scopus database, were analyzed. The hybrid OA papers were compared with the paywalled ones.

The journals were randomly selected from those with share of OA papers higher than some minimal value. More than 60 thousand research articles were analyzed in the sample, of which 24% under the OA modality.

Findings

We obtain at journal level that cites per article in both hybrid modalities (OA and paywalled) strongly correlate. However, there is no correlation between the OA prevalence and cites per article.

There is OA citation advantage in 80% of hybrid journals. Moreover, the OA citation advantage is consistent across fields and held in time. We obtain an OA citation advantage of 50% in average, and higher than 37% in half of the hybrid journals. Finally, the OA citation advantage is higher in Humanities than in Science and Social Science.

Research limitations

Some of the citation advantage is likely due to more access allows more people to read and hence cite articles they otherwise would not. However, causation is difficult to establish and there are many possible bias.

Several factors can affect the observed differences in citation rates. Funder mandates can be one of them. Funders are likely to have OA requirement, and well-funded studies are more likely to receive more citations than poorly funded studies.

Another discussed factor is the selection bias postulate, which suggests that authors choose only their most impactful studies to be open access.

Practical implications

For hybrid journals, the open access modality is positive, in the sense that it provides a  greater number of potential readers. This in turn translates into a greater number of citations and an improvement in the position of the journal in the rankings by impact factor.

For researchers it is also positive because it increases the potential number of readers and citationsreceived.

Originality /value

Our study refines previous results by comparing documents more similar to each other. Although it does not examine the cause of the observed citation advantage, we find that it exists in a very large sample.

URL : https://arxiv.org/abs/2201.09284

Prevalence and citation advantage of gold open access in the subject areas of the Scopus database

Authors : Pablo Dorta-González, Yolanda Santana-Jiménez

The potential benefit of open access (OA) in relation to citation impact has been discussed in the literature in depth. The methodology used to test the OA citation advantage includes comparing OA vs. non-OA journal impact factors and citations of OA versus non-OA articles published in the same non-OA journals.

However, one problem with many studies is that they are small -restricted to a discipline or set of journals-. Moreover, conclusions are not entirely consistent among research areas and ‘early view’ and ‘selection bias’ have been suggested as possible explications. In the present paper, an analysis of gold OA from across all areas of research -the 27 subject areas of the Scopus database- is realized.

As a novel contribution, this paper takes a journal-level approach to assessing the OA citation advantage, whereas many others take a paper-level approach. Data were obtained from Scimago Lab, sorted using Scopus database, and tagged as OA/non-OA using the DOAJ list.

Jointly with the OA citation advantage, the OA prevalence as well as the differences between access types (OA vs. non-OA) in production and referencing are tested. A total of 3,737 OA journals (16.8%) and 18,485 non-OA journals (83.2%) published in 2015 are considered. As the main conclusion, there is no generalizable gold OA citation advantage at journal level.

URL : https://arxiv.org/abs/1708.06242

Reconsidering the gold open access citation advantage postulate in a multidisciplinary context: an analysis of the subject categories in the Web of Science database 2009-2014

Authors : Pablo Dorta-González,  Sara M. González-Betancor, María Isabel Dorta-González

Since Lawrence in 2001 proposed the open access (OA) citation advantage, the potential benefit of OA in relation to the citation impact has been discussed in depth.

The methodology to test this postulate ranges from comparing the impact factors of OA journals versus traditional ones, to comparing citations of OA versus non-OA articles published in the same non-OA journals.

However, conclusions are not entirely consistent among fields, and two possible explications have been suggested in those fields where a citation advantage has been observed for OA: the early view and the selection bias postulates.

In this study, a longitudinal and multidisciplinary analysis of the gold OA citation advantage is developed. All research articles in all journals for all subject categories in the multidisciplinary database Web of Science are considered.

A total of 1,137,634 articles – 86,712 OA articles (7.6%) and 1,050,922 non-OA articles (92.4%)- published in 2009 are analysed. The citation window considered goes from 2009 to 2014, and data are aggregated for the 249 disciplines (subject categories).

At journal level, we also study the evolution of journal impact factors for OA and non-OA journals in those disciplines whose OA prevalence is higher (top 36 subject categories). As the main conclusion, there is no generalizable gold OA citation advantage, neither at article nor at journal level.

URL : https://arxiv.org/abs/1703.03220