A “basket of metrics”—the best support for understanding journal merit

Authors : Lisa Colledge, Chris James

Aim

To survey opinion of the assertion that useful metricbased input requires a “basket of metrics” to allow more varied and nuanced insights into merit than is possible by using one metric alone.

Methods

A poll was conducted to survey opinions (N=204; average response rate=61%) within the international research community on using usage metrics in merit systems.

Results

“Research is best quantified using multiple criteria” was selected by most (40%) respondents as the reason that usage metrics are valuable, and 95% of respondents indicated that they would be likely or very likely to use usage metrics in their assessments of research merit, if they had access to them.

There was a similar degree of preference for simple and sophisticated usage metrics confirming that one size does not fit all, and that a one-metric approach to merit is insufficient.

Conclusion

This survey demonstrates a clear willingness and a real appetite to use a “basket of metrics” to broaden the ways in which research merit can be detected and demonstrated.

URL : http://europeanscienceediting.eu/articles/a-basket-of-metrics-the-best-support-for-understanding-journal-merit/

17 octobre 2018

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

Authors : Adrian G Barnett, Scott R. Glisson, Stephen Gallo

Background

Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return.

Methods

We examined the peer reviewers’ scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures of reviewer disagreement: the standard deviation and range.

The outcome variable was the relative citation ratio, which is the number of citations from all publications associated with the application, standardised by field and publication year.

Results

There was a clear increase in relative citations for applications with a better mean. There was no association between relative citations and either of the two measures of disagreement.

Conclusions

We found no evidence that reviewer disagreement was able to identify applications with a higher than average return. However, this is the first study to empirically examine this association, and it would be useful to examine whether reviewer disagreement is associated with research impact in other funding schemes and in larger sample sizes.

URL : Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

DOI : http://dx.doi.org/10.12688/f1000research.15479.2

7 octobre 2018

How significant are the public dimensions of faculty work in review, promotion, and tenure documents?

Authors : Juan Pablo Alperin, Gustavo E. Fischman, Erin C. McKiernan, Carol Muñoz Nieves, Meredith T. Niles, Lesley Schimanski

Much of the work of universities, even private institutions, has significant public dimensions. Faculty work in particular is often funded by public funds, is aimed at serving the public good, and is subject to public evaluation.

To understand how the public dimensions of faculty work are valued, we analyzed review, tenure and promotion documents from a representative sample of 129 Canadian and American universities.

We found terms and concepts related to public and community are mentioned in a large portion of documents, but mostly in ways that relate to service—an undervalued aspect of academic careers.

Moreover, we find significant mentions of traditional research outputs and citation-based metrics. Such outputs and metrics reward faculty work targeted to academics, and mostly disregard the public dimensions.

We conclude that institutions that want to live up to their public mission need to work towards systemic change in how faculty work is assessed and incentivized.

URL : How significant are the public dimensions of faculty work in review, promotion, and tenure documents?

DOI : https://hcommons.org/deposits/item/hc:21015

1 octobre 2018

Plurality in multi-disciplinary research: multiple institutional affiliations are associated with increased citations

Authors : Paul Sanfilippo, Alex W. Hewitt, David A. Mackey

Background

The institutional affiliations and associated collaborative networks that scientists foster during their research careers are salient in the production of high-quality science. The phenomenon of multiple institutional affiliations and its relationship to research output remains relatively unexplored in the literature.

Methods

We examined 27,612 scientific articles, modelling the normalized citation counts received against the number of authors and affiliations held.

Results

In agreement with previous research, we found that teamwork is an important factor in high impact papers, with average citations received increasing concordant with the number of co-authors listed.

For articles with more than five co-authors, we noted an increase in average citations received when authors with more than one institutional affiliation contributed to the research.

Discussion

Multiple author affiliations may play a positive role in the production of high-impact science. This increased researcher mobility should be viewed by institutional boards as meritorious in the pursuit of scientific discovery.

URL : Plurality in multi-disciplinary research: multiple institutional affiliations are associated with increased citations

DOI : https://doi.org/10.7717/peerj.5664

24 septembre 2018

What increases (social) media attention: Research impact, author prominence or title attractiveness?

Authors : Olga Zagovora, Katrin Weller, Milan Janosov, Claudia Wagner, Isabella Peters

Do only major scientific breakthroughs hit the news and social media, or does a ‘catchy’ title help to attract public attention? How strong is the connection between the importance of a scientific paper and the (social) media attention it receives?

In this study we investigate these questions by analysing the relationship between the observed attention and certain characteristics of scientific papers from two major multidisciplinary journals: Nature Communication (NC) and Proceedings of the National Academy of Sciences (PNAS).

We describe papers by features based on the linguistic properties of their titles and centrality measures of their authors in their co-authorship network.

We identify linguistic features and collaboration patterns that might be indicators for future attention, and are characteristic to different journals, research disciplines, and media sources.

URL : https://arxiv.org/abs/1809.06299

24 septembre 2018

Do all citations value the same? Valuing citations by the value of the citing items

Authors : Cristiano Giuffrida, Giovanni Abramo, Ciriaco Andrea D’Angelo

Bibliometricians have long recurred to citation counts to measure the impact of publications on the advancement of science. However, since the earliest days of the field, some scholars have questioned whether all citations should value the same, and have gone on to weight them by a variety of factors.

However sophisticated the operationalization of the measures, the methodologies used in weighting citations still present limits in their underlying assumptions. This work takes an alternate approach to resolving the underlying problem: the proposal is to value citations by the impact of the citing articles.

As well as conceptualizing a new indicator of impact, the work illustrates its application to the 2004-2012 Italian scientific production indexed in the WoS.

The new indicator appears highly correlated to traditional field normalized citations, however the shifts observed between the two measures are frequent and the number of outliers not at all negligible. Moreover, the new indicator seems to show greater “sensitivity” when used in identification of the top-cited papers.

URL : https://arxiv.org/abs/1809.06088

5 septembre 2018

The Scientific Prize Network Predicts Who Pushes the Boundaries of Science

Authors : Yifang Ma, Brian Uzzi

Scientific prizes are among the greatest recognition a scientist receives from their peers and arguably shape the direction of a field by conferring credibility to persons, ideas, and disciplines, providing financial rewards, and promoting rituals that reinforce scientific communities.

The proliferation of prizes and links among prizes suggest that the prize network embodies information about scientists and ideas poised to grow in acclaim. Using comprehensive new data on prizes and prizewinners worldwide and across disciplines, we examine the growth dynamics and interlocking relationships found in the worldwide scientific prize network.

We focus on understanding how the knowledge linkages among prizes and scientists’ propensities for prizewinning are related to knowledge pathways across disciplines and stratification within disciplines.

We find several key links between prizes and scientific advances.

First, despite a proliferation of diverse prizes over time and across the globe, prizes are more concentrated within a relatively small group of scientific elites, and ties within the elites are more clustered, suggesting that a relatively constrained number of ideas and scholars lead science.

Second, we find that certain prizes are strongly interlocked within and between disciplines by scientists who win multiple prizes, revealing the key pathways by which knowledge systematically gains credit and spreads through the network.

Third, we find that genealogical and co authorship networks strongly predict who wins one or more prizes and explains the high level of interconnections among acclaimed scientists and their path breaking ideas.

URL : https://arxiv.org/abs/1808.09412