The F3-index. Valuing reviewers for scholarly journals

Posté le par

Authors : Federico Bianchi, Francisco Grimaldo, Flaminio Squazzoni

This paper presents an index that measures reviewer contribution to editorial processes of scholarly journals. Following a metaphor of ranking algorithms in sports tournaments, we created an index that considers reviewers on different context-specific dimensions, i.e., report delivery time, the length of the report and the alignment of recommendations to editorial decisions.

To test the index, we used a dataset of peer review in a multi-disciplinary journal, including 544 reviewers on 606 submissions in six years. Although limited by sample size, the test showed that the index identifies outstanding contributors and weak performing reviewers efficiently.

Our index is flexible, contemplates extensions and could be incorporated into available scholarly journal management tools. It can assist editors in rewarding high performing reviewers and managing editorial turnover.

URL : The F3-index. Valuing reviewers for scholarly journals

DOI : https://doi.org/10.1016/j.joi.2018.11.007

“Blacklists” and “whitelists” to tackle predatory publishing : A cross-sectional comparison and thematic analysis

Posté le par

Authors : Michaela Strinzel​, Anna Severin, Katrin Milzow, Matthias Egger

Background

Despite growing awareness of predatory publishing and research on its market characteristics, the defining attributes of fraudulent journals remain controversial.

We aimed to develop a better understanding of quality criteria for scholarly journals by analysing journals and publishers indexed in blacklists of predatory journals and whitelists of legitimate journals and the lists’ inclusion criteria.

Methods

We searched for blacklists and whitelists in early 2018. Lists that included journals across disciplines were eligible. We used a mixed methods approach, combining quantitative and qualitative analyses.

To quantify overlaps between lists in terms of indexed journals and publishers we employed the Jaro-Winkler string metric and Venn diagrams. To identify topics addressed by the lists’ inclusion criteria and to derive their broader conceptual categories, we used a qualitative coding approach.

Results

Two blacklists (Beall’s and Cabell’s) and two whitelists (DOAJ and Cabell’s) were eligible. The number of journals per list ranged from 1404 to 12357 and the number of publishers from 473 to 5638. Seventy-three journals and 42 publishers were included both in a blacklist and whitelist. A total of 198 inclusion criteria were examined.

Seven thematic themes were identified: (i) peer review, (ii) editorial services, (iii) policy, (iv) business practices, (v) publishing, archiving and access, (vi) website and (vii) indexing and metrics.

Business practices accounted for almost half of blacklists’ criteria, whereas whitelists gave more emphasis to criteria related to policy and guidelines. Criteria were grouped into four broad concepts: (i) transparency, (ii) ethics, (iii) professional standards and (iv) peer review and other services.

Whitelists gave more weight to transparency whereas blacklists focused on ethics and professional standards. The criteria included in whitelists were easier to verify than those used in blacklists. Both types of list gave relatively little emphasis to the quality of peer review.

Conclusions

There is overlap between journals and publishers included in blacklists and whitelists. Blacklists and whitelists differ in their criteria for quality and the weight given to different dimensions of quality. Aspects that are central but difficult to verify receive insufficient attention.

URL : “Blacklists” and “whitelists” to tackle predatory publishing : A cross-sectional comparison and thematic analysis

DOI : https://doi.org/10.7287/peerj.preprints.27532v1

Ten years of AoB PLANTS the open access journal for plant scientists: inception and progress since 2009

Posté le par

Author : Michael B Jackson

AoB PLANTS is a not-for-profit, open access, plant science journal and one of three peer-reviewed journals owned and managed by the Annals of Botany Company. This article explains events and thinking that led to the starting of AoB PLANTS and how the unique features of the Journal came to be formalized prior to its launch in September 2009.

The article also describes how the Journal’s management developed over the first 10 years and summarizes the Journal’s achievements in a decade where open access journals have proliferated despite subscription journals continuing to dominate the publishing of peer-reviewed botanical science.

URL : Ten years of AoB PLANTS the open access journal for plant scientists: inception and progress since 2009

DOI : https://doi.org/10.1093/aobpla/plz025

Over-optimization of academic publishing metrics: observing Goodhart’s Law in action

Posté le par

Authors : Michael Fire, Carlos Guestrin

Background

The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades.

Moreover, recent studies indicate that these metrics have become targets and follow Goodhart’s Law, according to which, “when a measure becomes a target, it ceases to be a good measure.”

Results

In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening.

In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists.

Measures such as a journal’s impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors.

Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department.

Conclusions

Academic publishing has changed considerably; now we need to reconsider how we measure success.

URL : Over-optimization of academic publishing metrics: observing Goodhart’s Law in action

DOI : https://doi.org/10.1093/gigascience/giz053

Open enough? Eight factors to consider when transitioning from closed to open resources and courses: A conceptual framework

Posté le par

Authors : Michael B. McNally, Erik G. Christiansen

Transitioning from closed courses and educational resources to open educational resources (OER) and open courseware (OCW) requires considerations of many factors beyond simply the use of an open licence.

This paper examines the pedagogical choices and trade-offs involved in creating OER and OCW. Eight factors are identified that influence openness (open licensing, accessibility and usability standards, language, cultural considerations, support costs, digital distribution, and file formats).

These factors are examined under closed, mixed and most open scenarios to relatively compare the amount of effort, willingness, skill and knowledge required.

The paper concludes by suggesting that maximizing openness is not practical and argues that open educators should strive for ‘open enough’ rather than maximal openness.

DOI : https://doi.org/10.5210/fm.v24i6.9180

Scientific misconduct and accountability in teams

Posté le par

Authors : Katrin Hussinger, Maikel Pellens

Increasing complexity and multidisciplinarity make collaboration essential for modern science. This, however, raises the question of how to assign accountability for scientific misconduct among larger teams of authors. Biomedical societies and science associations have put forward various sets of guidelines. Some state that all authors are jointly accountable for the integrity of the work.

Others stipulate that authors are only accountable for their own contribution. Alternatively, there are guarantor type models that assign accountability to a single author. We contribute to this debate by analyzing the outcomes of 80 scientific misconduct investigations of biomedical scholars conducted by the U.S. Office of Research Integrity (ORI).

We show that the position of authors on the byline of 184 publications involved in misconduct cases correlates with responsibility for the misconduct. Based on a series of binary regression models, we show that first authors are 38% more likely to be responsible for scientific misconduct than authors listed in the middle of the byline (p<0.01). Corresponding authors are 14% more likely (p<0.05).

These findings suggest that a guarantor-like model where first authors are ex-ante accountable for misconduct is highly likely to not miss catching the author responsible, while not afflicting too many bystanders.

URL : Scientific misconduct and accountability in teams

DOI : https://doi.org/10.1371/journal.pone.0215962

How to avoid borrowed plumes in academia

Posté le par

Authors : Margit Osterloh, Bruno S. Frey

Publications in top journals today have a powerful influence on academic careers although there is much criticism of using journal rankings to evaluate individual articles.

We ask why this practice of performance evaluation is still so influential. We suggest this is the case because a majority of authors benefit from the present system due to the extreme skewness of citation distributions. “Performance paradox” effects aggravate the problem.

Three extant suggestions for reforming performance management are critically discussed. We advance a new proposal based on the insight that fundamental uncertainty is symptomatic for scholarly work. It suggests focal randomization using a rationally founded and well-orchestrated procedure.

URL : How to avoid borrowed plumes in academia