Étiquette : Peer Review

Publish-and-Flourish: decentralized co-creation and curation of scholarly content

Authors : Emilija Stojmenova Duh, Andrej Duh, Uroš Droftina, Tim Kos, Urban Duh, Tanja Simonič Korošak, Dean Korošak

Scholarly communication is today immersed in publish or perish culture that propels noncooperative behaviour in the sense of strategic games played by researchers.

Here we introduce and describe a blockchain based platform for decentralized scholarly communication. The design of the platform rests on community driven publishing reviewing processes and implements incentives that promote cooperative user behaviour.

Key to achieve cooperation in blockchain based scholarly communication is to transform a static research paper into a modifiable research paper under continuous peer review process.

We describe and discuss the implementation of a modifiable research paper as a smart contract on the blockchain.

URL : https://arxiv.org/abs/1810.10263

Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

Authors : Adrian G Barnett, Scott R. Glisson, Stephen Gallo

Background

Decisions about which applications to fund are generally based on the mean scores of a panel of peer reviewers. As well as the mean, a large disagreement between peer reviewers may also be worth considering, as it may indicate a high-risk application with a high return.

Methods

We examined the peer reviewers’ scores for 227 funded applications submitted to the American Institute of Biological Sciences between 1999 and 2006. We examined the mean score and two measures of reviewer disagreement: the standard deviation and range.

The outcome variable was the relative citation ratio, which is the number of citations from all publications associated with the application, standardised by field and publication year.

Results

There was a clear increase in relative citations for applications with a better mean. There was no association between relative citations and either of the two measures of disagreement.

Conclusions

We found no evidence that reviewer disagreement was able to identify applications with a higher than average return. However, this is the first study to empirically examine this association, and it would be useful to examine whether reviewer disagreement is associated with research impact in other funding schemes and in larger sample sizes.

URL : Do funding applications where peer reviewers disagree have higher citations? A cross-sectional study

DOI : http://dx.doi.org/10.12688/f1000research.15479.2

Towards a Decentralized Process for Scientific Publication and Peer Review using Blockchain and IPFS

Authors : Antonio Tenorio-Fornés, Viktor Jacynycz, David Llop, Antonio A. Sanchez-Ruiz, Samer Hassan

The current processes of scientific publication and peer review raise concerns around fairness, quality, performance, cost, and accuracy. The Open Access movement has been unable to fulfill all its promises, and a few middlemen publishers can still impose policies and concentrate profits.

This paper, using emerging distributed technologies such as Blockchain and IPFS, proposes a decentralized publication system for open science.

The proposed system would provide (1) a distributed reviewer reputation system, (2) an Open Access by-design infrastructure, and (3) transparent governance processes.

A survey is used to evaluate the problems, proposed solutions and possible adoption resistances, while a working prototype serves as a proof-of-concept.

Additionally, the paper discusses the implementation, in a distributed context, of different privacy settings for both open peer review and reputation systems, introducing a novel approach supporting both anonymous and accountable reviews. The paper concludes reviewing the open challenges of this ambitious proposal.

URL : Towards a Decentralized Process for Scientific Publication and Peer Review using Blockchain and IPFS

Gender and international diversity improves equity in peer review

Authors : Dakota Murray, Kyle Siler, Vincent Lariviére, Wei Mun Chan, Andrew M. Collings, Jennifer Raymond, Cassidy R Sugimoto

The robustness of scholarly peer review has been challenged by evidence of disparities in publication outcomes based on author’s gender and nationality. To address this, we examine the peer review outcomes of 23,873 initial submissions and 7,192 full submissions that were submitted to the biosciences journal eLife between 2012 and 2017.

Women and authors from nations outside of North America and Europe were underrepresented both as gatekeepers (editors and peer reviewers) and last authors. We found a homophilic interaction between the demographics of the gatekeepers and authors in determining the outcome of peer review; that is, gatekeepers favor manuscripts from authors of the same gender and from the same country.

The acceptance rate for manuscripts with male last authors was significantly higher than for female last authors, and this gender inequity was greatest when the team of reviewers was all male; mixed-gender gatekeeper teams lead to more equitable peer review outcomes.

Similarly, manuscripts were more likely to be accepted when reviewed by at least one gatekeeper with the same national affiliation as the corresponding author. Our results indicated that homogeneity between author and gatekeeper gender and nationality is associated with the outcomes of scientific peer review.

We conclude with a discussion of mechanisms that could contribute to this effect, directions for future research, and policy implications. Code and anonymized data have been made available at https://github.com/murrayds/elife-analysis

URL : Gender and international diversity improves equity in peer review

DOI : https://doi.org/10.1101/400515

Systematic analysis of agreement between metrics and peer review in the UK REF

Authors : Vincent Traag, Ludo Waltman

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found.

However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review.

We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review.

In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF.

The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review.

To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

URL : https://arxiv.org/abs/1808.03491

Open access monitoring and business model in Latin America and Middle East: a comparative study based on DOAJ data and criteria

Authors : Ivonne Lujano, Mahmoud Khalifa

This research will focus on analyzing the state of open access journals in two regions of developing countries (Latin America and Middle East) according to two main aspects: a) business models and b) monitoring policies that journals implement to ensure the quality.

DOAJ alongside to other institutions has performed great efforts in order to enrich the movement of open access in developing countries. DOAJ is the largest database of peer reviewed open access journals. As March 2018 it has 11.250 journals, and more than 2.900.000 indexed articles from 123 countries.

Using the DOAJ database first, we identified the journals published in countries from the Latin America and Middle East. Then we extracted the data on APCs and submission charges to analyze the business models comparing this data with some other official documents.

We also analyzed some of the DOAJ’s data on monitoring policies, i.e. the review process for papers and the policy of screening for plagiarism. According to initial survey of business models implemented in open access journals in Latin America we found that only 5% of journals charge author fees (APCs and submission charges) being Brazil the country with the highest number of journals that adopt this policy.

Open access is the predominant business model in the majority of countries and it is mostly public funded. Regarding the Middle East region, we can list variant models depending on the economic conditions of each country. APCs and submission charges is growing trend in low economic countries, for example: Egypt, Sudan, North Africa States, however in high economic countries like Gulf States the authors get paid when publish a paper in a journal.

Most of the journals from Latin America (LATAM) implement double or simple blind peer review process and only four journals (published in Brazil and Argentina) carry out some kind of open peer review system. Concerning the policy of screening for plagiarism only 20% of journals state to use any type of software (open source, proprietary, free, etc.).

For journals in the Middle East (MENA), depending on DOAJ experience the types of peer-review are not quite clear for all journals’ editors. Some countries initiated to have policy for plagiarism.

Through the Higher Supreme of Universities in Egypt, screening for plagiarism checked for theses and faculty staff researches, however journals still not familiar with plagiarism detection software, and it requires high cost.

The research will find out deeper results about the two areas depending on DOAJ data analysis and other resources regarding the business model and journal monitoring.

URL : Open access monitoring and business model in Latin America and Middle East: a comparative study based on DOAJ data and criteria

Alternative location : http://library.ifla.org/id/eprint/2126

Post-publication peer review in biomedical journals: overcoming obstacles and disincentives to knowledge sharing

Authors : Valerie Matarese, Karen Shashok

The importance of post-publication peer review (PPPR) as a type of knowledge exchange has been emphasized by several authorities in research publishing, yet biomedical journals do not always facilitate this type of publication.

Here we report our experience publishing a commentary intended to offer constructive feedback on a previously published article. We found that publishing our comment required more time and effort than foreseen, because of obstacles encountered at some journals.

Using our professional experience as authors’ editors and our knowledge of publication policies as a starting point, we reflect on the probable reasons behind these obstacles, and suggest ways in which journals could make PPPR easier. In addition, we argue that PPPR should be more explicitly valued and rewarded in biomedical disciplines, and suggest how these publications could be included in research evaluations.

Eliminating obstacles and disincentives to PPPR is essential in light of the key roles of post-publication analysis and commentary in drawing attention to shortcomings in published articles that were overlooked during pre-publication peer review.

URL : Post-publication peer review in biomedical journals: overcoming obstacles and disincentives to knowledge sharing

DOI : https://dx.doi.org/10.31229/osf.io/8kxyz