Systematic analysis of agreement between metrics and peer review in the UK REF

Authors : Vincent Traag, Ludo Waltman

When performing a national research assessment, some countries rely on citation metrics whereas others, such as the UK, primarily use peer review. In the influential Metric Tide report, a low agreement between metrics and peer review in the UK Research Excellence Framework (REF) was found.

However, earlier studies observed much higher agreement between metrics and peer review in the REF and argued in favour of using metrics. This shows that there is considerable ambiguity in the discussion on agreement between metrics and peer review.

We provide clarity in this discussion by considering four important points: (1) the level of aggregation of the analysis; (2) the use of either a size-dependent or a size-independent perspective; (3) the suitability of different measures of agreement; and (4) the uncertainty in peer review.

In the context of the REF, we argue that agreement between metrics and peer review should be assessed at the institutional level rather than at the publication level. Both a size-dependent and a size-independent perspective are relevant in the REF.

The interpretation of correlations may be problematic and as an alternative we therefore use measures of agreement that are based on the absolute or relative differences between metrics and peer review.

To get an idea of the uncertainty in peer review, we rely on a model to bootstrap peer review outcomes. We conclude that particularly in Physics, Clinical Medicine, and Public Health, metrics agree quite well with peer review and may offer an alternative to peer review.

URL : https://arxiv.org/abs/1808.03491

Open access monitoring and business model in Latin America and Middle East: a comparative study based on DOAJ data and criteria

Authors : Ivonne Lujano, Mahmoud Khalifa

This research will focus on analyzing the state of open access journals in two regions of developing countries (Latin America and Middle East) according to two main aspects: a) business models and b) monitoring policies that journals implement to ensure the quality.

DOAJ alongside to other institutions has performed great efforts in order to enrich the movement of open access in developing countries. DOAJ is the largest database of peer reviewed open access journals. As March 2018 it has 11.250 journals, and more than 2.900.000 indexed articles from 123 countries.

Using the DOAJ database first, we identified the journals published in countries from the Latin America and Middle East. Then we extracted the data on APCs and submission charges to analyze the business models comparing this data with some other official documents.

We also analyzed some of the DOAJ’s data on monitoring policies, i.e. the review process for papers and the policy of screening for plagiarism. According to initial survey of business models implemented in open access journals in Latin America we found that only 5% of journals charge author fees (APCs and submission charges) being Brazil the country with the highest number of journals that adopt this policy.

Open access is the predominant business model in the majority of countries and it is mostly public funded. Regarding the Middle East region, we can list variant models depending on the economic conditions of each country. APCs and submission charges is growing trend in low economic countries, for example: Egypt, Sudan, North Africa States, however in high economic countries like Gulf States the authors get paid when publish a paper in a journal.

Most of the journals from Latin America (LATAM) implement double or simple blind peer review process and only four journals (published in Brazil and Argentina) carry out some kind of open peer review system. Concerning the policy of screening for plagiarism only 20% of journals state to use any type of software (open source, proprietary, free, etc.).

For journals in the Middle East (MENA), depending on DOAJ experience the types of peer-review are not quite clear for all journals’ editors. Some countries initiated to have policy for plagiarism.

Through the Higher Supreme of Universities in Egypt, screening for plagiarism checked for theses and faculty staff researches, however journals still not familiar with plagiarism detection software, and it requires high cost.

The research will find out deeper results about the two areas depending on DOAJ data analysis and other resources regarding the business model and journal monitoring.

URL : Open access monitoring and business model in Latin America and Middle East: a comparative study based on DOAJ data and criteria

Alternative location : http://library.ifla.org/id/eprint/2126

Post-publication peer review in biomedical journals: overcoming obstacles and disincentives to knowledge sharing

Authors : Valerie Matarese, Karen Shashok

The importance of post-publication peer review (PPPR) as a type of knowledge exchange has been emphasized by several authorities in research publishing, yet biomedical journals do not always facilitate this type of publication.

Here we report our experience publishing a commentary intended to offer constructive feedback on a previously published article. We found that publishing our comment required more time and effort than foreseen, because of obstacles encountered at some journals.

Using our professional experience as authors’ editors and our knowledge of publication policies as a starting point, we reflect on the probable reasons behind these obstacles, and suggest ways in which journals could make PPPR easier. In addition, we argue that PPPR should be more explicitly valued and rewarded in biomedical disciplines, and suggest how these publications could be included in research evaluations.

Eliminating obstacles and disincentives to PPPR is essential in light of the key roles of post-publication analysis and commentary in drawing attention to shortcomings in published articles that were overlooked during pre-publication peer review.

URL : Post-publication peer review in biomedical journals: overcoming obstacles and disincentives to knowledge sharing

DOI : https://dx.doi.org/10.31229/osf.io/8kxyz

Reviewing the review process: Investigation of researchers’ opinions on different methods of peer review

Author : Carolin Anna Rebernig

Peer review is considered the gold standard of scientific publishing. Trust in the traditional system of editor – blind-reviewer – author is still high, but it’s authority is in decline and alternative methods are on the rise.

The current study investigates opinions of alternative peer review methods, the arguments for and against, and the reasons why academics are searching for new approaches. The opinions were analysed by applying qualitative content analysis to online discussions.

The findings were interpreted using two different sociological theories: the Mertonian sociology of science and social constructivism.

The results of the study show that the most discussed method was also the most traditional one: closed pre-publication peer review comprised of single blind, double-blind and open peer review (non blinded).

Discussions of open peer review (both open publishing of reports and open discussions) were also common. All other alternative methods were discussed much less. But the discussions were lively and each method was discussed in both positive and negative terms.

The reasons for preferring certain methods were also manifold, but dominant topics were bias and fairness, quality issues (regarding reviews and publications), issues concerning human resources and communication and exchange among people.

The results of this study demonstrate that while ethical norms seems to be a scientific ideal, human nature makes it impossible to accomplish this goal.

URN : http://urn.kb.se/resolve?urn=urn%3Anbn%3Ase%3Ahb%3Adiva-14607

Scholarship as an Open Conversation: Utilizing Open Peer Review in Information Literacy Instruction

Author : Emily Ford

This article explores the ACRL Framework for Information Literacy’s frame, Scholarship as a Conversation. This frame asserts that information literate students have the disposition, skills, and knowledge to recognize and participate in disciplinary scholarly conversations.

By investigating the peer-review process as part of scholarly conversations, this article provides a brief literature review on peer review in information literacy instruction, and argues that by using open peer review (OPR) models for teaching, library workers can allow students to gain a deeper understanding of scholarly conversations.

OPR affords students the ability to begin dismantling the systemic oppression that blinded peer review and the traditional scholarly publishing system reinforce. Finally, the article offers an example classroom activity using OPR to help students enter scholarly conversations, and recognize power and oppression in scholarly publishing.

URL : http://www.inthelibrarywiththeleadpipe.org/2018/open-conversation/

Case for the double-blind peer review

Author : Lucie Tvrznikova

Peer review is a process designed to produce a fair assessment of research quality prior to publication of scholarly work in a journal. Demographics, nepotism, and seniority have been all shown to affect reviewer behavior suggesting the most common, single-blind review method (or the less common open review method) might be biased.

A survey of current research suggests that double-blind review offers a solution to many biases stemming from author’s gender, seniority, or location without imposing any major downsides.

URL : https://arxiv.org/abs/1807.01408

Ten considerations for open peer review

Authors : Birgit Schmidt, Tony Ross-Hellauer, Xenia van Edig, Elizabeth C Moylan

Open peer review (OPR), as with other elements of open science and open research, is on the rise. It aims to bring greater transparency and participation to formal and informal peer review processes.

But what is meant by `open peer review’, and what advantages and disadvantages does it have over standard forms of review? How do authors or reviewers approach OPR? And what pitfalls and opportunities should you look out for?

Here, we propose ten considerations for OPR, drawing on discussions with authors, reviewers, editors, publishers and librarians, and provide a pragmatic, hands-on introduction to these issues.

We cover basic principles and summarise best practices, indicating how to use OPR to achieve best value and mutual benefits for all stakeholders and the wider research community.

URL : Ten considerations for open peer review

DOI : http://dx.doi.org/10.12688/f1000research.15334.1