Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future

Since 2005, and with generous support from the A.W. Mellon Foundation, The Future of Scholarly Communication Project at UC Berkeley’s Center for Studies in Higher Education (CSHE) has been exploring how academic values—including those related to peer review, publishing, sharing, and collaboration—influence scholarly communication practices and engagement with new technological affordances, open access publishing, and the public good.

The current phase of the project focuses on peer review in the Academy; this deeper look at peer review is a natural extension of our findings in Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines (Harley et al. 2010), which stressed the need for a more nuanced academic reward system that is less dependent on citation metrics, the slavish adherence to marquee journals and university presses, and the growing tendency of institutions to outsource assessment of scholarship to such proxies as default promotion criteria.

This investigation is made urgent by a host of new challenges facing institutional peer review, such as assessing interdisciplinary scholarship, hybrid disciplines, the development of new online forms of edition making and collaborative curation for community resource use, heavily computational subdisciplines, large-scale collaborations around grand challenge questions, an increase in multiple authorship, a growing flood of low-quality publications, and the call by governments, funding bodies, universities, and individuals for the open access publication of taxpayer-subsidized research, including original data sets.

The challenges of assessing the current and future state of peer review are exacerbated by pressing questions of how the significant costs of high-quality scholarly publishing can be borne in the face of calls for alternative, usually university-based and open access, publishing models for both journals and books.

There is additionally the insidious and destructive “trickle down” of tenure and promotion requirements from elite research universities to less competitive and non-research-intensive institutions.

The entire system is further stressed by the mounting—and often unrealistic—government pressure on scholars in developed and emerging economies alike to publish their research in the most select peer-reviewed outlets, ostensibly to determine the distribution of government funds (via research assessment exercises) and/or to meet national imperatives to achieve research distinction internationally.

The global effect is a growing glut of low-quality publications that strains the efficient and effective practice of peer review, a practice that is, itself, primarily subsidized by universities in the form of faculty salaries. Library budgets and preservation services for this expansion of peer-reviewed publication have run out. Faculty time spent on peer review, in all of its guises, is being exhausted.

As part of our ongoing research, CSHE hosted two meetings to address the relationship between peer review in publication and that carried out for tenure and promotion. Our discussions included: The Dominant System of Peer Review: Types, Standards, Uses, Abuses, and Costs; A Very Tangled Web: Alternatives to the Current System of Peer Review; Creating New Models: The Role of Societies, Presses, Libraries, Information Technology Organizations, Commercial Publishers, and Other Stakeholders; and Open Access “Mandates” and Resolutions versus Developing New Models.

This report includes (1) an overview of the state of peer review in the Academy at large, (2) a set of recommendations for moving forward, (3) a proposed research agenda to examine in depth the effects of academic status-seeking on the entire academic enterprise, (4) proceedings from the workshop on the four topics noted above, and (5) four substantial and broadly conceived background papers on the workshop topics, with associated literature reviews.

The document explores, in particular, the tightly intertwined phenomena of peer review in publication and academic promotion, the values and associated costs to the Academy of the current system, experimental forms of peer review in various disciplinary areas, the effects of scholarly practices on the publishing system, and the possibilities and real costs of creating alternative loci for peer review and publishing that link scholarly societies, libraries, institutional repositories, and university presses.

We also explore the motivations and ingredients of successful open access resolutions that are directed at peer-reviewed article-length material. In doing so, this report suggests that creating a wider array of institutionally acceptable and cost-effective alternatives to peer reviewing and publishing scholarly work could maintain the quality of academic peer review, support greater research productivity, reduce the explosive growth of low-quality publications, increase the purchasing power of cash-strapped libraries, better support the free flow and preservation of ideas, and relieve the burden on overtaxed faculty of conducting too much peer review.”

URL : http://cshe.berkeley.edu/publications/publications.php?id=379

Towards Scholarly Communication 2.0: Pee…

Towards Scholarly Communication 2.0: Peer-to-Peer Review & Ranking in Open Access Preprint Repositories :

“In this paper we present our unified peer-to-peer review model for Open Access preprint repositories. Its objective is to improve the efficiency and effectivity of digital scholarly communication. The key elements of this model are standardized quality assessment instruments, public and private communication channels, special rankings and novel incentives. The model allows scholars to proficiently evaluate both the manuscripts and their peer reviews. These scrutinized manuscripts and peer reviews will then be made available to the relevant parties. These standardized quality assessments allow for new quality metrics for papers and peer reviews. The Reviewer Impact, which represents the peer review proficiency and peer review output of scholars, is one such metric. The model includes diverse rankings for scholars to appear in to receive better odds of having their own manuscripts noticed, read, peer reviewed and cited. Their specific ranking is proportional to their Reviewer Impact and the overall quality of their manuscripts. The Open Access preprint repository model is a suitable foundation for our model because of its high degree of accessibility, but little to no certification of its deposited manuscripts. With this combination we envision a novel, Open Access, peer-to-peer scholarly communication model that functions independently of, but not incompatibly with, the traditional journal publishing model: Scholarly Communication 2.0.”

URL : http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1681478

Combining peer review and metrics to ass…

Combining peer review and metrics to assess journals for inclusion in Scopus :

“Peer review has been in place for centuries as an accepted process to validate manuscripts submitted for publication in scientific journals. Yet a similarly rigorous assessment of content also happens a level up, when looking at the quality of journals that apply for indexing in bibliographic databases. Scopus, an abstract & citation database provided by Elsevier, indexing 18,000 scientific titles, is receiving an increasing number of title suggestions; in 2009 this grew to almost 5,000 in that year alone. Some of the suggested journals are dedicated to niche areas and/or are published in other languages than English. To ensure a fair and transparent evaluation of these titles and to address the rising interest in being indexed, Scopus redesigned its entire title evaluation process – basing it on a metrical scorecard and on the principles of peer review. By developing an online editorial system – the Scopus Title Evaluation Platform (STEP) – Scopus also created the prerequisite of an improved communication with publishers and editors about their journals.”

URL : http://www.ingentaconnect.com/content/alpsp/lp/2010/00000023/00000004/art00011