Archives des auteurs : Hans Dillaerts

Peer-review under review – A statistical study on proposal ranking at ESO. Part I: the pre-meeting phase

Posté le par

Author : Ferdinando Patat

Peer review is the most common mechanism in place for assessing requests for resources in a large variety of scientific disciplines. One of the strongest criticisms to this paradigm is the limited reproducibility of the process, especially at largely oversubscribed facilities. In this and in a subsequent paper we address this specific aspect in a quantitative way, through a statistical study on proposal ranking at the European Southern Observatory.

For this purpose we analysed a sample of about 15000 proposals, submitted by more than 3000 Principal Investigators over 8 years. The proposals were reviewed by more than 500 referees, who assigned over 140000 grades in about 200 panel sessions.

After providing a detailed analysis of the statistical properties of the sample, the paper presents an heuristic model based on these findings, which is then used to provide quantitative estimates of the reproducibility of the pre-meeting process.

On average, about one third of the proposals ranked in the top quartile by one referee are ranked in the same quartile by any other referee of the panel. A similar value is observed for the bottom quartile.

In the central quartiles, the agreement fractions are very marginally above the value expected for a fully aleatory process (25%). The agreement fraction between two panels composed by 6 referees is 55+/-5% (50% confidence level) for the top and bottom quartiles.

The corresponding fraction for the central quartiles is 33+/-5%. The model predictions are confirmed by the results obtained from boot-strapping the data for sub-panels composed by 3 referees, and fully consistent with the NIPS experiment. The post-meeting phase will be presented and discussed in a forthcoming paper.

URL : https://arxiv.org/abs/1805.06981

Shadow Libraries : Access to Knowledge in Global Higher Education

Posté le par

Author : Joe Karaganis

Examining the new ecosystems of access that are emerging in middle- and low-income countries as opportunities for higher education expand but funding for materials shrinks.

Even as middle- and low-income countries expand their higher education systems, their governments are retreating from responsibility for funding and managing this expansion. The public provision of educational materials in these contexts is rare; instead, libraries, faculty, and students are on their own to get what they need.

Shadow Libraries explores the new ecosystem of access, charting the flow of educational and research materials from authors to publishers to libraries to students, and from comparatively rich universities to poorer ones. In countries from Russia to Brazil, the weakness of formal models of access was countered by the growth of informal ones.

By the early 2000s, the principal form of access to materials was informal copying and sharing. Since then, such unauthorized archives as Libgen, Gigapedia, and Sci-Hub have become global “shadow libraries,” with massive aggregations of downloadable scholarly materials.

The chapters consider experiments with access in a range of middle- and low-income countries, describing, among other things, the Russian samizdat tradition and the connection of illicit copying to resistance to oppression; BiblioFyL, an online archive built by students at the University of Buenos Aires; education policy and the daily practices of students in post-Apartheid South Africa; the politics of access in India; and copy culture in Brazil.

URL : Shadow Libraries : Access to Knowledge in Global Higher Education

Alternative location : https://mitpress.mit.edu/books/shadow-libraries

Enhancing Institutional Publication Data Using Emergent Open Science Services

Posté le par

Authors : David Walters, Christopher Daley

The UK open access (OA) policy landscape simultaneously preferences Gold publishing models (Finch Report, RCUK, COAF) and Green OA through repository usage (HEFCE), creating the possibility of confusion and duplication of effort for academics and support staff.

Alongside these policy developments, there has been an increase in open science services that aim to provide global data on OA. These services often exist separately to locally managed institutional systems for recording OA engagement and policy compliance.

The aim of this study is to enhance Brunel University London’s local publication data using software which retrieves and processes information from the global open science services of Sherpa REF, CORE, and Unpaywall.

The study draws on two classification schemes; a ‘best location’ hierarchy, which enables us to measure publishing trends and whether open access dissemination has taken place, and a relational ‘all locations’ dataset to examine whether individual publications appear across multiple OA dissemination models.

Sherpa REF data is also used to indicate possible OA locations from serial policies. Our results find that there is an average of 4.767 permissible open access options available to the authors in our sample each time they publish and that Gold OA publications are replicated, on average, in 3 separate locations.

A total of 40% of OA works in the sample are available in both Gold and Green locations. The study considers whether this tendency for duplication is a result of localised manual workflows which are necessarily focused on institutional compliance to meet the Research Excellence Framework 2021 requirements, and suggests that greater interoperability between OA systems and services would facilitate a more efficient transformation to open scholarship.

URL : Enhancing Institutional Publication Data Using Emergent Open Science Services

Alternative location : http://www.mdpi.com/2304-6775/6/2/23

Dissertation-to-Book Publication Patterns Among a Sample of R1 Institutions

Posté le par

Authors: Karen Rupp-Serrano, Jen Waller

INTRODUCTION

A common concern about openly available electronic theses and dissertations is that their “openness” will prevent graduate student authors from publishing their work commercially in the future. A handful of studies have explored aspects of this topic; this study reviewed dissertation-to-book publication patterns at Carnegie Classification R1 academic institutions.

METHODS

This study analyzed over 23,000 dissertations from twelve U.S. universities to determine how frequently dissertations were subsequently published as books matching the original dissertation in pagination, chapters, and subject matter.

WorldCat and several other resources were used to make publication determinations.

RESULTS

Across the sample set, a very small percentage of dissertations were published as books that matched the original dissertation on pagination, chapters, and subject matter. The average number of years for dissertations in the study to be published as books was determined for broad subject categories and for select academic disciplines.

Results were compared across public and private institutions, and books that were self-published or published by questionable organizations were identified.

DISCUSSION

Dissertation-to-book trends occur primarily in the social sciences, humanities, and arts. With dissertations for which the author is actively working to publish as a book, the commonly offered 6- to 24-month embargo periods appear sufficient, provided that extensions or renewals continue to be available.

CONCLUSION

This study has implications for librarians providing services to graduate students, faculty advisors, and graduate colleges/schools in regard to dissertation embargo lengths, self-publishing, and what we have termed questionable publishers, as these areas continue to provide opportunities for librarians to educate these stakeholders.

URL : Dissertation-to-Book Publication Patterns Among a Sample of R1 Institutions

DOI : http://doi.org/10.7710/2162-3309.2187

 

Prepublication disclosure of scientific results: Norms, competition, and commercial orientation

Posté le par

Authors : Jerry G. Thursby, Carolin Haeussler, Marie C. Thursby, Lin Jiang

On the basis of a survey of 7103 active faculty researchers in nine fields, we examine the extent to which scientists disclose prepublication results, and when they do, why? Except in two fields, more scientists disclose results before publication than not, but there is significant variation in their reasons to disclose, in the frequency of such disclosure, and in withholding crucial results when making public presentations.

They disclose results for feedback and credit and to attract collaborators. Particularly in formulaic fields, scientists disclose to attract new researchers to the field independent of collaboration and to deter others from working on their exact problem.

A probability model shows that 70% of field variation in disclosure is related to differences in respondent beliefs about norms, competition, and commercialization. Our results suggest new research directions—for example, do the problems addressed or the methods of scientific production themselves shape norms and competition?

Are the levels we observe optimal or simply path-dependent? What is the interplay of norms, competition, and commercialization in disclosure and the progress of science?

URL : Prepublication disclosure of scientific results: Norms, competition, and commercial orientation

DOI : http://advances.sciencemag.org/content/4/5/eaar2133

Measuring Scientific Broadness

Posté le par

Authors : Tom Price, Sabine Hossenfelder

Who has not read letters of recommendations that comment on a student’s `broadness’ and wondered what to make of it?

We here propose a way to quantify scientific broadness by a semantic analysis of researchers’ publications. We apply our methods to papers on the open-access server arXiv.org and report our findings.

URL : https://arxiv.org/abs/1805.04647

Opium in science and society: Numbers

Posté le par

Authors : Julian N. Marewski, Lutz Bornmann

In science and beyond, numbers are omnipresent when it comes to justifying different kinds of judgments. Which scientific author, hiring committee-member, or advisory board panelist has not been confronted with page-long “publication manuals”, “assessment reports”, “evaluation guidelines”, calling for p-values, citation rates, h-indices, or other statistics in order to motivate judgments about the “quality” of findings, applicants, or institutions?

Yet, many of those relying on and calling for statistics do not even seem to understand what information those numbers can actually convey, and what not. Focusing on the uninformed usage of bibliometrics as worrysome outgrowth of the increasing quantification of science and society, we place the abuse of numbers into larger historical contexts and trends.

These are characterized by a technology-driven bureaucratization of science, obsessions with control and accountability, and mistrust in human intuitive judgment. The ongoing digital revolution increases those trends.

We call for bringing sanity back into scientific judgment exercises. Despite all number crunching, many judgments – be it about scientific output, scientists, or research institutions – will neither be unambiguous, uncontroversial, or testable by external standards, nor can they be otherwise validated or objectified.

Under uncertainty, good human judgment remains, for the better, indispensable, but it can be aided, so we conclude, by a toolbox of simple judgment tools, called heuristics.

In the best position to use those heuristics are research evaluators (1) who have expertise in the to-be-evaluated area of research, (2) who have profound knowledge in bibliometrics, and (3) who are statistically literate.

URL : https://arxiv.org/abs/1804.11210