Can Tweets Predict Citations? Metrics of Social Impact Based on Twitter and Correlation with Traditional Metrics of Scientific Impact

Posté le par

Background

Citations in peer-reviewed articles and the impact factor are generally accepted measures of scientific impact. Web 2.0 tools such as Twitter, blogs or social bookmarking tools provide the possibility to construct innovative article-level or journal-level metrics to gauge impact and influence. However, the relationship of the these new metrics to traditional metrics such as citations is not known.

Objective:

(1) To explore the feasibility of measuring social impact of and public attention to scholarly articles by analyzing buzz in social media, (2) to explore the dynamics, content, and timing of tweets relative to the publication of a scholarly article, and (3) to explore whether these metrics are sensitive and specific enough to predict highly cited articles.

Methods

Between July 2008 and November 2011, all tweets containing links to articles in the Journal of Medical Internet Research (JMIR) were mined. For a subset of 1573 tweets about 55 articles published between issues 3/2009 and 2/2010, different metrics of social media impact were calculated and compared against subsequent citation data from Scopus and Google Scholar 17 to 29 months later. A heuristic to predict the top-cited articles in each issue through tweet metrics was validated.

Results

A total of 4208 tweets cited 286 distinct JMIR articles. The distribution of tweets over the first 30 days after article publication followed a power law (Zipf, Bradford, or Pareto distribution), with most tweets sent on the day when an article was published (1458/3318, 43.94% of all tweets in a 60-day period) or on the following day (528/3318, 15.9%), followed by a rapid decay. The Pearson correlations between tweetations and citations were moderate and statistically significant, with correlation coefficients ranging from .42 to .72 for the log-transformed Google Scholar citations, but were less clear for Scopus citations and rank correlations. A linear multivariate model with time and tweets as significant predictors (P < .001) could explain 27% of the variation of citations. Highly tweeted articles were 11 times more likely to be highly cited than less-tweeted articles (9/12 or 75% of highly tweeted article were highly cited, while only 3/43 or 7% of less-tweeted articles were highly cited; rate ratio 0.75/0.07 = 10.75, 95% confidence interval, 3.4–33.6). Top-cited articles can be predicted from top-tweeted articles with 93% specificity and 75% sensitivity.

Conclusions

Tweets can predict highly cited articles within the first 3 days of article publication. Social media activity either increases citations or reflects the underlying qualities of the article that also predict citations, but the true use of these metrics is to measure the distinct concept of social impact. Social impact measures based on tweets are proposed to complement traditional citation metrics. The proposed twimpact factor may be a useful and timely metric to measure uptake of research findings and to filter research findings resonating with the public in real time.

URL : http://www.jmir.org/2011/4/e123/

Longitudinal Trends in the Performance of Scientific Peer…

Posté le par

Longitudinal Trends in the Performance of Scientific Peer Reviewers :

“Study objective : We characterize changes in review quality by individual peer reviewers over time.

Methods : Editors at a specialty journal in the top 11% of Institute of Scientific Information journals rated the quality of every review, using a validated 5-point quality score. Linear mixed-effect models were used to analyze rating changes over time, calculating within-reviewer trends plus predicted slope of change in score for each reviewer. Reviewers at this journal have been shown comparable to those at other journals.

Results : Reviews (14,808) were performed by 1,499 reviewers and rated by 84 editors during the 14-year study. Ninety-two percent of reviewers demonstrated very slow but steady deterioration in their scores (mean –0.04 points [–0.8%] per year). Rate of deterioration was unrelated to duration of reviewing but moderately correlated with mean reviewer quality score (R=0.52). The mean score of each reviewer’s first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%. Scores of the group stayed constant over time despite deterioration because newly recruited reviewers initially had higher mean quality scores than their predecessors.

Conclusion : This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.”

URL : http://www.annemergmed.com/article/S0196-0644(10)01266-7/fulltext

Toward a new model of scientific publishing discussion…

Posté le par

Toward a new model of scientific publishing: discussion and a proposal :

“The current system of publishing in the biological sciences is notable for its redundancy, inconsistency, sluggishness, and opacity. These problems persist, and grow worse, because the peer review system remains focused on deciding whether or not to publish a paper in a particular journal rather than providing (1) a high-quality evaluation of scientific merit and (2) the information necessary to organize and prioritize the literature. Online access has eliminated the need for journals as distribution channels, so their primary current role is to provide authors with feedback prior to publication and a quick way for other researchers to prioritize the literature based on which journal publishes a paper. However, the feedback provided by reviewers is not focused on scientific merit but on whether to publish in a particular journal, which is generally of little use to authors and an opaque and noisy basis for prioritizing the literature. Further, each submission of a rejected manuscript requires the entire machinery of peer review to creak to life anew. This redundancy incurs delays, inconsistency, and increased burdens on authors, reviewers, and editors. Finally, reviewers have no real incentive to review well or quickly, as their performance is not tracked, let alone rewarded. One of the consistent suggestions for modifying the current peer review system is the introduction of some form of post-publication reception, and the development of a marketplace where the priority of a paper rises and falls based on its reception from the field (see other articles in this special topics). However, the information that accompanies a paper into the marketplace is as important as the marketplace’s mechanics. Beyond suggestions concerning the mechanisms of reception, we propose an update to the system of publishing in which publication is guaranteed, but pre-publication peer review still occurs, giving the authors the opportunity to revise their work following a mini pre-reception from the field. This step also provides a consistent set of rankings and reviews to the marketplace, allowing for early prioritization and stabilizing its early dynamics. We further propose to improve the general quality of reviewing by providing tangible rewards to those who do it well.”
URL : http://www.frontiersin.org/computational_neuroscience/10.3389/fncom.2011.00055/full

Sharing and the Creative Economy Culture in the…

Posté le par

Sharing and the Creative Economy: Culture in the Internet Age :

“In the past fifteen years, file sharing of digital cultural works between individuals has been at the center of a number of debates on the future of culture itself. To some, sharing constitutes piracy, to be fought against and eradicated. Others see it as unavoidable, and table proposals to compensate for its harmful effects. Meanwhile, little progress has been made towards addressing the real challenges facing culture in a digital world. Sharing starts from a radically different viewpoint, namely that the non-market sharing of digital works is both legitimate and useful. It supports this premise with empirical research, demonstrating that non-market sharing leads to more diversity in the attention given to various works. Taking stock of what we have learnt about the cultural economy in recent years, Sharing sets out the conditions necessary for valuable cultural functions to remain sustainable in this context.”
URL : http://paigrain.debatpublic.net/docs/internet_creation_1-3.pdf
URL : http://www.amazon.com/Sharing-Culture-Economy-Internet-Age/dp/9089643850

Mathematicians’ Views on Current Publishing Issues A Survey…

Posté le par

Mathematicians’ Views on Current Publishing Issues: A Survey of Researchers :

This article reports research mathematicians’ attitudes about and activity in specific scholarly communication areas, as captured in a 2010 survey of more than 600 randomly-selected mathematicians worldwide. Key findings include:

  • Most mathematicians have papers in the arXiv, but posting to their own web pages remains more common;
  • A third of mathematicians have published papers in open access (OA) journals, with speed of publication being seen as the primary advantage over traditional journals, but there is substantial philosophical opposition to OA journal models that charge author fees;
  • Tenure and promotion criteria influence publishing decisions even among most tenured faculty members;
  • Mathematicians want to keep more rights to their publications than they have been allowed, but they have a high success rate in negotiating with publishers for more;
  • Online collaboration tools, such as Google Groups, are not yet widely used for research but their use is expected to rise in the near future.

Reasons behind the mathematics culture of openness were also explored.”

URL : http://www.istl.org/11-fall/refereed4.html

Citations to Wikipedia in Chemistry Journals: A Preliminary Study

Posté le par

Wikipedia has been the subject of an increasing number of studies. Many of these have focused on the quality of Wikipedia articles and the use of Wikipedia by students. Little research has focused on the use of Wikipedia by scholars. This study helps to fill that gap by examining citations to Wikipedia in chemistry journals from three major publishers over a five year period.

The study reports the number of citations to Wikipedia and describes how Wikipedia is being cited. The results show that, while only a small percentage of all articles contained a citation to Wikipedia, it is in fact being cited as a credible information source in articles in major chemistry journals.

URL : http://www.istl.org/11-fall/refereed2.html