Publish or perish, and get cited, too

Peter Lawrence decries the state of impact factors and grant applications in a Current Biology editorial, "The mismeasurement of science."

In an opening sure to capture my attention, he quotes from Leo Szilard:

Answer from the hero in Leo Szilard's 1948 story "The Mark Gable Foundation" when asked by a wealthy entrepreneur who believes that science has progressed too quickly, what he should do to retard this progress: "You could set up a foundation with an annual endowment of thirty million dollars. Research workers in need of funds could apply for grants, if they could make a convincing case. Have ten committees, each composed of twelve scientists, appointed to pass on these applications. Take the most active scientists out of the laboratory and make them members of these committees. ...First of all, the best scientists would be removed from their laboratories and kept busy on committees passing on applications for funds. Secondly the scientific workers in need of funds would concentrate on problems which were considered promising and were pretty certain to lead to publishable results. ...By going after the obvious, pretty soon science would dry out. Science would become something like a parlor game. ...There would be fashions. Those who followed the fashions would get grants. Those who wouldn't would not."

His argument is that science "is being damaged by attempts to measure the quantity and quality of research," particularly those based on impact factors and number of publications. His argument is that if you judge merit based on publication quantity, you encourage "gatecrashing" -- including names in author lists that have no input or familiarity with the work. Citations are based "more by visibility or convenience than by the content or quality of the work," because only a small proportion of cited papers in any research have actually been read.

The article raises several good points, including the effects of all these things on the scientific workplace, training of students, and gender balance.

UPDATE (8/14/2007): Comments from The Other 95% on the problems measuring citation rates for ecologists and systematists. A quick summary: taxonomic papers are rarely cited by people doing research on model organisms. I would say that my experience confirms this -- last year I asked a group of graduate students to describe the natural environment of their model organisms, and none of them could do it. These weren't anthro students -- they would know better, and besides, their study organisms live in their natural environments!

I think paleoanthropologists have some of the same problems -- assessments of citation rates are highly skewed toward the (relatively) few papers that are cited highly by geneticists. Those are mostly "convenience" citations -- probably 1000 papers have the exact same introductory paragraph, citing exactly the same papers, which the authors have never read. Also, the "second tier" of journals for genetics (Nature Genetics, PLoS Genetics, etc.) has much higher citation numbers than the equivalent journals in anthropology. So in that sense, there is a higher variance in citation rates for important papers in paleoanthropology, with a bigger discrepancy between the few "winners" and all the others. Still, there aren't as many of us, and we write more independently -- although that has been changing during the past 15 years or more.

Also, much more from David Colquhoun, describing the criteria for "publication quality" at Imperial College London, and including a demonstration that impact factor is uncorrelated with citation numbers for individual papers.

On the other hand, a paper in 1981 in a journal with an 'unacceptable' impact factor of 3.56 has had over 15000 citations (Hamill et al. , 1981). This paper would have earned for Sakmann a publication score of a miserable 0.71, less than 100th of our perspective in Science.

References:

Lawrence PA. 2007. The mismeasurement of science. Curr Biol 17:R583-R585. doi:10.1016/j.cub.2007.06.014