Peer review reviewed

There's an article about the problems with peer review by Alison McCook in The Scientist. I think it's a good summary of some of the difficulties with the review system with an exploding number of journal submissions, and it has many good quotes from journal editors.

Lawrence, based at the MRC Laboratory of Molecular Biology at Cambridge, UK, says his earlier papers were always published because he and his colleagues first submitted them to the journals they believed were most appropriate for the work. Now, because of the intense pressure to get into a handful of top journals, instead of sending less-than-groundbreaking work to second- or third-tier journals, more scientists are first sending their work to elite publications, where they often clearly don't belong.
Consequently, across the board, editors at top-tier journals say they are receiving more submissions every year, leading in many cases to more rejections, appeals, and complaints about the system overall. "We reject approximately 6,000 papers per year" before peer review, and submissions are steadily increasing, says Donald Kennedy, editor-in-chief of Science. "There's a lot of potential for complaints."

Since the interviews are mainly with big-time journal editors (Science, JAMA), they focus on the pressure to exaggerate findings and to burn papers from competitors. There's a great table that lists acceptance rates for some of these journals -- Science is less than 8 percent, PLoS Biology around 15 percent.

This was interesting to me:

Indeed, an abundance of data from a range of journals suggests peer review does little to improve papers. In one 1998 experiment designed to test what peer review uncovers, researchers intentionally introduced eight errors into a research paper. More than 200 reviewers identified an average of only two errors. That same year, a paper in the Annals of Emergency Medicine showed that reviewers couldn't spot two-thirds of the major errors in a fake manuscript. In July 2005, an article in JAMA showed that among recent clinical research articles published in major journals, 16% of the reports showing an intervention was effective were contradicted by later findings, suggesting reviewers may have missed major flaws.

A lot of the comments concerning signed reviews are that junior scientists will be afraid to write critical reviews of senior figures. There is some point to that, but there is no shortage of junior people willing to write letters when some senior person publishes an error, either.

And there are always blogs!

I don't really think that too many of these issues affect anthropological journals, although probably the press of submissions does at a slightly smaller scale. I do think that peer review would be improved by signing reviews, and I almost always give the editor the choice whether to keep mine anonymous. I hardly have to sign them, though; everybody knows they're twice as long as anybody else's!