I had a question tonight from a reader who is a student at a university somewhere else. Without going into the details, this student was in a seminar where her colleagues were all espousing a position that has been for some time a fad in paleoanthropology. My correspondent did exactly what any good student should do: she found the relevant literature, which seemed to contradict the faddish seminarians.
What to do? Well, I mean besides flashing the hawk signal.
How could these seemingly earnest students have slavishly adopted the fad as their own? Did some professor mislead them?
I thought of that question when I was reading a post by Jeremy Fox: "Can the phylogenetic community ecology bandwagon be stopped or steered? A case study of contrarian ecology". Fox is writing about a particular branch of ecology, in which a growing group of researchers have been using phylogenetic methods to study the factors that determine the species composition of ecological communities. For the details of the methodology and substance of Fox's critique, you'll have to read the whole post.
The interesting part is the literature analysis. Fox notes the publication two years ago of a strong critique of the fad methodology, published in the leading field-specific journal. It makes a unique case study in citation practices:
Even if you dont agree with M&Ls critique, I hope youll agree that this is an interesting case study of a contrarian attempt to stop or steer an ongoing bandwagon. You have a situation where lots of people are pursuing a particular question using a particular approach. But then someone well-known publishes a serious, easy-to-understand critique of that approach in a very prominent venue, and suggests an alternative approach. What happens next?
He discovers that much research plodded along as if no critique had ever been published. The most common kind of citation for the critique is in papers about totally different aspects of ecology, citing the paper as a review. Specialists within the relatively narrow area of the critique have shunted it aside. As Fox concludes, a paper can't stop a bandwagon. It can't even redirect one.
This isn't entirely a fair comparison, considering that research already underway may be hard to alter in response to critical research. Doubtless the researchers on this bandwagon can defend their preferred approach. Still, they should cite and take seriously a prominent methodological criticism.
As for my correspondent, I wrote back:
The quote you pulled is a very good one. I'm afraid you've discovered an inconvenient truth about academics: very few of them are good scholars who read and take seriously the primary literature.
Scientists are supposed to act like proper skeptics, but certain ideas seem to run away with them. I think that failure to read is the biggest reason why bandwagons get going. I've seen it so many times: Instead of engaging with a critical paper, an entire crowd of scientists cite some lame -- and often wrong -- secondary description of the paper. They take the incorrect description because it's short, because it accords with their prior conceptions, and most importantly because reviewers don't demand accuracy.
Oh, about that hawk signal -- one of my other readers today noticed that I am also a superhero. No, really -- if you're a long-time reader, you will no doubt remember:
Yep, I deliver a mighty lecture, an' pack a mighty punch.
UPDATE (2012-10-09): Jeremy Fox writes:
Just saw your discussion of my recent post on the phylogenetic community ecology bandwagon and pushback against it. Glad you liked the post. I don't know that I'm quite as pessimistic as you about the effect Mayfield and Levine, or anyone, to affect the ultimate course of that bandwagon. I think it's still fairly early days, and it's promising that some folks are already publishing papers based on Mayfield & Levine's ideas.
I thought you might be interested in a recent paper that a commenter on the post pointed out to me, and which I'm embarrassed to admit I hadn't seen. It's a review of 17 high profile papers that were subsequently rebutted, asking how often the rebuttals were cited and how they affected the citation patterns of the rebutted papers.
http://www.esajournals.org/doi/pdf/10.1890/ES10-00142.1 (it's open-access)If anything, the review suggests that Mayfield and Levine is an unusually high impact rebuttal. Most rebuttals seem to have very little impact. Which is indeed a rather depressing conclusion. I'm planning a follow-up post on this paper as soon as I can carve out some time.