John Bohannon is a reporter for science magazine, who has been engaged in an investigative report for the last year about “open access” journals: “Who’s afraid of peer review?”. Bohannon’s project was relatively simple, although clearly very work intensive at a massive scale: send a fake paper to hundreds of these journals and see how often they accept it. More than half the time, the paper riddled with obvious mistakes and not a few unethical elements was accepted for publication.
Journals offering and “open access” option have multiplied over the past several years, and have come to include a large number published by a variety of reputable and non-reputable companies throughout the world. Some companies publish hundreds of “journals”, with titles that may sound convincingly real but that are not associated with any scientific society or recognized scientists as editors. I have received hundreds of solicitations from such journals in my email box, as have most other people working in science, so much so that the phenomenon has become known as “open access spam”.
What does this say about the open access movement in general? Bohannon notes that one of the highest profile open access publications, PLOS ONE, which publishes tens of thousands of papers per year had a very rigorous standard of review:
The rejections tell a story of their own. Some open-access journals that have been criticized for poor quality control provided the most rigorous peer review of all. For example, the flagship journal of the Public Library of Science, PLOS ONE, was the only journal that called attention to the paper's potential ethical problems, such as its lack of documentation about the treatment of animals used to generate cells for the experiment. The journal meticulously checked with the fictional authors that this and other prerequisites of a proper scientific study were met before sending it out for review. PLOS ONE rejected the paper 2 weeks later on the basis of its scientific quality.
As an associate editor for PLOS ONE, I can reinforce that. The papers I have been involved with, both as an editor and as an author, have been reviewed just as rigorously as those submitted to field-specific journals.
If we are going to rely on a system in which scientific papers are reviewed by anonymous peers, then journal editorial boards need to be set up with a high standard of professionalism, packed with recognized authorities in the field.
And quite frankly, I think even this expectation of professionalism is not enough. Anonymous peer review in some cases does protect reviewers who express negative opinions about a paper, but has the pernicious effect of divorcing the review process from other aspects of a scientist’s reputation. Anyone who has submitted a paper to a scientific journal has experienced a good reviews, and reviews of the other kind as often useless as bad. Journals would be more accountable, and the entire review process would be more collegial, if reviewers’ names appeared in the published article.
As it stands, the expectation of anonymity allows the entire system of reputation to be abused by companies who start new “journals” on a simple theme: a straightforward trade in which the publisher agrees to provide a traceable publication in exchange for some fee. Any administrator or person looking to confirm the line in a researcher’s CV will find the paper, professionally formatted and ready for download. There are hundreds of universities in the world paying researchers and looking for tangible ways to make sure that their money is well spent. Inside the mainstream of science, there are traditional journals, recognized experts, and an entire system of credentialing beyond the PhD. But there is a large gray area outside the mainstream in which paying for a CV line is a necessary investment in job security.
“Open access spam” is only one aspect of this black market of scientific reputation. The entire phenomenon of “junket” scientific conferences, where abstracts are never reviewed and where presenters are not really expected to show up and give a talk, is another aspect especially common in fields where publication in a “scientific proceedings” is equivalent to journal publication.
The system as a whole is designed around one central deceit: convincing the people paying the bills that researchers are actually doing research. This is a larger problem in countries where plagiarism and research fraud have already been shown to be much more widespread; the “fake paper” publication model is not tied uniquely to open access. In many cases, university administrators may be complicit in this system, understanding with a wink that those CV lines really don’t mean very much. The point is to keep the money coming, which at lower-tier universities and research institutes internationally is often done with minimal oversight.
With that in mind, there is really nothing new here. Why do researchers submit their work to traditional journals and not expect to be paid, regardless of how interesting or original the work is? It’s because the authors actually are being paid, just in scientific recognition instead of money.
But the traditional system of scientific publication has been undergoing a crisis. Traditional subscription journals have historically controlled quality by relying on their real customers: university libraries. Publishers have had to persuade libraries to pay for the journals, and libraries have often listened to their patrons about which journals are worthwhile. Now, this system is collapsing under the high costs of traditional journals. In particular the move to provide libraries with huge “packages” of journals included for a single multi-thousand dollar fee has removed quality control from the subscription process for many specialized journals. Meanwhile, funding agencies and universities have been pressured to show that their research efforts are useful to the public, driving the growth of the open access movement.
These dynamics may have created a situation in which a reporter can send a fake article to lots of “open access” journals, but the problem is not original to the open access movement. And that’s why we see Bohannon’s results applying not only to shady companies in developing countries, but also to well-established publishers like Sage and Elsevier. This is not about access to results, it is about selling credentials for a fee.
UPDATE (2013-10-03): A reaction to Bohannon’s report from open access pioneer Michael Eisen: “I confess, I wrote the Arsenic DNA paper to expose flaws in peer-review at subscription based journals”.
None of this will stop anti-open access campaigners (hello Scholarly Kitchen) from spinning this as a repudiation for enabling fraud. But the real story is that a fair number of journals who actually carried out peer review still accepted the paper, and the lesson people should take home from this story not that open access is bad, but that peer review is a joke. If a nakedly bogus paper is able to get through journals that actually peer reviewed it, think about how many legitimate, but deeply flawed, papers must also get through. Any scientist can quickly point to dozens of papers including, and perhaps especially, in high impact journals that are deeply, deeply flawed the arsenic DNA story is one of many recent examples. As you probably know there has been a lot of smoke lately about the reproducibility problem in biomedical science, in which people have found that a majority of published papers report facts that turn out not to be true. This all adds up to showing that peer review simply doesnt work.
Does he go too far? I think a paper will be fundamentally more credible when reviewers are willing to attach their names to it. Is there any doubt about this? If not, why don’t we make this a part of the system?