Should journals do post-pub review?

Nikolai Slavov recently published an opinion piece in eLife arguing the advantages of post-publication review of scientific papers: “Point of view: Making the most of peer review”

As with referee reports, I often read blogs and comments and find many—but not all—of them helpful. However, just as the scientific community is failing to take full advantage of the time and expertise that goes into referee reports, I feel that we are also failing to take advantage of the possibilities offered by post-publication comments on papers; indeed, many legitimate comments remain ignored by authors, journals and universities (PubPeer, 2014). By neglecting these comments we are missing an opportunity to reduce the time and resources that are spent trying to repeat and build on experiments that are not reproducible (Freedman et al., 2015).

Slavov’s piece suggests that journals should find ways to facilitate and track the post-publication commentary that follows their papers.

Of course, every post-publication comment on a paper does not have to lead to the publication of a formal challenge (such as a Technical Comment in Science or a Brief Communication Arising in Nature) or a retraction. However, I feel that many of the legitimate concerns about papers that are being raised in blogs and other platforms are being ignored by journals, so there is a clear need to make sure that comments that satisfy some basic criteria (see below) are acted upon by journals. The peer review process, the reproducibility of published results, and the scientific community as a whole stand to benefit tremendously from a more inclusive consideration of post-publication comments.

Most journals maintain reputation by perpetuating a fiction that they have thoroughly vetted papers during the editorial process. Authors who survive the editorial process rightly expect some protection from the journal from some kinds of critical attention. Of course, if the journal ignores well-justified critical commentary, the journal’s reputation must suffer.

As it is, unless a paper is very badly flawed, the journal and authors usually have shared interests with respect to commentary. Critical commentary on the papers must to some extent be adversarial to the journal as well as to the authors. Meanwhile, additional laudatory commentary must be independent (and therefore not promoted by the journal), or else it seems self-serving to both the journal and the authors.

In reality, no paper that answers the concerns of every possible reviewer. Commentary has a real value, because it can bring the attention of the community and the authors to unexamined aspects of the work. Many journals used to create a space for such commentary accompanying a paper, by inviting independent scholars to comment on a peer-reviewed draft of the article. Some journals still do this, including Current Anthropology.

Should journals take on even more responsibility than this? So far, journals have done very poorly managing commenting on published articles. Comments on journal sites often mix of one or two critical comments, sometimes very formally written, with many low-information comments and trolls.

Research by my UW-Madison colleagues Dominique Brossard and Dietram Scheufele has shown that readers’ perception of an online article is affected by the comments that follow it (“Online communication biases upon the public perception of science”). If a journal is going to publish comments after an online article appears, those articles will affect readers’ perception of the paper. Authors should not be protected from critical commentary, but authors deserve strong moderation to enforce some rigor and to filter out obvious trolls.

In other words, effective post-publication commentary is not going to happen alongside published papers unless journals invest a great deal of time and assume responsibility for post-publication editorial work. That’s not very different from the current system of technical comments, although actually investing effort into managing commentary would be a substantially faster system than the current one.

Speed in the service of better science is no vice.

I recently published a technical comment in Science, in response to a paper on a fossil hominin discovery. I submitted the comment on the same day the paper was published. The comment did not appear in Science until more than three months later. That delay is unnecessary in the internet era—it is an artifact of the space and scheduling limitations of paper, so that no article will end up with more than one or two published technical comments, and no paper issue of the journal will have more than one or two total. In practical terms, prolonged delays in publishing critical comments shield the journal and authors from answering criticism while they are promoting their work to the media.

Is there a solution? Not yet.

The PubPeer model has potential, because readers can return to a single place to find comments on published articles, and commenters can develop a reputation on the site. Commentary on the site can be rapid, which makes real exchanges possible. But PubPeer itself lacks the kind of moderation and links to most journals that would be necessary to actually draw readers into the commentary. So far, the site lacks the critical mass to raise standards of scientific discussion on most topics.

There is no effective answer yet, but many are approaching the topic of post-pub review with the same questions.