Open peer review: a note from a physicist

I’d like to take note of this post by Sabine Hossenfelder, “Open peer review and its discontents”. She reflects on a growing cultural divide in science between those who see science as an open conversation and those who see it as a curated hierarchy.

[Science journalist Philipp] Hummel wrote by email he found my blogpost very useful and that he had also contacted the author asking for a comment on my criticism. The author’s reply can be found in Hummel’s article. It says that he hadn’t read my blogpost, wouldn’t read it, and wouldn’t comment on it either because he doesn’t consider this proper ‘scientific means’ to argue with colleagues. The proper way for me to talk to him, he let the journalist know, is to either contact him or publish a reply on the arxiv. Hummel then asked me what I think about this.
To begin with I find this depressing. Here’s a young researcher who explicitly refuses to address criticism on his work, and moreover thinks this is proper scientific behavior. I could understand that he doesn’t want to talk to me, evil aggressive blogger that I am, but that he refuses to explain his research to a third party isn’t only bad science communication, it’s actively damaging the image of science.

I get concerned when I hear of early career scientists behaving this way. Obviously I’m not a physicist but the same dynamic can occur in biological anthropology. Some scientists are markedly open and see the public discussion of their ideas as helpful, others react defensively.

I can handle senior scientists having this reaction. I have heard from a number of them that open discussion of their research, sharing of data and public engagement is “not the proper way to conduct science”. That’s OK, these old fogies will be gone before too long. What worries me is that they may be replaced by like-minded mid-career people who have long been waiting for “their turn” to reap the benefits of status.

I’m not talking about revealing ideas before they are fully developed. Every scientific idea undergoes a birth process. The birth canal, to strain an analogy, is tortuous and the idea must deform and shift as it passes through. The idea shouldn’t be reviewed until a researcher reaches a stage where review will be productive and helpful. So much research today is developed by teams of researchers, and engineering a collaborative paper requires trust between people who may not have worked together before. This trust can only build in a confidential environment where frank discussion of ideas is possible.

Many scientists now publish pre-prints at the time they are ready to receive such commentary. Others submit papers for confidential pre-publication review and discuss them only after they are published. But in either case, once a scientist is ready to present ideas, he would be a fool not to listen to critique and commentary in every form available.

Returning to Hossenfelder’s post, the issue of open review has two sides. A researcher should be willing to hear commentary on her work, adjusting it where appropriate. But outside experts should be equally willing to offer frank commentary. As it stands, few ideas are subject to robust review:

[M]ost of my colleagues in theoretical physics entirely ignore papers that they think are wrong. They are convinced that in the end only truth will prevail and thus practice live-and-let-live. I used to do this too. But look at the evidence: it doesn’t work. The arxiv now is full with paid research so thin a sneeze could wipe it out. We seem to have forgotten that criticism is an integral part of science, it is essential for progress, and for cohesion. Physics leaves me wanting more every year. It is over-specialized into incredibly narrow niches, getting worse by the day.

We get more of what we reward, and less of what we punish. As science stands today, there is no reward for good review of work, either pre-publication or post-publication. The scientists who carry out review for journals, on blogs, preprint servers, or elsewhere, are performing a charitable service. It is foolish to expect more and better review, whether open or not. So authors should be eager to take what they can get.