Conference criticisms

Science News has a piece that gives a critical view of our practice of flying thousands of people to a distant city just for scientific sessions: “Weighing the costs of conferencing”. Much of the article focuses on the energy costs of travel. But the critique by John Ioannidis, known for his demonstrations of non-replicable results in the medical literature, is more interesting:

Conferences promote a bulk production of abstracts, with no or superficial peer review, Ioannidis contends, leading to mediocre curriculum vita building. Moreover, published abstracts may live in perpetuity online or in citations, especially if the work it describes never makes its way into a full paper within a peer reviewed journal. So premature and potentially inaccurate findings can be communicated widely, he observes and in such tight word budgets that important caveats may be edited out.

We need to get people together sometimes, but we should focus on how best to create value for attendance. Serving on the program committee for a large conference can be a thankless task, especially if the organizers are stuck with a format that doesn’t allow for genuine exchange of information. We need to recognize a wider range of participation than the 12-minute presentation, particularly since the abstracts for such presentations must be ready months in advance. Ioannidis is correct that these are “mediocre CV building” exercises.

UPDATE (2012-04-09): Reader Roland Kuhn writes:

Re "Conference criticisms" and with respect to one of Ioannidis's points, I wish people wouldn't assume that other scientific communities have the same practices and sociology as their own. Ioannidis refers to "a bulk production of abstracts, with no or superficial peer review. In computational linguistics, conference submissions are thoroughly reviewed (as I think I've written you before). The three or four top conferences always assign three different reviewers, and the rejection rate is high. One of my colleagues is area chair for machine translation for this year's NAACL conference, which will take place in Montreal in June. He has told me that the ACCEPTANCE rate for papers submitted to the conference as a whole is 19% this year. In my community, conference papers are eagerly read, and the best ideas from them often end up being incorporated in other people's work within a few months. By contrast, it is journal papers that are seen as "mediocre curriculum vitae building", in the words of Ioannidis.

Man, I wish I went to computational linguistics meetings. Imagine having interesting papers in advance of presentations! Imagine having every presentation vetted to ensure it was good science! Imagine people competing on the basis of scientific quality instead of last-minute presentations. Work you can incorporate into your own research within a few months!

I will say, it greatly changes the tone of a conference when most attendees are not presenters. And it greatly changes the tone of a field when professionals aren’t able to attend the meetings because they can’t get funding for a plane ticket without giving a presentation of some kind. So I’m not sure high rejection rates will be a good solution in many cases. But it’s clear that a conference volume or scientific output is only as valuable as the standard of preparation that goes into it.