metascience

Christopher Reddy, from Woods Hole Oceanographic Institution, comments on his experience doing science around the Deepwater Horizon oil accident in the Gulf of Mexico two years ago: "How Science Failed During the Gulf Oil Disaster". His essay concentrates on the competing interests of scientists, journalists and policymakers.

We had published the study a little more than two months after gathering the data — lightning fast for a scientific paper. But when I was the academic liaison at the oil spill’s headquarters the following month, I learned that those on the front line weren’t impressed by the publication of a paper a month after the crisis was over. Crisis responders often must make decisions on the spot, with imperfect information, even if it is risky.

During a crisis, “peer review is the biggest problem with academia” Juliette Kayyem, who was an assistant secretary at the Department of Homeland Security during the Deepwater Horizon and teaches crisis response at Harvard, told me.

In this case, the scientists and bureaucrats both wanted peer review to validate straightforward buck-passing reciprocity. The government is often willing to do something expensive that might fail, as long as they can pin the blame on scientists; scientists will shoulder the blame as long as peer review covers them from other scientists' criticism.

Missed is the sad fact that peer review is only as good as the probability of drawing two thoughtful reviewers from the pool.

Reddy and many others did a lot of good science during the Deepwater Horizon spill. In particular, they were able to discover and quantify some of the different dynamics of oil in deep water, including the formation of a deep water oil plume. As a study in mismatched priorities, Reddy describes his experience working on the problem, which drew overhyped attention from the press:

Government responders and industry had to respond to the press about the plumes, rather then focusing on higher priorities such as capping the well. And the public had to respond to these reports, too. I recall one Gulf resident asking me if he should sell his house and move away.

The investigation of the plume was where the most novel science was to be found, but was not the central issue for the engineers and other workers tasked with ending the spill and minimizing damage to shoreline ecosystems.

I wish I could say I wasn’t thinking about scooping my peers, confirming the plume, and publishing a top-notch science paper, but that wouldn’t be true. In fact, I called an editor of a journal from the bow of a boat asking him if he was interested in our findings.

Reddy's essay lacks a clear moral, but he is revealing about his motives. He began by criticizing other scientists who drew press attention to the idea of a deep water plume, then joined them in the chase to find it.

Science News has a piece that gives a critical view of our practice of flying thousands of people to a distant city just for scientific sessions: "Weighing the costs of conferencing". Much of the article focuses on the energy costs of travel. But the critique by John Ioannidis, known for his demonstrations of non-replicable results in the medical literature, is more interesting:

Conferences promote “a bulk production of abstracts, with no or superficial peer review,” Ioannidis contends, leading to “mediocre curriculum vita building.” Moreover, published abstracts may live in perpetuity online or in citations, especially if the work it describes never makes its way into a full paper within a peer reviewed journal. So premature and potentially inaccurate findings can be communicated widely, he observes — and in such tight word budgets that important caveats may be edited out.

We need to get people together sometimes, but we should focus on how best to create value for attendance. Serving on the program committee for a large conference can be a thankless task, especially if the organizers are stuck with a format that doesn't allow for genuine exchange of information. We need to recognize a wider range of participation than the 12-minute presentation, particularly since the abstracts for such presentations must be ready months in advance. Ioannidis is correct that these are "mediocre CV building" exercises.

UPDATE (2012-04-09): Reader Roland Kuhn writes:

Re "Conference criticisms" and http://www.sciencenews.org/view/generic/id/339633/title/Weighing_the_cos... with respect to one of Ioannidis's points, I wish people wouldn't assume that other scientific communities have the same practices and sociology as their own. Ioannidis refers to "a bulk production of abstracts, with no or superficial peer review”. In computational linguistics, conference submissions are thoroughly reviewed (as I think I've written you before). The three or four top conferences always assign three different reviewers, and the rejection rate is high. One of my colleagues is area chair for machine translation for this year's NAACL conference, which will take place in Montreal in June. He has told me that the ACCEPTANCE rate for papers submitted to the conference as a whole is 19% this year.

In my community, conference papers are eagerly read, and the best ideas from them often end up being incorporated in other people's work within a few months. By contrast, it is journal papers that are seen as "mediocre curriculum vitae building", in the words of Ioannidis.

Man, I wish I went to computational linguistics meetings. Imagine having interesting papers in advance of presentations! Imagine having every presentation vetted to ensure it was good science! Imagine people competing on the basis of scientific quality instead of last-minute presentations. Work you can incorporate into your own research within a few months!

I will say, it greatly changes the tone of a conference when most attendees are not presenters. And it greatly changes the tone of a field when professionals aren't able to attend the meetings because they can't get funding for a plane ticket without giving a presentation of some kind. So I'm not sure high rejection rates will be a good solution in many cases. But it's clear that a conference volume or scientific output is only as valuable as the standard of preparation that goes into it.

From "Social anthropology: Past and present" [1]:

The thesis I have put before you, that social anthropology is a kind of historiography, and therefore ultimately of philosophy or art, implies that it studies societies as moral systems and not as natural systems, that it is interested in design rather than in process, and that it therefore seeks patterns and not scientific laws, and interprets rather than explains. These are conceptual, and not merely verbal, differences. The concepts of natural system and natural law, modelled on the constructs of the natural sciences, have dominated anthropology from its beginnings, and as we look back over the course of its growth I think we can see that they have been responsible for a false scholasticism which has led to one rigid and ambitious formulation after another. Regarded as a special kind of historiography, that is as one of the humanities, social anthropology is released from these essentially philosophical dogmas and given the opportunity, though it may seem paradoxical to say so, to
be really empirical and, in the true sense of the word, scientific.

This passage is often cited in anthropological theory courses as an early statement of how cultural anthropology came to be seen by its practitioners as an interpretive and fundamentally humanistic discipline. The end of the passage, in which Evans-Pritchard presages the social anthropologists of the future will mainly be humanists, is indeed a polemic for an interpretive approach. But his argument for humanism is not actually anti-science in today's terms; instead it is anti-normative.

As he described the agenda of a humanistic anthropology, Evans-Pritchard effectively described what later would be known as "historical science". Evolutionary biology, for example, is fundamentally historical rather than experimental. "Laws" are a part of evolutionary biology only in the sense that they may provide useful generalizations about the outcomes of historical (and contingent) natural processes. After the passage above, Evans-Pritchard described a research agenda for social anthropology basically akin to evolutionary biology:

What more do we do, can we do or should we want to do in social anthropology than this? We study witchcraft or a kinship system in a particular primitive society. If we want to know more about these social phenomena we can study them in a second society, and then in a third society, and so on, each study reaching, as our knowledge increases and new problems emerge, a deeper level of investigation and teaching us the essential characteristics of the thing we are inquiring into, so that particular studies are given a new meaning and perspective. This will always happen if one necessary condition is observed: that the conclusions of each study are clearly formulated in such a way that they not only test the conclusions reached by earlier studies but advance new hypotheses which can be broken down into fieldwork problems.

You can see that Evans-Pritchard equated a scientific approach with a positivist approach. In those days, the equation was not unreasonable. Although philosophers of science had long been probing alternatives to positivism, most working scientists -- and particularly anthropologists and archaeologists -- used a kind of naive positivist epistemology. In Evans-Pritchard's view, this kind of inquiry had tainted anthropological inquiry throughout its history by encouraging anthropological hubris. If anthropologists could find and understand natural laws of culture, they could improve the effectiveness of social policy.

This normative element in anthropology is, as we have seen, like the concepts of natural law and progress from which it derives, part of its philosophical heritage. In recent times the natural-science approach has constantly stressed the application of its findings to affairs,the emphasis in England being on colonial problems and in America on political and industrial problems. Its more cautious advocates have held that there can only be applied anthropology when the science is much more advanced than it is today, but the less cautious have made far-reaching claims for the immediate application of anthropological knowledge in social planning; though, whether more or less cautious, both have justified anthropology by appeal to utility. Needless to say, I do not share their enthusiasm and regard the attitude that gives rise to it as naive. A full discussion of it would take too long, but I cannot resistthe observation that, as the history of anthropology shows, positivism leads very easily to a misguided ethics, anaemic scientific humanism or - Saint Simon and Comte are cases in point - ersatz religion.

If the lecture had stopped here, it might have been remembered as an early statement in favor of anthropology as a humanistic science, rather than as humanities opposed to science. The lecture was nine years before the famous "Two cultures" lecture by C. P. Snow, but obviously takes a similar theme. But Evans-Pritchard did not take the daring route of redefining anthropological science. Instead, he observes that most future anthropologists would no longer be drawn from the sciences at all (emphasis added):

There is, however, an older tradition than that of the Enlightenment with a different approach to the study of human societies, in which they are seen as systems only because social life musthave a pattern of some kind, inasmuch as man, being a reasonable creature, has to live in a world in which his relations with those around him are ordered and intelligible. Naturally I think that those who see things in this way have a clearer understanding of social reality than the others, but whether this is so or not they are increasing in number, and this is likely to continue because the vast majority of students of anthropology today have been trained in one or other of the humanities and not, as was the case thirty years ago, in one or other of the natural sciences. This being so, I expect that in the future there will be a turning towards humanistic disciplines, especially towards history, and particularly towards social history or the history of institutions, of cultures and of ideas. In this change of orientation social anthropology will retain its individuality because it has its own special problems, techniques and traditions. Though it is likely to continue for some time to devote its attention chiefly to primitive societies, I believe that during this second half of the century it will give far more attention than in the past to more complex cultures and especially to the civilizations of the Far and Near East and become, in a very general sense, the counterpart to Oriental Studies, in so far as these are conceived of as primarily linguistic and literary -- that is to say, it will take as its province the cultures and societies, past as well as present, of the non-European peoples of the world.

Not a bad prediction. Evans-Pritchard did not anticipate that Orientalism would give rise to a backlash, and that anthropology would become much more reflexive and inward-looking, focused on subcultures within Western societies nearly as much as non-European peoples. But the field's actual history followed from Evans-Pritchard's basic prediction about the students of the future. Anthropology began to draw students who did not speak the language of science, and thus became more humanistic. The human sciences always have had use for cultural information, drawing in anthropologists concerned with psychological and sociological interests, but leaving students in anthropology often as a residue of those with more humanistic than scientific interests.

A science of culture could be, and was partially, constructed along the lines of a historical science as Evans-Pritchard nearly described, but that science has been attempted more often in psychology or biology than in anthropology.

References

Sabine Hossenfelder reviews a recent book by Michael Nielsen, Reinventing Discovery: The New Era of Networked Science.

Collective intelligence, Michael argues, works by bringing together many people's "microexpertise," that is a specialized knowledge in a specific area. New software can tell them when their microexpertise is needed and where and how they can add their contribution. To that end, it is preferable if problems are brought into a modular structure, so that parts can be tackled independently. Suitable tools, some of which already exist, then allow scientists to scale up collaborations, helping them solve problems much faster, wasting less time and effort. These are exciting developments for every scientist that promise to make scientific research smoother, faster and less frustrating.