John Ioannidis is well known as a critic of the way science has usually been practiced. I’ve linked to his work before (“Link: John Ioannidis and the scientific workforce core”, “Conference criticisms”), although I haven’t written directly about his most famous work on the reasons why most scientific findings go unreplicated.
Ioannidis has a recent article in PLoS Medicine that discusses ways to change the scientific research enterprise. He focuses on two criticisms of the present scientific research practice: Most published articles are based on samples that lack sufficient power to justify the claimed associations or effects, and a large majority of resources (he cites 85% as an estimate) are “wasted”. Based on these considerations, he writes about several ways that scientific research could be changed to increase the pace of genuine discovery and reduce the rate of spurious results.
The current system values publications, grants, academic titles, and previously accumulated power. Researchers at higher ranks have more papers and more grants. However, scholars at the very top of the ladder (e.g., university presidents) have modest, mediocre, or weak publication and citation records . This might be because their lobbying dexterity compensates for their lack of such credentials, and their success comes at the expense of other worthier candidates who would bring more intellectual rigor and value to senior decision making; equally, it could be because they excel at the bureaucratic work necessary to keep the mind-boggling academic machine going, and their skills enable more scientifically gifted colleagues to concentrate on research. The current system does not reward replication—it often even penalizes people who want to rigorously replicate previous work, and it pushes investigators to claim that their work is highly novel and significant . Sharing (data, protocols, analysis codes, etc.) is not incentivized or requested, with some notable exceptions –. With lack of supportive resources and with competition (“competitors will steal my data, my ideas, and eventually my funding”), sharing becomes even disincentivized. Other aspects of scientific citizenship, such as high-quality peer review, are not valued. Peer review can be a beneficial process, acting as a safety net and a mechanism for augmenting quality. It can also be superficial, lead to only modest improvements of the reviewed work, and allow for the acceptance of blatantly wrong papers ,. That it is so little valued and rewarded is not calculated to encourage its benefits and minimize its harms.
I don’t endorse all of his arguments, which are heavily skewed toward the biomedical research fields. These areas are after all his specialty, and more “pure science” areas of research such as paleontology and astronomy are not his aim. On the other hand, physics and genetics share a great deal in the way they approach new scientific research projects today: Both fields have come to rely more strongly upon large-scale collaborations, both have begun to institute more quality assurance in their software and tool development, and both have adopted more open access protocols toward data collection. Both also have recognized that very stringent standards on statistical significance are necessary to reduce the reporting of results that will almost certainly fail to be replicated in larger-scale studies.
Paleoanthropology has begun to embrace a scientific approach with wider-scale collaboration and data sharing. I think we are fortunate in being a relatively small field in which the contributions of peer reviewers and editors are much more recognized than in many fields. I think this recognition would be even greater if the reviews were less anonymous, and I have encountered an increasing number of reviewers who have refused anonymity (often including myself). But even more important than peer review is a greater discussion and conversation about research directions and priorities. If we can get students and professionals collaborating with each other on larger-scale research datasets instead of working alone in isolation, we’ll increase the replicability of results and improve the significance of everyone’s work.
Ioannidis JPA (2014) How to Make More Published Research True. PLoS Med 11(10): e1001747. doi:10.1371/journal.pmed.1001747