Probing the 'seven biggest problems facing science'

Vox writers Julia Belluz, Brad Plumer, and Brian Resnick compile the results of a survey they sent to scientists around the world, with the single question: “If you could change one thing about how science works today, what would it be and why?” Their resulting article points to several issues of common concern: “The 7 biggest problems facing science, according to 270 scientists”.

Many of these are issues that I’ve discussed here on the weblog over the years. Paleoanthropology of course has its own distinctive spin on several of the big challenges facing scientists worldwide, but in general there is a lot of overlap.

What I find interesting about the list is thinking about the ways that global challenges for science may be overlooked in the study of human origins. For example, the second problem discussed by Belluz, Plumer and Resnick is “Too many studies are poorly designed. Blame bad incentives.” They’re referring directly to issues like p-hacking and fishing for positive results, exacerbated by the use of too-small samples with weak statistical power.

Our respondents suggested that the two key ways to encourage stronger study design — and discourage positive results chasing — would involve rethinking the rewards system and building more transparency into the research process.
"I would make rewards based on the rigor of the research methods, rather than the outcome of the research," writes Simine Vazire, a journal editor and a social psychology professor at UC Davis. "Grants, publications, jobs, awards, and even media coverage should be based more on how good the study design and methods were, rather than whether the result was significant or surprising."
Likewise, Cambridge mathematician Tim Gowers argues that researchers should get recognition for advancing science broadly through informal idea sharing — rather than only getting credit for what they publish.

For many questions, paleoanthropology is the very definition of small samples with weak statistical power. This has its downside. Journals sometimes publish results on fossil hominins that would be meaningless in most fields, simply because everyone acknowledges the difficulty of doing better.

Many paleoanthropologists have seriously grappled with the problem of small samples. We do not shy away from replication studies and negative results. A negative result to a well-formulated research question can be just as interesting in the context of human evolution as a positive one. Papers that report a failure to replicate earlier results, or that emphasize how methods fail to support conclusions, are a regular feature of paleoanthropology research. Robust published debates are very common. These are all areas in which the study of human evolution is ahead of most other fields of science.

However, even though paleoanthropology may be better than many other fields when it comes to replication, we face the problem that some measurements cannot be replicated by independent scientists because the data are inaccessible. This is where, as a field, we fail to meet the standards that are common in other areas of science.