The broken grant review system

For a Sunday morning read, there’s Gina Kolata’s article in the NY Times: “Grant System Leads Cancer Researchers to Play It Safe”:

Yet the fight against cancer is going slower than most had hoped, with only small changes in the death rate in the almost 40 years since it began.
One major impediment, scientists agree, is the grant system itself. It has become a sort of jobs program, a way to keep research laboratories going year after year with the understanding that the focus will be on small projects unlikely to take significant steps toward curing cancer.
These grants are not silly, but they are only likely to produce incremental progress, said Dr. Robert C. Young, chancellor at Fox Chase Cancer Center in Philadelphia and chairman of the Board of Scientific Advisors, an independent group that makes recommendations to the cancer institute.
The institutes reviewers choose such projects because, with too little money to finance most proposals, they are timid about taking chances on ones that might not succeed.

The story goes through many of the problems with the current system of peer review, and profiles a few researchers whose “transformative” grants haven’t been funded. This kind of story runs a risk: if you applied for grants and didn’t get them, are you just whining? I mean, ninety percent of grant applications are not funded, so if you’re even average good, you’ve got to submit 10 before you have any expectation of return.

The other risk of the story is that readers will draw that conclusion that there are so many deserving grants that go unfunded, all we need to do is raise the funding level. This overlooks the very high costs of the current review system – flying all those people around for panels, handling the paperwork, paying the grant agencies’ bureaucracies, plus the support staff at universities and other institutions to collect the grant money – “indirect” costs of grants that cover administration now amount to nearly half of federal grant budgets. There’s no question that this money could be better allocated.

The question raised by Kolata’s article: Measured by progress in treatments, the money spent by NIH looks like a poor investment. Is there some way that we could invest the money in grants to yield faster progress? Many researchers (some disappointed in their pursuit of promising ideas) think that more money should be given to projects with higher risks of failure.

I view this as a basic reality of science: If you want to discover something new, you have to falsify some existing ideas. So if we want to make scientific progress, we have to invest money to disprove things that many people already believe to be true. It is a scientific mistake to allocate money to projects that do not challenge existing ideas.

As a practical matter, how much money should we invest in risk versus small incremental, low-risk projects? After all, there are ideas, and then there are ideas – well-entrenched, hidebound, review panels goosestepping together ideas. I think a little R. A. Fisher is in order. Think of cancer research as a search space, where research advances are moving toward some (perhaps several) peaks. Now, the question is what kind of change will bring you closer to a peak?

Well, it depends how close to the peak you already are. If you’re already close to a peak, a very large change can do nothing but take you further from it. An incremental change will be roughly halfway likely to take you up a small amount; halfway likely to take you down. So if you think you’re already very close to an acceptable cure rate for cancer, you should focus on the very small incremental improvements.

On the other hand, if you’re far from a peak, a large change is not unlikely to take you closer to the top, and it will certainly get you there a lot faster.

But worse, what if you’re climbing some low peak, where the outcome – even if you do everything entirely right – is poor? A small incremental change will never do any better than the short peak you’re on. But a large change has some chance of finding some other peak – a radical shift in treatment that would bring much better outcomes. The likelihood of that for any particular change may be very low. But your coverage of the search space would be vastly larger – making progress much faster.

That is, assuming you think that the present state of cancer science is far from the best possible treatment plan.

It seems to me that anytime there is a lot of government money involved in research, it’s because the political process has come to some consensus that major, transformative changes are necessary. If you’re not funding large, transformative research, you’re wasting the people’s money.

I’m remembering a story from earlier this year, that concluded that humanities grant panels rank “what is fascinating” over what is likely to be true. I wonder if NIH might do better to import some of those humanities panels to review their grants?

UPDATE (2009-06-28): Yes, I realize I suddenly switched from Fisher’s mutation analogy to Wright’s rugged landscape metaphor.