Did Neandertals evolve in a population sink?

Different populations of humans live at markedly different densities. Human populations historically have had very different growth rates. Although Pleistocene humans did not live in cities, the archaeological record suggests that they also had a huge variety of densities and growth rates in different places. Some populations were relative sources, generating an excess of people who migrated outward to other places. Other populations were sinks, with populations unable to maintain their numbers without immigration from elsewhere.

Today, Europe is a population sink. The fertility rate of indigenous Europeans is below replacement in nearly every European country. Demographic projections indicate that the population will shrink in the coming century, except to the extent it is supported by immigration from outside of Europe. The evolutionary effect of a population sink is to reduce the genetic differentiation between the sink and source population.

A population sink may have other, longer-term evolutionary effects. Europe in the Pleistocene was periodically a challenging place for humans to live. The people who remained there and elsewhere across the temperate zone of Central and Western Eurasia were the Neandertals. I have often wondered whether Neandertal evolution was influenced by negative population growth and immigration from larger human populations to the south, such as the populations of Africa or South Asia.

The Neandertals appear to have been cold-adapted, and other aspects of their anatomy may have been adaptive to the temperate Eurasian ecology. Local adaptation suggests that Neandertal genes were not constantly swamped by immigration from elsewhere. But does that preclude a strong effect of migration on other genes? And is it possible that changes in the ecology of Europe during the last glacial may have intensified the population sink, drawing a higher fraction of migration from outside of Europe? To try to answer these questions, we need to examine some population genetics. Holt and Gomulkiewicz (1997) addressed the problem of local adaptation in sink populations.

Now suppose the local population is a "sink" population, that is, one that would go extinct deterministically without recurrent immigration (Smida and Ellner 1984; Pulliam 1988; Holt 1993). If we apply the prior rule of thumb to such a population, it leads to an absurd conclusion. While zero immigration provides the widest scope for local adaptation, a sink population without immigration goes extinct. And there is, of course, little scope for evolution in extinct populations (564).

The authors propose that this paradox may be resolved by an appeal to absolute fitness instead of relative fitness, and population numbers instead of gene frequencies.

That is music to my ears, since I think the dynamics of population size have been very important to human evolution. Maybe the evolution of the European Neandertals was determined by the demographic effects of new alleles, in opposition to the effects of immigration. Genetic evidence from Neandertals and later Europeans might test that hypothesis, at least applied to the Neandertals of the last glaciation.

Holt and Gomulkiewicz showed with very simple mathematics that the condition for a rare locally adaptive gene to increase in a sink population is that the absolute fitness of the gene be greater than one – called “unity” by the mathematically-minded. What that means is that carriers of the gene must increase in absolute numbers within the sink population. If they don’t then the gene will not invade the population.

When I read this, I thought there must be some oversight or mistake. Why couldn’t an allele be superior to the wild-type, and therefore increase in relative frequency, even if its absolute fitness were less than unity?

The answer is twofold. First, the sink population is shrinking. The fixation probability of a selected mutation is 2(s + r) in a population of changing size. That’s a bonus of population growth (as we illustrated in 2007) but it’s a deficit in a shrinking population, where r is a negative number. If s is bigger than r, then the new mutation will have a chance of beating genetic drift – and that condition means that the absolute fitness of s will be greater than unity.

Second, the wild-type allele is constantly refreshed by immigration from outside the sink. It can’t drop down to zero, even though it tends to decline locally. The mutant allele can’t be introduced from outside. Its fate will be extinction.

Now since a sink population by definition declines in numbers without immigration, this seems to be a paradox. If the fitness of the gene were a constant, then local adaptation would be possible only if the gene causes the population not to be a sink anymore. From the discussion:

The second implication is that local adaptation (fitness maximization) only occurs if mutants arise that can permit the population to persist without immigration. In other words, adaptive evolution in a sink habitat is essentially synonymous with niche evolution (570).

That seems like a trap – a source-sink metapopulation with local adaptations couldn’t exist. But the logic assumes that the locally adaptive allele has the same fitness effect when it is common as when it is rare. That need not be the case: for example, a locally adaptive allele might have high fitness when the local population is scarce, but lower fitness when it becomes common. In that case, the locally adaptive allele might come to an equilibrium frequency where the sink population is still propped up by immigration. But the “sink” in that scenario would only be so with reference to excess migration from the source. If immigration were stopped, the “sink” would shrink to a lower population size but not to extinction.

The authors covered this point by considering density-dependence:

There is also the possibility of temporal fluctuations in the sink population, so that it is self-sustaining at some times but sustained only by immigration at others. This is the most probable scenario for Europe, and I will consider it further in a second post, which draws on some additional theoretical work for support.

Finally, Holt and Gomukiewicz made an interesting observation. An allele must have an absolute fitness greater than unity to invade a sink population. The fact that population is a sink means that the wild-type allele has an absolute fitness less than unity. So for local adaptation to occur, the difference in absolute fitnesses of the two alleles must exceed the deficit of reproduction in the sink. This leads to the following:

This observation suggests that mutants of small effect on individual fitness typically cannot invade a sink habitat. Only mutants of sufficiently large positive effect on absolute fitness can increase when rare. Part of the conventional wisdom of evolutionary biology has long been that adaptive evolution largely rests on the accumulation of genetic changes, each of small effect. The above models clearly highlight one circumstance, namely, sink populations maintained by immigration, where this mode of adaptive evolution is precluded (570).

Put these things together, and what do they add up to?

1. Local adaptation in marginal environments, like Pleistocene Europe, should be strongly mutation-limited. Only the new mutations with the strongest effect on fitness will be able to invade the population. These are intrinsically rare.
2. Local adaptation will depend strongly on the periods in which the local population is growing and large. These may be when the local climate is most similar to the center of the species’ range – in our case, subtropical Africa. But when the ecology is most similar to the source population, the selection pressure for change is the weakest.
3. Thus, local adaptation should be strongest for features of the environment that continued to constrain growth even during when the regional climate was mild. For example, Neandertal body form may not be an adaptation to cold per se, as much as it is an adaptation to winter, which continued even during interglacials.
4. Holt and Gomulkiewicz’s model assumes that the selected allele is a new mutation in the sink population. But it may be more likely that migration of low-frequency alleles from the source population may introduce adaptive alleles. Such alleles would be most likely to succeed if they were reintroduced multiple times. Hence, polymorphisms that actually occurred at appreciable frequencies in the source (to be introduced many times) would be most likely to contribute to local adaptations in the sink. To the extent that morphology reflects such alleles, we might find many of the features of Pleistocene Europeans in other Pleistocene populations at low frequencies.

What does this theoretical perspective tell us? Part or possibly all of the European population may have remained a sink, even over the evolutionary timescale, even while developing local adaptations. That’s a hypothesis, and it remains to figure out what kind of observations would test it.

The scenario predicts two things:

a. Locally adaptive alleles must not have been near fixation. If they were, the population was not a sink.

b. Ecological changes that reduced the absolute fitness of locally adapted alleles would have increased the importance of immigration. For example, glacial maxima might actually reduce the distinctiveness of the European population by making the persistence of local adaptations less likely.

If we don’t observe these effects, it would argue that the Neandertals had successfully established niche differentiation, with an evolutionary trajectory independent of other human populations.

UPDATE (2009-03-22):

I should also point to the longer paper by Gomulkiewicz and colleagues (1999), which covers the same theoretical ground in more detail. The earlier paper is a lot easier to get the general argument, which is based on a very simple algebraic formulation. But the longer paper covers more cases.

In particular, they examine what happens to a sink population after an adaptive allele is introduced, if then migration from the source is removed. Since the criterion for selection to succeed on the adaptive allele is that its absolute fitness is greater than unity (when rare), they find the expected: the sink population stabilizes at a persistent, though smaller, size.

Also, they show that the immigration rate determines the long-term survival of selected alleles in the sink population. That’s not unexpected either, but they give a clear statement of it in the discussion:

Density regulation has a number of potentially important effects on local adaptation in a black-hole sink. First, density dependence causes local fitnesses in the sink to depend on the immigration rate. In the absence of a locally favored allele and given stable dynamics, increasing immigration increases the equilibrium sink size. This demographic effect in turn depresses local absolute fitness, including the fitness of individuals that carry the favored allele. It is through this ecological effect that immigration impedes local adaptation rather than because migration swamps out local adaptation, as is assumed in standard treatments (294).

The authors manage to induce chaotic dynamics on the size of the sink population with relatively simple assumptions concerning density dependence. In the end, that has minor effects on the frequency of a selected allele in that population, but has very large effects on the population size. These dynamics are determined by the way that immigration depends on the current size of the sink population.

They find that density dependence also can determine the success of a locally adaptive allele:

Variable fitness is a complicated topic examined by Kimura in a number of papers. It’s interesting that density dependence can by itself induce the same kind of variation in the effect of selection as stochastic variation in the environment. If you’re thinking about Neandertals in terms of a distinction between the influence of climatic change and interaction with other populations, there may be no difference between these forces with respect to population dynamics.

References

Gomulkiewicz, R., Holt, R. D., & Barfield, M. (1999). The Effects of Density Dependence and Immigration on Local Adaptation and Niche Evolution in a Black-Hole Sink Environment. Theoretical Population Biology, 55(3), 283–296. https://doi.org/10.1006/tpbi.1998.1405

Holt, R. D., & Gomulkiewicz, R. (1997). How Does Immigration Influence Local Adaptation? A Reexamination of a Familiar Paradigm. The American Naturalist, 149(3), 563–572. https://doi.org/10.1086/286005