Should we be surprised if Neandertals, Denisovans, and modern humans didn’t form stable hybrid zones?

Anthropologists have come a long way toward recognizing the complexity of modern human origins and dispersal. Ten years ago, I was one of a relative few who still maintained that mixture with Neandertals was important to our evolution. Over the last six years, the scale of Neandertal genetic introgression has been quantified by genetic comparisons from ancient DNA. We know that Neandertal ancestry contributed a small fraction of the genomes of living people in much of the world.

The amount of introgression varies across different parts of the genome. The Neandertal fraction of ancestry is lower in regions that are rich in genes. In some large blocks of the genome, nobody sampled today has any Neandertal DNA. Together, these observations suggest that some Neandertal DNA didn’t work quite as well in recent populations. Modern human samples from the Upper Paleolithic of Europe had a higher fraction of Neandertal ancestry in gene-rich parts of the genome, indicating that this fraction was not excluded during the initial hybridization, but instead declined over time under natural selection. At the same time, some Neandertal-derived genes were adaptively useful in later populations and have increased in frequency far beyond the average amount of genetic introgression across the genome.

In 2006, before any nuclear DNA results from Neandertals were known, Greg Cochran and I published a paper, “Dynamics of Adaptive Introgression from Archaic to Modern Humans”, that examined the way that natural selection and hybridization would interact as Neandertal genes were absorbed into modern human populations. The paper included some simple population genetic modeling and drew upon a rich series of examples from non-human animal and plant species.

The early 2000s marked a time when genetic data increased the power of molecular ecology and biogeography. Geneticists moved toward larger-scale sampling of mtDNA in natural populations. More important, they started to develop the nuclear markers that are necessary to quantify and study hybridization across the genome. The first useful cases were those involving domesticated and model species, for which nuclear markers had already been studied for commercial purposes. But wild species were following. Already, it was clear that hybridization and introgression were widespread phenomena. Many natural and domesticated populations actually derive a small fraction of their DNA from a history of hybridization with distantly related populations, subspecies or species. Greg and I drew upon this literature and projected what this phenomenon would predict for modern humans and Neandertals.

The evidence has of course changed quite a lot since then. In some ways, we predicted the outcome of later research that followed the publication of the draft Neandertal genome in 2010. Of course we failed to foresee many of the more interesting findings, but I’m proud that we captured the central idea of how a small fraction of the genome might behave in the later human population. More important from the standpoint of biogeography and ecology, the evidence on non-human species has vastly increased. Phylogeographers have documented mixture and introgression in well-known domesticates and their close relatives, in cases of biological invasions, and in many other cases where natural populations did not undergo any recent biogeographic disruption. Whole-genome techniques have brought hybridization and introgression to the attention of evolutionary biologists in a way that was not true even ten years ago.

The cool thing about having massive genetic data is that we can imagine answering questions that wouldn’t have been possible for anyone to ask in the past. One of those was asked this spring in a commentary by Ajit Varki, “Why are there no persisting hybrids of humans with Denisovans, Neanderthals, or anyone else?”

The title is unfortunate, because of course the vast majority of people in the world, possibly all of them, are descendants of hybrids with Denisovans, Neandertals, or someone else. But the article is not about that aspect of hybridization and its later history. Instead, Varki considers that stable hybrid zones exist today among many other closely related species or subspecies, and he asks a very direct question: Why did modern humans not form stable hybrid zones with contemporary archaic human populations?

Current genomic and archaeological data indicate that BMHs arose in Africa ∼100,000–200,000 y ago and spread across the planet (including the rest of Africa), encountering other extant hominins like Neanderthals, Denisovans, archaic African hominins, and possibly other lineages from earlier diasporas of Homo erectus. Although genomic evidence indicates interbreeding, the number of functional genes incorporated is limited, resulting in a “leaky replacement” (3), without persistence of true hybrids. Thus, our single BMH (sub)species was the “winner” in every contact/replacement event, spanning tens of thousands of years. I cannot find any other example wherein a single (sub)species from one geographic origin completely replaced all extant cross-fertile (sub)species in every planetary location, with limited introgression of functional genetic material from replaced taxa, and leaving no hybrid species. Typically, one instead finds multiple cross-fertile (sub)species, with hybrid zones in between.

Actually, in our 2006 review, Greg and I pointed to some instances of “extinction by hybridization”, in which an invasive population is proliferating at the expense of a local endemic population but absorbing genes from that endemic in the process. One well-known case is mallard ducks in New Zealand, which have been absorbing genes from the native New Zealand brown ducks even as the brown ducks have declined in numbers. This is not a rare phenomenon in nature. The phrase “extinction by hybridization” brings 1450 results on Google Scholar. Our 2006 paper benefited substantially from the classic 1996 review by Rhymer and Simbeloff, but examples of the phenomenon have massively increased since then. Varki’s opinion piece neglects this area of genetic evidence, which is understandable considering its short length.

Varki is quite correct that there is an interesting contrast. Some closely related species and geographic variants within species seem to have formed stable hybrid zones. Others have failed to attain such spatial equilibria and one species or variant displaces the others. Humans during the Late Pleistocene were an extreme example of the latter case.

Hybrid zones are epiphenomena of the evolutionary forces of migration and selection. If a hybrid zone between two species or genetically differentiated subspecies persists over evolutionary time, it is because selection maintains it. The ecological circumstances of the resident population may disfavor the traits of hybrids, impeding gene flow deep into the resident population’s range. In such cases, a hybrid zone may be a demographic sink, absorbing migration from both source populations. Selection may also favor reinforcement, in which the members of a population favor mating with other members of their own population at the expense of hybrids. “Extinction by hybridization” occurs when selection is weak relative to gene flow from the invading population over the entire range occupied by the resident population.

We don’t see stable Neandertal and Denisovan hybrid zones in today’s world because when those populations existed, selection did not oppose modern human dispersal and gene flow into their geographic ranges. There’s nothing particularly curious about that in itself. What may seem puzzling is that the expansion of modern humans into these geographic ranges followed several hundred thousand years of existence of Neandertals and Denisovans in those parts of the world.

But it is possible to read too much into the apparent long persistence of the Neandertals and Denisovans. DNA tells us that the Neandertals were a highly structured population during the later part of their existence. Different regions of the Neandertal range contained inbred regional populations with a history of very low gene flow among them. These Neandertal populations were nearly as different from each other as the most different living human groups were in the 1400s. Mitochondrial DNA suggests that the later Neandertals in central and western Europe were not simple descendants of the earlier population of Neandertals there, they may instead have had much more ancestry from the eastern part of the Neandertal range. Neandertals did not persist in Europe so much as they were continually supplanted by other Neandertals. The entry of modern humans may simply have been one more step in a series of partial population replacements.

With this population dynamic unfolding, it would be impossible for a stable hybrid zone to persist for long.

However, that raises a contradiction that we should consider. Ancient DNA shows that human populations were much more strongly differentiated in the Pleistocene than in recent times. Ancient DNA makes it apparent that regional-scale population replacements may have been very common during Pleistocene human evolution. Ancient DNA also shows that Pleistocene human populations were strongly inbred, with little genetic variation compared to recent human populations. How can all three of these observations be true?

In widely-traveled medium-sized mammals like hominins, strong genetic divergence is unlikely to have resulted from strong geographic isolation alone, unless selection helped to maintain it. Strong selection fitting populations to local and regional ecology would explain how regional differences could persist, and yet would also generate demographic replacements when regional climatic, ecological, or cultural factors changed. If selection played a role in generating and maintaining the genetic divergence of these populations, then there probably were hybrid zones among human populations during some parts of the Pleistocene.

I’ll add one observation: Cultural adaptation can allow invasive human populations to attain parity in the face of other human populations with long genetic adaptations to particular ecologies. Many archaeologists have pondered the possible cultural factors underlying the invasion of modern humans. An interesting question is whether cultural differences may have made crucial differences to even earlier population replacements.

References

Hawks, J., & Cochran, G. (2006). Dynamics of adaptive introgression from archaic to modern humans. PaleoAnthropology, 2006, 101-115.

Rhymer, J. M., & Simberloff, D. (1996). Extinction by hybridization and introgression. Annual Review of Ecology and Systematics, 83-109.

Varki, A., 2016. Why are there no persisting hybrids of humans with Denisovans, Neanderthals, or anyone else? Proceedings of the National Academy of Sciences, 113(17), E2354-E2354.