Laurent Excoffier and colleagues' work has investigated how range expansions may have affected human genetic diversity. I've commented on this work several times ("One model, hold the extra parameters", "The Neandertal mtDNA story, 2004 edition, "Surfing and recent selection). They have applied a "geographically explicit" model to questions of human population history, modeling how populations expand and interact in the face of a simulated model of the Old World.
In the past, I've found some things I like in this work, and other points where I disagree with the models' assumptions. Personally, I like to examine analytical models first, because the assumptions are often much more explicit, so we can see more easily how the results follow from them.
This week, a new paper by Mathias Currat and Excoffier in PNAS claims to find evidence for some degree of reproductive incompatibility between Neandertals and modern humans. This is another case where I think the approach is very clever but I disagree with the model's assumptions. I just don't believe that today's distribution of genetic variation can tell us about "reproductive incompatibility" with Neandertals or other archaic people.
Today, if you take a large random sample of people within the continental U.S. and look for a DNA legacy from Precolumbian American people, you will find it in your sample at a level somewhat under 2 percent. This percentage results from the differential growth of European and African-derived peoples during the last 500 years of American history. Whatever else it may be, the current percentage is notevidence for hybrid incompatibility of the world's populations before 500 years ago.
It's not a perfect analogy. Today, Native American ancestry is heterogeneous in the continental U.S., with some people carrying very high fractions. After 30,000 years, such heterogeneity would likely have balanced out. With the Neandertals, we are looking at a much longer history, and different events.
I would contend that the events that have affected today's representation of Neandertal-derived genes were demographically larger than those leading to the European colonization of the Americas. The contraction of the European population during the Last Glacial Maximum, the subsequent movements of Late Upper Paleolithic and settlement of Mesolithic peoples, followed by the introduction of agriculture and waves of population growth and invasions, have partially erased the genetic patterns of the initial Upper Paleolithic. We know that the mtDNA complement of Europe changed markedly both before and after the Neolithic. Today's Europeans are not the people who encountered the Neandertals 35,000 years ago. The genes of those initial Upper Paleolithic people may be almost as rare today as Neandertal genes.
Range expansions and surfing
Nevertheless, I think the analysis in this paper gives us some valuable information about how populations may have interacted at that final stage of population mixture among archaic populations.
A range expansion occurs when a population that is initially limited to some small area begins to expand outward across a larger area. The expansion may include interbreeding with other populations who already occupy those areas, for example, the movement of Neolithic agriculturalists into Europe. Or the range expansion may go into territory where nobody lives, like the initial habitation of the Americas some 14,000 years ago.
Range expansions can distort allele frequencies beyond the pattern expected in a random-mating population. As the population pushes its boundary outward, individuals at the frontier carry with them a slightly skewed sample of the alleles in the population as a whole. Pushing further and further along, this skewed sample gives rise to a founder effect. This phenomenon has been called "allele surfing", by analogy with a spreading wave of population expansion.
When a population expands its range into that of another population, the invaders usually mate with the natives. As the "wave" of migration continues to spread, more and more of the natives' genes are picked up into the expanding population. As a result, you expect to see a gradient of genetic contribution from the original native population, higher and higher as you look farther from the invaders' point of origin.
Currat and Excoffier  assume that a group of 50 people originated in Northeast Africa 50,000 years ago and then began to spread throughout the Old World. This population (moderns) expands into the range of another human population (Neandertals) by virtue of a higher carrying capacity: in fact they assume that modern humans existed at four times the density of Neandertal populations. The modern human value is set at 1 person per 10 square kilometers, which is very low compared to ethnographically described hunter-gatherers. The population as a whole is made up of demes that occupy an area 100 km on a side (in some trials, four times as many demes 50 km on a side). The outcome is inevitable: the higher carrying capacity leads modern humans to replace Neandertals, while incorporating some amount of Neandertal ancestry.
Any model is unrealistic to some extent. An unrealistic model generally leads to results that are very different from reality. In modeling, there's a common strategy to deal with this problem: Leave one free parameter and change it until the results fit reality. In this case, the free parameter is migration rate, the probability that an individual will move to an adjoining deme. Currat and Excoffier used values for this parameter that caused their "modern" population to displace Neandertals in Europe over a span of 6000 years. The value that made this dispersal speed was 20 percent per generation for the dispersing modern human population.
I'm a little concerned that a whole literature of geographically explicit population models has emerged in human genetics without any apparent reference to the anthropological literature on human demography. If you know ethnography, a migration rate of 20 percent per generation over 100 km distances seems very high. It's more than double the observed rate of intertribe marriages among precontact Aboriginal Australian people, for example. The value of one person per 10 square kilometers for population density is near the low end ever observed for hunter-gatherers. If it's a stretch to make a model fit with parameters found in known hunter-gatherers, that's when I go back to the drawing board. But then, my philosophy about this is different from most human geneticists. I'm an anthropologist.
Anyway, with these values the result is foreordained: modern humans will replace Neandertals, and fast. What Currat and Excoffier observe in their simulated populations is that the modern humans tend to pick up a larger fraction of Neandertal genes, especially in Europe. How can we explain why our population today has a relatively small fraction of Neandertal genes? In particular, how can we explain why Europeans have no more Neandertal genes than any other population? They conclude that some kind of reproductive incompatibility must have existed.
Where I think the method falls short
I think this paper would be perfectly reasonable if I was willing to assume that the range expansion of modern humans was the last major event in our evolution. If this were true, then echoes of this range expansion would be the most highly visible today — just as astronomers can still find echoes of the Big Bang in the cosmic microwave background.
But I would offer that our genetic diversity today is not the result of a single Big Bang of movement out of Africa. Many population movements of comparable or even larger scale have happened during the last 30,000 years.
The paper presents the hypothesis of reproductive incompatibility as an attempt to solve two problems: First, Chinese, New World peoples, Southeast Asians and Europeans today have approximately the same amount of Neandertal ancestry. Second, the amount of Neandertal ancestry in Europe is only around 2-4 percent. A 6000-year wave of population growth and mixture as modern humans entered Europe might have left more Neandertal genes, and a higher proportion in this Neandertal-rich area of the world than in East Asia.
Here's how I currently see those problems. Europeans today are not the Europeans of the past. They have undergone massive population movements and replacements since the initial Upper Paleolithic people encountered Neandertals. That's not only the result of archaeology, it's also clear from the paleogenetics. If we recognize this subsequent history, then we will find it easy to explain why the rest of the population outside Africa has basically the same small amount of Neandertal ancestry: they received a massive influx of genes from some West Asian population with Neandertal mixture. Europe also got these genes, mostly long after the initial Upper Paleolithic.
So I don't think the present fraction of Neandertal genes tells us anything about sex between Neandertals and humans, except that it happened. Many times. Hooba-hooba.
There is some irony in the timing of this publication, since only last week PNAS published a paper claiming that today's African populations derive some of their DNA from a population fully twice as different from non-Africans as Neandertals were.
I don't fully believe that, either.
- . Strong reproductive isolation between humans and Neanderthals inferred from observed patterns of introgression. Proceedings of the National Academy of Sciences. 2011;108(37):15129 - 15134.