Sample sizes and the "Neandertal haplogroup"

10 minute read

I have an excellent e-mail question about last week’s Neandertal mtDNA paper, which has provoked a lot of commentary.

I just skimmed over your comments on the recent paper and I have a couple questions. First, how many Neanderthals did they receive mitochondrial DNA from? I think I read somewhere that it was fewer than ten.
Second if that is true, what the hell does it mean? I wouldn’t try and predict anything based on even fifty humans from that long ago much less 8 or 9 in genetic terms. I don’t think that anyone else would either unless they are grandstanding. You can’t prove a negative so they really can’t say that no modern humans have any Neanderthal DNA. Did they study Neanderthals from Asia? I just think they don’t have a good enough sample and until we can resequence a Neanderthal nucleus and bring the little tyke to term and wait for him or her to marry then wait for those kids to have kids will we really be sure we’ve got the goods.

Krause et al. (2007) list 15 Neandertal partial mtDNA sequences. Ten of these at that time presented relatively long portions, including the central Asian Okladnikov and Teshik Tash specimens, Mezmaiskaya, Feldhofer 1 and 2, Vindija 75 and 80, Scladina, Monte Lessini, and El Sidrn 1252. The same paper lists five additional specimens for which only a very short sequence had been recovered (just enough to diagnose as part of the Neandertal clade), including Vindija 77, El Sidrn 441, Engis 2, Rochers de Villeneuve, and La Chapelle-aux-Saints.

We do not know that every Neandertal belonged to the same mtDNA clade as those 15 sequences. Some of them may have looked different, possibly including the new clade otherwise present in later Upper Paleolithic and living people. But based on the 15 sequences we have, we can say that a large fraction of Neandertals must have carried the “Neandertal haplogroup.” Exactly how large a fraction depends on what we are willing to believe about contamination, preservation, and the randomness of our sample.

Now, let’s consider the question: Can we predict anything about Neandertal evolution and relationships based on this small, possibly unrepresentative sample of mtDNA?

The answer is that it doesn’t matter very much whether we have 5 sequences or 500. If 15 out of 15 specimens from different sites across Europe preserve a single mtDNA haplogroup, we can’t say it was universal, but we can say it was common. If 40 out of 50, or 400 out of 500 specimens had the same haplogroup, that would increase the precision, but not change the basic fact: Neandertals had at least one common haplogroup that is now so rare it has never been found in a sample of 100,000 or more people. We deserve some explanation.

The possible explanations are:

  1. Random genetic drift
  2. Accelerated genetic drift due to demographic turnover
  3. Population extinction and replacement
  4. Natural selection


Random genetic drift is fairly easy to refute, although it might not appear so at first. In favor of drift: There were few Neandertals, and the population size of the succeeding Upper Paleolithic, up through the Last Glacial Maximum, was also small—the best estimates are on the order of 2000 people for Western Europe and 5000 for continental Europe to the Urals (Bocquet-Appel et al.2005). There would have been perhaps twice or more that number across the entire Neandertal range. The effective population size represented by this population would have been smaller; perhaps 3000–5000 for Neandertals and Aurignacian-era people, only half, or around 2000, females. Genetic drift in this small mtDNA population would have been much stronger than for autosomal genes, and very much stronger than in most recent human populations.

But when we plug these numbers into a model of random genetic drift, it starts to appear very unlikely that drift alone could explain the observations. Let’s assume (falsely) that our Neandertal genetic samples all dated to 40,000 years ago, and the female effective size was 2000 individuals between then and 15,000 years ago, and that the population of Neandertal country were a random mating pool. Following these assumptions, on averageall the mtDNA genomes at 15,000 years ago would descend from only 4 or 5 ancestral copies in the population 40,000 years ago. If these five ancestral copies were, by chance, a different haplogroup from the 15 copies we’ve already found, then drift could explain the data.

However, this still doesn’t appear very likely. So far, every one of the Neandertals shares a single haplogroup. The frequency of this haplogroup was apparently very high, making it very unlikely that all five ancestral copies would have belonged to some other haplogroups of which we have never found any trace.

Notice that this argument does not depend very much on the number of Neandertal mtDNA sequences that we have found. The fact that there are 15 helps to constrain the frequency of the haplogroup within the population 40,000 years ago, in our model. That frequency is unlikely to be less than around 85%, assuming random sampling. But suppose there were only five. We would still know that the Neandertal haplogroup was very common in its population, even if we thought it was only 50%. It would still be unlikely to draw four or five ancestral copies and have all of them be some other haplogroup that we haven’t found.

This gives us a considerable confidence margin against drift. We need it. After all, the Neandertals were not randomly sampled at a single time, and it is possible that some of them actually carried a human-like mtDNA sequence, which we now falsely interpret as contamination. But even with these shadows hanging over us, it would still be unlikely that none of the ancestors of today’s mtDNA variation were like the Neandertal haplogroup.

Also, the population was not a random-mating pool. When we add geographic structure to the story, which tends to reduce the importance of genetic drift, we find that the possibility that drift alone is almost zero, and it remains very unlikely that a single migration of modern humans interbreeding with Neandertals under random drift could explain the observations, either (Currat and Excoffier2004).


It is at this point that most geneticists turn to the hypothesis of complete Neandertal extinction. They have a point. Genetic drift apparently cannot explain what we have observed, In their point of view, if genetic drift alone cannot explain the Neandertal mtDNA disappearance, then the only other random process at hand is extinction.

I think that hypothesis is false. It does not account for morphological similarities between Neandertals and later people, genetic evidence that suggests a strong ancient population structure with introgression, or with the apparent behavioral continuity in the Upper Paleolithic.

Happily, I don’t have a commitment to random processes. Instead, I think that the mtDNA evolution of Europe was driven by nonrandom processes of demographic turnover and selection.

Demographic turnover

Here we come to an important point. No one believes that later Europeans evolved from earlier Neandertals by a random process of genetic drift. Yet that is precisely the hypothesis that most studies have set up to refute. Without question it is valuable to set up boundary conditions under the hypothesis of random genetic drift. But the time has come to investigate more interesting models.

Personally, I am surprised that more complicated metapopulation dynamics have not gotten more attention as an explanation for the Neandertal mtDNA results. Population sources and sinks are a hot topic in biology, and you would think that anthropologists would have picked up on this. To my knowledge, the only time anyone has examined a population sink model was in 2001, when Milford Wolpoff and I worked with mathematician Per Enflo on such an idea for Neandertals (Enflo et al.2001). This idea deserves a fuller treatment (I think I’ll suggest it as a project for one of my classes this year!).

In a nutshell, a population sink is a region where the average rate of reproduction is below replacement levels. This region can remain populated only if individuals migrate in from other places. The places that reproduce above replacement are called population sources. The continual migration from sources to sinks creates a genetic gradient. Individuals sampled at any given time in the population sink are overwhelmingly likely to have ancestors not in the sink but in one or more source populations.

Europe today is a population sink. The population of the continent does not produce enough children to replace itself, and immigration from other parts of the world is high. There are several reasons to suggest that Europe may have been a population sink in prehistory as well. In Neandertal and Upper Paleolithic times, climate fluctuations created unique challenges in Europe, where caloric expenditures were high and food harder to obtain than some other regions.

Continual migration into Europe would provide a simple explanation for why none of today’s mtDNA haplogroups derive from the European Neandertals. The mtDNA population of 15,000 years ago had a few ancestors 40,000 years ago, and none of these ancestors lived in the sink population—all came from the source population in Africa or West Asia. The Neandertal mtDNA variation would have been a short-lived phenomenon, continually being turned over from source populations. Some Neandertal genes would have survived in Europe for hundreds of thousands of years, but some would have come in with more recent migrants from the population source.

There are points that argue against this source-sink hypothesis. The Neandertal-human divergence time for mtDNA is not very different than that estimated for the autosomal genome. If a European population sink had made genetic drift more powerful, that should have affected mtDNA more than the autosomes, so we might expect a more recent mtDNA divergence. Still, there is nor reason why the source-sink dynamic need have been constant over Neandertal evolution, and there may have been multiple sources in the Pleistocene, not only Africa and West Asia. Investigating the boundary conditions of the source-sink model and its correspondence to autosomal genetic results would be helpful.

I should note that mtDNA is not special. Neandertals had lots of traits that are now very rare. The horizontal-oval, or “bridged” mandibular foramen is a prominent example. Out of the relatively small sample of Neandertal mandibles, half have this derived form. Fewer than one percent of recent European mandibles have this form. As for mtDNA, a once-common variant is now very rare. And as for mtDNA, we deserve some explanation. A source-sink model would appear consistent with the continued evolution of such traits during the Upper Paleolithic—a time when the extinction and replacement hypothesis predicts no change in these characters.

Natural selection

The other nonrandom hypothesis is natural selection, which would presumably have favored one or more modern human types while eliminating the original Neandertal haplogroup. I won’t say much about that hypothesis here, since I discussed it in my initial post about the whole-mtDNA-genome sequencing. Selection has a leg up over the other hypotheses now because it seems like there’s good evidence it happened.

Still, selection on mtDNA alone could not explain the total pattern of observations about Neandertals. Physical traits that were once frequent in Neandertals were much less common or absent in later Europeans, and some continued to reduce in frequencies over time. To explain these changes, we must invoke either selection on other traits, or continued demographic turnover in the post-Neandertal population (probably more immigration into Europe) or both.

So selection on mtDNA has never been a sufficient or necessary hypothesis, even if we assume that other genes carried by Neandertals still survive. But given the current evidence that suggests something distinctive about the mtDNA of recent humans, natural selection may receive renewed attention as a factor in the disappearance of the Neandertal mtDNA haplogroup.


   Bocquet-Appel JP, Demars PY, Noiret L, Dobrowsky D. 2005. Estimates of Upper Palaeolithic meta-population size in Europe from archaeological data. J Archaeol Sci 32:1656–1668. doi:10.1016/j.jas.2005.05.006.

   Currat M, Excoffier L. 2004. Modern humans did not admix with Neanderthals during their range expansion into Europe. PLoS Biol 2:e421.

   Enflo P, Hawks J, Wolpoff MH. 2001. A simple reason why Neanderthal ancestry can be consistent with current DNA information. Am J Phys Anthropol 114:S62.

   Krause J, et al. 2007. Neanderthals in central Asia and Siberia. Nature 449:902–904. doi:10.1038/nature06193.