The Neandertal mtDNA story: 2004 edition

7 minute read

Note: I wrote this post in 2005. We have learned vastly more about Neandertal genetics since then. These two papers are important to the history of discoveries about Neandertals, but their results have been superseded by more recent work. I have left this post as a historical document, and more recent posts give a more modern view of Neandertals and their contribution to living and ancient people.

This week, two new papers add to our knowledge about the genetic variation of Neandertals and their genetic relationships–at least for their mitochondrial DNA–with succeeding populations of modern humans. Both of them also are freely available from PLoS, with links below.

Serre and colleagues expanded on the sample of four mtDNA sequences by attempting to find DNA traces in the 24 Neandertals fossils and 40 early modern human remains. The previous sample included two fossils from Feldhofer cave, one fossil from Vindija cave, and one from Mezmaiskaya cave in the Caucasus. For these four samples, the results have been relatively clear: the 4 Neandertals have mtDNA that is very closely similar, and as a group they differ from all known modern human sequences.

The new research used the knowledge that known non-Neandertal sequences are very divergent from modern humans as a way to identify Neandertal-like sequences by probing directly for them. This allowed a relatively rapid characterization of whether fossils preserved evidence of Neandertal affinities, without labor-intensive cloning or sequencing of ancient DNA. Out of the samples that they examined, 4 Neandertal remains and five early modern humans were preserved sufficiently to suggest that DNA might have survived. The 4 Neandertals included in two specimens from Vindija, Vi-77and Vi-80, Engis 2, and La-Chapelle-aux-Saints. All four of the specimens appeared to preserve sequences similar to the known sample of Neandertal sequences. In contrast the five modern humans, including Mladec 25c, Mladec 2, Cro-Magnon, Abri Pataud, and La Madeleine, did not amplify using the Neandertal primer despite the appearance of DNA preservation. Sarah and colleagues took their results as evidence that Neandertal-like DNA sequences disappeared along with the Neandertals, and were not preserved in any known modern human remains.

There are some weaknesses in the study design, which the authors acknowledge. Although they surveyed modern human remains for Neandertal-like sequences, they were unable to look for modern-like sequences in Neandertal remains. There is indirect evidence to suggest that they wouldn’t have found such sequences anyway, since the four Neandertals and that appeared to present evidence of DNA preservation all amplified with the Neandertal primer. But those who have questioned Neandertal results because they could not distinguish modern-like sequences from contamination have not had their questions answered here. The fact that none of the modern human remains preserved Neandertal-like sequences is the important result. This adds to evidence from one prior study (Caramelli et al. 2003), which sequenced two individuals from Paglicci cave dating to around 24,000 years ago in southern Italy. Like the modern remains in this study, the Paglicci remains showed no evidence of Neandertal contribution.

The second study, by Currat and Excoffier, attempts to model of the movement of modern humans into Europe as they replaced the Neandertals. The motivation for this study was the unrealistic nature of earlier attempts to model this movement and its effects on DNA variation. In essence, earlier analysts like Nordborg (1998) assumed an island model in which the Neandertals and modern humans comprised two gene pools from which later European DNA might or might not have been drawn. Given this type of simple model, it is possible to work out the minimum ratio of Neandertal to modern human contribution that would be possible given the observed absence of Neandertal mtDNA sequences in the succeeding population of Europeans. In other words, these attempts were ways to determine the prior probability of certain mixtures of ancient populations on the basis of the observed frequency of their alleles in recent humans: of course, under the special case where the observed frequency of one of their alleles is zero.

Two pieces of evidence have tended to improve our ability to make these kinds of estimates. One of them is the increasing availability of samples immediately succeeding the disappearance of the Neandertals. Presumably, if Neandertals mitochondrial DNA survived it would have a higher frequency in these early populations than it does in living Europeans. At present, the frequency of Neandertal-like sequences in living Europeans is zero, and the number of early modern Europeans with Neandertal-like sequences also may be zero, although it is possible that at least one specimen does present such a combination (Hawks and Wolpoff 2001).

The second source of evidence concerns the possible demography of ancient humans. It is in this realm that the paper by Currat and Escoffier makes its contribution. The study incorporates two assumptions that attempt to model the complex interaction of Neandertal and early modern populations. The first assumption is that the two populations did not mix instantaneously with each other, but instead coexisted alongside each other for a long period or time. The second assumptions is that the two populations did not mix throughout their entire range over this time period, but instead mainly interacted along a wave front of dispersing modern humans. The effective both of these assumptions is to increase the expected importance of Neandertal genes in the modern human gene pool. The assumption of a long coexistence gives Neandertals added opportunities to mate with modern humans, while the assumption of a spreading wave of moderns adds the potential that some areas of Europe would have relatively high Neandertal contributions even if the overall level of Neandertal genetic contribution was low. The authors actually ran several different demographic scenarios to find a range of results.

They found that all of their models are united in expecting a greater contribution of Neandertal mtDNA than the simple instantaneous mixture model. In the slightly-backward logic of the problem, the data (which indicate no evidence of Neandertal mtDNA in living Europeans) make at least one of the following conditions unlikely: that Neandertals had a substantial level of mtDNA exchange with modern human populations, or that Neandertals survived alongside modern humans for a substantial length of time. In other words, under the assumptions of the models used here, Neandertals are very unlikely to have mixed with modern humans spreading into Europe to replace them.

Of course, the problem with any of these studies is in the assumptions that they require. For my part, I think the demographic assumptions of Currat and Excoffier are relatively unobjectionable, and certainly an improvement over those used by Serre and colleagues as well as many earlier studies. But neither of the studies cites the really important paper in this regard, that of Mandersheid and Rogers (1996). In that paper, for the first time, the authors worked out the genetic consequences of demographic assertions about the Neandertal-modern transition. They found that Neandertal genes were quite unlikely to have gone extinct in an expanding post-Neandertal population, unless the Neandertals themselves had failed to contribute to that population. After this paper, the consequences of the Neandertal problem under purely neutral assumptions (like those used by Nordborg 1998) were really irrelevant anyway. Currat and Excoffier add a dimension of complexity to the problem, and therefore a further note of realism, but their conclusion was foregone.

The real elephant under the rug of these papers (or as I’ve said elsewhere, the 800-pound gorilla) is natural selection. Both of the papers rely on the assumption that mtDNA is neutral. This is, in a sense, necessary to the papers’ existence, since without this assumption mtDNA may be considered to be completely uninformative about the Neandertal problem. But there are good reasons to think that mtDNA has been under positive selection recently in human prehistory. Most notable among these reasons is the fact that human mtDNA violates every test of neutrality. Also suggestive is the limited mtDNA variation among the known Neandertal sequences–a suggestion that the positive selection that has affected human mtDNA recently may be just the most recent of several episodes throughout human evolution. Until papers like these take the issue of selection seriously, there is little chance of finding consensus on the Neandertal genetic problem.


Caramelli, D., Lalueza-Fox, C., Vernesi, C., Lari, M., Casoli, A., Mallegni, F., ... & Bertorelle, G. (2003). Evidence for a genetic discontinuity between Neandertals and 24,000-year-old anatomically modern Europeans. Proceedings of the National Academy of Sciences, 100(11), 6593-6597. doi:10.1073/pnas.1130343100

Currat, M., & Excoffier, L. (2004). Modern humans did not admix with Neanderthals during their range expansion into Europe. PLoS Biology, 2(12). doi:10.1371/journal.pbio.0020421

Serre, D., Langaney, A., Chell, M., Teschler-Nicola, M., Paunovic, M., Mennecier, P., ... & Pääbo, S. (2004). No evidence of Neandertal mtDNA contribution to early modern humans. PLoS Biology, 2(3), 313-317. doi:10.1371/journal.pbio.0020057