There's a new paper in PLoS ONE by Virginie Fabre, Silvana Condemi and Anna Degioanni, titled "Genetic evidence of geographical groups among Neanderthals." I think this is an ambitious paper -- it uses 12 mtDNA sequences recovered from Neandertal fossils to compare different phylogeographic scenarios for Neandertal populations.
The authors applied several different models to the data, attempting to find a population history that matches the geographic distribution of mtDNA diversity in Neandertals. They found that a model in which Neandertals had been part of three long-standing geographic populations was a better fit than others. Here's the relevant part of the abstract:
In this paper we used a new methodology derived from different bioinformatic models based on data from genetics, demography and paleoanthropology. The adequacy of each model was measured by comparisons between simulated results (obtained by BayesianSSC software) and those estimated from nucleotide sequences (obtained by DNAsp4 software). The conclusions of this study are consistent with existing paleoanthropological research and show that Neanderthals can be divided into at least three groups: one in western Europe, a second in the Southern area and a third in western Asia. Moreover, it seems from our results that the size of the Neanderthal population was not constant and that some migration occurred among the demes.
I like the study, and I have no strong objections to the conclusion. It has always seemed sort of likely on morphological grounds that Neandertals may have had modest geographic differentiation. Amud 1 doesn't look like a French Neandertal; nor does Teshik Tash. So I'm inclined to think the results are not too surprising.
Still, the data have some big weaknesses. Phylogeography is a tall order when we only have 12 sequences.
Many have pointed out, going back to McCown and Keith (1939), that time is another possible cause of morphological differentiation of Neandertals. The mtDNA sequences cover a wide range of times -- the Scladina sequence comes from roughly 100,000 years ago, the others cover the span from 50,000 down to 29,000 years ago. Why not test temporal groups instead of geographic groups? Temporal clusters might reflect interglacial colonizations, differential gene flow, or natural selection. There is a good precedent -- last year a report of complete mtDNA sequences from woolly mammoths found evidence for geographic structure among mtDNA lineages, one of which apparently replaced the other (Gilbert et al. 2008).
Time is just one example of an alternative model for variation. But I think it helps to clarify the basic problem of the a priori models -- you have to draw boundaries between the specimens somewhere. In the current paper, Fabre and colleagues divided the samples into one, two, or three groups. The one-group model amounts to a simulation of panmixia. The other models are a little like the setup of a Bayesian STRUCTURE analysis -- how well does the sample fit a model in which the the latter as the more likely null hypothesis.
But unlike STRUCTURE, in this case, each specimen had to be assigned deliberately to one group or another. That's why the authors generated three different versions of their three-group model -- in each version, the boundaries between groups were drawn in slightly different places.
That's not a criticism of the paper; it's just an inherent property of the method. There's no better way to come up with boundaries of the groups, and I've done similar things in earlier work. It's rational to have the groups contiguous with respect to geography, but without clear isolating barriers, no special reason why the groups should be bounded along any particular line.
In this case, one of the three-group models provided a substantially better fit between simulated data and the observed mtDNA sequences. So the paper concludes that three groups are supported.
Thinking about it, I would probably use the data to test a slightly different set of hypotheses.
I would start with an analytical approach, explicitly testing the hypothesis of panmixia; then explicitly testing isolation-by-distance. Panmixia should be easy to refute -- if you don't then phylogeography is a non-starter. I see isolation-by-distance as the appropriate null hypothesis, and while the time-dispersion of the samples makes life a little more complicated, a simple test of IBD would be straightforward. I don't expect you need simulations for either of these tests, although you could use simulations to explicitly include the ages of the specimens in the test.
One reason to start with IBD is that the specimens are heterogeneously distributed through space. Since there are some large gaps in the geographic distribution, the observed sequences may tend to clump into groups even if no real boundaries between groups existed. The best-supported model in the paper divides the sample with a clump of Italian and Croatian specimens, a large gap between Ukraine and Uzbekistan, and most of the specimens in one large group. That looks like a pattern that might be consistent with IBD, complicated by the actual ages of the specimens.
At any rate, the conclusion in the paper should make one set of people nervous: those who think that Paleolithic archaeological industries reflect populations. I can't see any obvious alignment between these three "groups" of Neandertals and well-known cultural units at any time interval. There are some localized and relatively long-lasting industries or variants within the boundaries of some groups, and there are others that span the boundaries.
Now, if these groups really reflect long-standing population boundaries -- spanning some 100,000 years in the model -- then we might expect it would have been hard to exchange information across them.
The same should have been true of Africa, which I've mentioned shows evidence for population differentiation going far back into the Late Pleistocene if not earlier. In Africa, the MSA shows both long-standing variations in different regions and relatively rapid temporal fluctuations within regions. The same general picture holds for the Mousterian, although one may argue whether the correspondence is exact. In any event, the African regional variants show no obvious correspondence to the genetic differentiation of African populations today. Maybe that's because of subsequent changes in the African population -- today's differences don't necessarily reflect those of MSA populations.
But suppose we take the Neandertal model seriously. Information transfer in living people occurs on a much more rapid timescale than genetic exchange. That cannot always have been true in human evolution -- it isn't generally true of other primates, where long-distance information transfer basically depends on the transfer of individuals from their natal groups. What should an intermediate stage look like, in which the amount of information transfer may be less than in recent human groups (with writing, accounting and vastly more people), but the pace of transfer may have been comparable? I doubt they would correspond well to genetic populations over much longer timescales, although they may be limited by them to some extent.
Are these Neandertal races?
I raised the question in my class today. If these really are groups of Neandertals, occupying different geographic ranges for a hundred thousand years, what do we call them? I thought it was a good lead-in to talk about species concepts in paleoanthropology, and of course it is.
If these aren't species, why aren't they? Presumably because we think that genetic exchanges across this range would have been likely. You can test the hypothesis by comparison with living humans and other primates. Mitochondrial phylogeography of human populations includes some long-standing population structure going back more than 60,000 years. Within great apes, there are long-standing subspecies that go back much further, hundreds of thousands of years. In humans, we tend to call the resulting groups races or populations. Among great apes, we tend to call them subspecies.
So are these subspecies of Neandertals? Races? Geographic populations? I wouldn't interpret further without really determining the nature of the boundaries here. As I mentioned earlier, I think the null hypothesis is isolation-by-distance. It's conceivable that the Neandertal population was patterned in a similar way to recent humans -- although considering our rapid recent evolution, I wouldn't be quick to assume that human differentiation is a good model.
One other thing. Let's assume that the Neandertals really were differentiated from each other, and that the groups proposed by Fabre and colleagues are generally right. In that case, the Neandertal Genome Project has been concentrating on an individual from the Southern subpopulation, a subpopulation otherwise very far the population interface between Neandertals and other humans before 45,000 years ago. Hence, that sequence may be a bad place to look for evidence of interactions between Neandertals and modern humans. Genetic exchanges are more likely to have happened across long-standing areas of contact -- which in Fabre and colleagues' best model, would likely involve the Western or Eastern subpopulations.
That's entirely speculative on my part, but it does seem to be one implication of the model.
Fabre V, Condemi S, Degioanni A. 2009. Genetic evidence of geographic groups among Neanderthals. PLoS ONE 4:e5151. doi:10.1371/journal.pone.0005151
Gilbert MTP and 32 others. 2008. Intraspecific phylogenetic analysis of Siberian woolly mammoths using complete mitochondrial sequences. Proc Nat Acad Sci USA 105:8327-8332. doi:10.1073/pnas.0802315105