For most of their prehistory, humans were highly mobile hunter-gatherers. We can expect that Neandertals were also highly mobile, at least compared to sedentary post-agricultural human populations. Great apes are our closest living relatives, but they live in tropical forests -- a pretty different environment than the Neandertals. There are constraints on ape mobility, including difficulty of locomotion, habitat complexity, and extreme territoriality, that might not have constrained ancient humans, including Neandertals.
We might then consider the population structure of other highly mobile large mammals. Brown bears have been sympatric with humans in Europe since the Middle Pleistocene. Bear ecology has similarities and differences from Neandertals -- bears were omnivores accentuating meat consumption to a similar extent, but did not live in groups. Like Neandertals they may have exploited edges between habitat types, although brown bears are effective in open country as well.
For bears, like other European mammals, one of the most important questions is what happened to their population during the Last Glacial Maximum (LGM). The LGM was only around 18,000 years ago, so it's not an issue for Neandertals who were long gone by that time. But because the LGM is relatively recent, we have a relatively large representation of bear mitochondrial genetics spanning that time interval. So it gives us a chance to look at the relationship of population structure and genetic diversity in a large, mobile, European mammal. The bear comparison also lets us consider the effects of a smaller sample on our conclusions about ancient population structure and dynamics.
Brown bears are very common in archaeological and subfossil paleontological faunal lists. During the LGM, brown bears are known from northern Spain and Moldova. Evidence from today's bears suggests the occupation of at least four refugia (Sommer and Benecke 2005) -- basically Iberia, Italy, the Balkans and the Carpathians. These four areas can be expected to have housed substantial diversity during the LGM. The subsequent recolonization of northern Europe may be the largest factor organizing the present pattern of genetic variability, with the differential expansion of lineages through space.
Interspecific patterns of recolonization from refugia
Taberlet and colleagues (1998) collated phylogeographic evidence from 10 European species, ranging from plants to large mammals and including brown bears, to trace the likely pathways of postglacial recolonization of Europe. They found evidence for the importance of three refugia -- basically Iberia, Italy, and the Balkans. But most interesting, they found that each of their 10 species showed different patterns of postglacial expansion dynamics.
It seems that each taxon has responded independently to Quaternary cold periods, and therefore is largely a unique case with its own history. For example, if we compare lineages present in Italy and in the Iberic peninsula, they are closely related in Ursus (less than 1% of sequence divergence in the cytochrome b gene) but much more distantly related in Crocidura (6.4%), in Arvicola (7.6%) and in Triturus (8.5%), while the Sorex species considered here exhibit two lineages in each of these two refugia. Populations occurring in France come either from a refugium in the Iberic peninsula (e.g., Arvicola sapidus, Triturus marmoratus), or from a refugium in the Balkans (e.g., Chorthippus parallelus, Fagus sylvaticus).
...[T]he results obtained in Europe and North America (Zink 1996) suggest that congruence is the exception at the continental scale. The consequence of an independent history for each taxon is that assemblages of plants and animals comprising particular communities are not stable over time, an observation consistent with previous findings based mainly on fossil pollen data (Bennett 1990) (Taberlet et al. 1998:459).
Before going on to cite their conclusion, I want to note one possibility that they don't consider -- namely, that the species have similar dynamics of range constriction and expansion but that the mtDNA evidence represents these dynamics with substantial variance.
One aspects of that study stands out as interesting as applied to the Neandertals. Although the species did not share any single pattern of expansion from refugia, one aspect was shared: species did not expand from Italy. The authors speculated that the Alps are an effective barrier to rapid recolonization of northern Europe from Italian refugia, and indeed most northern species were recolonized either from Iberia or from the Balkans, or both. Thus Italy today contains many endemic lineages that were stuck in Italy during the LGM or other contractions, and never left. The possibility of an Italian-Croatian population of Neandertals was raised by Fabre and colleagues (2009). Was recolonization from this population possible during warmer phases of the Pleniglacial? If not, this population of Neandertals may have been exceptionally variable -- containing many long-standing endemic variants compared to other Neandertal populations. It may also have been substantially divergent from those other populations. Since Vindija is the most important source of the Neandertal genome, it's an important aspect of biogeography to try to understand.
Recolonization by brown bears
So much for the general pattern of recolonization. Now back to brown bears.
Sommer and Benecke (2005:161) considered further the present population of European brown bears and likely refugia in southern Europe. They returned to the genetic data developed in earlier studies by Taberlet and colleagues to conclude:
It is possible to detect three different glacial refugia from their data: (i) the Iberian Peninsula (Spain), (ii) the Italian Peninsula and (iii) the Balkans (Bulgaria/Greece). Furthermore, the investigation into the mitochondrial DNA of brown bears in Europe (Taberlet & Bouvet, 1994) shows four main points:
1. The individuals of southern Scandinavia originated from the Iberian Peninsula are closely related to the individuals from the Balkans and the Italian Peninsula, and form a 'western lineage'.
2. The bears of northern and eastern Scandinavia, from the Baltic States, from north-western Russia and the Carpathians differ with a sequence divergence of 7.13% from those individuals in the western lineage (Fig. 5).
3. Based on their genetic similarity, the brown bears from northern and eastern Scandinavia, the Baltic States, and north-western Russia are designated as 'eastern lineage' and a glacial refuge in eastern Europe is assumed to be the origin of this genotype (Hewitt, 1999).
4. Within the mitochondrial DNA of brown bears from the Carpathians, three different genotypes can be identified, whereas the genotype of bears from north of the Carpathians (Slovakia) is distributed throughout bears from Norway, Finland, the Baltic States and north-western Russia (Fig. 5).
They used these observations to argue for a refuge in the Carpathians during the LGM, which seems eminently reasonable based on their observations.
They did not point out (but I will add) that the expansion from an Iberian refugium toward Scandinavia mirrors the pattern of expansion of Magdalenian assemblages after the LGM. The recent literature has described this as a slow and tentative process of expansion (e.g., Jochim et al. 1999), but it was nonetheless as fast or faster than accomplished by small mammals, and may have mirrored the movements of the Magdalenians' large mammal prey animals. That human movement may also explain the distribution of mtDNA haplogroup H in Europe, which Pereira and colleagues (2005) attributed to a post-glacial recolonization from Iberia northeastward. This is not a new idea; Cavalli-Sforza wrote about this direction of postglacial migration some 30 years ago.
Later, Sommer and Nadachowski (2006) extended the map of refugia to take in more species, using faunal records from LGM archaeological sites. The map below helps to put these observations into context:
The possible ranges of human occupation and mammal refugia seem very extensive across southern Europe but are not necessarily contiguous. For example, the Alps form a partial barrier around the northern part of Italy, and the Pannonian Basin might partially cut off from Italy/Dalmatia as well. But it's not hard to imagine a large mammal like a bear (or a human) traversing the distances between such refugia, or walking along corridors between them such as the coasts.
More samples, more complexity
We have to remember that the interpretation of semi-isolated refugia has been based on the pattern of genetic variation in living species in Europe. But geographic differentiation need not only have occurred because populations were once fragmented during glacials. Differentiation may also be a product of range expansion, selection, or later interaction with other species, including humans. Today's differentiation is not necessarily a trace of refugia in the past.
So it becomes important to test the hypothesis of semi-isolated refugia, by looking at the variation of ancient DNA sequences. Last year, a study by Valdiosera and colleagues did exactly that -- looking at new sequence data from a larger set of brown bear subfossil remains from Iberia.
Here's a paragraph from the discussion of that paper:
Under traditional glacial refugia hypotheses (4, 17), the extant brown bear phylogeographic structure derives from ancestral glacial refugia: the western lineage originating from Iberia, Italy, and the Balkans, and the eastern lineage possibly derived from a Carpathian refugium (14, 16). In contrast to such a strict refugial model, but in concordance with a continuous European prehistoric population, we have identified a sequence from a Pleistocene Iberian brown bear from Arlanpe site (the Basque country) that belongs to the eastern clade. In our analyses, such a phylogenetic assignment is supported by maximal posterior probabilities (Fig. 1 A). This pattern is further supported by three Pleistocene brown bear sequences from Valdegoba (northern Spain), which cluster with a previously published sequence from Atapuerca (northern Spain) and with several sequences from modern Italian and Balkan bears. Furthermore, AMOVAs suggest little geographic substructure among Spanish and European Pleistocene populations. These new data confirm the lack of phylogeographic discontinuity in European brown bears before the LGM (23). Although Spanish and European Holocene populations appear geographically differentiated in our AMOVAs, a recent study has suggested that gene flow could have continued from the Pleistocene to the Holocene (20). An Iberian brown bear, dated to the time of the LGM from the site of Atapuerca in Burgos in the north of Spain, was more closely related to Italian/Balkan bears than to the Iberian ones. Moreover, during the Holocene in Mont Ventoux (southern France) three mitochondrial groups are found between 1,570 to 6,525 years B.P.: one belonging to the Iberian group, another one to the Italian/Balkan one, and yet a third one not associated with any of the three main glacial refugia (20). Note, however, that support for the Spanish and the Italian/Balkan clades are low in our tree. In this study, we have found three different individuals from Valdegoba, a Late Pleistocene site also in Burgos, that group together with the sample from Atapuerca (Valdiosera et al. 2008: emphasis added).
I think this study is so interesting because of the way it shows the influence of sample size on the phylogeographic interpretation. Consider how the conclusions of the study would have been different if the sample had been smaller. The authors found one Iberian Pleistocene bear that belonged to a clade otherwise comprising bears from Austria, Germany and Russia. This one bear is their clearest indication of ancient movement between plausible refugia. Had they not found a sequence in this bear, the evidence favoring two distinct refugia would have been much stronger.
Likewise, their sample includes three bears from Pleistocene France that belong to a clade of their own. This diversity no longer exists among today's bears -- at least, not the ones sampled up to now. If this region of France happened not to have produced bear remains, we would not have any evidence of this divergent clade at all. Again, the record would suggest that present bears derived from two largely isolated refugia. As it is, either another French refugium existed or the Pleistocene Iberian population harbored more diversity than present bears of Iberia.
That last element, a reduction of diversity over time, is also suggested by the pattern of variation between Pleistocene and Holocene bear remains. It has a lesson for the interpretation of human variation -- some human mtDNA haplogroups have reduced in frequency in recent Europeans, others have apparently increased. In the case of humans, we may be looking at selection. Brown bears have had a smaller effective size than humans during the last 10,000 years, so we might be looking at an actual reduction in numbers or geographic range.
In any event, with the bears every additional sample carries information about ancient population structure. We can expect that the addition of more Neandertal mtDNA samples will likewise add information about Neandertal population structure. The addition of samples is more likely to confuse a simple story than to confirm it, although either is possible.
Valdiosera and colleagues conclude that brown bears were actually highly mobile during the LGM, moving easily across a range that, although limited compared to earlier and later time periods, extended from east to west across southern Europe. It's hard to believe that Neandertals weren't capable of similar movement. On the other hand, chimpanzees are likely capable of long-distance movement but still have substantial population differentiation. This may be because intervening groups prevent individuals from moving long distances. So the dispersal character of Neandertal populations may have depended upon their social dynamics, an aspect of behavior that we are poorly situated to test.
Pereira L and 12 others. 2005. High-resolution mtDNA evidence for the resettlement of Europe from an Iberian refugium. Genome Res 15:19-24. doi:10.1101/gr.3182305
Sommer RS, Benecke N. 2005. The recolonization of Europe by brown bears Ursus arctos Linnaeus, 1758 after the Last Glacial Maximum. Mammal Rev 35:156-164. doi:10.1111/j.1365-2907.2005.00063.x
Sommer RS, Nadachowski A. 2006. Glacial refugia of mammals in Europe: evidence from fossil records. Mammal Rev 36:251-265. doi:10.1111/j.1365-2907.2006.00093.x
Taberlet P, Fumagalli L, Wust-Saucy A-G, Cosson J-F. 1998. Comparative phylogeography and postglacial colonization routes in Europe. Mol Ecol 7:453-464. doi:10.1046/j.1365-294x.1998.00289.x
Valdiosera CE and 10 others. 2008. Surprising migration and population size dynamics in ancient Iberian brown bears (Ursus arctos). Proc Nat Acad Sci USA 105:5123-5128. doi:10.1073/pnas.0712223105