I pointed a couple of weeks ago to the Les Rois Neandertal paper by Ramirez Rozzi and colleagues.
In the new article section of Journal of Human Evolution, Shara Bailey, Tim Weaver and Jean-Jacques Hublin have a paper that examines the Les Rois sample (among many others) in terms of dental discrete traits. Basically, they set up a discriminant function that can tell Mousterian Neandertals from later Upper Paleolithic people with around 89 percent classification accuracy, and then they applied it to Aurignacian and Châtelperronian dental remains. Here's what they concluded about Les Rois (p. 13):
The analysis of the entire Les Rois sample (n = 15) showed that they have an overwhelmingly ‘modern’ signal. Fourteen of the ﬁfteen individuals had high posterior probabilities of belonging to the Upper Paleolithic modern human group (>80%). This is not unexpected considering the teeth are associated with an Aurignacian industry (Dujardin, 2000) and most come from unit B, which has been dated to 28,715 ± 145 BP using AMS C14. One individual (Mandible B), however, was classiﬁed as Neandertal with a low posterior probability (54%) based on 13 traits.
Ramirez Rozzi (pers. comm.) has recently suggested that Mandible B represents a Neandertal, based largely on the asym-
metrical P4 together with some aspects of the corpus. Trinkaus (2007) has also argued that the specimens from Les Rois are mixed in morphology. While the P4 is asymmetrical, a large sagittal crack in the crown exaggerates this feature and the remaining aspects of the tooth are distinctively not Neandertal-like (it lacks a transverse crest and multiple lingual cusps: Fig. 6). In the end, we do not consider the posterior probability of 54% to be compelling enough to conclude, based on dental traits, that there were Neandertals present at Les Rois.
Ramirez Rozzi et al. (2009) made their conclusions about the Les Rois specimens mainly based upon perikymata packing patterns, secondarily supported by tooth sizes and the dental nonmetrics used here. So they are really concluding the same thing about mandible B. What's interesting is that Ramirez Rozzi and colleagues find several other dental individuals from the earlier unit with similar enamel formation patterns, and which they claim are also Neandertals.
In some senses, Les Rois is a good case study for how more complete specimens come to dominate the discussion to the exclusion of other fragmentary remains. Mandible B is itself a tiny piece of a skeleton, but because it has several different anatomical elements on that little piece, the two papers can conduct this kind of in-depth analysis. But the status of the other isolated teeth are equally important -- if Ramirez Rozzi and colleagues are right, they might be enough to establish the earlier Les Rois sample as standing apart from other early Aurignacian-associated samples. If we're talking about a single specimen, however well documented, the situation is somewhat different.
As everybody knows, my null hypothesis is that the samples are mixed in their morphological pattern. On that topic, the more interesting implication of the study by Bailey and colleagues (2009) is that when they applied their discriminant function to Châtelperronian samples, they got a similar classification frequency to Neandertals as for earlier Neandertal samples. And when they applied their function to Aurigacian samples, they got a similar classification frequency to modern humans as for later Upper Paleolithic samples. Under the hypothesis of a mixed transition from earlier Neandertals through later Châtelperronian Neandertals into early Aurignacian and into the Upper Paleolithic, you'd expect the intermediate time steps to give a lower classification frequency -- more Neandertal-like in Aurignacian than later; more modern-like in Châtelperronian than earlier. Bailey and colleagues found that their function classified 85 percent of Aurignacian dental individuals as modern (29/35) compared to 89 percent of later Upper Paleolithic individuals (56/63). Those numbers aren't significantly different, and given the variability in completeness of assigned specimens, I wouldn't go farther.
One possible criticism of the paper is that the morphological pattern that makes up the discriminant mostly consists of traits that are shared by both groups but differ in frequencies. It is still quite possible to use the traits to discriminate individual specimens, but somewhat harder to interpret what a change in trait frequencies means in genetic terms. Bailey and colleagues recognize this issue, and raise it in their discussion of the Oase 2 specimen, for example (p. 12):
Given that the cranium of Oase 2 is clearly not that of a Neandertal (Rougier et al., 2007), the assignment of this individual to the Neandertal group was unexpected. Trinkaus (2007) has suggested that, while essentially ‘modern,’ both Oase 1 and 2 exhibit a mosaic of cranio-dental features, some of which are archaic (e.g., dental proportions, long and ﬂat frontal bone), and others apparently derived towards anatomically modern humans (parietal curvature, absence of supraorbital torus) or towards Neandertals (unilateral lingual bridging of the mandibular canal).
It is important to note that the dental traits aligning Oase 2 with Neandertals are archaic in nature, as they are observed in other fossil hominins as well (Bailey, 2002b, 2006; Martínon-Torres et al., 2007). It is unfortunate that incisor morphology could not be assessed (teeth are missing), and that the upper M1s are too worn to ascertain occlusal polygon shape and occlusal polygon area, since these are features that are likely derived for the Neandertals/Neandertal lineage (Bailey, 2004; Gómez-Robles et al., 2007). Considering that some of the most diagnostic features of the upper dentition could not be assessed and that our approach is not 100% accurate, we caution against over-interpreting the classiﬁcation of Oase 2.
That's the basic problem of comparing Neandertals with humans, which we encounter genetically as well as morphologically. The groups differ in the frequencies of traits, but not often in the exclusive presence of distinctive ones. Sometimes, what once looked like a distinctive trait is then found in the other group -- so it's not distinctive anymore! Some researchers focus on "distinctive combinations" of traits, which tend to include a mixture of primitive and derived morphologies. But differences in trait frequencies automatically lead to differences in the combination of traits. Sometimes combinations are disproportionately represented (that is, putting traits together separates the samples more than considering them individually), but it is unclear how much of a trait combination may be explained by a history of inbreeding (in isolated populations) and how much may be explained by pleiotropy (of a few genes that differ in frequency).
You'd think this problem would be easier than it is. But look at Les Rois -- a fragmentary but interesting sample, with a blend of morphologies in different specimens. How do we interpret the similarity of Les Rois mandible B to Neandertals? Is it a Neandertal? If we considered the dental nonmetric features alone, as Bailey and colleagues suggest, the specimen looks like a Neandertal but with really rather weak evidence. If we add the perikymata and size data, the similarities with Neandertals are increased, and other teeth from the site also tend to look more Neandertal-like. But we know that perikymata patterns vary in recent human populations. How should we consider this variation as we compare the Les Rois teeth -- should we consider Europeans only, or modern humans more broadly? How did those traits change within the last 30,000 years, and is that relevant to the 10,000 years before?
Well, it's certainly enough to keep things interesting. I'm raising a lot more questions than offering answers. I like the approach Bailey and colleagues have taken because it includes much of the available sample in a way that can be considered as a unit. But then when we return the the issue of particular specimens and the possible patterns of genetic causation of the traits, we are left with the same problems as before.
Bailey SE, Weaver TD, Hublin J-J. 2009. Who made the Aurignacian and other early Upper Paleolithic industries. J Hum Evol (in press) doi:10.1016/j.jhevol.2009.02.003