Look to the baboons; there will you your insights find!

Clifford Jolly's review article in the 2001 Yearbook of Physical Anthropology pretty much covers every aspect for which baboons make an analogy for human evolution. These include Jolly's own "seed-eaters" hypothesis, the implications of baboon diversity for early hominid diversification, the spread of features across geographically dispersed populations, and the implications of baboon hybrids for hypotheses of modern human origins.

Jolly spends a lot of time talking about the implications of hybrid sense and population replacement for the evolution of early Homo. It's interesting in its new thinking, and well worth going over. There's no sense reviewing it all, since you could just read it, but I'm working on modern human origins problems myself right now and found the following passages relevant:

The fragments of Neandertal mtDNA sequence (Krings et al., 1997; Hoss, 2000; Ovchinnikov et al., 2000) suggest the point at which the Neandertal story can be linked to the analogous history of baboons. Discussion of the Neandertal mtDNA sequence has focused mainly on its relatively ancient separation from the root of all extant human sequences, and its implications for a Neandertal genetic contribution to modern human populations. From the baboon (or chimpanzee, or gorilla) perspective, however, the separation is not very ancient. It is comparable to 600 ka divergences between olive and hamadryas baboon mtDNA haplotypes, and much more recent than, e.g., the Guinea-hamadryas split. Mitochondrial diversity in Papio may be analogous to the condition in Homo before the "event" (generally interpreted as an "out-of-Africa" expansion of a relatively small subpopulation) that eliminated most of the diversity from its collective mitochondrial (and Y-linked, and autosomal) gene-pool. Unfortunately, investigation of continent-wide genetic phenostructure in Papio is still in its earliest stages, so we cannot pursue the analogy further in this direction. We can, however, make some suggestions based on work in contemporary zones of hybridization, especially the Awash anubis-hamadryas hybrid zone. For example, we can conclude that unless an undocumented, radical genetic event occurred in the 600 ka since they shared mtDNA ancestry with the Neandertals, premodern humans were certainly able to interbreed with them and produce viable, fertile, offspring, as hamadryas and anubis baboons do (Jolly 2001:198).

Jolly notes that evidence of hybrids may only occur within a hybrid zone itself, which suggests difficulty in examining the existence of such scenarios in fossil contexts.

For the human case, this has an important implication: demonstrating phenotypic distinctness (lack of overlap) of Neandertal
and "modern" samples drawn from areas remote in time and space from the zone of contact does not disprove the occurrence of interbreeding at the interface. It also means that the Lagar Velho child, if indeed it is a hybrid, is a rare and valuable find, even though it is irrelevant to the Neandertal "species question," and does not tell us whether Neandertals (or other "archaic" humans) contributed genes to the Upper Paleolithic, or the extant, human gene-pool. Not that these are equivalent, as is often implied; there was ample opportunity for the loss of a few stray Neandertal genes from European Upper Paleolithic populations when the latter shrank and were replaced by food-producing peoples (Jolly 2001:198-199).

This is probably true from a morphological point of view -- morphological mixture will be evident only at the time and place where different populations were clearly in contact. It is less true of individual alleles or features, which might well intergrade much more extensively depending on their selective dynamics. The least persistent evidence of mixing will be features that are substantially multigenic -- which of course probably includes most of the anatomical features with which anthropologists are familiar. On the other hand, individual features or genes might well be expected to persist long after Neandertals themselves disappeared. This appears to be the case for the features examined by Frayer (1993). In that case, the finding of actual "hybrid" individuals is not so relevant: what is important is the observation of changes in trait frequency within larger, more temporally-dispersed samples.

Jolly later discusses the dynamics of hybrid zones and their application to the discovery of Neandertal-modern mixture:

In the Neandertal case, the fact that the interface moved historically from east to west indicates that the pressure of gene-flow was greater in that direction; if a hybrid zone existed, the genes in it were contributed disproportionately by "moderns." "Neandertal morphological genes" may have been removed by natural selection from a narrow zone of hybridization, or been swamped by differential genetic inflow, or perhaps they simply died out with their carriers without any hybridization at all. Any combination of these factors could have contributed to their disappearance. A much more fine-grained temporal record of the transition would be necessary to decide between these alternatives, and the precise scenario is immaterial both for the eventual outcome, and for the so-called "species question" (Jolly 2001:199)
What is important, and hotly contested, is whether Neandertals (and other archaics) contributed any genes to the gene-pool of the human population who succeeded them. This would imply a
flow of genes from the marginal hybrid zone into the expanding modern population: swimming, as it were, against the tide. The important question is not whether Neandertals could have passed some genes by hybridization to incoming Afro-Arabians; they almost certainly could. It is certainly not the neoessentialist (Cartmill, personal communication) red herring of whether or not they were "really" different species. The important questions are purely empirical: first, whether they actually did contribute any distinctive alleles to the incoming population, and second, whether any of these have survived post-Pleistocene upheavals in the human gene-pool. The first question can only be answered by genetic investigation of the DNA of post-Neandertal fossil humans (cf. Hawks and Wolpoff, 2001); the second by trawling the extant human gene-pool itself (ibid.).

Of course, the question of whether they were "really" different species may be deprecated by those who deal with the fuzzy boundaries between species in nature, but "species" is a term that carries loaded meaning for most biologists. Calling Neandertals a different species is tantamount to asserting their irrelevance to the ancestry of recent humans. This is the point behind the "assimilation" model of human origins -- modern humans actually were a different "thing" than Neandertals, and when the two "came into contact," one group "assimilated" the other. If "assimilation" didn't carry this meaning, there would be no reason to talk about it as a model separate from multiregional evolution, or restricted gene flow and isolation by distance, or "mostly Out of Africa", or any number of other names. All these models agree on the presence of Neandertal genes in later people. Where they disagree is in the emphasis. None of them disagree that Neandertals had an evolutionary history different from other regions. But they disagree about just what kind of history that was. So the question of "species" is a central one, not one that can be shoved under a rug.

From that perspective, the important questions are not merely empirical. They are also conceptual. The "assimilation" model depends on a rather complex conceptual scenario. It envisions the differentiation of Neandertal (along with other archaics) and modern populations over some substantial time. During this time, the evolving modern population within Africa gathered steam for its ultimate dispersal, while the Neandertals and other archaics proceeded along their own unique evolutionary trajectories. Finally, the reestablishment of contact among these populations led to the genetic assimilation of most archaic groups and the establishment of a majority-African gene pool throughout the world. In this hypothesis, the initial isolation (which may have been partial or complete) is essential to the ultimate result. Only if Neandertals had become relatively isolated and divergent could their ultimate assimilation make any sense. Thus, the conceptual basis of Neandertal assimilation is their initial speciation, or if not "speciation" in the formal sense, at least their origin as a distinct population and divergence through substantial isolation.

Now this scenario may or may not have happened; we really don't have the data to test it in comparison to its less conceptually elaborate alternatives. But it is not mindless essentialism to note that this hypothesis depends for its reality on a certain historical identity for Neandertals. This identity is potentially testable. Some may find it distasteful to argue about what a species is, since the concept is so variable and messy in its application to living populations, let alone fossil ones. But that doesn't allow us to avoid the issue: what we call things has meaning, particularly to those outside the intricate details of the fossil record.

Jolly points to the baboon example as having important implications for this case of modern human origins. I agree. However, pointing out the analogy between baboon hybrid zones and the Neandertal-modern transition does not make the latter a case of the former. I have no doubt that Jolly would agree quite fully, but it is worth pointing out nonetheless.

In the meantime, the baboons do give a clear notion of the direction that we should look for evidence of Neandertal-modern interactions:

So far, almost all genetic systems investigated in extant humans show no signs of a Neandertal inheritance, but perhaps we need to be more selective in our search. A moving hybrid zone may leave in its wake a few neutral markers derived from the retreating population (Arntzen and Wallis, 1991), but these are likely later to be eliminated by drift. Most likely to survive and be incorporated are genes for traits strongly favored by local conditions (and "hitch-hiking" markers linked to these). Some years ago, a popular work (Kurtén, 1971) plausibly suggested that Neandertals were blond and blue-eyed in adaptation to cloudy, periglacial Europe, while incoming "moderns" had the darker pigmentation of a subtropical people. Perhaps we should survey nordic Europeans for unusually "deep" diversity in noncoding genetic elements closely linked to loci determining pigmentation... (ibid.).

Of course by the publication of this review, precisely that had been attempted by the survey of MC1R variation by Rosalind Harding and colleagues (2000), finding that the so-called ginger allele may be ancient enough to have come from European Neandertals (reviewed here).

Jolly continues with an interesting hypothesis about possible immunological retentions from Neandertals:

Less fancifully, Parham et al. (1994; and Parham, personal communication) speculatively identified a possible Neandertal legacy: an allele of the human MHC system that is found at low frequency in the old Neandertal range. It is remarkable for its inferred ancient separation from other alleles, which themselves form a tight, young clade. MHC alleles are among the likeliest genes to pass through a semipermeable hybrid zone, since selection favors immunological diversity per se, so if the interpretation is confirmed it would set a likely upper limit on the Neandertal genetic contribution to extant Europeans (ibid.).

Much of interest here from the perspective of interbreeding among archaic human groups. Immensely important stuff, and the earlier parts fo the article are just as essential. Please read it.

On a side-note, I also found this interesting mis-citation of myself:

...the uniquely human, culture-driven, in situ conversion of Neandertals to "moderns" (Hawks and Wolpoff 2001) without any appreciable population movement or "gene-flow" is now hard to reconcile with the rather short timescale of replacement (Churchill and Smith 2001), and has been abandoned by its original formulators (Jolly 2001:198).

</p>

Funny, I do remember myself ever writing about the "in situ conversion" of Neandertals without gene flow. Nor do I often see gene flow written with scare quotes, since it is given the chapter in most genetics textbooks. Oh well, the idea that people sneakily abandon their theory is seems to be quite the growing meme lately. I guess the only way to avoid it is to keep oneself from "originally formulating" anything.

References:

Jolly CJ. 2001. A proper study for mankind: analogies from the papionin monkeys and their implications for human evolution. Yrbk Phys Anthropol 44:177-204.