Genetic discord

I ran across this paper from a few years ago by John Avise and DeEtte Walker, which considers the implication of reticulation-based species concepts for mtDNA-generated phylogenies.

After quoting Dobzhansky on natural categories, they point to the central problem with using mtDNA phylogenies to define species: a clonally inherited gene does not easily lend itself to testing horizontal gene transfer:

In this same spirit, we ask here whether biotic discontinuities as seen through the eyes of laboratory-based mitochondrial geneticists tend to bear resemblance in number and composition to the biological units currently recognized as taxonomic species. There are additional reasons for interest in the outcome. First, discontinuities might be evident in local biotas (the nondimensional species perception) but may blur when geographic variation is taken into account. Molecular phylogeographic studies address this issue, because they explicitly analyze spatial variation (6, 7). Second, under the biological species concept (BSC), a sexual species usually is perceived as a reproductive community whose gene pool retains coherency primarily via the bonds of interbreeding and genetic exchange (1, 8); however, mtDNA molecules are transmitted asexually, and matrilines are nonreticulate. Thus, any genuine unities within (and discontinuities between) groups of organisms in mtDNA genotype cannot be attributed to "horizontal" patterns of contemporary lineage anastomosis via mating per se. Instead, they must be caused by "vertical" connections (and partitions) in matrilineal phylogenies. However, vertical connections themselves are functions of the demographic histories of population units demarcated by temporally extended patterns of interbreeding and gene flow.

I think this passage puts the situation more direly than deserved -- after all, every gene is vertically inherited. Mitochondrial DNA is no exception. It can be transferred by gene flow just as surely as any autosomal gene.

No, the key difference is that clonal inheritance leaves mtDNA with a greatly reduced effective size compared to autosomal (or X-linked) genes. This means that a given amount of gene flow is vastly less effective at dispersing mtDNA variants. Hence mtDNA (and Y chromosomes) have much higher FST (at equilibrium) than other genetic markers.

In other words, longstanding populations within a species will tend to look more divergent considering only their mitochondrial DNA than considering their autosomal genes. We can see this pattern when considering differences among subspecies of chimpanzees and other hominoids. The subspecies are highly distinct from each other considering only their mtDNA, with long divergence times ranging higher than a million years. The other uniparentally inherited genetic system, the nonrecombining portion of the Y chromosome (NRY) shows a similar pattern -- subspecies of chimpanzees are highly distinct, sharing no NRY lineages (Stone et al. 2001). In contrast, there is substantially more sharing of variants at autosomal sites (Fischer et al. 2004). Chimpanzee subspecies share many fewer autosomal variants than are shared among human groups, but they share many more autosomal than mtDNA or Y chromosome variants. Gorilla genes follow a similar pattern: mtDNA indicates very strong divergence between western and eastern gorillas, while autosomal genes show evidence for recurrent gene flow between them up to 150,000 years ago (Thalmann et al. 2007).
Avise and Walker compared mtDNA phylogenies for vertebrates with commonly accepted taxonomic species, finding roughly twice as many deep mtDNA phylogroups as taxonomic species. They consider that these generally represent historical patterns of demography and constrained gene flow within species.

Coalescent patterns in gene trees are related intimately to historical patterns in population demography (7, 21, 22). In particular, tight connections among nonanastomose [nonreticulating] genotypes suggest recent lineage coalescence to a shared ancestor, likely because of relatively small evolutionary effective population sizes that cause extant lineages to have shallow temporal depth. Conversely, large genetic gaps between gene-tree branches suggest long-standing historical population separations. In support of this likelihood, nearly all of the deep phylogenetic disjunctions registered in the intraspecific mtDNA gene trees in this review involved regionally separate populations.

This is basically saying that regional differentiation within species is an important source of genetic variability. They mention that male-mediated dispersal would create patterns not easily tested with mtDNA; this is one factor but broadly, any single gene will create a phylogeny that is potentially discordant with others in various ways.

References:

Avise JC, Walker D. 1999. Species realities and numbers in sexual vertebrates: Perspectives from an asexually transmitted genome. Proc Nat Acad Sci USA 96:992-995. Abstract

Fischer A, Wiebe V, Pääbo S, Przeworski M. 2004. Evidence for a complex demographic history of chimpanzees. Mol Biol Evol 21:799-808. doi:10.1093/molbev/msh083

Stone AC, Griffiths RC, Zegura SL, Hammer MF. 2002. High levels of Y-chromosome nucleotide diversity in the genus Pan. Proc Nat Acad Sci USA 99:43-48. doi:10.1073/pnas.012364999

Thalmann O, Fischer A, Lankester F, Pääbo S, Vigilant L. 2007. The complex evolutionary history of gorillas: insights from genomic data. Mol Biol Evol 24:146-158. doi:10.1093/molbev/msl160