Ancient genetic introgression between cave hyenas and spotted hyenas

I’ve been writing about ancient mixture between species for a long time now. Since the reporting of the first Neandertal genome in 2010, a lively field of ancient genomics has been bringing new DNA analyses of other extinct species. They’ve turned up ancient gene flow between species again and again, in extinct lineages like mammoths and straight-tusked elements, in living and extinct canids, in baboons, and pretty much wherever a geographically widespread mammal has some species diversity.

An interesting paper last year by Michael Westbury and coworkers presented interpretation of the genomes of cave hyenas from Europe and China: “Hyena paleogenomes reveal a complex evolutionary history of cross-continental gene flow between spotted and cave hyena”.

It may not seem like a surprise that hybridization and ancient resulting ancient gene flow is part of the story of these hyenas. At this point, it would be more newsworthy to find that hybridization never happened. I am interested in this paper because it illustrates a shift in our understanding of ancient mixture in many lineages where mitochondrial-only sampling has only recently started to give way to whole-genome approaches. I’m also interested because of the pattern of hyena evolution has some parallels with hominins.

This paper is not about every part of the hyena phylogeny. The surviving members of the hyena family belong to four genera, the brown hyena Parahyaena brunnea, striped hyena Hyaena hyaena, the aardwolf Proteles cristata, and the spotted hyena Crocuta crocuta. Brown and striped hyenas were historically classified together; they are sister species today and diverged from a common ancestor around 4.5 million years ago. This paper covers only the genus Crocuta. Part of a paragraph of the introduction helps to understand the taxonomy in this genus:

Although now restricted to sub-Saharan Africa, the genus [Crocuta] once had a much more extensive range, occupying most of Eurasia, from the British Isles to the far east of Asia. Initially, because of distinct morphologies, Eurasian cave and African spotted hyena lineages were considered distinct taxa. Cave hyenas had shorter distal limb elements than the extant species, indicating less cursorial ability. They also had a less trenchant check tooth morphology, indicating less active hunting and meat eating and a greater reliance on scavenging for their nutritional needs. Eurasian cave hyenas have further been split into European (C. crocuta spelaea) and Asian (C. crocuta ultima) subspecies. These classifications have, however, met with some resistance, with differences being attributed to phenotypic plasticity caused by different climates. This skepticism was further supported by a study using short fragments of mtDNA, which found African spotted hyenas to be intermingled within Eurasian cave hyenas’ mitochondrial haplogroups. This result indicated that, when only considering these mtDNA fragments, cave and spotted hyena appear to be inseparable taxa.

Due to the previous disagreements about taxonomy, the paper throughout refers to “spotted hyenas” and “cave hyenas”, with no further reference to their species names.

The earlier study of ancient mtDNA was by Nadin Rohland and coworkers in 2005: “The Population History of Extant and Extinct Hyenas”. The trend in phylogeography at that time was to interpret geographically widespread mtDNA clades as evidence for population dispersal at around the time of the inferred age of the clade. The paper found that one mtDNA clade was shared by many extant spotted hyenas in Africa and ancient cave hyenas in Eurasia, while there was strong mtDNA clade separation between southern and northern parts of the spotted hyena range in Africa. The paper also found that the deepest clade separation was between one divergent mtDNA clade in East Asian cave hyenas and the rest of the spotted and cave hyenas in Africa and Europe.

Rohland and coworkers interpreted these results as the product of three successive invasions of Africa by spotted hyenas from a Eurasian origin. The paper suggested that the results did not support the traditional separation of cave hyenas and spotted hyenas into different subspecies or species, because the mtDNA clades did not line up with those classifications.

The more recent 2020 work by Westbury and colleagues was able to partially replicate the mtDNA results from the earlier work. They confirmed a highly divergent mtDNA clade represented by East Asian cave hyenas, and found a clade that includes some Eurasian and some African individuals. But the nuclear DNA data shows that this mtDNA pattern is unrepresentative of the genome as a whole. The Eurasian and African samples are strongly separated from each other in their nuclear genomes. The estimate of divergence time between African spotted hyenas and Eurasian cave hyenas is around 2.7 million years ago. European and East Asian samples are also substantially separated but with a greater apparent level of intermixture throughout their evolutionary histories.

The pattern from the nuclear genome suggests that the mtDNA clade shared by European and some African spotted hyenas reflects introgression rather than a dispersal. Westbury and coworkers probed this hypothesis using the D-statistic approach applied to the nuclear genome data. They found that spotted hyenas from Somalia and Ghana have more similarity with the extinct Eurasian cave hyenas. That similarity is not limited to European cave hyenas, it is also shared to some extent with East Asian cave hyenas. This confirms that gene flow from cave hyenas into spotted hyenas is the likely explanation for the mtDNA pattern and is reflected by a fraction of the nuclear genome as well. The clade age for the mtDNA clade shared by African and some Eurasian hyenas is less than a half million years.

The paper goes to some effort to try to understand the timing and pattern of gene flow. These aspects are less persuasive because they rely heavily on the assumption that two genomes that share a similar amount of ancestry with a third genome must also reflect the same mixture event. So for example, the European cave hyenas seem to belong to two groups that differ genetically in a comparison where the African spotted hyenas are removed. The African spotted hyenas that show evidence of introgression from Eurasian cave hyenas have a similar degree of genetic similarity to both “branches” of the European hyenas. The study then concludes that the gene flow must have occurred before these two European “branches” became separated from each other. That is one possible interpretation of the data. But other hypotheses, such as long-term gene flow involving both these groups, are also possible.

Where Rohland and coworkers had interpreted the mtDNA tree to reflect a Eurasian origin for spotted hyenas, the newer paper favors an African origin. In part this shift reflects the new genetic data. The nuclear genome divergence of the African and Eurasian populations was already in place long before the common mtDNA ancestor of all the known African and European mtDNA clades. This means that mtDNA introgression and replacement must have happened, no matter whether the populations originated from African or Eurasian ancestors.

But it is actually a change in the fossil record that matters more to the origin of Crocuta. Westbury and coworkers point to fossil Crocuta dietrichi from the Pliocene of Kenya and Namibia as an argument in favor of the African origin of the genus. In their account, the dispersal of Crocuta into Eurasia sometime around 2 million years ago is a parallel to the dispersal of Homo. When Rohland and coworkers did their work in 2005, it was thought that Crocuta had an Asian origin because the earliest known representative was Crocuta sivalensisfrom the Siwalik Hills. But Werdelin and Lewis in 2012 reassessed the fossil that represented Crocuta sivalensis, finding it isn’t Crocuta at all but the extinct genus, Pliocrocuta, which is related to the stem of Crocuta and the large extinct species Pachycrocuta brevirostris and not a close relative of modern Crocuta.

What does it mean if Crocuta and Homo dispersed around the same time? I’m not convinced that what seems like a coincidence in time tells us very much. For one thing, we’re talking about dispersals that may have been a half million years or more apart, anytime between 2.7 million and 1.8 million years. That’s not much of a coincidence.

A second challenge for any interpretation is the observation that Crocuta was invading a geographic range already occupied by closely related species that were potential competitors. The initial dispersal of Homo into Eurasia seems to have been into a range where no close relatives, such as Australopithecus existed.

There are some who would say that Homo had become part of the carnivore guild, making their entry into Eurasia highly meaningful in terms of interpreting hominin ecology. I’m skeptical. Early Homo erectus may have been an effective hunter and scavenger, but so far there is no evidence that Early Pleistocene Homo lived anywhere that plant foods would have been scarce or seasonally unavailable. Homo succeeded by encompassing greater dietary breadth, including meat, not by becoming an obligate carnivore. This makes the relevance of interactions with carnivores very different for Homo than for Crocuta.

What, in the end, do the data have to tell us about species or subspecies in Crocuta? The divergence and strong maintenance of genetic separation between African spotted hyenas and Eurasian cave hyenas for more than 2 million years supports the traditional taxonomy separating these into Crocuta crocuta and Crocuta spelaea. The two East Asian samples are substantially different from the European samples, and that may suggest a further separation in the taxonomy would be appropriate. But more samples chosen between these locations would be valuable in understanding the pattern of diversity of Eurasian cave hyenas.

The mtDNA of African spotted hyenas appears to have been totally replaced by Eurasian mtDNA lineages within the last million years. This replacement is similar to the pattern we see in Neandertals, where an African-aligned clade replaced a previous Denisova-aligned clade sometime after 350,000 years ago. The limited nuclear diversity of spotted hyenas, and the fact that near-identical mtDNA occurs across some large geographic distances, both make it unlikely that a much larger sample of extant spotted hyenas would recover any further deep mtDNA clades. It’s more than a little remarkable to realize that at least one Eurasian mtDNA clade was flowing south from cave hyenas into spotted hyenas at around the same time that a hominin mtDNA clade was flowing north into Neandertals.