Culture, mathematical models, and Neandertal extinction

10 minute read

I am not philosophically opposed to building a mathematical model of Neandertal populations. Some of my best work has involved mathematical model-building. Models have an important place in helping us to understand evolutionary history. But when it comes to understanding Neandertal and modern human interactions, we have had lots and lots and lots of models and few testable predictions.

When you assume that modern human populations grew faster than Neandertal populations, you will conclude that modern human populations could have out-reproduced the Neandertals. This is not a very deep piece of circular logic. and so I get a little frustrated at the number of papers that really say nothing more than this.

Find x meme

Modeling is a start, but cultural systems are complicated. Clever innovations can help a population grow, but a population can co-evolve with its culture, yielding not only more growth but also greater difference in growth between populations. A cultural innovation can be tied to a particular landscape or raw material substrate, making it difficult to apply outside the context where it was invented.

The populations that we call “modern humans” really did out-reproduce the Neandertals. That’s why living people have only a small fraction of Neandertal ancestry today. But is culture a sufficient explanation? Were modern humans just smarter than Neandertals? Or were other factors important to the interactions between these populations?

Differential equations and Neandertals

I’m taking up this subject today in response to a paper by William Gilpin, Marcus Feldman and Kenichi Aoki, who have investigated a differential equation model for population growth with feedbacks from competitive interactions. The abstract summarizes the paper well:

Archaeologists argue that the replacement of Neanderthals by modern humans was driven by interspecific competition due to a difference in culture level. To assess the cogency of this argument, we construct and analyze an interspecific cultural competition model based on the Lotka−Volterra model, which is widely used in ecology, but which incorporates the culture level of a species as a variable interacting with population size. We investigate the conditions under which a difference in culture level between cognitively equivalent species, or alternatively a difference in underlying learning ability, may produce competitive exclusion of a comparatively (although not absolutely) large local Neanderthal population by an initially smaller modern human population. We find, in particular, that this competitive exclusion is more likely to occur when population growth occurs on a shorter timescale than cultural change, or when the competition coefficients of the Lotka−Volterra model depend on the difference in the culture levels of the interacting species.

The Lotka-Volterra system of differential equations is one in which two components of a system change over time, in such a way that the amount of change in one component depends upon the magnitude of the other component. It has been most famously applied as a model for predator and prey populations, where predator population growth is coupled to prey population size. The behavior in this system can be cyclical—for example, if a predator population crashes, the growth of a prey population can resume, driving subsequent growth in the predator population.

This kind of feedback between components of the system is determined by the differential equation coefficients. The utility of this kind of system is that varying the coefficients allows us to investigate the conditions under which the system can exhibit stable, cyclical, or degenerate behavior.

Gilpin and colleagues assume the Neandertal and modern human populations to have been in a competitive interaction, where the rate of growth of the Neandertal population is smaller when the modern human population is larger, and vice-versa. Each population grows logistically up to a carrying capacity, which is determined by a parameter that they refer to as “culture level”. This “culture level” increases with population size, and it increases faster for the modern human population than for Neandertals—in other words, in this model Neandertals are stupid.

In this system, modern human populations grow faster when they wipe out the local Neandertals. They innovate faster when the population gets larger. If the difference in the rate of change of “culture level” is sufficiently large, Neandertals are doomed.

In principle, the model allows the authors to investigate the way that a second parameter, “culture level”, might constitute an advantage if Neandertals had a lower rate of increase. But the paper does not actually deploy this model in a way that would inform us about the importance of this second parameter. As a result, the conclusion is boring. If we assume that Neandertals were stupid, we don’t need a differential equation to tell us what happened to them.

Neandertal in a suit

Is “culture level” relevant?

The problem is not that we lack models to show how culture may have helped modern humans beat the Neandertals. The problem is that the archaeological data suggest that culture alone may be a poor explanation.

For one thing, the Neandertals persisted in Europe and central Asia long beyond the entry of modern humans into Asia. Initial modern humans in Asia exhibited no obvious cultural superiority over other Middle Paleolithic people, who were presumably archaic humans. “No cultural superiority” is maybe an understatement: Archaeologists have trouble finding any consistent material culture differences between people in West Asia before 50,000 years ago.

Tens of thousands of years later, when modern humans did start to enter Europe, they seem to have mixed with Neandertals more extensively. The later Neandertals were making symbolic artifacts, using pigments, feathers and other ornaments. The people who made the earliest Aurignacian, often assumed to be the earliest modern humans in Western Europe, did not have the intensity of symbolic artifacts of later Aurignacian and Gravettian people. Instead they seem to have been sparse and little different in most cultural practices from Neandertals.

In other words, at the critical time when modern humans entered Europe and their population apparently grew, there was little cultural difference between them. There is even less evidence that there was any cultural advantage to modern humans who spread across southern Asia prior to 50,000 years ago.

What gives? If we assume that “culture level” was a continuous variable, and that “modern humans” had a higher rate of increase than Neandertals, we get a very simple pattern. The data are not a simple pattern. So the “culture level” model seems like a bad model to account for the complexity of what actually happened.

Vital forces

“Culture level” is an archaeological version of “vital force”. Plenty of archaeologists think there is something special about “modern human behavior”, and believe that a “spark” entered the human population. After this vital spark entered the modern population, they were able to grow their population, spread around the world, and conquer the earth with their cultural adaptability. Some have written that this “spark” was a key mutation, some believe it was fully human grammar, some believe that it was a special demographic or ecological setting.

They are not thinking like biologists. The evolution of human cognition was not magic, and it was not caused by a “spark”.

I don’t object to the idea that Neandertals may have been cognitively different than modern humans—in fact, I think this is likely. The idea that Neandertals were fixed for stupid and modern humans fixed for smart is biologically incredible. Instead, we need to consider that if many Neandertals had challenges learning to work with some cultural innovations, many modern humans should have had such challenges as well. Key innovations, if rare, must have been stochastic.

To understand human cognitive evolution, we must consider how specific behaviors may have contributed to reproductive success. Useful cultural innovations tend to be transferred readily across groups, and so make unlikely vehicles for a continuous growth of one population at the expense of another. Was the ability to learn some cultural behaviors heritable? If so, it is unlikely that the ability to learn two behaviors was equally heritable; some must have been more influenced by genes than others. To the extent that behaviors are learned by exposure to skilled individuals, this exposure causes the selection in favor of the ability to learn to weaken as the trait becomes more commonly expressed.

Cultural selection to enforce cultural uniformity can be very effective in fixing cultural traits in a population, but is not especially likely to enhance the traits that really matter for adaptation to new environments. Indeed, cultural selection in many recent human groups has been depressingly conservative, preventing innovation and reducing population growth by imposing various handicaps.

In the real world, some archaic people—including the Neandertals—really were more successful than most early modern human groups. Neandertals as a population contributed more DNA to people around the world than their “conquerors”, the Upper Paleolithic people of Europe. Some modern human populations have massively grown during the last 50,000 years at the expense of others, often for cultural reasons. When we look at the diversity of those situations, we can see that culture is not easily broken down into a linear variable.

Non-linear culture

What makes culture more than a simple system of accumulating knowledge?

There are conditions under which a cultural system may be hard to transfer across groups. A cultural system may have co-evolved with some genetic variants, like dairying and lactase persistence. The cultural traits in such a system, even if learned, might not have had the advantage for Neandertals as for the modern humans that developed it. A cultural system may rely upon some elaborate codification of social behavior, like religious rules, that are not readily adopted by new cultural groups. Again, the behaviors that seem tied to reproductive advantage in such a system may not be as advantageous to people who lack the essential cultural background.

What we lack is some empirical demonstration of what cultural factors among Late Pleistocene people actually led to higher reproductive success. Archaeologists have proposed several, but have tested few. Most, like evidence for symbolic behavior, have subsequently been found in the Neandertals themselves, making them poor explanations for Neandertal extinction.

Later modern humans exhibited greater material culture diversity and more symbolic expression than earlier modern humans, thousands of years after the Neandertals were gone. This is true not only in Europe and central Asia, it is also true in other places long after the first modern humans appeared there: in southern Africa, west Asia, and southeast Asia. In every part of the world, the evidence for elaborate symbolic culture occurs long after the earliest evidence of modern humans. And in most of these areas, the first evidence for symbolic culture occurs substantially before the earliest evidence of modern humans.

To me, this means it was not just “culture level” that made a difference. I can imagine that there may have been some specific aspects of culture, which may not have been archaeologically visible, that made a key difference. Archaeologically visible material culture may reflect demographic growth, but not necessarily the key aspects that mattered to the initial dispersal of populations.

But I can also imagine that non-cultural factors were more important. For example, disease has been a key factor underlying the survival or replacement of populations during the past 10,000 years. It is not a stretch to imagine that disease influenced the Neandertals and other archaic peoples differently from each other and from modern humans. The evidence for selection on genes related to immunity from these archaic humans is now strong, and some of these may reflect pathogens or parasites that were important at the time of population contacts among these people.

Exploration needed

This is why we need more data, more exploration, more archaeology. I don’t mind if people continue to think of mathematical models, as they may help us to understand which factors are important. Listing all the possible factors doesn’t necessarily get us closer to a test of which of those factors were crucial to our evolution, and including every factor in a model will make it untestable.

But we are past the point where a simple model is going to tell us something we don’t already know. Neandertals are gone. Their cultures did not persist. Yet they are among our ancestors. What is necessary is to test models against the timeline of modern human dispersal as we currently understand it, and to take note of those predictions that we have not yet observed. It is the novel predictions of a model that make it valuable to the future.


Gilpin W, Feldman MW, Aoki K. 2016. An ecocultural model predicts Neanderthal extinction through competition with modern humans. Proc Nat Acad Sci USA (online) doi:10.1073/pnas.1524861113