Burtsev and Turchin (2006) present the results of simulations of cooperative behavior in self-interested agents. This is a well-established subject, and their contribution is that their strategies are "evolved" from basic behavioral elements within their simulations, instead of being assumed a priori.
In our model, agents are endowed with a limited set of receptors, a set of elementary actions and a neural net in between. Behavioural strategies are not predetermined; instead, the process of evolution constructs and reconstructs them from elementary actions. Two new strategies of cooperative attack and defence emerge in simulations, as well as the well-known dove, hawk and bourgeois strategies. Our results indicate that cooperative strategies can evolve even under such minimalist assumptions, provided that agents are capable of perceiving heritable external markers of other agents.
This to me is one of the most interesting aspects of the model: behavioral traits gain random associations with recognizable phenotypes, and individuals shape their behavior according to the phenotypes that they detect around them.
Each agent has external phenotype that is coded by a vector of integer values (markers). The markers do not influence behaviour but function only as indicators of similarity....All of our simulations were started with an initial population of agents that were unaware of markers (the matrix coefficients connecting input from markers to actions were preset to zero). Thus, the use of markers in a population had to evolve from a blank slate. Because markers and behaviours are not linked (apart from both being inherited from the ancestors), agents can lose cooperative behaviours by mutation while retaining 'in-group' markers. Thus, the structure of the model allows free-riders to arise.
This "phenotypic association" vector is suggestive. Of course, for real animals it would probably be more effective to recognize the behaviors themselves as signs. But this depends on multiple opportunities -- you have to see somebody else's behavior at least once to judge it. If there were external manifestations associated with behaviors, it would give the opportunity to decide before an interaction what the other individual's likely strategy would be.
But then, selection would favor mimicry -- free-riders with the phenotypes of cooperators, for example. This force will tend to limit the degree of association between observable traits and behaviors...
That observable traits that indicate relatedness will also tend to indicate similarity in cooperator phenotypes. In other words, you can figure that your relatives will tend to act like you, and also tend to look like you. And as a bonus, if you are sharing with a relative, you are increasing your inclusive fitness.
They find that the evolution of different strategies depends on carrying capacity, and some new strategies emerged. The comparison of these is worth reading, but a bit too long and involved to quote at length. This part is important:
Our results have important implications for the evolution of territoriality in animals (and private property in humans). With a few exceptions, theorists have paid little attention to the role that cooperation may have in the evolution of territoriality. Our study suggests that cooperative defence of territory can radically change the course of evolution in resource-rich (C > C2) environments. When the amount of resource becomes large enough to support more than one agent, and too large for a single agent to monopolize, solitary bourgeois are replaced by cooperative starlings, provided that agents can recognize in-group members. The starling strategy does not take over completely, however, but coexists with other strategies in a complex dynamical way.
The "starling" strategy is a mobbing strategy, in which small animals cooperate to attack and drive away large solitary predators.
One limitation of these kinds of simulations is that they don't include reproductive boundaries. For this reason, they don't really distinguish models of within-species cooperation from between-species mutualism. Different strategies like "hawk" and "dove" might really represent predators and prey species, or they might represent contrasts of competitive behavior within a species.
So the appearance of stable strategies at any given level of possible complexity might be a constraint on natural communities, but the level of that constraint may not be immediately obvious. To that end, this study has a very large possible set of strategies (more than 101000 combinations), which means it is sampling a richer set of behaviors than most simple game-theoretic models.
Burtsev M, Turchin P. 2006. Evolution of cooperative strategies from first principles. Nature 440:1041-1044. DOI link