The PNAS early edition is like a one-stop shop today. Here's a paper where the authors set up an experimental model for language origins:
The emergence of simple languages in an experimental coordination game
Reinhard Selten and Massimo Warglien
We investigate in a series of laboratory experiments how costs and benefits of linguistic communication affect the emergence of simple languages in a coordination task when no common language is available in the beginning. The experiment involved pairwise computerized communication between 152 subjects involved in at least 60 rounds. The subjects had to develop a common code referring to items in varying lists of geometrical figures distinguished by up to three features. A code had to be made of a limited repertoire of letters. Using letters had a cost. We are interested in the question of whether a common code is developed, and what enhances its emergence. Furthermore, we explore the emergence of compositional, protogrammatical structure in such codes. We compare environments that differ in terms of available linguistic resources (number of letters available) and in terms of stability of the task environment (variability in the set of figures). Our experiments show that a too small repertoire of letters causes coordination failures. Cost efficiency and role asymmetry are important factors enhancing communicative success. In stable environments, grammars do not seem to matter much, and instead efficient arbitrary codes often do better. However, in an environment with novelty, compositional grammars offer considerable coordination advantages and therefore are more likely to arise.
The subjects were given a small set of letters to work with, and a relatively large set of graphical symbols (I'll call them icons) to encode with the letters. The icons are illustrated in the paper; they sort of look like Kanzi-like ideograms. The players had to combine letters to attempt to communicate icons as they appeared on a screen, sending the letters across an anonymous channel with another player. Each player's correspondence of icons and letter-combinations amounts to a code. For this system to work as a communication between the two subjects, they had to arrive at some conventions, so that they would share the same code. The form and effectiveness of the conventions is what the researchers were trying to study.
Out of that abstract, this sentence strikes me as interesting: "Our experiments show that a too small repertoire of letters causes coordination failures."
Why should that be? In the experiment, the use of letters carried a cost to the sender -- like sending a telegram, charged by the letter. So short messages carry a higher payoff. But when there are very few letters to choose from, it is impossible to form many unique messages, unless the messages combine a lot of letters. So players supplied with a "small repertoire" of letters should try to economize communications by packing as many different combinations into as few letters as possible. And that leads to confusion, since messages will be minimally redundant and hard to remember. Give them more letters, and they can make many more combinations with shorter message lengths. Problem solved.
This is essentially an information theoretic explanation for why a large phoneme count is advantageous for human language, which the authors point out:
From a logical point of view, two symbols are sufficient for the construction of a code. In principle, communication could be based on a binary code. However, it seems to be the case that the availability of a sufficiently large variety of letters not only makes it easier to achieve communication efficiency but also has a cognitive effect that facilitates linguistic coordination (Selten and Warlien 2007:7363).
That's not a new insight, but it is interesting that, in these experiments, the value of a relatively large letter diversity is derived from the explicit per-letter cost of transmission together with an apparently greater comprehensibility of messages in systems that employed greater letter diversity. Both the learnability and transmission cost factors work in the same direction in the context of this experiment.
One essential ingredient to a successful game was, in a sense, hidden: two players trying to develop a common code could achieve a higher payoff in the long run if they very quickly differentiated their roles into a single "teacher" and a single "learner." In the context of the game, if both players tried to adjust to each others' codes, they would end up changing past each other, and mismatches still persisted. But if one player consistently changed and the other remained constant, every change tended to increase payoff by increasing the consistency between the two players' use of codes.
Last, the icons themselves were chosen by the experimenters to exhibit some similarities in form that might have been exploited by the players in forming their codes. For example, the symbol set included open circles and triangles, circles and triangles with dots in them, circles and triangles with plus signs in them, etc.
Some players ended up forming codes that randomly assigned sequences (generally pairs) of letters to these icons, irrespective of their shape. Every icon gets essentially a random one or two letters.
Some players arrived at codes in which all circles were encoded by a letter combination beginning with, say, "R", and all triangles by a combination beginning with, say, "S". The researchers considered these codes to be "grammatical", because the letter combinations had a clear syntactic structure with one position assigned to indicate the shape of the icon, and later positions other information.
Some players formed codes that were grammatical, assigning a position to the shape of the icon, and also assigned the same letter for any shape having the same inner sign. So every icon containing a plus sign might have a "M" as the second letter, and every one containing a dot might have a "Z" as the second letter. The researchers considered these grammars to be "compositional", because the two positions each encoded separate information, which when brought together communicated combined information in a consistent way across all icons.
In the context of the experiment, using a grammatical form for the code was somewhat costly, since it predicated that a certain number of letters be used for each figure regardless of the actual frequency with which they were presented.
By the most complicated phase of the experiment, which involved thirty-six different icons to encode with 10 letters, almost everybody who came up with a grammatical code also made it compositional. It just seems like if people were going to systematize their code by shape, they did it, well, systematically -- using different letters to encode different kinds of information about the icons independently. Since the players built up to this phase of the experiment with simpler systems of icons, the experimenters could observe what happened as things got more complicated. At the earlier stages, there were a number of different noncompositional grammars used, and all of them were chucked when things got complicated.
But still, in a majority of cases, the pairs of players still didn't arrive at a single code by the end of the trials. The researchers were interested that compositional grammars were successful in this phase of the experiment, which they interpreted as involving "novelty" since each icon was presented only once. Clearly there was a premium on a system that could get things right that hadn't been practiced, and only the "compositional" alternative really allowed that to happen in this context. The authors conclude:
(iii) In stable environments as those considered in point ii, grammar does not matter much, and efficient arbitrary codes often do better. However, compositional grammars have the advantage of being more easily extendable to broader environmental demands. Noncompositional grammars are more fragile and are easily lost if new conditions have to be met.
(iv) In an environment with novelty, in the sense that often the need arises to express something that never has been expressed before, compositional grammars offer considerable coordination advantages. Therefore, under such circumstances, compositional grammars are more likely to arise. In this respect, our findings parallel and complement hypotheses proposed in the literature on language evolution (28, 29). In our experiments, all subjects have grammatical competence but they make relatively little use of it unless pressure of novelty gives them an incentive to do so (Selten and Warglien 2007:7365).
Well, this experiment isn't enough to demonstrate all that, but the cost and incentive structure makes it an interesting addition to theoretical arguments favoring these points.
Selten R, Warglien M. 2007. The emergence of simple languages in an experimental coordination game. Proc Nat Acad Sci USA 104:7361-7366. doi:10.1073/pnas.0702077104