Weaver (2005) examined the size ratio of the cerebellum and neocortex in fossil hominid brains. The division between the cerebellum and the cortex is one of the few features of the brain that is readily identifiable from the internal surface of the skull.
Recent neuroanatomical studies and radiographic observations have demonstrated that the cerebellum plays a role in many cognitive functions. Moreover, the cerebellum has reciprocal connections, through the thalamus, with each of the major neocortical regions listed by Holloway as having changed in the course of human cognitive evolution (Weaver 2005:3576).
Weaver found that the relative size of the cerebellum was not the same in all fossil hominids. "Relative size" in this context was the ratio of the actual cerebellar size to the size predicted from a regression on net brain size in recent humans. The study considered only around twenty fossil specimens across the past three million years, but did observe some significant differences over time. These included a significant decrease in relative cerebellar size across the Pleistocene from the Middle Pleistocene into Neandertals (La Chapelle-aux-Saints and La Ferrassie). The Cro Magnon 1 specimen is equal to the Neandertals in this measure.
She discusses the manner of the change as follows:
A decrease in a ratio can be achieved in two ways: (i) by decreasing the denominator (CBLM volume) or (ii) by increasing the numerator (NetBrain volume). The data indicate that the decrease in CQ seen in the early archaic H. sapiens and Late Pleistocene humans is due to an increase in the NetBrain. On the other hand, a slight decrease in NetBrain volume in recent humans is accompanied by a significant increase in cerebellum volume (3578).
It is not obvious what the importance of these changes might be. Weaver gives some examples of the ways the cerebellum coordinates with the neocortex to perform certain functions, but there is no single answer that is likely to explain the recent increase in cerebellar volume. To explain the most recent change, Weaver suggests that there may have been an increasing need for "complexity management" that itself changed during the course of the Upper Paleolithic.
But there are some weak points in the data that should make us hesitate to accept the pattern. The paper does not present information that would demonstrate whether the relationship between cerebellar volume and net brain volume is linear; a nonlinear relationship might result in a bias toward smaller relative cerebellar size in large crania like the Neandertals and Cro Magnon 1. Nor is the possible role of sex examined.
But I think the most important thing to consider is that some today's humans are probably more closely related to Upper Paleolithic Europeans than they are to each other. If there were significant volumetric differences in the cerebellum between this ancient European specimen and recent Europeans, then there almost certainly are significant differences among living human groups. As far as I can tell, living humans across cultures have a very similar ability to "manage complexity." In this study, the relative cerebellar volume among human populations is not examined, so it is not possible to say whether the differences between ancient groups are large or small relative to living human between-group variability. So the possibility that ancient humans and Neandertals may have been different in their relative cerebellar volume may not say much about their mental functions or capabilities.
Weaver AH. 2005. Reciprocal evolution of the cerebellum and neocortex in fossil humans. Proc Nat Acad Sci USA 102:3576-3580.