Semenza et al. (2006) examined the mathematical abilities of aphasics with hemispheral dominance to determine which side of the brain is used for mathematical abilities. They found that the side with the language capability was always (at least in their sample) the one with math -- even when the language functions were lateralized on the right instead of the left side:
The main purpose of the present study was however to learn how mathematical functions are located in the brain with respect to language. Right hemisphere aphasia is a rare phenomenon and eight cases, including all the most classic varieties, represent a considerable sample. The assessment of language and calculation in this case series seems to suggest that, as a rule, the two functions share the same hemisphere. According to a more cautious interpretation, these data may just reveal that in case of right brain lateralisation of language some math sub-processes migrate with the language functions, whereas others remain located in the left hemisphere. How this would happen could be demonstrated only by studying cases of right hemisphere aphasia, in unambiguously right handers, fully assessed on math functions, suffering a further injury to the left hemisphere or submitted to a Wada test: the possibility of carrying out such study is obviously remote and the results would be only partially revealing. The present study shows however an incidence of numerical and calculation disorders in right hemisphere aphasia that is similar to that expected in aphasia resulting from left hemisphere damage. This seems enough evidence to suggest that, in right lateralisation for language, a considerable amount of math functions migrate to the same side. The reason for this anatomical proximity may lie in the fact that, as recently suggested , a primitive computational mechanism capable of recursion, thus constituting an open-ended and limitless system of communication needed for language and calculation, has evolved in the dominant hemisphere for reasons independent of both functions.
They argue that this connection is not merely because language itself is necessary for math:
[L]anguage and calculation abilities have been shown to dissociate at several levels, even though aphasia is most often accompanied by acalculic disorders ,  and .
So they suggest that the two require common neural substrates that allow recursion. Personally, I'm guessing there is more to it than "recursion" for this connection with the linguistic areas to hold in this way. But then, I think "recursion" has become a bit of a catchall term for "syntax-ordering functions".
Semenza C and 10 others. 2006. Is math lateralised on the same side as language? Right hemisphere aphasia and mathematical abilities. Neurosci Lett (in press) DOI link