Perils of modeling

1 minute read

You’re not coming here for economic analysis, but I found this Wired article on quants, risk, and the financial crisis useful:

Bankers should have noted that very small changes in their underlying assumptions could result in very large changes in the correlation number. They also should have noticed that the results they were seeing were much less volatile than they should have beenwhich implied that the risk was being moved elsewhere. Where had the risk gone?
They didn't know, or didn't ask. One reason was that the outputs came from "black box" computer models and were hard to subject to a commonsense smell test. Another was that the quants, who should have been more aware of the copula's weaknesses, weren't the ones making the big asset-allocation decisions. Their managers, who made the actual calls, lacked the math skills to understand what the models were doing or how they worked. They could, however, understand something as simple as a single correlation number. That was the problem.

These models are not so different from genetic analyses, and in fact phenotype prediction on the basis of genome-wide SNP data or sequences will likely involve many of the same problems. In particular, the problem of testing for correlations with limited data is one that I run up against with phenotype evolution quite often.

On the same topic, Nassem Taleb’s essay, “The Fourth Quadrant” is also useful in understanding the present difficulties.