Risk management and intangibles

Steven Pearlstein's column in the Washington Post this morning is a good illustration of the limitations of the quantitative approach - in this case, The Art of Managing Risk. Pearlstein talks with risk management expert Vince Kaminski:

As Kaminski sees it, the first problem is that the models these systems are based on, while potentially useful, have serious limitations that are too often ignored.
The data that go into them, he says, are so aggregated and "averaged" that they disregard outliers and abnormalities that turn out to be important. There are also risks -- like risk to reputation -- that are ignored because there is no data set by which to quantify them.
Moreover, by relying heavily on past patterns of behavior, they are often useless in dealing with the new products and new markets that are most often the source of the trouble.
Most importantly, Kaminski says, the models have been unable to capture the cascading effect as problems spread, confidence is undermined and people start to act irrationally.
. . .
"What matters in terms of managing risk," Kaminski says, "isn't the model -- it's the intuition, judgment and experience to spot the risks as they are developing, and the character to be able to stand up to very aggressive and successful commercial people and say, 'Enough is enough.' "

In other words, the key is tacit knowledge and understanding the intangibles, like reputation risk and human behavior.

The same can be said of other "quant" approaches. Years ago, I was in a lecture by a forecasting expert who was comparing large econometric models with "expert" opinion. He said that over iterations of revising a forecast, the models and the experts tend to converge. One person asked, "is that because the experts modify their judgments in light of the results of the models?" "No," he replies, "it is just as likely that the modelers tweak the model based on the experts judgments."

Metrics are important -- but the numbers don't always capture everything. As we improve our measurements of intangible, let us always remember that part of our quest for information will lie outside the realm of the quantifiable.