A Note on Forecasting

Here is something that I wrote in the comments at Protein Wisdom in response to this comment by BLT in CO. BLT was responding to the decision that Ray Nagin made to keep New Orleans evacuees from returning with the threat of Rita looming. I'm posting it here because it was really long and kind of post worthy:

I’ve noticed a pattern with Colorado winter weather forcasting: At some point each winter a storm which is predicted to drop several inches (a small storm) will instead unleash a foot or two, catching everyone by surprise and embarrassing the forecasters.

Then for the next month or more, every storm that comes near Colorado will be hailed as a major snowfall producer (!!) yet these storms will typically produce only an inch or two.

I’m wondering if the same will now hold true for hurricanes: that massive (and potentially unnecessary) evacuations will be made for even the smallest, just in case.

I responded:

"BLT, your illustration strongly emphasizes some key difficulties when trying forecast: (1) your model is only as good as your assumptions and (2) determining your risk threshold. Being a guy who makes his living building forecasting tools, let me try to expand on your thought(s).

In the last 20 years, with the proliferation of multifaceted and relational databases, we’ve been able to expand the quantity of assumptions that a given model can operate under. Despite this monumental improvement, the human decision of deciding what information is relevant or not is still the crux of the problem.

For instance, if I want to forecast GE’s ability to pay off its creditors, the 3 year incremental increase/decrease in borrowing combined with sales pipeline data would be relevant, where-as the ratio of executives wearing brown versus black shoes would not be relevant. Bits of information in my example here are easy to assume, but fringe pieces that can influence final results are more difficult to predict. Plus, the ability to understand and account for all the variables involved is a Herculean task.

Naturally, this leads to the next point, if I am unable to account for all variables, choosing to ignore outliers, how much risk am I willing to tolerate that one of these variables can, pardon the pun, put me under water?

The aggregate risk tolerance of a population tends to move in a herd from too much to too little. This manifests itself in the various market bubbles and their subsequent explosion (or really implosion). In the Dot.com bubble, everyone saw people making money off of small companies with aspirations of being the next “Netscape.” It was irrational to think that 30 companies a week were capable of producing a product that was as revolutionary (hindsight shows us that even Netscape was unable to sustain itself), however folks threw caution to the wind and most were left holding nothing but their asses after these companies imploded. Their irrational behavior that once caused them to completely ignore risk (i.e. hearing quotes like “This market is different"), now causes them to knee-jerk in the opposite direction. Some folks that were trading penny stocks in 2000 are now afraid to move money into anything other than CDs in 2005.

With total consistency, whether you are talking about markets, hurricane preparations or snow storms, the fundamental character of humans never change. They get greedy as hell, they ignore risk and lose it, then instead of looking back critically, they try to shift blame on someone or something else. The breadth of the pendulum swing caused by these knee jerk reactions is the only unknown, and it is dependant on willingness of the outliers to listen to those in the middle who are able convince the extremists on either end of the fallacious assumptions their arguments are based on.

With Katrina, Jeff has done a great job at showing how much the media spectacle has sensationalized events to the point where very few people have been able to understand any truth that happened there. It is utterly repulsive to see it revealed, and all the media has accomplished, by scaring the living bejesus out of people, is the assurance that next time, people will be so unwilling to take risks that they will go to silly extremes to protect from the next “Big One.” Mayor Nagin will likely order an evacuation (school buses and all), and if the storm misses or loses steam at the 12th hour and produces a little afternoon shower, you can bet the house that the media will be standing downtown with their cameras, passive-aggressively mocking the overreaction. These spineless vultures did it with NASA in Challenger and Columbia disasters, and there is no reason to assume they won’t do it again here.

Note: I’m venturing close to building a strawman argument with my examples in market behavior above, as they mostly reflect some extreme outliers; however I do think that the behavior is consistent with actual results that history has proven correct.

Ultimately, this comment is a forecast, and we’ll see if I’m wrong. The perfidy and shamelessness of the media has yet to disappoint me in my lifetime."