Wednesday, April 5, 2017

Mixed Methods Forecasting

There’s a nice piece up today at Bloomberg by Noah Smith on the utter failure of macroeconomists to develop a functioning forecasting model.  Complex models can be calibrated using past data but fail out of sample going forward.  The literature Smith cites doesn’t look at heterodox models (all those stock-flow consistent entries), but I’d wager they’d fail the same tests.

My take—aside from a general disbelief in the theory that underlies DSGEism—is that the objective is unrealistic.  I can’t imagine there will ever be a single model that does what we want for forecasting.  What I can imagine is a set of models that do the job, providing you know which model to call on when.  That ability to size up a situation, see what’s important and pick the model appropriate for it, relies on judgment, a qualitative assessment of a single, immensely complex moment.  Judgment can’t be bottled and marketed as an off the shelf model; it’s a different kind of process.  But it can be learned, more or less the way it’s learned in other realms of life, through apprenticeship, practice and reflection.

Think of a medical analogy.  Would you want to replace an MD (or PA or NP) with a machine that followed algorithmic instructions to diagnose patients and prescribe treatment?  (This is not a hypothetical question; we’re on the cusp of doing something like this.)  I’d say, in general, no.  There is a valuable role for all sorts of tests and decision algorithms based on them.  In the end, however, care will still be better if a knowledgeable human assesses the overall condition of a patient and turns to the diagnostic models that seem to make the most sense under the circumstances—and then interprets the output to see if it still makes sense.  Maybe at some point in the future AI can completely replace human judgment, but I wouldn’t go there yet.

An economy is even more complex than a single human body (from a treatment/intervention point of view), there are vastly less observations to calibrate on, and AI, as represented by economic models, is still further from displacing qualitative assessment.  What’s needed is a mixed approach.

2 comments:

Spencer England said...

I have a theory that we can never forecast recessions.

Generally, recessions occur when the business community makes a mistake like overestimating demand growth.

Consequently, for the consensus to forecast a recession it has to forecast that the consensus will be wrong-- a very doubtful development. Sure, a few forecasters can do this, but the great majority can not.

Thornton Hall said...

In the technical literature what you describe is called "an educated guess." I suspect you're right and that's all we can do.

You are wrong, however, to suggest that the way to present these guesses is to chose some collection of Newtonian math which generates the guess as it's "solution" specified to the hundredths place and illustrated with multiple Cartesian Planes. That's not a guess. That's fraud.