1. Set up a model.
2. Derive an implication from your model.
3. Select/create a data set.
3a. Modify/transform the data set according to assumptions from your model. (optional)
4. Apply causal inference tests.
5. If the result is consistent with the implication from Step 2, claim support for your model.
5a. If the result is not consistent, keep it secret and then go back and tweak the model or the data set. Rinse and repeat until the result is consistent.
For the vast majority of the economics profession, this is regarded as a scientific procedure. Richard Feynman would beg to differ.
I found this choice RF quote from Paul Romer’s post about Feynman Integrity:
It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty–a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked–to make sure the other fellow can tell they have been eliminated.
Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can–if you know anything at all wrong, or possibly wrong–to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.You don’t have to look hard to see that Feynman’s view of science is rather far removed from usual econometric practice. Note in particular the obligation of the research to report “other causes that could possibly explain your results”. If there are plausible theories other than yours that “are consistent with” those little significance asterisks you’re so proud of, you need to specify them. The more of them there are, and the more plausible they are, the less claim your particular model has on our acceptance. Of course, there’s also a responsibility to report all the empirical strategies you tried that didn’t give you the results you were looking for. These are not “blind alleys”; they are possible disconfirmations, and you owe it to yourself and your readers to report them and explain why you think their negative verdicts should be set aside—if in fact they should.
Finally, Feynman’s subtle problem is familiar to anyone who reads widely in the econometric literature. The researcher encounters a problem, creates a theory to explain the problem and then tests the theory (or tries to produce results “consistent with” it), and when it works claim a sort of victory. But at a deep level this is a type of overfitting that impedes the ultimate purpose of scientific investigation, to develop an understanding of the world we can rely on in new situations.
Not all econometric work is guilty of the sins Feynman describes. There’s lots of good stuff out there! But there’s also a lot of deceptive stuff and no filter that tries to uphold scientific standards.
13 comments:
Ceteris paribus the implication IS consistent with the data...
I've always felt that far too many economists think it a success if they can predict what happened last year, based on the data from previous years - but are unconcerned with their model's ability to predict future events. It's easy to predict the past...
Whoops, I made a type... let's try again:
RepubAnon, out-of-sample tests are pretty common in finance. I'd even dare to claim that if you have a model that claims to predict but the model doesn't perform well out-of-sample, you'll have a very small probability of getting it published anywhere good.
KV, based on my admittedly skimpy reading of the finance literature, I think you're right, and my hunch would be that the same holds for other fields that overlap with practical or speculative decision-making. An example I know better is exchange rate theory, where out of sample prediction is everything. But in vast swaths of the model-plus-test literature -- take household behavior for instance -- my criticism applies all too well.
Based on the title, I expected the post to focus on a different methodological sin: the reversal of the "burden of proof," a.k.a. "my model IS the null hypothesis," characteristically committed with language like "the data fail to reject the model at the 95% confidence level." See? The data "are consistent" with my model. Happily, this sort of thing seems to be confined to certain disreputable corners of the literature which shall remain nameless. (Macro! Macro! Macro!) (Hush, you.)
Uhhh... Feynman is a physicist, right?
Are we studying what happens when two economic agents collide at high energies?
Or are you trying to say true things about human beings in an evolving social ecology?
Would you ask a physicist where to take a woman on a date?
Thornton, Would you take an economist out on a date? Would that depend on the economist's level of research validity? Or, would it depend upon the economist's professional position, i.e. who pays his salary?
That suggests an interesting area of study within the economics profession. "Character of Published Research and Who Funds the Chair" Of course this leaves out all economists employed in private industry and finance. Economists holding both an academic position and professional consultation gigs may be a special case requiring independent research. Or that group can be included in a multi variate design study. The null hypothesis: Academic economists whose positions are not funded by private sources, or hold chairs not named for high rolling donors, publish the same level of disputable findings as do academic economists who accept think tank grants or hold chairs funded by specific donors or their funded Trusts.
The profession gets what is paid for. The question is often, but who is it that pays the bill and what ideological perspective is expected there from?
The research I want to see is this: percentage of undergraduate physics majors that get PhDs in fields that study living things.
I might date an economist, but I would never date a physics major who looked at humans and said: I'll bet when they trade things it's just like how the quantity of water and water vapor is stable at any given pressure and temperature.
KV: Of course, out of sample prediction can have the same problem: Run OOS, find your model failed to generalize, tweak, rinse, repeat, OOPS!
That is a very difficult problem for a reviewer to detect, and basically relies on the integrity of the author to report.
"I might date an economist, but I would never date a physics major who looked at humans and said: I'll bet when they trade things it's just like how the quantity of water and water vapor is stable at any given pressure and temperature."
Maybe not, but economists have been saying the same thing for years now. I don't know how many times I've heard economists claim that a system has attained equilibrium in a system with perfect information, often without evidence of equilibrium and in the face of strong evidence of imperfect information. Are you saying you wouldn't date a physicist who has read too much economic theory?
Kaleberg: "Are you saying you wouldn't date a physicist who has read too much economic theory?"
Aha!! That implies the difference between the two groups. If a scientific group, in this case physicists, is accustomed to reading mostly research focused on data collection and analysis would they be likely to read material that is described as research, but is almost exclusively focused on theorizing and modeling? And vice versa. And which comes first, the theory or the data?
PS: it's surprisingly difficult (using Google) to find out the undergraduate major of famous economists. But if the Saltwater vs Freshwater debate seems to miss the point entirely, it might be interesting to note that Paul Romer and John Cochrane both majored in physics.
Post a Comment