Internal/External, Validity/Consistency

As every consultant knows, all the mysteries of the universe can be revealed in a two by two matrix.  We divide the cases up one way and then some other way.  That gives us four cells and vast, remunerable wisdom.

Here is my version for economics.  One way of dichotomizing how much faith we should put into hypotheses is between reasoning and evidence.  Reasoning is about consistency.  An inconsistent argument is at war with itself in some way and should be regarded with suspicion.  The other criterion is evidence.  Evidence either adds to or detracts from the validity of an argument.  Ideally a hypothesis should be strong on both fronts, although we know our powers of formulating and testing hypotheses are incomplete, especially in social sciences like economics.  We don’t necessarily rip up and burn theories that have consistency or validity problems, but we take those problems seriously.  Or should.

The other dimension is internal/external.  Internal means “with respect to this particular empirical study or body of theory” and external “with respect to all the rest of the empirical cases and theory out there”.  Each piece of work needs to be judged on its own terms, but research and analysis do not occur in a vacuum.  We also have to be mindful of the empirical world outside our particular sample, and we should respect the models developed by other researchers, especially when they have done well on consistency and validity tests.

The distinction between internal and external validity is familiar from statistics.  Internal validity is about the reliability of our claims based on the quality of our data and power of our analytical tools.  External validity is about how well a given study generalizes to the larger universe of cases we care about.  There has been a move toward experimental and quasi-experimental methods in economics, because studies designed around experiments have more internal validity for questions about causation.  This is controversial, however, because the constraints entailed in setting up or finding experiments often detract from external validity.

But the same tension exists on the theoretical side, consistency.  An extremely important example can be found in the debate surrounding so-called microfoundations in macroeconomics.  (I say so-called because, strictly speaking, representative agent models are not microfounded.)  Models incorporating intertemporal utility maximization are typically preferred by economists because they  possess more internal consistency, where “internal” means “within economics”.  Given that the models do not distinguish themselves empirically, their main defense, in fact, is exactly this type of consistency.  But wherever there’s an internal, there’s an external.  External consistency in this context refers to consistency with the models promulgated outside economics, for instance in psychology, social psychology, neuropsychology and sociology.  Given that academia is siloed, most economists give little thought to this, but they should.  The assumption that there is such a thing as utility, that individuals maximize it, and that they do so over their life cycle is radically inconsistent with the understanding of human behavior one finds in these other social sciences.  That’s a problem.

External inconsistency is not necessarily fatal.  One can always try to make the case that I’m right and you’re wrong.  That sometimes happens when the findings in one scientific field contradict and undermine what practitioners in another field believe.  It could happen here too.  But (1) economists seem unaware of the inconsistency, and (2) a cursory look at the state of knowledge suggests that the basis for the economics way of modeling behavior is its convenience, while other social and natural scientists have actually gone out and tested different behavioral hypotheses.

The irony is that we have now had more than two decades of prominent work in behavioral economics that has established the second point conclusively but hasn’t made a dent in what passes for a “consistent” model.  Yet.