Replication in Economics

Despite the voodoo noise, a different and lot more interesting debate about replication in economics has started on Environmental Economics. The debate on Env-Econ is a response to a post on Market Movers, which opens like this:

Falsifiability and replicability are key cornerstones of any academic research. If you’re running an empirical study, and your results aren’t replicable, your study is largely worthless.

First of all, the claim that falsifiability is a key cornerstone of academic research is simply not true, or at least not agreed upon. I understand falsifiability in the Popperian sense:

For Popper, a theory is scientific only if it is refutable by a conceivable event. Every genuine test of a scientific theory, then, is logically an attempt to refute or to falsify it, and one genuine counter-instance falsifies the whole theory.

(Read more about Popper and his ideas at the Stanford Encyclopedia of Philosophy.) Popper’s philosophy of science has been subject to extended debates. Thomas Kuhn, for example, claims that science consist of problem solving within a paradigm, and that paradigms change through scientific revolutions. In particular, Kuhn did not think that a statement had to be falsifiable to be scientific, as Popper did.

I am more willing to agree on the importance of replication. In the hard sciences like physics and chemistry, replication was truly important and researchers (ideally) remained sceptic towards new results until they had been replicated independently from the initial study. Since most research in physics and chemistry are conducted in laboratories, controlled experiments could fairly easily be replicated and results compared.

In economics, however, only recently have researchers started to conduct controlled experiments in laboratories. Tim Haab writes (On replicability in economics and the validation of models)

With economics models, at least until recently, we don’t have labs.  We are working with real observations from highly complex systems.  As such, we are forced to rely on modeling by assumption and measurement through statistical force rather than isolated direct observation.  This makes external validation of economic models extremely difficult.

This is changing over the past decade or so with the advent of experimental economics research in which researchers are testing fundamental economic results in a controlled setting.  Unfortunately, these experiments often suffer from over control which makes generalizability and practical applicability of laboratory results difficult at best. […]

To me, lack of replicability is not a condemnation of economics as a field, rather a challenge to the field to continue the unending pursuit of defensibility.

As Tim also points out, the post on Market Movers is primarily concerned with duplication of results (instead of independent replication), which requires the same data set and statistical tools. Such research is quite rare in economics, and it has different reasons (see Tim’s post). What is more common in economics, however, is that researchers collect their own data and see if they can find the same kind of conclusions as those available in the literature. John Whitehead puts it like this:

Why bother with replication with someone else’s data when you can publish your own study with your own data? The only time you ask for someone else’s data is if you really think they’ve made a horrible mistake or if you have an ax to grind.

However, I do agree with Market Movers that in general, replication and duplication is more important than what it seems to be in academia today.

Hat-tip: Env-Econ


Tags: , , , , , , ,

%d bloggers like this: