Posts Tagged ‘Economics’

Mathiness in Economics

June 9, 2015

In the Papers & Proceedings section of American Economic Review, Paul Romer* describes, after revealing formal errors in a couple of recent prestigious articles, what he calls the new equilibrium in economics:

[…] empirical work is science; theory is entertainment. Presenting a model is like doing a card trick. Everybody knows that there will be some sleight of hand. There is no intent to deceive because no one takes it seriously.

At a seminar I attended recently, there was debate about the concept of rationality in economics. It was pointed out that any behavior (in the given case discussed, but the claim holds more generally, if not fully) can be rationalized if just the model is rich enough. But then rationality becomes worthless because we cannot falsify it, to use a Popperian term. This may be to take it too far; one can set up a model for rational behavior and find conflicting behavior. While we cannot conclude that behavior was irrational, we can conclude that the model we not rich enough.

Romer discusses mathiness in economics further on his blog.

*Romer, PM (2015) Mathiness in the Theory of Economic Growth, American Economic Review: Papers & Proceedings 105(5): 89-93.

Advertisements

IKEA economics

February 9, 2011

I knew IKEA was a crazy place, but it gets crazier. Ironically, when it rains, IKEA reduces the price of umbrellas with half. I find it hard to believe that IKEA do not know basic economics. Rain presumably increases the willingness to pay for umbrellas, so a rational  reaction to rain would be to increase the price.  (What about efficiency?) Perhaps the stunt is meant to get people to smile while they’re waiting at the register. Or perhaps it is just the occasional diversion IKEA executives afford themselves on a boring day. Or, perhaps it is meant to attract gamblers? The rain may stop before you reach the counter.

The Early, British Hegemony in Economics

November 30, 2010

I’m still reading New Ideas from Dead Economists by Todd Buchholz (progressing slowly; see Living Among the Dead). Although I haven’t seen much to the promising new ideas yet, Buchholz give a great, historical account of the development of economics.

The father of Economics was, as every economist know, Adam Smith, at least if we are talking about economics as its own, scientific disipline (and we are!). Adam Smith was from Scotland. Given that and Brittain’s position as world leader (in politics, trade, military, you name it), it comes as no surprise that all the early, great economists were British. They were also all rather close; this is how Buchholz begins the chapter on John Stuart Mill:

Almost all renowed British economists since Adam Smith have been linked through close friendships. Remeber that Smith’s good friend David Hume was a “godfather” to Thoms Malthus, who was an intimate friend with David Ricardo, whose comrade James Mill encouraged his economics. James begot John Stuart Mill. A slight break occurs since Mill did not befriend his successor Alfred Marshall. But Marshall learned from Mill’s works (and from the economist F.Y. Edgeworth, nephew or Ricardo’s friend Maria Edgeworth) and then thaught Keynes, who dominated British economics until World War II and produced numerous prominent disciples [p. 91].*

No surprise, perhaps, that the early development of a new field has a geographical structure, so to speak; after all, they had to learn from each other and compete for the same, few positions. Anyway, that it was a hegemony is beyond doubt:

In 1848, Mill published his chief work on economics, Principles of Political Economy. For decades it dominated the book market like monopolies Mill discussed within its pages. Oxford relied on the Principles until 1919, probably because its successor was written by Marshall, a Cambridge man. Indeed, the works of all the great economists illuminate long paths. [Here it comes:] From 1776 to 1976, just five books regined over economics in nearly unbroken succession: Smith’s Wealth of Nations, Ricardo’s Principles, Mill’s Principles, Marshall’s Principles, and Samuelson’s Economics. What they lack in imaginative titles, they make up in endurance [p. 102].

Looks like I just got five new books on my ‘buy and read’ list. Perhaps a tall order, but 200 years of economics, almost 90% of its history, in just five books sounds rather cheap. (But how many volumes?)

* New Ideas from Dead Economists, Revised Edition, Todd G. Buchholz, 1999, Penguine Books.

Big, Unsolved Problems in Economics

November 23, 2010

Earlier this year, experts gathered at the hub of the universe (which is Harvard, it seems) to suggest and debate the big, unsolved problems in the social sciences, economics between them. From a press release:

Initiated and funded by the non-profit Indira Foundation, this effort was inspired by David Hilbert, who challenged the world to solve 23 fundamental mathematical problems in 1900. Since then, mathematicians have solved 10 of the now-famous ‘Hilbert Problems’, creating new fields of knowledge along the way.

“Hilbert made two powerful observations,” said Nicholas Nash, a member of the Indira Foundation. “First, having important, unsolved problems is essential to the vitality of a discipline. And, as important, by identifying those problems, we can inspire future generations to solve them.”

Taleb was there, not surprisingly, and suggested the ‘Black Swan problem’:

How can we be robust against “Black Swans”; that is, how can we (1) identify domains where these consequential rare events play a large role (these are too rare for any statistical models track them properly), and (2) instead of predicting Black Swans, build systems and societies that can resist their shocks.

King suggested the problem of international institutions:

What is the relationship between strong international institutions and international cooperation? Do strong international institutions lead to or result from international cooperation?

King also suggested a methodological problem:

A major methodological problem is how to avoid (or ameliorate) post-treatment bias in big social science questions. Post-treatment bias occurs when the causal ordering among predictors is ambiguous or wrong or when, in an attempt to control for confounding variables, one controls away a consequential variable.

Nobody Reads the Introduction

May 27, 2010

The ‘unofficial’ (it’s not linked to from his official page) homepage of Ken Judd was brought to my attention. Ken Judd demonstrates the resistance against computational methods he has met in economics. In particular, he shows the ignorance and insulting attitude displayed by editors of Econometrica and members of the Econometric society. Of course, it’s a serious matter, and as far as I can tell, Ken’s anger and frustration is warranted. It is entertaining too, when Ken reports some of the stupidity he has to deal with:

The lead article of the silver anniversary volume of Econometrica began with an inaccurate description of the math programming literature, ignoring methods developed over 30 years ago. Trying to defend this paper, [a former/current North American council member of the Econometric Society] said, “It really did no harm since no one reads introductions.” I agree that experts in a field likely don’t read introductions, but is that the only audience? My reply was “I bet many graduate students read introductions.” I guess the lesson here is that readers should not trust introductory material in Econometrica articles.

A Farewell to Alms by Gregory Clark (Part Three)

May 21, 2010

Its been a while since I finished the third and last part of A Farewell to Alms; let me knot down some thoughts. I’ve commented on part one and two earlier; after discussing the pre-industrial world and the Industrial Revolution, Clark devotes part three to the development in the world after the Industrial Revolution. While all parts of Clark’s book is interesting, the final part was the part I enjoyed the most. Perhaps because he discusses current issues and more familiar problems, or perhaps because of his ideas are in fact interesting. It could also be Clark’s powerful expression and how he violently dismisses the economic mainstream.

Clark begins part three by discussing world growth since 1800: While some societies has become very rich, others are still poor, many even poorer than before the Industrial Revolution. Clark calls the development the Gread Divergence and raises the puzzle Why did it happen? Clark seem to have the answer. Before he gets to it, however, he wastes no time characterizing the economic mainstream for failing to understand and deal with the observed divergence:

Commentators, having visited climate, race, nutrition,  education, and culture, have persistently returned to one theme: the failure of political and social institutions in poor countries. Yet, as we shall see, this theme can be shown to manifestly fail in two ways. It does not describe the anatomy of the divergence we observe: the details of why poor countries remain poor. And the medicine of institutional and political reform has failed repeatedly to cure the patient […] Yet, like the physicians of the prescientific era who prescribed bloodletting as the cure for ailments they did not understand, the modern economic doctors continue to prescribe the same treatment year after year through such cult centers as the World Bank and the International Monetary Fund. If the medicine fails to cure, then the only conclusion is that more is needed [p. 328].

The main source of the divergence, Clark claims, is differences in efficiency. In particular, the difference is ‘rooted in an inability to effectively employ labor in production’ (p. 329). As he does throughout the book, Clark justifies his claim with ample evidence and a number of examples. Seldom has I read a book which collects, discusses, and uses as much data to drive the arguments home. From the evidence, Clark concludes:

Thus the crucial variable in explaining the success or failure of economies in the years 1800-2000 is the efficiency of the production process within the economy. Inefficiencies in poor countries took a very specific form: the employment of extra production workers per machine without any corresponding gain in output per unit of capital [p. 351].

Clark moves on to discuss evidence of differences in the quality of labor across societies. These differences, however, did not stem from the Industrial Revolution; rather, they were inbuilt in the societies from the Malthusian era. So what changed during the Industrial Revolution to produce the Great Divergence? There are three main reasons, according to Clark (see pp. 365-366). The first is that, under the Malthusian trap, differences in the quality of labor had no effect on the average income, only the population level. When population and income were decoupled, the differences in quality of labor expressed itself through the Great Divergence. Secondly, modern medicine reduced the subsistence wage such that populations could expand despite lower incomes than earlier. Thirdly, modern production techniques increased the wage premium for high quality labor.

When it comes to the causes of differences in quality of labor, there is no satisfactory theory, according to Clark (p. 370).

Arriving to the final chapter, rather exhausted, I must admit, after Clark had laid out his theory of world economic history in three, extensive parts, I ended up underlining almost the entire chapter. I cannot put it all in here, but the chapter, titled ‘Conclusion: Strange New World,’ begins like this:

God clearly created the laws of the economic world in order to have a little fun at economists’ expense [p. 371].

The value of a strong, opening line cannot be underestimated! Some more bashing on mainstream economics cannot hurt, either, seemingly:

Our economic world is one that the deluge of economics journal articles, working papers, and books devoted to ever more technically detailed studies of capital markets, trade flows, tax incidence, sovereign borrowing risk, corruption indices, rule of law serves more to obscure that to illuminate […] The great engines of economic life in the sweep of history demography, technology, and labor efficiency seem uncoupled from these quotidian economic concerns [p. 372].

In the end, Clark makes an interesting observation on the research into happiness; income seem to have only a slight effect on happiness. Clark offers the potential conclusion that humans may be programmed to be strivers for the relative rather than the absolute; the contented ones have been sorted out. At any rate, if happiness was our measuring rod, it leads to somewhat surprising conclusions (see pp. 376-377; perhaps the surprise is that happiness is not what society strives for).

Although Clark’s outlook is a simple one; the Industrial Revolution is the one, significant event in world economic history, he makes room for further dwellings on the issue in the last paragraph:

World economic history is […] full of counterintuitive effects, surprises, and puzzles. It is intertwined with who we are and how our culture was formed. No one can claim to be truly intellectually alive without having understood and wrestled, at least a little, with these mysteries of how we arrived at our present affluence only after millennia in the wilderness, and of why it is so hard for many societies to join us in the material Promised Land [p. 377].

Related posts:

Journal Submission Strategies

October 27, 2009

What sparked this post was a discussion with a fellow PhD-student, where I was told that advice from several senior researchers was not to submit a basically finished manuscript because it would have to go to an only okay journal; not a top journal. Instead, the manuscript should be totally reworked and then sent to a top journal. One top journal publication is supposedly more worth than five ‘other’ publications.

I don’t understand. Anyway, it motivated me to read a recent article from the B.E. Press by Heintzelman & Nocetti on journal submission strategies. The article starts out with two quotes from famous economists:

Start with a higher-quality outlet than your eventual target […] The professional returns to choosing a better journal are higher. But a strategy of aiming high requires thick skin; the acceptance rate at major economics journals is around 10 percent. Thus, it pays to have a ‘submission tree’ in mind, a sequence of alternative outlets for your work. – Daniel S. Hamermesh [see Heintzelman & Nocetti 2009 for the reference]

Give each of your papers a shot or two at the top journals, such as the AER, JPE, or QJE. Even if you are not confident in the paper, it is worth a try for two reasons. First, as author, you are not in the best position to judge its quality; some people are too fond of their own work, and some are too hard on it. Let the editors decide. Second, the editorial process is highly imperfect. The bad news is that some of your best articles may end up getting rejected from the top journals. The good news is that you may get lucky, and some of your so-so articles may end up published in top journals simply because they hit the editor’s desk when he is in a good mood. – Gregory Mankiw [p. 1]

Fair enough; these advices does not say only to go for the top publications. More interesting, perhaps, is footnote 2 on page 2, which refers to ‘Oswald (2007)’ [again, see Heintzelman & Nocetti 2009 for the reference], which ‘shows that the best (most-cited) articles in middle-tier journals are often ‘better’ than the least-cited papers in top-tier journals.’

Heintzelman & Nocetti 2009 moves on to show that Hamermesh’s and Mankiw’s advices holds up well in their analysis.

Given the long reviewing times in most journals, however, [the advices] may not be well suited for young, untenured, professors who are more likely to be impatient and risk averse. These authors should instead consider submitting to lower tier journals first [p. 3].

And then move up the ladder?

Heintzelman & Nocetti also brings advice for less gifted authors (read: me):

[A]uthors of papers that are not of the highest quality, and especially those without an established reputation, will lean towards lower tier outlets [p. 3].

The part on reputation is somewhat unsettling. Anyway, the ‘senior’ advice my fellow student got seems to be B.S.

(Somewhat) related post:

The Economics of Uncertainty, or How I’m Connected to Borch and Clive Granger

October 5, 2009

Something funny happened this morning. A small chain of events led me to be in the possesion of Clive W. Granger‘s copy of Karl Henrik Borch‘s The Economics of Uncertainty (1968).

What’s so funny about that? Well, Karl Henrik Borch was a Norwegian economist who had a big influence on the development of my school, the Norwegian School of Economics and Business Administration; one of the auditoriums there is named after him. Academically, Borch played an important role in the development of insurance economics. Clive W. Granger was an economist from UC San Diego. He was a giant in econometrics and was awarded the Prize in Economic Sciences in Memory of Alfred Nobel in 2003. When I was a grad student at UCSD in 2007/2008, I was fortunate enough to meet Clive.

This morning I came across some leftover books from Clive’s office outside the Econ department at UCSD (I’m visiting here). Among them, I found Borch’s book. It feels almost morbid to have this book, but also, as a friend told me, it was a find and I am very happy with it.

Krugman Picks Up the Ball

September 29, 2009

After I started discussing cap-and-trade, Krugman saw the need to explain the economics behind it (not the best ‘popular’ explanation of an economic idea, maybe, but fair enough; from the top of his head, I guess).

Anyway, in Krugman’s picture it is fairly easy to see the equivalence of a cap on emissions, which limits the amount of pollution allowed, and a tax on emissions, which increases the costs of pollution and thus indirectly limits pollution. A tax would be set equal to the permit price, polluters would pollute until their marginal benefit of more polluting activity equals the tax, and the deadweight loss would be the same as in Krugman’s picture.

As Kolstad pointed out, a cap may be better because the market knows best how to price pollution (a bureucrat would need to know the marginal benefits curves of all polluters to set the right tax). The right (or ‘optimal’) cap level, however, needs to be set by a bureucrat, and that is not necessarily any easier. (‘Really low’ is perhaps good enough in the current situation, though.) Main point is, there’s a lot of uncertainty around these things; how much do we need to reduce pollution, how much should we spend, what are the benefits; it goes on and on.

Hat-tip: Env-econ.

UPDATE: Jim Roumasset makes a lot of sense over on Env-Econ:

Taxes are better we are told because they generate more revenue. In contrast, cap and trade is said to be better because its primary purpose is to control pollution, whereas the primary purpose of emission taxes is to raise revenue. I’m afraid that these propositions cloud the waters.

[…] In the world of perfect competition, controlling quantity with price (Pigouvian taxation) is exactly equivalent to controlling price with quantity via transferable and auctioned permits. This remains true even if there is uncertainty about marginal damage costs but not about the marginal benefits of emissions (Weitzman, 1974).

[…] The equivalence perspective is also useful for understanding the implications of taxes vs. permits for revenue. In the world of certainty, there are none! Again, a specific tax on all emissions is equivalent to auctioning the permits. Same price, same quantity, same revenue. […] cap and trade can be designed to match the revenue-raising implications of carbon taxes and vice versa.

So much for blackboard economics. In the real world we have uncertainty about both costs and benefits. Clearly it is possible to design hybrid schemes that are superior to either taxes or permits, but I don’t think we have strong results about the optimal hybrid scheme.

Advice to Phd Students

September 27, 2009

I came across a post with advice for graduate students on Greg Mankiw’s blog which links to a lot of interesting reading. Among Don Davis’s advice on finding research topics, I found the following phrase:

Most of economics is boring. No, I don’t mean this in the way that the public at large means it; on the contrary, I think that economics done well can be beautiful and fascinating. What I mean is that most writing on economics is boring because: (1) It does not address interesting questions; (2) It has nothing new to add that is itself important; or (3) Even if the researcher does in fact have something new and important to say, the researcher does such a poor job of articulating this that the reader has little chance of figuring this out.

I take this as another push toward working on writing well, and an indication that writing is very important when it comes to contributing to a science. Science is social, and contributing in a social setting means communicating; doing it well means communicating well, and writing is the way the important communication happens in economics. (I mean, a lot of communication goes on in seminars, on conferences, and workshops, and it is important to do that well too, but when it comes to contributing to the science, it’s the writing that matters, not your slick tounge.)

Truth Versus Precision in Economics by Thomas Mayer

September 12, 2009

TruthVersusPrecisionMy initial interest in Thomas Mayer’s Truth Versus Precision in Economics was spured when it was mentioned alongside McCloskey’s The Rhetoric of Economics in a footnote in a paper I read; the paper refered to it as a justification to accept unconventional p-values (probability of sampling error) in evaluating regression results.* Anyway, I picked it up at the library and was soon enthralled by Mayer’s sympathetic ideas.

The main claim in Truth Versus Precision is that economics is a victim of the principle of the strongest link, which leads to increased rigouization and decreased real-world relevance.

Mayer argues persuasively that economists has incentives to spend too much time on formalism, and that the formally explicit parts of arguments thus gets too much attention. Weaker parts of arguments are usually tended to by arm-waving. Strong, mathematically explicit arguments are subject to relatively much attention and are thus made stronger; weaker, implisit or verbal arguments receives less attention and remains weak. Further, the strength of a chain of arguments is often measured by the strength of the strongest argument, counter to the proverb that a chain is no stronger than its weakest link:

I call this procedure of focusing attention on the strongest part of an argument, and then attributing its strength to the entire argument, the ‘principle of the strongest link’ [p. 57**].

Mayer further suggests that economists preoccupation with formalism governs the prestige ranking of economics fields:

The prestige ranking of economics runs: first, formalist theory; second, empirical science theory; third, policy-advicing and data gathering, and fourth, history of economic thought and methodology [p. 46].

Mayer is a macroeconomist, and naturally parts of Truth Versus Precision discusses problems in macroeconomics. In particular, he argues that the foundations of new classical economics are questionable and concludes:

[N]ew classical theory is another example of the principle of the strongest link. Its advocates rightly take pride in the rigour of their deductive chains. But a rigorous deduction from a questionable premise, accompanied by no adequate tests of the conclusions, does not guarantee truth [p. 120].

Mayer also discusses the problems surrounding empirical testing in economics, for example that many focuses solely on Type I errors, that regressors with insignificant coefficients are excluded, problems with pre-testing of data, and confusions between statistical and substantive significance (see pp. 134 – 139). Finally, he discusses problems surrounding robustness tests (or rather, the lack thereof) (see pp. 142 – 147). He concludes the chapter on emprical testing accordingly:

[M]ost econometric testing is not rigorous. Combining such tests with formalized theoretical analysis or elaborate techniques is another instance of the principle of the strongest link. The car is sleek and elegant; too bad the wheels keep falling off [p. 149].

In the last chapter, Mayer discusses possible remedies. He calls for less abstraction and less formality; more replications and retests; proper use of statistical tests; care for data and awareness of anecdotal evidence; he wants journals to act as communication devices (not archives); critical evaluations of conflicting evidence; less focus on formal techniques in graduate training programmes; and more focus on writing skills.

All in all, I find many of Mayer’s arguments persuasive; they align with my feeling of unease when it comes to mathematical economics (note; I’m a mathematical economist myself). Some of Mayer’s critique also align with some of McCloskey’s critique. However, a professor at my school told me that Mayer was out-dated already in 1993 (the year of publication), and mentioned an article by Alexander Rosenberg from 1983 as evidence: Rosenberg discusses new classical economics. Notwithstanding, I still think there is something to Mayer’s critique, and as I said, it resonates with my own attitude towards economics. A more recent treatise discussing the very modern development of economics would be useful; have economics ridden itself of the principle of the strongest link? I need to find out.

* See p. 157; as far as I can see the only place in the text that actually argues for unconventional p-values, but not unconditionally.

** Page numbers refer to the paperback edition.

Related posts:

A Vertical Division of Economics

August 26, 2009

Alright, this book (Truth Versus Precision in Economics, Tom Mayer) is so interesting that I’ve put Shakey aside for a while (in the middle of ‘Ohio’). I’m marking pages all the time.

In chapter 2, Two Types of Mainstream Economics, Mayer suggests to divide up economics vertically:

The fields that are often loosely referred to as ‘the sciences’ divide their subject matter not only horizontally by fields, e.g. biology and geology, but also vertically by the level of abstraction, e.g. physics and engineering, and physiology and medicine. In doing so they make room for different criteria at each level. An engineering paper need not be as rigorous as a physics paper. […] In the social sciences we divide fields horizontally, but not vertically, so that economics comprises mathematical formalism, empirical science work, and applications to specific practical problems. Hence we are tempted to apply inappropriate standards of rigour [p. 24, paperback edition].

On the following pages, he discusses the difference between formalistic and empirical economics, particularly in terms of rigour. He concludes chapter 2:

Although there is no sharp line of demarcation between hypotheses that are close to the observational level and those that are at a high level of abstraction, it is useful to classify the work of economic theorists into two categories, formalist theory and empirical science theory, because economists invoke widely different criteria in evaluating theories. Much misunderstanding is caused if one type of theory is judged by the criteria applicable to the other type [p. 36].

In the next chapter, he discusses whether there is too much of one type of economics and why that might be so:

We need both formalistic theory and empirical science theory. How much of each is more difficult to decide. While there are some reasons suggesting that formalist economics currently receives too little emphasis, there are stronger reasons to think the opposite is true. Professional self-interest, a belief that formalist economisits are much abler, and certain confusions all induce our profession to over-emphasize the importance of formalist theory relative to empirical science theory [p. 52].

Related post:

Economics: A Science or Not?

August 23, 2009

I came across this truly interesting book, Truth Versus Precision in Economics (1993) by Thomas Mayer. I’m still only on p. 16, but endnote 1 in chapter one provided fodder for my thinking about economics and science:

I see no purpose in discussing whether economics actually is a science. Philosophers have not succeeded in finding a criterion that distinguishes science from non-science […], and the question whether a field is an empirical science may even lack clear meansing […]. Fortunately, nothing hinges on whether one calls economics a science or not, and the question can be left to lexicographers. Knowing whether economics is a science would not allow us to decide whether  it should use the same methods as the natural sciences, since not all sciences necessarily us the same methods. What methods economics should use can be decided better by looking at specific methods and specific problems than by talking in general about ‘scientific method’. Similarly, knowing whether economics is a science would not allow us to say whether it provides answers that deserve a high degree of credence. The science of weather forecasting does not, while the non-science of history does [p. 8, paperback edition].

I agree that nothing really important hinges on whether economics belong to the (hard) sciences or not (what matters is that it is scientific). But, Mayer doesn’t seem to recognize that the English science has lost it’s propper meaning (here’s McCloskey’s explanation): Science means ‘systematic inquiry’ in any other language.

Related posts:

Green Bias

August 13, 2009

Economists often tend to think of biologists as tree-huggers or similar kinds. Of course, there’s something to it. Most researchers tend to work on issues that interests them, and of course the ‘intrinsic’ interest is an asset in the research. The researcher works harder. But is it also a problem? But of course. It influences the research agenda, it may bias results. Maybe more importantly, are biologists aware of such problems? Do they care?

Sometimes, I suspect economists to kind of use the tree-hugger characteristic against biologists. Is it sometimes because biology is a proud memeber of the hard sciences? Something econoics, notably, is not, traumatically enough.

Last night, I talked to an ecologist about this. He agreed that there might be something to my agenda; the research agenda of biologists are often coloured green; they may end up with biased results. But, he contended, economists aren’t necessarily any better. (He’s seen a lot of [environmental] economists make biological claims that are plain wrong!) He’ s probably right. People tend to what they care for & care for what they tend to.

What kind of bias does the ‘intrinsic’ outlook of the economist introduce? (I don’t know; I am [supposed to be] one. Is it blinding to care about efficiency & trade-offs?)

The ecologist slid off the hook, though. Economists doing the same mistake doesn’t free the biologists of guilt.

The Cult of Statistical Significance by Ziliak & McCloskey

July 28, 2009

The Cult of Statistical SignificanceLast year, Stephen T. Ziliak and Deirdre N. McCloskey published The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. I finished it a while ago now (during my trip to Alaska, in fact; a lot of time available on those planes), and I want to discuss it here.

Before checking out my amateur opinions, however, it may be wise to check out some the reviews that Ziliak kindly has gathered on his homepage. I, for example, found the one from Science (by Theodore Porter) interesting. Only read the review in Journal of Economic Literature (by Saul Hymans) if you don’t plan to read the book itself. You may settle with the following, closing paragraph:

Despite my firm belief that most applied econometricians would benefit from adopting the methodological position presented by Ziliak and McCloskey and that economics as science would be improved significantly thereby, I can’t close without something of a rebuke. As often happens when someone is pushing what the mainstream considers an extreme or fringe position, the arguments become narrowly and harshly focused. This comes through too often in Ziliak and McCloskey. In its particularly narrow perspective, their treatment of the professional accomplishments of a number of exceptionally gifted economists is simply unjustified. Included among such economists are Gary Becker, Trygve Haavelmo, Harold Hotelling, Lawrence Klein, and Paul Samuelson. It is especially unfortunate, for example, that Ziliak and McCloskey misrepresent the significance of Haavelmo’s pathbreaking article of 1944 and never even mention his major contribution of 1947, a piece which Ziliak and McCloskey should find quite simpatico [p. 503].

The ‘review’ in the Journal of Economic Methodology (by Tom Engsted) is a full-length article discussing a debate surrounding the Ziliak & McCloskey book.

So, I hope you’re fed up already: Here are my (largely unqualified) opinions on and comments to this book. As (almost) always; first things first: The Cult of Statistical Significance carries some important messages (despite it’s terrible title; maybe they were inspired by the dreadful Andrew Keen). The Cult tells us that statistical significance is not the same thing as substantive significance, that statistical significance is often misused and appears to be misunderstood by many a scientist (and particularly economists), and that only by attending to quantitative, scientific magnitude and judgement will sciences like medicine (yes, medicine), economics, and other statistically confused fields be able to move ‘into the age of science and humanity’ (p. 251; all page references are to the paperback edition). Alright, maybe the last one there may be discussed (and the quote is slightly out of context); notwithstanding, Ziliak and McCloskey certainly feels that way, wants their readers to feel that way, and bring a lot of good arguments to the table.

A quick, non-technical update on statistical significance is found here, by the way. From the first paragraph:

In normal English, “significant” means important, while in Statistics “significant” means probably true (not due to chance). A research finding may be true without being important. When statisticians say a result is “highly significant” they [should] mean it is very probably true.

Everyone who understands statistical significance understands that substantial significance is something else and the more important of the two. The disagreement would be whether statistical significance is misused, misunderstood, or both. Perhaps, then, Ziliak and McCloskey’s crown argument is their study of the practice with statistical significance in the American Economic Review during the 1980s. (To those unaware; a publication in the AER is among the most prestigious and important things an economist can achieve, particularly in terms of their career.) Their findings, discussed in chapter 6 (pp. 74-78) and published in the Journal of Economic Literature in 1996, is discouraging. The best economists (that is, those publishing in the AER in the 1980s) misuse statistical significance to a large degree.

Next, Ziliak and McCloskey do something odd. Faced with arguments from colleagues that best practice had improved since the 1980s, perhaps partly because of their 1996 article, they go ahead and do another study of the practice in the AER in the 1990s:

We are very willing to believe that since the 1980s our colleagues have stopped making an elementary error and especially that we have changed their minds. But being readers of typical economics articles since that first decace of personal computers [the 1980s] we seriously doubted that fishing for significance had much abated. […] And so in a second article, published in 2004, we [reapplied our study from 1996] to all the full-length empirical articles of the next decade of the AER, the 1990s [p. 79].

There’s a logical flaw here. If Ziliak and McCloskey caused the change with their 1996 article, it won’t show up in a study of the 1990s! And probably not of the first decade of the new millenium either; changes take a while, often a generation or so (just ask Thomas Kuhn, rephrased by Paul A. Samuelson in 1999: ‘Science advances funeral by funeral’). And indeed, Ziliak and McCloskey(or, McCloskey and Ziliak, as the reference will show) doesn’t find much of an improvement in the 1990s compared to the 1980s.

Of course, Ziliak and McCloskey have ideas on why scientific practice in the AER has not changed:

Significance unfortunately is a useful means toward personal ends in the advance of science – status and widely distributed publications, a big laboratory, a staff or research assistants, a reduction in teaching load, a better salary, the finer wines of Bordeaux. Precision, knowledge, and control. In a narrow and cynical sense statisitcal significance is the way to achieve these. Design experiment. Then calculate statistical significance. Publish articles showing “significant” results. Enjoy promotion.

But it is not science, and it will not last [p. 32].

They may be right. But, one cannot forget the position of the AER among economists. Because of it’s career generating potential, economists are likely to mimic it, both methodologically and rhetorically.

Still, Ziliak and McCloskey’s empirical evidence of the poor econometric practice in the AER is striking and convincing. I am now skeptic towards empirical economists. And not only economists: Ziliak and McCloskey summarizes evidence of bad practice in psychology, medicine, ecology and several other fields. One gets the impression that only the ‘hard’ sciences got it right (and I who used to think that medicine was a hard science).

In Ziliak and McCloskey’s view, one man is responsible for most of the statistical mess in economics and other fields: R. A. Fisher, whose Statistical Methods for Research Workers (1925), which went through no less than 14 editions, laid down the foundations for much of the later, statistical practice in many applied statistical fields. Ziliak and McCloskey attacks Fisher hard, in an almost distasteful way (they even accuse him of ‘outright, scientific fraud’ in an endnote somewhere, I’m sure, but I wasn’t able to find back to it), and large parts of the second half of the book they devote to attack Fisher in various ways and contrast him to (their hero, it appears) William Sealy Gosset, better know as ‘Student’ (look up Student-t). Maybe even more absurd is the blame put on the philosophical trends of the time:

One reason for the success of the Fisherian program against more logical alternatives […] is that the Fisherian program emerged just as neopositivismand then falsificationism emerged in the philosophy of science. It would have fallen flat in philosophically more subtle times, such as those of Mill’s System of Logic Ratiocinative and Inductive (1843) or Imre Lakatos’s Proofs and Refutations (1976). No serious philosopher nowadays is a positivist, no seroius philosopher of science a simple falsificationist. But the philosophical atmosphere of 1922-62 was perfect for the fastening of Fisher’s grip on the [statistically confused] sciences [p. 149].

Toward the end, after the very interesting chapter 23 (pp. 238-244), Ziliak and McCloskey get almost out of hand. On pages 249-250, they propose a “Statement on the proprieties of Substantive Significance” which they want editors, administrators and scientists to sign. The language of their ‘statement’ is, however, to involved and won’t hold as a standalone statement. I don’t understand the purpose of the statement when it isn’t self-containted, and otherwise just repeats a message that has been pounded upon throughout the book. And they keep pounding it, increasing their volume:

The textbooks are wrong. The teaching is wrong. The seminar you just attended is wrong. The most prestigious journal in your scientific field is wrong [p. 250].

Got it.

I had high expectations for this book, particularly because McCloskey’s name was on the cover. I was sligthly disappointed, however. By close examination, I discovered that McCloskey’s name is put last and that they’ve ignored the alphabetical order of names: Ziliak is the main author. It is obvious in some places, Ziliak don’t have McCloskey’s Economical Writing under his skin (neither do I, of course, but I would expect McCloskey to). This is a minor issue, certainly, but I looked forward to some persuasive, well-written, and witful prose of the kind McCloskey promotes in her Economical Writing; most of the time, it didn’t happen.

UPDATE: I have a hundred things to say about this book, but I cannot say them all at once. My above review is bad, I know, and I apologize. I confused the important things I wanted to say with the unimportant ones. I may get back to the book in later posts, but for now, I suggest the reader rather reads my discussion of the debate between Spanos and Ziliak & McCloskey. It presumable gives a better idea of what’s important in the book and yields more enjoyable and interesting reading.

And, for the record, let me again point out that I agree with Ziliak & McCloskey on their main point in The Cult, as stated in their reply to Spanos:

Statistical significance is neither necessary nor sufficient for substantive scientific significance.