Posts Tagged ‘Thomas Kuhn’

Gould on Biased Reporting in Science

July 5, 2013

At Bookish, a used book store in Berkeley, I picked up Stephen Jay Gould’s essay collection An Urchin in the Storm. In an essay on the fundamental problems with human sociobiology, a discipline that tries to explain social behavior with Darwinian selection, Gould has an interesting comment on the problem with biased reporting in science.

Few observers outside science (and not nearly enough researchers inside) recognize the severe effects of biased reporting. The problem is particularly acute, almost perverse, when scientists construct experiments to test for an expected effect. Confirmations are joyfully reported; negative results are usually begrudgingly admitted. But null results-the failure to find any effect in any direction-are usually viewed as an experiment gone awry. Meticulous scientists may report such results, but they disappear forthwith from the secondary literature (and are almost never reported in the press). Most scientists probably don’t publish such results at all-who has the time to write up ambiguous and unexciting data? And besides, they rationalize, maybe next week we’ll have time to do the experiment again and get better results. I call such nonreporting perverse because we cannot gauge its depth and extent. Therefore, we do not know the proper relative frequency of most effects-a monumental problem in sciences of natural history, where nearly all theoretical claims are arguments about relative frequencies, not statements about exclusivity [p. 37, 1988 paperback edition].

Incidentally, I just work through a referee report that asked for more motivation for a model complication which did not affect the main results much. I cannot help but suspect that if the complication had lead to larger effects, the referee would be less inclined to ask for further motivation. The irony: The referee agrees that the complication is novel and leads to a more realistic model, however slightly.

In introducing sociobiology, Gould mentions Kuhn. I sense a sarcastic tone in the final remark.

Thomas Kuhn’s seminal work, The Structure of Scientific Revolutions, affected working scientists as deeply as it moved those scholars who scrutinize what we do. Before Kuhn, most scientists followed the place-a-stone-in-the-bright-temple-of-knowledge tradition, and would have told you that they hoped, above all, to lay many of the bricks, perhaps even set the keystone, of truth’s temple-the additive or meliorist [the doctrine that the world tends to become better or may be made better by human effort, definition from] model of scientific progress. Now most scientists of vision hope to foment revolution. […] We are therefore awash in revolutions, most self-proclaimed [p. 27].

Progress and Its Problems by Larry Laudan

June 12, 2013

From time to time, I read books on philosophy of science. A good while ago, I read Progress and Its Problems by Larry Laudan. The book has the subtitle Towards a Theory of Scientific Growth. I have had this book for a long time, but has hesitated to read it after I found Kuhn vs. Popper by Steve Fuller an unsettling read. (I actually wrote a longer review of Kuhn vs. Popper than what I posted here, but never got around to edit the last part of it properly; I guess I should, as my ‘review’ is actually just a weird sort of summary.)

Anyway, another reason for my hesitation was that I was quite frustrated with philosophy of science and did not realize it was my understanding (or depth) which frustrated me. Most who has tried to produce knowledge (to be a scientist) and tried to understand Popper and Kuhn must agree that both their theories are artificial. Laudan, however, presents a theory for scientific growth which makes good sense and agrees well with empirical (anecdotal?) knowledge of scientific development.

ProgressAndItsProblemsThe central element in Laudan’s theory is the research tradition:

[…] I propose that the rationality and progressiveness of a theory are most closely linked-not with its confirmation or its falsification-but rather with its problem solving effectiveness. I shall be arguing that there are important nonempirical, even “nonscientific” (in the usual sense), factors which have-and which should have-played a role in the rational development of science. I shall suggest, further, that most philosophers of science have mistakenly identified the nature of scientific appraisal, and thereby the primary unit of rational analysis, by focusing on the individual theory, rather than on what I call the research tradition. This study will show, moreover, that we need to distinguish between the rationality of acceptance and the rationality of pursuit if we are to make any progress at reconstructing the congitive dimensions of scientific activity [p. 5,* italics in original]

Laudan aims to shift the research focus from a search for truth (which we cannot identify anyway) to a focus on progress:

[…] the rationale for accepting or rejecting any theory is thus fundamentally based on the idea of problem-solving progress.  If one research tradition has solved more important [scientific] problems than its rivals, then accepting that tradition is rational precisely to the degree that we are aiming to “progress,” [that is], to maximize the scope of solved problems. […], the choice of one tradition over its rivals is a progressive (and thus rational) choice precisely to the extent that the chosen tradition is a better problem solver than its rivals [p. 109].

Unfortunately, I do not have the time to give a decent and comprehensive account of Laudan’s ideas, for that, I must refer you to the book. (I am not even sure a decent account of short length is probably; the book is perhaps as brief as it can be. Laudan mostly writes economically.) Some key parts that to some degree can be studied out-of-context: The discussion of anomalous problems (pp. 26-ff). On problem solving and ambiguous tests (pp. 42-ff). The deconstruction of Kuhn and Lakatos (pp. 73-ff). On the progressiveness of ad hoc modifications (p. 115). The discussion of rationality at the beginning of chapter four (pp. 121-ff) should be read by every rational scientist, and perhaps in particular economists for whom rationality has such an central, theoretical role. On scientific revolutions, and Kuhn again (pp. 133-ff). Finally, on the justification for scientific research (pp. 224-225).

Some further interesting points: The note on on why Adam Smith wrote his treatise on moral philosophy (to resolve tensions between his economic theory and the Newtonian thesis of a balance of forces in nature (endnote 10 to chapter 2, p. 230). The (long) note on Foucault (“[…] Foucault has benefited from that curious Anglo-American view that if a Frenchman talks nonsense it must rest on a profundity which is too deep for a speaker of English to comprehend[!]”) (endnote 12 to chapter 6, p. 241). Again finally, the note on sociology of knowledge is also great (endnote 29 to chapter 7, pp. 244-245). Why do so many nonfictional writers put so much of interest in small print at the back? Who started this odd tradition?

I should have written a proper review of Laudan when I had it fresh in mind. What I can say is that it reinserted a feeling of aim and purpose into my own work as a researcher (something neither Kuhn nor Popper will likely do for you). It also felt like some sort of closure, as my thirst for further insights into the philosophy of science has since dried up(?). My unread volumes on Popper and Feyerabend will likely remain unread for a while still. But, in parts Laudan only sketches out his ideas. Some day I will most likely try and follow some of the loose ends; perhaps there are some interesting problems at the end of some of them? (A [long run] better solution would of course be to befriend someone in the philosophy department, but who has the [short run] courage for that?)

I am trailing off. Let me rather conclude with a sobering economic comment on research funding from Laudan’s epilogue:

Far too much scientific research today is devoted to problems which are as cognitively trivial as they are socially irrelevant. If the “pure” scientist is to deserve the generous support presently being lavished on him [Laudan might be thinking of English college professors here], he must be able to show that his problems are genuinely significant ones and that his program of research is sufficiently progressive to be worth gambling our precious and limited resources on it [p. 225].

* Page numbers refer to the 1978 paperback edition.

Kuhn vs. Popper by Steve Fuller, Part 2

October 19, 2010

Part 2? Take Two, rather (This is Take One). It surprises me how difficult it is to get to grips with this book, particularly given its apparent brevity (the main body of the book runs through page 215 in a relative small format). Of course, I’m not even an amateur philosopher of science, but still.

A part of the difficult lies in the chaotic or at least hidden structure of the book. Fuller announces his motives in the introduction (‘to recapture the full range of issues that separate [Kuhn and Popper],’ see p. 3*). The ‘full range’ is presumably a lot of material; the already mentioned brevity is thus surprising. But Fuller do not list nor declear the ‘issues’ he wants to address. There seem to be no plan or structure. Rather, he seems to move from issue to issue in a haphazard fasion, and the motive or aim of the discussion is often out of sight and elusive. The conclusion of the book is also something of an anti-climax. The last chapter seemingly only discusses one of the issues separating Kuhn and Popper; there are no final remarks, no conclusion, or anything that resembles a closure.

Kuhn vs. Popper did increase my understanding and knowledge of the ideas of both Kuhn and Popper, and also how their ideas connect to the ideas of other important thinkers. Perhaps more importantly, Fuller has helped me see the important differences between Kuhn and Popper. Throughout the book, for one thing, Fuller comes up with comparative statements.

Kuhn and Popper represent two radically different ways of specifying the ends of inquiry: What drives our understanding of reality? Where is the truth to be found? [p. 56].

Kuhn was indeed authoritarian and Popper liertarian in their attitudes to science. This point has been largely lost, if not inverted, by those who regard ‘Kuhn vs Popper’ as a landmark in 20th-century philosophy of science [p. 13]

Popper was a democrat concerned with science as a form of dynamic inquiry and Kuhn an élitist focused on science as a stabilising social practice. Nevertheless, they normally appear with these qualities in reverese. How can this be? [p. 68].

To dig deeper into these differences, one has to dig into the actual ideas. Kuhn first:

For Kuhn, science begins in earnest with the adoption of a ‘paradigm’, which means both an exemplary piece of research and the blueprint it provides for future research […] Kuhn deliberately selects the phrase ‘puzzle-solving’ (as in crossword puzzles) over ‘problem-solving’ to underscore the constrained nature of normal science […] A ‘revolution’ occurs [upon a ‘crisis’] when a viable alternative paradigm has been found. The revolution is relatively quick and irreversible. In practice, this means that an intergenerational shift occurs [pp. 19-20].

An important aspect of Kuhn’s philosophy of science is how history is rewritten after a scientific revolution, such that the scientific development appears streamlined and meaningful. In Kuhn’s view, Fuller writes,

[…] the secret of science’s success – its principled pursuit of paradigmatic puzzles – would be underminded if scientists had the professional historian’s demythologised sense of their history. After all, in the great scheme of things, most actual scientific work turns out to be inconsequential or indeterminately consequential [p. 20].

Another important feature of Kuhn’s ideas regards how people become scientists. One becomes a scientist through a “conversion experience or ‘Gestalt switch,’ whereby one comes to see the world in a systematically different way” (p. 21).  These features, combined with the conservative flavor of Kuhn’s The Structure of Scientific Revolutions, led Popperians to liken Kuhn to ‘religious and politcal indoctrinators’ (p. 21).

But of course, this was not how Structure was read by most of its admirers – if they actually did read the book. For while Kuhn’s examples are drawn almost exclusively from the physical sciences, these are the disciplines that have probably paid the ‘least’ attention to Structure, even though Kuhn himself was qualified only in physics. Kuhn’s admirers are to be found instead in the humanities and the social and biological sciences [p. 21].

Kuhn’s admirers persisted in wrenching Structure from its original context and treating it as an all-purpose manual for converting one’s lowly discipline into a full-fledged science. These wishful readings of Structure have been helped by its readers’ innocence of any alternative accounts of the history of science – often including their own – with which to compare Kuhn’s [p. 22].

When Fuller turns to discuss Popper, his sympathies with Popper become obvious:

[Popper] was always a ‘philosopher’ in the grand sense, for whom science happened to be an apt vehicle for articulating his general world-view [pp. 22-23].

For the ‘grand philosopher,’ philosophy of science is only a reflection of more fundamental attitudes:

Once Popper’s philosophy of science is read alongside his political philosophy, it becomes clear that scientific inquiry and democratic politics are meant to be alternative expressions of what Popper called ‘the open society’ [p. 26].

Popper grew up intellectually among the positivists in the Vienna Circle, but disagreed with them on their attitude towards the role of logical deduction.

For the positivists, deduciton demonstrates the coherence of a body of thought, specifically by showing how more general knowledge claims explain less general ones, each of which provide some degree of confirmation for the more general ones. For Popperians, deduction is mainly a tool for compelling scientists to thest th econesequences fo their general knowledge claims in particular cases by issuing predictions that can be contradicted by the findings of empirical research. This is the falsifiability principle in a nutshell [p. 25].

Fuller neatly sums up the difference between the 20th century’s giants in the philosophy of science:

Whereas actual scientific communities existed for Popper only as more or less corrupt versions of the scientific ideal, for Kuhn the scientific ideal is whatever has historically emerged as the dominant scientific communities [p. 6].

* Page numbers refer to the Icon Books 2006 paperback edition.

Related post:

Kuhn vs. Popper by Steve Fuller, Part 1

September 25, 2010

In Kuhn vs. Popper, Steve Fuller discusses and compares the two most important philosophers of science in the 20th century; Thomas Kuhn and Karl Popper. The book is not what I expected it to be (what I expected is not entirely clear, but that is not the point). The book as a broader view than I expected, for example, revisiting philosophers from Plato to contemporary and, to me, new names like Rorty, and shuffles over themes from theology to nazism. The broad view is indeed intentional and announced in the introduction:

This book is designed to recapture the full range of issues that separate these two self-styled defenders of science [Kuhn and Popper, of course].  Many of the issues plumb the depths of the Western psyche: What is the relationship beteween knowledge and power? Can science bring unity to knowledge? Can history bring meaning to life? At the same time, these issues are entangled in more secular concerns about economy and society, politics and war  most of which are still very much with us today [p. 3].*

While I cannot say I found every twist of the book equally interesting, it did help me understand the the ideas of Kuhn and Popper better than before and even correct things I got wrong. For one thing, I used to think of Popper as the normative and Kuhn as the descriptive; Fuller claims I was wrong:

It was not, as is often said, that Kuhn was more ‘descriptive’ and his rivals [the Popperians] more ‘prescriptive’ with respect to the history of science [p. 208].

Thomas Kuhn (1922 - 1996)

According to Fuller, the Popperians appeared ‘perversly contrarian’ to the public when they imposed a normative perspective on the history of science; to the public, science’s authority was self-evicent. Kuhn, on the other hand, ‘never articulated the norm’ he imposed when he selected and arranged the historical examples he used in his arguing (p. 209). In other words, one needs a rather good idea of the history of science in order to see and understand that Kuhn presents only the examples that fits with his theory; according to Fuller, hsitory is abound with examples less in accordance with Kuhn’s theory.

A thing I do not like about the book is that it is densely written, and in order to follow it, one need to at least be acquainted with the theories of Kuhn and Popper up front; Fuller only briefly summarizes the ideas. I find this surprising, as the cover is littered with acclaim from newspapers like the Economist and the Financial Times with the obvious purpose to attract the general reader. According to Fuller, this involved writing, presupposing knowledge of the issue at hand was a trademark of Popper, and, I’m afraid, many philosophers:

Popper and Adorno [another philosopher Popper locked horns with] shared the critic’s tendency [not entirely clear to me what ‘the critic’s tendency’ is supposed to point to here] to presuppose that the audience already knows the target of criticism in some detail, so that one’s own discourse becomes a series of reflections on the hidden opponent. This feature made it frustrating for listeners who sought  constructive advice on the conduct of social research [p. 154].

Frustrating indeed, do you listen, Fuller? (I guess not.)

Now, while flipping through the book, I realize I’ve underlined too many quotable passages and marked too many passages to address them all in one post. I will thus try to focus my opinion of this book in a later post; stay tuned.

* To be sure, page numbers refer to the Icon Books 2006 paperback edition.

Related post:

Dense Philosophy of Science: A Necessity?

September 13, 2010

What we call the ‘modern’, and distinctly Western, sensibility emerged as people tried to organise the conduct of the sciences in light of second-order condsiderations of what might be common to all the sciences. The result was a Galilean zeal for spotting latent contradictions between bodies of knowledge, the pretext for eliminating the social, lingusitic and practical barriers to their proper integration into one system of thought. Popper promoted a version of this strategy in his attack on the ‘myth of the framework’, the Kuhnian idea that the presence of incommensuarable theories rendered any explicit normative comparison so difficult that one simply had to wait for history to take its course, as individuals come to adopt one or another theory for their own reasons. In contrast, Popper argued that if the incommensurable theories are truly scientific, they aspire to universality, which means that there will be cases that they have yet to explain or predict. These cases may then serve as relatively neutral ground for designing a crucial experiment to decide amonst the theories.*

I’m reading Steve Fuller’s truly interesting account of ‘The Struggle for the Soul of Science,’ the legendary debate between Kuhn and Popper. Although I enjoy the book, I cannot help but marvel at the dense style Fuller, and many in his field, pledge to. I wrote an essay once, on what I’ve late learned is known as the underdetermination hypothesis (data can be explained by any number of mutually incompatible theories, see, f.eks. pp. 61-62 in Kuhn vs. Popper). I wrote rather dense myself, I seem to remember (I’ll post an example if I can dig it up). I wonder if a complicated style is necessary to philosophy of science, if it is impossible to iron it out more, if it’s somehow inherent to the subject. Philosophy of science is, in a weird way, depending on itself for its own existence. (See; I came up with something rather dense just writing about how dense it is!)

* Steve Fuller (2003), Kuhn vs. Popper, pp. 66-67, Icon Books 2006 edition.

IIFET 2010

August 18, 2010

I attended IIFET 2010 in Montpellier, France, this summer. I had a great time, meeting colleagues from around the world. IIFET stands for the International Institute of Fisheries Economics and Trade and organises biennial conferences. I’ve been to a couple of conferences, but never to an all-fisheries one; it was a pleasing experience (I’m a fisheries guy).

In addition to the social/network aspect of going to conferences (which is, perhaps, 50% of my motivation to attend conferences), I appreciate the plenary sessions. I observe that many conference participants skip the plenary sessions, but I cannot understand why. The conference organizers have, probably in most cases, tried tried their best to get the best and most interesting plenary speakers. Most times, invited keynote speakers have important and interesting things to tell you (it cannot be everyday they have the attention of an entire research field, and if they realize, they will struggle to make their talk matter). A missed plenary is a missed oportunity to learn from and listen to someone important who has something important to say.

One of the plenary speakers at IIFET 2010 was Anthony Scott, Professor Emeritus at the University of British Columbia, 87 years old, and a living legend in fisheries economics.* In his talk, he applied Thomas Kuhn’s ideas about scientific revolutions to fisheries economics. Truly amazing. Not only the idea itself, which seem obvious once it is formulated but still requires a wide perspective and intimate and detailed knowledge of the whole field to be formulated, but also that he remains relevant and contributes 55 years down the road. I just hope someone is responsible enough to make sure that paper gets published in one form or another.

The other speaker in the same plenary session was the IIFET 2010 Distinguished Service Award winner Susan Hanna. (Anthony Scott was given the IIFET Fellow Award, by the way.) Her talk was another plenary I wouldn’t have missed. She critisized how the current practice (meaning everything from publishing and hiring practices down to choice of words and metaphors) in fisheries economics, and in much of economics for that matter, failed to fulfill the ideal of any research activity; to be relevant in the real world, to inform policy, and to educate the public. I just hope everyone listened really carefully.

UPDATE: As Ann Shriver, IIFET’s Executive Director, writes in the comments, Susan Hanna’s presentation is available at the IIFET home page. It gets really interesting from slide 10 and onwards. Ann Shriver also confirms that both Scott’s and Hanna’s papers will be published.

* In 1955, Scott published the influental and cornerstone-article ‘The Fishery: The Objectives of Sole Ownership’ in the Journal of Political Economy. From a (fairly) recent praise-speak:

In present day fisheries economics, we accept [Scott’s] 1955 conclusions without question […] So, we can argue that [the] 1955 article provided the foundation for present day fisheries economics. It was, in fact, an article well ahead of its time.

Legendary status is further confirmed by the Anthony Scott fonds in  the UBC Archives; a collection of all his letters, reports, lecture notes, and even desktop diaries(!), no less than 7.15 meters of ‘textual materials.’

A Report from Montréal

July 1, 2010

‘This week, I’m attending the Fourth World Congress of Environmental and Resource Economists in Montréal. It’s been great so far, except my own presentation was a bit of a mess. I presented in the morning session on the first day and I was exhausted from the travel. That’s just an excuse, but it is what it is. Been to a lot of interesting sessions. The difficult thing is always to choose between the sessions. Yesterday afternoon, for example, I had to drop Thomas Kuhn for the session on Ethics and Social Norms.

Yesterday, AERE, EAERE, and FEEM presented their FEEM 20th Anniversary Prize to Resources for the Future and Marty Weitzman. Worthy winners, for sure. I attended Weitzman’s presentation on Tuesday and it was great, as expected. Depressing conclusions on the dismalness of economics (when it comes to climate change, that is), but important research nonetheless, the way I see it.

I’m discovering Montréal a little bit at the time. As Tom Sterner said in the first plenary session, it was a great idea to have a jazz festival in parallell with the conference, and I wholeheartedly agree. I just wish I had more time to enjoy the festival; the conference and surrounding arrangements takes 15 hours a day. My hotel, I have discovered, I just at the intersection between the shopping district, the jazz festival area, the Chinese quarter, and a more questionable and run down area with strip joints on every corner. To get to the conference site, I have to traverse the strip club area, which certainly adds to my ‘daily’ experience of Montréal.

Gotta run; more session to attend to, more people to meet, more fun to be had.

The Cult of Statistical Significance by Ziliak & McCloskey

July 28, 2009

The Cult of Statistical SignificanceLast year, Stephen T. Ziliak and Deirdre N. McCloskey published The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. I finished it a while ago now (during my trip to Alaska, in fact; a lot of time available on those planes), and I want to discuss it here.

Before checking out my amateur opinions, however, it may be wise to check out some the reviews that Ziliak kindly has gathered on his homepage. I, for example, found the one from Science (by Theodore Porter) interesting. Only read the review in Journal of Economic Literature (by Saul Hymans) if you don’t plan to read the book itself. You may settle with the following, closing paragraph:

Despite my firm belief that most applied econometricians would benefit from adopting the methodological position presented by Ziliak and McCloskey and that economics as science would be improved significantly thereby, I can’t close without something of a rebuke. As often happens when someone is pushing what the mainstream considers an extreme or fringe position, the arguments become narrowly and harshly focused. This comes through too often in Ziliak and McCloskey. In its particularly narrow perspective, their treatment of the professional accomplishments of a number of exceptionally gifted economists is simply unjustified. Included among such economists are Gary Becker, Trygve Haavelmo, Harold Hotelling, Lawrence Klein, and Paul Samuelson. It is especially unfortunate, for example, that Ziliak and McCloskey misrepresent the significance of Haavelmo’s pathbreaking article of 1944 and never even mention his major contribution of 1947, a piece which Ziliak and McCloskey should find quite simpatico [p. 503].

The ‘review’ in the Journal of Economic Methodology (by Tom Engsted) is a full-length article discussing a debate surrounding the Ziliak & McCloskey book.

So, I hope you’re fed up already: Here are my (largely unqualified) opinions on and comments to this book. As (almost) always; first things first: The Cult of Statistical Significance carries some important messages (despite it’s terrible title; maybe they were inspired by the dreadful Andrew Keen). The Cult tells us that statistical significance is not the same thing as substantive significance, that statistical significance is often misused and appears to be misunderstood by many a scientist (and particularly economists), and that only by attending to quantitative, scientific magnitude and judgement will sciences like medicine (yes, medicine), economics, and other statistically confused fields be able to move ‘into the age of science and humanity’ (p. 251; all page references are to the paperback edition). Alright, maybe the last one there may be discussed (and the quote is slightly out of context); notwithstanding, Ziliak and McCloskey certainly feels that way, wants their readers to feel that way, and bring a lot of good arguments to the table.

A quick, non-technical update on statistical significance is found here, by the way. From the first paragraph:

In normal English, “significant” means important, while in Statistics “significant” means probably true (not due to chance). A research finding may be true without being important. When statisticians say a result is “highly significant” they [should] mean it is very probably true.

Everyone who understands statistical significance understands that substantial significance is something else and the more important of the two. The disagreement would be whether statistical significance is misused, misunderstood, or both. Perhaps, then, Ziliak and McCloskey’s crown argument is their study of the practice with statistical significance in the American Economic Review during the 1980s. (To those unaware; a publication in the AER is among the most prestigious and important things an economist can achieve, particularly in terms of their career.) Their findings, discussed in chapter 6 (pp. 74-78) and published in the Journal of Economic Literature in 1996, is discouraging. The best economists (that is, those publishing in the AER in the 1980s) misuse statistical significance to a large degree.

Next, Ziliak and McCloskey do something odd. Faced with arguments from colleagues that best practice had improved since the 1980s, perhaps partly because of their 1996 article, they go ahead and do another study of the practice in the AER in the 1990s:

We are very willing to believe that since the 1980s our colleagues have stopped making an elementary error and especially that we have changed their minds. But being readers of typical economics articles since that first decace of personal computers [the 1980s] we seriously doubted that fishing for significance had much abated. […] And so in a second article, published in 2004, we [reapplied our study from 1996] to all the full-length empirical articles of the next decade of the AER, the 1990s [p. 79].

There’s a logical flaw here. If Ziliak and McCloskey caused the change with their 1996 article, it won’t show up in a study of the 1990s! And probably not of the first decade of the new millenium either; changes take a while, often a generation or so (just ask Thomas Kuhn, rephrased by Paul A. Samuelson in 1999: ‘Science advances funeral by funeral’). And indeed, Ziliak and McCloskey(or, McCloskey and Ziliak, as the reference will show) doesn’t find much of an improvement in the 1990s compared to the 1980s.

Of course, Ziliak and McCloskey have ideas on why scientific practice in the AER has not changed:

Significance unfortunately is a useful means toward personal ends in the advance of science – status and widely distributed publications, a big laboratory, a staff or research assistants, a reduction in teaching load, a better salary, the finer wines of Bordeaux. Precision, knowledge, and control. In a narrow and cynical sense statisitcal significance is the way to achieve these. Design experiment. Then calculate statistical significance. Publish articles showing “significant” results. Enjoy promotion.

But it is not science, and it will not last [p. 32].

They may be right. But, one cannot forget the position of the AER among economists. Because of it’s career generating potential, economists are likely to mimic it, both methodologically and rhetorically.

Still, Ziliak and McCloskey’s empirical evidence of the poor econometric practice in the AER is striking and convincing. I am now skeptic towards empirical economists. And not only economists: Ziliak and McCloskey summarizes evidence of bad practice in psychology, medicine, ecology and several other fields. One gets the impression that only the ‘hard’ sciences got it right (and I who used to think that medicine was a hard science).

In Ziliak and McCloskey’s view, one man is responsible for most of the statistical mess in economics and other fields: R. A. Fisher, whose Statistical Methods for Research Workers (1925), which went through no less than 14 editions, laid down the foundations for much of the later, statistical practice in many applied statistical fields. Ziliak and McCloskey attacks Fisher hard, in an almost distasteful way (they even accuse him of ‘outright, scientific fraud’ in an endnote somewhere, I’m sure, but I wasn’t able to find back to it), and large parts of the second half of the book they devote to attack Fisher in various ways and contrast him to (their hero, it appears) William Sealy Gosset, better know as ‘Student’ (look up Student-t). Maybe even more absurd is the blame put on the philosophical trends of the time:

One reason for the success of the Fisherian program against more logical alternatives […] is that the Fisherian program emerged just as neopositivismand then falsificationism emerged in the philosophy of science. It would have fallen flat in philosophically more subtle times, such as those of Mill’s System of Logic Ratiocinative and Inductive (1843) or Imre Lakatos’s Proofs and Refutations (1976). No serious philosopher nowadays is a positivist, no seroius philosopher of science a simple falsificationist. But the philosophical atmosphere of 1922-62 was perfect for the fastening of Fisher’s grip on the [statistically confused] sciences [p. 149].

Toward the end, after the very interesting chapter 23 (pp. 238-244), Ziliak and McCloskey get almost out of hand. On pages 249-250, they propose a “Statement on the proprieties of Substantive Significance” which they want editors, administrators and scientists to sign. The language of their ‘statement’ is, however, to involved and won’t hold as a standalone statement. I don’t understand the purpose of the statement when it isn’t self-containted, and otherwise just repeats a message that has been pounded upon throughout the book. And they keep pounding it, increasing their volume:

The textbooks are wrong. The teaching is wrong. The seminar you just attended is wrong. The most prestigious journal in your scientific field is wrong [p. 250].

Got it.

I had high expectations for this book, particularly because McCloskey’s name was on the cover. I was sligthly disappointed, however. By close examination, I discovered that McCloskey’s name is put last and that they’ve ignored the alphabetical order of names: Ziliak is the main author. It is obvious in some places, Ziliak don’t have McCloskey’s Economical Writing under his skin (neither do I, of course, but I would expect McCloskey to). This is a minor issue, certainly, but I looked forward to some persuasive, well-written, and witful prose of the kind McCloskey promotes in her Economical Writing; most of the time, it didn’t happen.

UPDATE: I have a hundred things to say about this book, but I cannot say them all at once. My above review is bad, I know, and I apologize. I confused the important things I wanted to say with the unimportant ones. I may get back to the book in later posts, but for now, I suggest the reader rather reads my discussion of the debate between Spanos and Ziliak & McCloskey. It presumable gives a better idea of what’s important in the book and yields more enjoyable and interesting reading.

And, for the record, let me again point out that I agree with Ziliak & McCloskey on their main point in The Cult, as stated in their reply to Spanos:

Statistical significance is neither necessary nor sufficient for substantive scientific significance.

Replication in Economics

February 23, 2009

Despite the voodoo noise, a different and lot more interesting debate about replication in economics has started on Environmental Economics. The debate on Env-Econ is a response to a post on Market Movers, which opens like this:

Falsifiability and replicability are key cornerstones of any academic research. If you’re running an empirical study, and your results aren’t replicable, your study is largely worthless.

First of all, the claim that falsifiability is a key cornerstone of academic research is simply not true, or at least not agreed upon. I understand falsifiability in the Popperian sense:

For Popper, a theory is scientific only if it is refutable by a conceivable event. Every genuine test of a scientific theory, then, is logically an attempt to refute or to falsify it, and one genuine counter-instance falsifies the whole theory.

(Read more about Popper and his ideas at the Stanford Encyclopedia of Philosophy.) Popper’s philosophy of science has been subject to extended debates. Thomas Kuhn, for example, claims that science consist of problem solving within a paradigm, and that paradigms change through scientific revolutions. In particular, Kuhn did not think that a statement had to be falsifiable to be scientific, as Popper did.

I am more willing to agree on the importance of replication. In the hard sciences like physics and chemistry, replication was truly important and researchers (ideally) remained sceptic towards new results until they had been replicated independently from the initial study. Since most research in physics and chemistry are conducted in laboratories, controlled experiments could fairly easily be replicated and results compared.

In economics, however, only recently have researchers started to conduct controlled experiments in laboratories. Tim Haab writes (On replicability in economics and the validation of models)

With economics models, at least until recently, we don’t have labs.  We are working with real observations from highly complex systems.  As such, we are forced to rely on modeling by assumption and measurement through statistical force rather than isolated direct observation.  This makes external validation of economic models extremely difficult.

This is changing over the past decade or so with the advent of experimental economics research in which researchers are testing fundamental economic results in a controlled setting.  Unfortunately, these experiments often suffer from over control which makes generalizability and practical applicability of laboratory results difficult at best. […]

To me, lack of replicability is not a condemnation of economics as a field, rather a challenge to the field to continue the unending pursuit of defensibility.

As Tim also points out, the post on Market Movers is primarily concerned with duplication of results (instead of independent replication), which requires the same data set and statistical tools. Such research is quite rare in economics, and it has different reasons (see Tim’s post). What is more common in economics, however, is that researchers collect their own data and see if they can find the same kind of conclusions as those available in the literature. John Whitehead puts it like this:

Why bother with replication with someone else’s data when you can publish your own study with your own data? The only time you ask for someone else’s data is if you really think they’ve made a horrible mistake or if you have an ax to grind.

However, I do agree with Market Movers that in general, replication and duplication is more important than what it seems to be in academia today.

Hat-tip: Env-Econ

What is Science?

January 15, 2009

Despite my earlier efforts, the strange usage of ‘science’ in the English language still obstructs the discussion over on Climate Progress. John McCormick writes

Here is a definition of the word ‘science’

“1. the systematic observation of natural events and conditions in order to discover facts about them and to formulate laws and principles based on these facts. 2. the organized body of knowledge that is derived from such observations and that can be verified or tested by further investigation. 3. any specific branch of this general body of knowledge, such as biology, physics, geology, or astronomy.”

Academic Press Dictionary of Science & Technology […]

Economics does nto [sic] fit the definition of science, in my opinion. So, scientific norms do not apply [when it comes to economics.]

McCloskey tracks the current use of ‘science’ back to 1867 (p. 20 in ‘The Rhetoric of Econmics,’ 2nd ed.). Earlier ‘science’ meant ‘studies,’  in line with its counterpart in other Indo-European languages. The weird thing is that today,  its counterpart in most languages hasn’t really changed meaning; it means ‘systematic inquiry’ and is not explicitly chained to ‘natural events.’ It is thus used to describe, e.g., philosphy and studies of poetry and language, as well as physics and chemistry. It is thus absurd that economics is a science in other languages, but not in English. What are we supposed to make of this? I let McCloskey explain (p. 21).

The point is that the foreigners have gotten it right. […] “Economics is a science” should not be the fighting words they are in English. The fighting lacks point because, as our friends across the water could have told us, nothing important depends on its outcome. Economics in particular is merely a disciplined inquiry into the market for rice or the scarcity of love. Economics is a collection of literary forms, some of them expressed in mathematics, not a Science. Indeed, science is a collection of literary forms, not a Science. And literary forms are scientific. […] The idea that science is a way of talking, not a separate realm of Truth, has become common among students of science since Thomas Kuhn.[*]

So, what’s important is that economics is scientific. Economics might not be a science in the U.S., but it is certainly scientific and scientific norms do apply.

* Thomas Kuhn (1922 – 1996) was maybe the most influental philosopher of science in the twentieth century. Anyone slightly interested in science, philosophy or generally should read his book ‘The Structure of Scientific Revolutions.’