At Bookish, a used book store in Berkeley, I picked up Stephen Jay Gould’s essay collection An Urchin in the Storm. In an essay on the fundamental problems with human sociobiology, a discipline that tries to explain social behavior with Darwinian selection, Gould has an interesting comment on the problem with biased reporting in science.
Few observers outside science (and not nearly enough researchers inside) recognize the severe effects of biased reporting. The problem is particularly acute, almost perverse, when scientists construct experiments to test for an expected effect. Confirmations are joyfully reported; negative results are usually begrudgingly admitted. But null results-the failure to find any effect in any direction-are usually viewed as an experiment gone awry. Meticulous scientists may report such results, but they disappear forthwith from the secondary literature (and are almost never reported in the press). Most scientists probably don’t publish such results at all-who has the time to write up ambiguous and unexciting data? And besides, they rationalize, maybe next week we’ll have time to do the experiment again and get better results. I call such nonreporting perverse because we cannot gauge its depth and extent. Therefore, we do not know the proper relative frequency of most effects-a monumental problem in sciences of natural history, where nearly all theoretical claims are arguments about relative frequencies, not statements about exclusivity [p. 37, 1988 paperback edition].
Incidentally, I just work through a referee report that asked for more motivation for a model complication which did not affect the main results much. I cannot help but suspect that if the complication had lead to larger effects, the referee would be less inclined to ask for further motivation. The irony: The referee agrees that the complication is novel and leads to a more realistic model, however slightly.
In introducing sociobiology, Gould mentions Kuhn. I sense a sarcastic tone in the final remark.
Thomas Kuhn’s seminal work, The Structure of Scientific Revolutions, affected working scientists as deeply as it moved those scholars who scrutinize what we do. Before Kuhn, most scientists followed the place-a-stone-in-the-bright-temple-of-knowledge tradition, and would have told you that they hoped, above all, to lay many of the bricks, perhaps even set the keystone, of truth’s temple-the additive or meliorist [the doctrine that the world tends to become better or may be made better by human effort, definition from dictionary.com] model of scientific progress. Now most scientists of vision hope to foment revolution. […] We are therefore awash in revolutions, most self-proclaimed [p. 27].