Fixing science, in part

By Razib Khan | May 19, 2011 2:32 pm

The GiveWell Blog has some suggestions for “Suggestions for the Social Sciences”. Here is the big one:

Our single biggest concern when examining research is publication bias, broadly construed. We wonder both (a) how many studies are done, but never published because people don’t find the results interesting or in line with what they had hoped; (b) for a given paper, how many different interpretations of the data were assembled before picking the ones that make it into the final version.

The best antidote we can think of is pre-registration of studies along the lines, a service of the U.S. National Institutes of Health. On that site, medical researchers announce their questions, hypotheses, and plans for collecting and analyzing data, and these are published before the data is collected and analyzed. If the results come out differently from what the researchers hope for, there’s then no way to hide this from a motivated investigator.

As the example of the NIH illustrates this is not just a social science problem, it is rife in any science which utilizes statistics. Statistical methods have become metrics to attain by any means necessary, when in reality they should be guidelines to get a better grasp of reality. The only solution to the problem of conscious and unconscious bias in statistical sciences seems to me to be radical transparency of some sort. There’s a fair amount of science ethnography which suggests that how science is done departs greatly from the clean and rational enterprise which one might presume based on the final product. The only way to clean up some of the natural human bias in the enterprise is to shed some light on it.

CATEGORIZED UNDER: philosophy, Psychology, Social Science
  • Brent Michael Krupp

    In case you or your readers don’t already read xkcd, I’m reminded of a recent comic:

  • Joshua Zelinsky

    In one of Ben Goldacre’s books he cites a study about publication bias which shows that there’s a publication bias in studies about publication bias. Given reasonable assumptions about distributions, people aren’t publishing studies which would show only a small bias. I don’t k now if this is hilarious or just demonstrates how severe a problem publication bias is.

  • Razib Khan

    lol. comment of the week i think!

    is there a bias in the bias of the studying bias?

  • DK

    Statistical methods have become metrics to attain by any means necessary, when in reality they should be guidelines to get a better grasp of reality.

    So true. Humans seem to have a proclivity for numerology, no matter how educated they are. Even in protein crystallography, arguably the most precise of all biological sciences, there is a lot of bullshit associated with “meeting expected targets”.

  • Pingback: Friday Fluff – May 20th, 2011 | Gene Expression | Discover Magazine()

  • Clark

    This could be done if the NIH and NSF just started demanding it as a requirement for getting a grant.

  • Justin Loe

    “In his recent Inaugural Lecture entitled “Analysing the data you haven’t got,” Professor Sutton illustrated how he developed tools for identifying and quantifying bias in systematic reviews through work done on anti-depressants.”

    Of course, this may or may not hold up over time.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar