The punch first culture

By Razib Khan | July 26, 2012 10:07 am

Dr. Daniel MacArthur has a nice write-up in Nature, Methods: Face up to false positives:

Few principles are more depressingly familiar to the veteran scientist: the more surprising a result seems to be, the less likely it is to be true. We cannot know whether, or why, this principle was overlooked in any specific study. However, more generally, in a world in which unexpected results can lead to high-impact publication, acclaim and headlines in The New York Times, it is easy to understand how there might be an overwhelming temptation to move from discovery to manuscript submission without performing the necessary data checks.

This is not just an issue in genomics. I’ve discussed it before as being a major problem in psychology. Though the infamous centenarian study will do nothing for the careers of the scientists involved, I do wonder what the effects of publishing large numbers of false positive results in science are on an individuals’ career when it isn’t so inexpertly executed (i.e., in this particular case the technical errors were so glaring that the authors should never have submitted their findings). I wonder because apparently major newspapers are now running with stories which they know are highly likely to be exaggerations or misrepresentation to induce pageviews, and then subsequently ‘correcting’ them. More specifically, the number of corrections has been rising rapidly.

CATEGORIZED UNDER: Science
MORE ABOUT: Science
  • Chad

    Its a systematic failure at multiple levels.

    We can start from the bottom and work our way up.

    Grad Student or Post Doc, given a project where they are insufficiently trained or severely lack an understanding of a topic and not given the time and encouragement necessary to develop the right skills. I have engaged in a lot of discussions over at Seqanswers.com, a forum on next generation sequencing. It is absolutely amazing how many come looking for answers, data already in hand, but absolutely no idea how to even start analyzing the data. Even more disturbing is when they come with data, absolutely no replicates, asking how to analzye data using various programs without replicates.

    Professor, doesn’t understand or know the technical details of a project. Gives a project to a student or post doc unfamiliar with the technical details, but expects them to get results quickly without the necessary time to learn. When given results, does not have the knowledge to double check it, it slips by.

    Professor and entire lab, feels pressured to publish and publish quickly in top journals. Needs the paper out now, publishes before it is ready.

    Reviewers, probably don’t understand the technical aspects themselves. Busy. Skims paper, doesn’t have time or ability to seriously critique or double check results.

    Journals, wants to publish the next headline, sensationalizes false results that have slipped through the system.

  • Justin Loe

    There was a report in some article that about 30-40% of the papers in Nature and Science are never replicated. It is sobering to consider that either those results are false or scientists have no incentive to verify and replicate other researcher’s results.
    I found this reference, which isn’t the article I referred to above, but it makes a similar point: http://www.nature.com/nature/journal/v483/n7391/full/483531a.html

  • Chad

    #2

    Its almost guaranteed that it gets far worse for non-Nature/Science/Cell papers that are lower impact and where there is less visibility and interest. Unfortunately, except where there is high interest (human evolution/cancer) or strong economic incentive (cancer/biomedical, etc) there is no benefit to most researchers replicating another’s results unless they were unaware of them or because they have reason to suspect that those results are incorrect. Due to the nature of getting science published, it is almost impossible to publish results that simply replicate what is already published. Even if you can, it will be in a journal of such low impact so as to be of not worth your time.

    There is good reason to think a vast amount of the scientific literature is simply incorrect, typically due to poorly designed and controlled experiments. There is a strong aversion to statistics in regular biology. I sometimes listen to This Week In Virology, a podcast done by Virologists on Virology. Several times I have heard them openly state that if you have to do statistics or anything other than a t-test that you are doing it wrong. That has to be one of the most harmful attitudes in Biology, one that has led to countless incorrect results.

  • DK

    Grad Student or Post Doc, given a project where they are insufficiently trained or severely lack an understanding of a topic and not given the time and encouragement necessary to develop the right skills.

    It’s a feature, not a bug. Most common exploitation: a clueless one does 10 difficult experiments, obtains 10 different results. Then comes the boss and cherry picks the one that “confirms the model”; writes the paper.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com

ADVERTISEMENT

See More

ADVERTISEMENT

RSS Razib’s Pinboard

Edifying books

Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »