Fixing Science, Not Just Psychology

By Neuroskeptic | May 27, 2013 8:56 am

Neuroskeptic readers will know that there’s been a lot of concern lately over unreproducible results and false positives in psychology and neuroscience.

In response to these worries, there have been growing calls for reform of the way psychology is researched and published. We’ve seen several initiatives promoting replication and, to my mind even more importantly, registration of studies to prevent bad scientific practice in future.

But the problem is not limited to psychology. Concern is growing too in cancer biology, as revealed in a new study from the MD Anderson Cancer Center in Texas: A Survey on Data Reproducibility in Cancer Research.

The researchers polled all of the nearly 3000 staff at the center. Unfortunately, just 15% responded, but of those that did, 55% reported having been unable to reproduce a published result, but only 33% of those published it.

This joins other reports into the poor reproducibility of preclinical cancer research. A 2011 article surveyed pharmaceutical industry scientists who’d attempted to reproduce published findings (with a view to making drugs out of them):

In almost two-thirds of the projects, there were inconsistencies between published data and in-house data that either considerably prolonged the duration of the target validation process or, in most cases, resulted in termination of the projects…

I’d be amazed if that problem is limited to cancer drug discovery, either.

The problem is formal and systemic. In a nutshell, false-positive results will be a problem in any field where results are either ‘positive’ or ‘negative’, and scientists are rewarded more for publishing positive ones. Unless you are much more careful than we are today.

Poor reproducibility has little to do with the subject of the science. Whether you study study minds, mice, or molecules, if you measure them and analyze them statistically, you are all in the same boat. It’s the structure of the incentives.

So we shouldn’t single anyone out for blame.

However, this doesn’t mean that those at whom fingers are pointed should get defensive. That’s a natural, but regrettable, response.

Yes, it’s rather unfair that at the moment, people seem to be demanding more of, say, social psychologists than of cognitive psychologists. We ought to demand the highest standards of both of them.

But raising the standard has to start somewhere, so why not with social psychology? “We’re no worse than anyone else” is a comforting mantra, but a poor defence. And any field or speciality that puts its house in order will soon be in a position to say “We’re better than you”.

ResearchBlogging.orgMobley, A., Linder, S., Braeuer, R., Ellis, L., & Zwelling, L. (2013). A Survey on Data Reproducibility in Cancer Research Provides Insights into Our Limited Ability to Translate Findings from the Laboratory to the Clinic PLoS ONE, 8 (5) DOI: 10.1371/journal.pone.0063221

  • JonFrum

    The point of publishing your work has always been to announce new knowledge of the way the world works. Negative results simply rule out claimed knowledge – they don’t tell us the right answer. For that reason, I’d be hesitant to publish papers that simply give negative results – even if it’s in answer to a positive claim.

    Still, the point is well taken. Perhaps a registry of follow-up replication could be published. Simply a place for labs to publish the fact that they’ve made a full effort to replicate results and failed. This isn’t a science publication- it’s a technical, methods-based matter. This could be done online, with a page for each publication. That way, anyone interested in a paper could follow up on it and see if anyone else has had problems reproducing results.

    Of course, this, too could be abused, with rivals claiming to be unable to replicate each other’s results. Then again, if you do claim that you can’t replicate results, and someone else does, you look like an incompetent.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      But whether a result is “positive” or “negative” is subjective.

      If I measure brain activation in response to X as Y, that is an observation, and a fact.

      I might have reasons to hope I would see Y and then call it “positive”, or again it might be unexpected or uninteresting to me, and I’d call it “negative”.

      But those are just my subjective feelings towards the result (even if other people share them.) A result should not be published just because I (or many people) don’t like it.

  • paulbenedict

    The problem exists in the failure of our current philosphies of science. Currently we are in a hypothesis + experimentation = proof mode of scientific theory. This is far too broad. The KIND of hypothesis matter: http://www.nolanchart.com/article10257-global-warming-was-never-scientific.html

    • http://tw.gs/P4t204 Marcia Kane

      ᴊᴜsᴛ ʙᴇғᴏʀᴇ I sᴀᴡ ᴛʜᴇ ᴄʜᴇᴄᴋ ғᴏʀ $6725, I ᴅɪᴅɴᴛ ʙᴇʟɪᴇᴠᴇ …ᴛʜᴀᴛ…ᴍʏ ʙʀᴏᴛʜᴇʀ ᴀᴄᴛᴜᴀʟʏ ᴇᴀʀɴɪɴɢ ᴍᴏɴᴇʏ ᴘᴀʀᴛ ᴛɪᴍᴇ ғʀᴏᴍ ᴛʜᴇʀᴇ ᴄᴏᴍᴘᴜᴛᴀʀ.. ᴛʜᴇʀᴇ ғʀɪᴇɴᴅs ᴄᴏᴜsɪɴ ʜᴀᴢ ᴅᴏɴᴇ ᴛʜɪs ғᴏʀ ᴏɴʟʏ 21 ᴍᴏɴᴛʜs ᴀɴᴅ ᴀᴛ ᴘʀᴇsᴇɴᴛ ᴛᴏᴏᴋ ᴄᴀʀᴇ ᴏғ ᴛʜᴇ ᴅᴇᴘᴛ ᴏɴ ᴛʜᴇʀᴇ ᴄᴏᴛᴛᴀɢᴇ ᴀɴᴅ ʙᴏᴜɢʜᴛ ᴀ ɢᴏʀɢᴇᴏᴜs Fɪᴀᴛ Pᴀɴᴅᴀ. ᴡᴇ ʟᴏᴏᴋᴇᴅ ʜᴇʀᴇ Mojo50­.c­o­m

  • Jud

    Topical soap box BS from Discover.

    The problem is that we have huge amounts of ‘research’ being funded by corporations that have no interest to disclose unsavory data. Whether it is side effects, no effects, or weak effects, they just don’t have an incentive to let everyon see the bad studies. Psychology has gotten a huge amount of funding from Insurance companies that wanted a cheaper alternative to pharmaceuticals and so they have pushed bogus treatments. Pharmaceutical companies similiarly push drugs on evidence that has been carefully selected to market their products. This isn’t science. The problem is not “how do we reform science” its… when do we let the science actually begin.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Topical? I’ve been saying this for 4 years, long before it was topical. It’s my topic.

      As for what you say about drugs and therapies that’s all true and the solution is… mandatory clinical trial registration and reporting. Drug companies have no incentive to be honest so we must force them.

      • Jud

        I agree

    • Andrea Kuszewski

      “Psychology has gotten a huge amount of funding from Insurance companies that wanted a cheaper alternative to pharmaceuticals and so they have pushed bogus treatments.”

      Do you have a citation for that claim, Jud?

      • Jud

        Economic factors also shaped the creation of the DSM-III and later editions. The cost of treatment, for example, was an important variable in the creation of the DSM-III categories. While, early on, psychotherapy was generally an out-of-pocket expense, health insurance companies and the government began to cover the cost of treatment during the 1960s [22]
        . This development has provided positive influences on the field of psychology and increased access to
        mental health care. However, this institutional change also created an external and artificial pressure on theory development in psychology. As stated by Mayes and Horwitz, ‘‘The rise of third party payers contributed top ressures to change the dynamic model:
        the continua and symbolic mechanisms of dynamic psychiatry did not fit an insurance logic that would only pay for the treatment of discrete diseases (2005).’’ When patients were no longer paying for their own treatment, power was given to their insurance providers to dictate what types of treatment would be compensated, pushing psychology and psychiatry towards conceptualizations of mental illness that more clearly fit theneeds of the insurance industry. Such
        events created a context in which it has become preferable to understand mentalillnessinthe sameway asphysicalillness–that is, with clear-cut categories and cures – without fully weighing the evidence for and against this position.”

        “The influence of corporate and political interests on models of illness in the evolution of the DSM”

        http://www.sciencedirect.com/science/article/pii/S0924933811000083

        As for specific funding… i know the Rand corporation funded a massive insurance study that has basically shaped these opinions over the last three decades. The study was conducted in the 70′s and 80′s and is still a huge focus in the industry.

  • http://www.buberel.org/ Jason Buberel

    If I had a billion dollars…

    I would create the “National Center for Result Reproduction in Science” and staff it with world-class laboratory technicians. We’d choose the highest profile or most dubious published findings, and attempt full reproductions.

    (adds item to his to-do list)

    • http://bekolay.org/ Trevor Bekolay

      It’s already happening! Minus the billion dollars. https://www.scienceexchange.com/reproducibility

      • http://www.buberel.org/ Jason Buberel

        Very cool – thanks for the URL. Now, about that billion dollars…hmm…

  • Pingback: Fixing Science, Not Just Psychology – Discover Magazine (blog) | newzbuff.com

  • Pingback: Fixing Science, Not Just Psychology - Discover ...

  • Pingback: Fixing Science, Not Just Psychology : Neuroskeptic | Social Fobi - Det Du Behöver Veta

  • Pingback: Fixing Science, Not Just Psychology : Neuroskeptic | Technology n Science News and Latest Updates

  • Pingback: Fixing Science, Not Just Psychology : Neuroskep...

  • Pingback: Monday Miscellany » Ashley Miller

  • Pingback: Links 6/3/13 | Mike the Mad Biologist

  • Pingback: Best of replication & data sharing Collection 2 | Political Science Replication

  • Pingback: "Is Psychology Science?" Is The Wrong Question - Neuroskeptic | DiscoverMagazine.com

  • Pingback: "Is Psychology Science?" Is The Wrong Question | Science

  • Pingback: Hormones and Women Voters: A Very Modern Scientific Controversy - Neuroskeptic | DiscoverMagazine.com

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »