Fixing Science’s Chinese Wall

By Neuroskeptic | December 21, 2013 5:59 am

In science we very often want certain things to be true.

Sometimes, this desire comes from noble reasons: we want a vaccine to work, because that would save many lives. Other times our motivations are less altruistic: we want our vaccine to work because then we will get lots of citations, a promotion, and a raise.

fixing_science

Scientists wanting things to be true is a fact, and a problem. Which is why science has evolved methods to prevent our desires from distorting the findings. For instance, we use double blind trials to ensure that the researcher’s preferences and expectations can’t influence the outcome measurements. We use statistical inference with an agreed standard (p = 0.05) to determine the significance of findings, rather than leaving it up to individuals to pick what they want to consider signal and what noise.

These systems are designed to protect observations from the whims of the observers. As Feynman famously said, “Science is a way of trying not to fool yourself… and you are the easiest person to fool.” It is rather like how in business, banks and other firms are required to have a ‘Chinese Wall‘ – a cordon around certain information to ensure that it can’t reach certain people. This is intended to prevent conflicts of interest influencing those branches of the company whose job is to provide impartial advice to the public or other departments.

However, there is a gap in science’s Wall between data and desire – the publication process. Whether the results of a particular experiment get published in a peer-reviewed journal depends upon at least three people:

- the researcher(s) themselves, who decides which results to submit, in what form
- the journal editor(s), who decides whether submissions are ‘interesting’
- the peer reviewer(s), who advises the editor on whether submissions are ‘good’

All of these people are scientists, with their own agenda and desires. They get to decide the fate of scientific results – knowing what those results are. So the desires of researchers get to influence what’s published, and the result is publication bias. This influence would be considered unscientific if it were allowed to happen within the conduct of a given experiment – but once the results are in, we allow it.

There is an alternative. Journals ought to peer review and accept articles before the work has been begun (preregistration with pre-peer review.) This ensures that science is published or not on the basis of the strength and originality of the methods, independent of what the results happen to be.

This would do for the later stages of the scientific process what double-blinding and randomization have done for the earlier, experimental stages. The Chinese Wall would be complete.

  • Chris Chambers

    Thanks for linking to the Cortex initiative. As we close out the year, one of the most encouraging developments from my point of view has been the uptake of Registered Reports (RRs) by several other journals. So far, four outlets in addition to Cortex have taken on RRs, including:

    - Attention, Perception & Psychophysics (launched): http://download.springer.com/static/pdf/98/art%253A10.3758%252Fs13414-013-0502-5.pdf?auth66=1387805581_f12bb96639423ff7e180f7e25e9eb126&ext=.pdf
    - AIMS Neuroscience (launched): http://www.aimspress.com/home.jsp?journalID=39
    - Experimental Psychology: officially approved, launch pending
    - Human Movement Science: officially approved, launch pending

    In addition, the following journals are currently considering RRs, and a couple of these have already approved the format unofficially (with announcements pending).

    - Addiction
    - BMC Psychology
    - Cognition and Emotion
    - Drug and Alcohol Dependence
    - eLife
    - Experimental Brain Research
    - International Journal of Radiation Oncology • Biology • Physics
    - NeuroImage
    - Neuropsychologia
    - Peace and Conflict: The Journal of Peace Psychology
    - PLOS ONE

    2014 should be an exciting year as we watch this new format further spread, and we also start to see the first published studies coming through.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      That’s great news!

      eLife really ought to go for it – if they want to claim the moral high ground over Science and Nature, this would be the best way to do it.

    • Speldosa

      Oh yes! This is great!

  • Pingback: From the Neuroskeptic blog | The Fibromyalgia Perplex

  • Pingback: @Neuroskeptic: Fixing Science’s Chinese Wall : Martin Larsson

  • Matthew Slyfield

    “We use statistical inference with an agreed standard (p = 0.05)
    to determine the significance of findings, rather than leaving it up to
    individuals to pick what they want to consider signal and what noise.”

    P values tell you absolutely nothing and your agreed to standard can be generated from any data set.

    http://wmbriggs.com/blog/?s=p+value&x=0&y=0

  • Kehvan

    Now think about how certain scientist, first claimed global warming, but now only assert climate change.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »