Unilaterally Raising the Scientific Standard

By Neuroskeptic | February 3, 2013 10:51 am

For years, I and others have been arguing that the current system of publishing science is broken. Publishing and peer-reviewing work only after the study’s been conducted and the data analysed allows bad practices – such as selective publication of desirable findings, and running multiple statistical tests to find positive results – to run rampant.

So I was extremely interested when I received an email from Jona Sassenhagen, of the University of Marburg, with subject line: Unilaterally raising the standard.

Sassenhagen explained that he’s chose to pre-register a neuroscience study on a public database, the German Clinical Trials Register (DRKS).

His project, Alignment of Late Positive ERP Components to Linguistic Deviations (“P600″), is designed to use EEG to test whether the brain generates a distinct electrical response – the P600 - in response to seeing grammatical errors. The background here is that the P600 certainly exists, but people disagree on whether it’s specific to language; Sassenhagen hopes to find out.

By publicly announcing the methods he’ll use before collecting any data, Sassenhagen has, in my view, taken a brave and important step towards a better kind of science.


Already, most journals require trials of medical treatments to be publicly pre-registered, and the DRKS is one such registry. This study, however, is ‘pure’ neuroscience with nothing clinical about it, so it doesn’t need to be registered – Sassenhagen just did it voluntarily.

Further, I should point out that he offered to pre-register his data analysis pipeline too by sending it to me. Unfortunately, I didn’t reply to the email in time… but that was purely my fault.

I very much hope and expect that others will follow in his footsteps. Unilaterally adopting preregistration is one of the ways that I’ve argued reform could get started. As I said:

This would, at least at first, place these adopters at an objective disadvantage. However, by voluntarily accepting such a disadvantage, it might be hoped that such actors would gain acclaim as more trustworthy than non-adopters.

Pre-registration puts you at a disadvantage – insofar as it limits your ability to use bad practice to fish for positive results. It means you can’t cheat, essentially, which is a handicap if everyone else can.

I don’t know if this is the first time anyone’s opted in to registering a pure neuroscience study, but it’s certainly the first case I know of it being done for an entirely new experiment.

There have, however, recently been many pre-registered attempts to replicate previously published results e.g. the Reproducibility of Psychological Science; the ‘Precognition’ Replications; and an upcoming special issue of Frontiers in Cognition.

Replications are good, registered ones doubly so – but they’re not enough to fix bad practice on their own. To do that we need to work on the source, original scientific research.

CATEGORIZED UNDER: EEG, FixingScience, science
  • http://www.blogger.com/profile/08010555869208208621 The Neurocritic

    Isn't there a massive study pre-registration database already in place (or one that provides infrastructure)? That would be research grants. Some funding agencies make PIs more accountable than others for carrying out their experiments as proposed.

    There could be JIT (just-in-time) submission of analysis pipeline if your grant is funded. NIH and others moved away from requiring methodological and analytic details in order to determine funding. No more nitpicking about stimulus duration and ISI.

    But if your grant is funded, your detailed methods go into the NIH RePORTER database (or ones set up by NSF, others), as do study results. Yearly (or quarterly) progress reports are required, which include details like statistics. These are posted online and made publicly available.

  • http://petrossa.me/ petrossa.me

    Works fine for middle of the road studies, i'd hate however to prepublish my groundbreaking results which will lead to a cure for normalcy.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    The Neurocritic: Yes, grant awarders a potentially valuable source and already provide much information.

    However, different bodies have different requirements and I think we need a better way of linking them to the relevant publications.

    If all major funders came together and said, you must publish your protocol online, and here's a central repository for them… that would go a long way.

  • Jona Sassenhagen

    On a minor note, at least this study is quite cheap to run. It's basically a class project; my subjects are attending for course credit, I'm using idle lab time, and EEG studies aren't too expensive anyway.

  • Anonymous

    I applaud Jona's idea. The damage done by current research practices is immense. Of the various estimates of the percentage of false research findings, I tend to believe the more pessimistic ones. Scientists really have no choice but to find some positive result if they want to get a publication/degree/job/grant/tenure. The system is completely f***ed up. For the life of me, I don't understand why NIH doesn't take some serious action to establish pre-registration for basic science.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Anonymous: Quite right. I'm optimistic that change will happen in the next 10-15 years. The success (and the sheer pace) of change in e.g. Open Access and the emerging replication/replicability movement is very encouraging.

  • Anonymous

    Is getting class credit for participating in a study IRB approved? Seems like a case of undue influence.

  • http://twitter.com/jdottan JT

    I'm hoping I haven't left this comment before (trying to minimize redundancy), but how do longitudinal studies fit into pre-registration, and even more generally, how do they fit into the conversation about reforming our science? My thinking is that the grant proposals to fund these studies serve as a pretty decent roadmap for what the researchers intend to do, but I could see how some might regard that as insufficient.

  • Y.

    Grants cannot serve as a registry because the level of specificity in most grants is currently not enough. Also, not every study is directly funded by a specific grant. For example, a behavioral study involving 20 subjects can be run essentially for free.

  • http://twitter.com/jdottan JT

    Forgot to mention, the Open Science Framework (http://openscienceframework.org/) also makes it relatively simple to pre-register studies for those who are looking to do that.

  • DS

    Neuroskeptic

    Have you given any thought about the menacing “pilot study”. I see these little beauties as unsung contributors to the positive findings filter.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    DS: They're an issue, yes. My view is that any study which is intended to be published, should be pre-registered. You should be able to run all the pilots you like but you have to decide in advance whether it's a private pilot or a publishable study.

  • Jona Sassenhagen

    I think it would be sufficient to use pre-registration (and I agree this should include the whole analysis pipeline, going beyond what I've done here) as the watershed between first-tier confirmatory science, and second-tier explorations.
    And we'd simply have to emphasize that science is science, and exploration is exploration.

    Speaking from personal experience, writing up your methods in advance, blind, without having the data in question in front of you, is hard, and running the experiment twice, once before and once after pre-registration, is more costly than some researchers might be able to afford money- or time-wise. And that shouldn't be prohibitive for publication. It should just be prohibitive for publication as real science.

  • T

    Dr Sassenhagen,
    is awarding students course credits for participating in your experiment ethical, or is it perhaps undue influence?
    And how come the experiment did not have go through ethics committee?
    This would not be possible where I work…

  • Anonymous

    This a great idea. There needs to be an attitudinal shift to the point where not pre-registering makes you presumptively suspicious, where doing so is in and of itself a shameful practice.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    T: Giving course credit is a very common practice.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Jona: I agree. If you do a study and want to go beyond your registered methods, there's nothing wrong with that (and many discoveries have come that way!), but it should be clear to everyone that that's what you were doing.

  • Jona Sassenhagen

    I've possibly translated the term wrongly. Basically, students taking methods classes are required to also, in some way, log some lab hours, which could include constructing or conducting experiments, and/or contributing participant hours. As far as I know, this is fairly standard procedure (“everybody does it”); if you google “course credit eeg experiment”, you will 1. find many experiments reporting that the subjects participated for course credit, 2. see that many courses require their students to get some lab time (up to 10s and 100s of lab hours).
    Here is a typical example: http://psychology.ucdavis.edu/labs/ranganath/articles%5CgotlibcE98.pdf
    p. 435

    In this case, I basically asked the students in my methods class if they wanted to see an EEG experiment from the participant side and they thought it would be a good idea, so I came up with something simple for them to experience. And since I had been talking about the difference between exploratory and confirmatory science anyways, I decided to pre-register it, too.
    Ethics approval is another topic; you will find that standards vary wildly here. I must admit I do not have much to say here.

    Also, I haven't graduated yet, no Dr.! I won't be an MD doctor, either. My degree will most likely be in linguistics.
    Are you in medicine?

  • Jona Sassenhagen

    I think we need a paradigmatic shift where it is understood that if your results couldn't be reproduced by recording an additional 15 subjects and handing an undergrad student the MATLAB scripts you used to get the results the first time, you better have a good excuse at hand.

    But being a poor undergrad student who cannot spare the money nor time, or investigating a fleeting phenomenon (rare fatal brain lesion?), is a good excuse, and shouldn't be prohibitive for getting (your exploratory investigation) published.
    It's only when you're making grand claims that you need to be able to back them up by e.g. following up your exploratory investigation with a pre-registration and a (trivial) self-replication.

  • DS

    Neuroskeptic

    I think the problem with pilot studies goes beyond issues solvable by preregistration. They are small studies which by themselves are statistically under-powered but potentially can have the unfortunate consequence of directing research along paths by which systematic error leads to positive results.

  • Jona

    Isn't the problem restricted to when what amounts to a pilot study is sold as being a confirmatory, solid result?

    (Thorough, whole-pipeline) pre-registration allows us to tell the confirmatory from the exploratory science.

  • T

    Thank you for your response. I may be oversensitive to the ethics part, I am not even allowed to have lab members participate in my experiments.

  • http://www.blogger.com/profile/10671627506175498768 Sergei

    I see one problem with this approach. Many articles are the result of secondary inquiries after there is a lot of data to experiment with.
    Maybe in neuroscience you can narrow data gathering, but in social psychology you can go for a large-scale sample and whole test battery. You can pre-register your intention (hypothesis) and method (including result analysis), however many go on gathering some extra info after the main experiment phase (e.g. additional tests). It often happens that the most interesting data comes not from your initial research design but from this unplanned total. Then you write an article that was not even planned in the first place… :)

  • Jona Sassenhagen

    You're actually highlighting a benefit, not a drawback.

    1. if your effect derived in exploratory analysis is truly substantial, it will usually be trivial to reproduce. If your effect depends on the sample at hand, it probably isn't true.
    2. you may still do and report all the exploratory stuff you want, but now, you have to be honest about it being exploratory.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »