Interview: Raising the Scientific Standard

By Neuroskeptic | March 23, 2013 2:35 pm

Last month, I blogged about Jona Sassenhagen, the University of Marburg neuroscientist who decided to have a go at Unilaterally Raising the Scientific Standard.

To recap, Sassenhagen chose to pre-register a neuroscience study on a public database, the DRKS. The methods he planned to use, the major data analysis steps, and the predicted results are all here; they’ve been up there for over two months, i.e. before the data was collected.

This is unusual. Pre-registration is something I have long argued should be mandatory, as I think it would guard against the bad practices that distort science, but currently it’s obligatory only in the case of clinical trials, not ‘basic’ research.

The project, Alignment of Late Positive ERP Components to Linguistic Deviations (“P600″), was designed to test whether the brain generates a distinct electrical response – the P600 - specifically in response to grammatical errors.

Now, Jona reports that he’s completed the study, written it up, and submitted it for peer review (already!). He kindly agreed to answer some questions about the whole thing…

Overall, would you recommend pre-registration? Do you regret not doing it previously?

I’d recommend it for everything, even for purely exploratory analyses. I regret not having done it before. As much as possible, I’ll register everything I’ll be doing in the future.

I have two papers right now that are confirmatory, but I can’t submit them because the paper we made the predictions in is getting rejected. Had I pre-registered, that would not have been so problematic.

However, I also think it’s not enough for confirmatory studies to pre-register outcomes and the basic methods. For truly confirmatory studies, we need full pre-registration of the whole analysis pipeline, like Chris Chambers’ proposal at Cortex.

Did you find that pre-registering meant the study was more effort or less effort than the normal approach?

The pre-registration itself was some effort to begin with. But besides that, it did not change the amount of work actually done on the study itself – it just changed the order. It meant more work up front, but less work afterwards.

It was substantially more work before; I made sure my scripts and my manuscript were all written up when the first participant rolled in. But we submitted a week after the last participant had been recorded, 3 weeks after recording had started.

The study took less time to run than usual because since I knew I couldn’t cheat later, I decided on a really strong effect, and therefore, my power analysis told me I wouldn’t need too many subjects. Also, I was quite enthusiastic about the whole adventure… I was doing a lot of night shifts!

Did you consult with your colleagues before pre-registering? If so, what did they think about it?

I started doing it by myself, but when I ran into problems during the pre-registration process (some of the terminology didn’t make much sense to me), I would ask my co-workers, and they liked the idea, though they were not too enthusiastic. It’s still some ways to go before I can convince my lab to follow me down that road.

The general perspective was that it wasn’t a bad thing to do, but nobody really got the point at first. Later, my office mate got the point when I sent him your blog post.

My PI got the point afterwards, when we were finishing up the paper. At one point she asked, “This sounds a bit explorative to me, what if a reviewer doubts if this is what we truly went out to test?” And I reminded her that if a reviewer did decide to question us there, we could point to the pre-registration, where I had said that I’d test it. That’s when she warmed up to the idea.

But whatever her personal feelings, she gave me her full support here, running the study on short notice and keeping up with the rapid pace of finishing up the manuscript after data collection had finished!

Did you feel that the published protocol was ‘tying your hands’ in data collection or analysis?

Well, the pre-registration procedure I went through is still quite vague. It’s the standard clinical trial registration procedure, and it’s definitely a lot better than nothing. But it’s not too strict. I had to note down my main outcome and a very basic sketch of the methods. There was still considerable wiggle room…

With full pre-registration of methods, this would not have been possible this easily. Which means, I think, you will need to do more work up front, including running a few pilot test subjects and testing your full scripts. If you do a full pre-registration of methods, you will not be able to get by without running a few preliminary test subjects. I’m confident that it would be impossible to run anything but the most trivial confirmatory study without having some sample data available to prep up your scripts.

Also, we would most likely see a shift towards mostly investigating reliable, strong effects. Telling the world you expect to find X, and then having to report Y… that idea was quite stressful to me. On the other hand, you can’t argue that the pre-registration forced me to go with the mainstream. On the contrary. I’m decisively going against the consensus here (which associates the P600 with high-level combinatorial processing). This is in part why I decided to pre-register; so everybody would be able to tell I didn’t cheat to prove my (controversial) point.

Finally, Jona also asked me to forward a question to Neuroskeptic readers. He’s willing to make the raw data and his analysis scripts publicly available. Is anyone able to recommend a platform that hosts such things?

CATEGORIZED UNDER: EEG, FixingScience, methods, select, Top Posts
  • Pingback: Interview: Raising the Scientific Standard | WorldTopStories.com

  • knowyourbrain

    Who is the Neuroskeptic? Are you anonymous like Neurobonkers? How do we even know if you are different people? Have you seen my new blog?

    http://www.tritonia.me

  • tbekolay

    I’ve actually done this recently with a paper submission of mine, and I think it’s a very nice set of tools for making data and analysis scripts available.

    I put the code in a Github repository. I put the raw data in figshare.

    Having the code in a legitimate version control system means that people can look at my code, use it in their own analyses, and propose changes or point out issues like any other piece of software. Figshare is really nice because it’s citable, also has versioning, and offer permanent unlimited storage for data made publicly available. It also has a decent API; I use the API in my set of scripts so that downloaded the raw data is as easy as running a Python script. And, you know, having the bandwidth to download over a gig of data files.

  • http://twitter.com/davenuss79 Dave Nussbaum

    I would guess OSF would have that: http://openscienceframework.org/

  • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

    Okay, I’ve registered at figshare and will take a look at OSF. Thanks for those!

  • http://twitter.com/jdottan Joseph Tan

    This is really good stuff. I’m wondering how to handle longitudinal projects, as some have collected data decades before the current people running analyses started on the projects. How would pre-registration work there?

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      I think those studies kind of take care of themselves, because if they’ve been going on for that long, they’re a big deal and everyone in the field will already know about them.

      But there’s also a question of what to do about registering (or not) the analysis of a body of data which already exists. Ideally I would want to preregister everything, even if all you do is download a public dataset and start analyzing it… that would be rather difficult, however.

      • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

        Or simulation studies. It works out if your analyses require 5 weeks on a supercluster, but what if you do something small-scale? What if your data is Tweets, or Facebook likes?
        There’s many, many cases where pre-registration doesn’t trivially solve the problem. However, nobody’s claimed it’s enough to solve all problems. Just that it’s necessary to solve some.

      • http://twitter.com/jdottan Joseph Tan

        That makes sense to me, was just curious since it seems like those studies aren’t usually referenced in the overall “reform” discussion, but perhaps it is because they do take care of themselves.

  • Pingback: For Preregistration in Fundamental Research : Neuroskeptic

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »