Preregistration …Problem?

By Neuroskeptic | April 29, 2013 4:04 pm

A new blog called neurorant has launched a broadside at preregistration, my current hobbyhorse. (This will be the last post on the topic for a while, honest.)

Neurorant’s criticism is anonymous, unscholarly, snarky, and not published in a peer reviewed journal… but it also has some bad points:

Pre-registration of studies is meant to stop unconstrained post-hoc analyses that end up finding differences that are actually just noise in the data.

That’s one of the motivations; but equally important, it will facilitate the publication of negative findings. Indeed, there seems to be a common misconception that registration is essentially about prevention, a kind of scientific contraceptive intended to eliminate bad results from the gene pool.

Pre-registration will act to discourage exploratory science… In fMRI, for example, the really interesting developments almost all use existing data… i.e. working on defining and understanding functional brain networks using data that has not been acquired for the purpose (e.g., generic resting state data). There’s no way to pre-register for this kind of research.

Let’s consider the example of the 1000 Connectomes Project, (1000C) an existing dataset of the kind in question. Anyone can download this, and analyze it. Should those analyses be preregistered?

My answer is: they already are. Or rather, they’re not, and everyone knows it, so everything’s above board, which is exactly how exploratory analyses would be under a registration regime. We all know that any 1000C finding might be a picked cherry from a rich fruit basket.

The trouble with the current system is that planned and exploratory analyses are confused. All registration does, at its core, is make the distinction clear.

Incidentally, I’m strongly in favor of fMRI data sharing.

It will slow science down for two reasons: i) there will end up being several rounds of review of the pre-registration phase so these will have to be written carefully, slowly. (If there isn’t any review of pre-registration then the whole thing can’t work).

This is another common misconception.

Registration doesn’t need to be linked to any kind of review. In clinical trial registration, the model for the whole thing, there isn’t any. You just submit. Likewise, anyone could just register their non-clinical work today, unilaterally. Jona Sassenhagen has, along with various others e.g. the Bem replication experiments.

No-one formally reviewed those notices of registration. The judge is the reader of the eventual results, who can compare them to the registered protocol and decide if they think the final cherries were subject to picking, or whether (most likely negative) results are going unreported.

That said, I believe that uniting preregistration and peer review (as implemented e.g. like this) would be a great idea, and that it could actually save time overall. This, however, is registration plus reform of peer-reviewed publishing. Plain vanilla registration is a lot simpler.

ii) In fast moving fields, there will be analysis and theoretically developments throughout the planning/data acquisition phase that will mean that if you’re taking pre-registration seriously, you have to re-register and collect more data, not just repurpose what you have. This may also involve people collecting even more boring FMRI datasets that all show the same thing (this already happens by the way), which swallows precious research resources.

I think the concern here is that registration will mean that if people have some data (or are halfway through collecting it) and want to try a new, unregistered analysis on it, they’d have to register and run a wasteful new study, rather than just repurposing the existing data. If that’s the worry, then it’s largely groundless. The authors in this case could do it, and publish it, as an exploratory analysis… which after all is what it is.

It’s true that if registration were combined with peer review, our hypothetical researchers might have to convince the reviewers (and the journal) to let them bundle the new analysis into the same paper as the registered stuff. But if it doesn’t get allowed, they could just publish it elsewhere. Whether or not to allow a large “Secondary Results” section of a paper is a matter for the journal; it’s a publishing decision.

Neurorant’s third point is that like an AAA credit rating, preregistration could be exploited by fraudsters. Well, I’m sure that will happen – unfortunately, fraudsters exploit systems. It’s what they do. They’re doing it currently. I’m not aware of an increase in fraud in US clinical trials after they were all made preregistered by law, although that would be interesting to investigate.

Finally (and I detect a mischievous grin peering from between these lines)

I don’t think dodgy results in some papers have that big an effect, or not enough to lead to any drastic, restrictive countermeasures. In fact, it’s possible that having dodgy papers is good for science as a whole, maybe it’s like stochastic resonance in motor systems, a bit of noise may help the field.

Well, fair enough. Maybe it depends on the field of science you inhabit; some are worse-hit than others. But ultimately, I think this is a question of how you feel about science today:

Personally, I’m not smiling.

  • Pingback: Preregistration …Problem? | TopStoriesDaily.com

  • Y.

    No, don’t stop writing about pre-registration, because it’s the only solution to the disastrous state of science today. Research has become a sad joke. If pre-registration ever becomes a reality, you and some other bloggers should receive much of the credit.

  • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

    “No-one formally reviewed those notices of registration”
    I must mention Sandra Böttcher at the DRKS here, who invested some of her patience into reviewing the formalities of my submission. Of course, she didn’t conduct peer review, so this does not actually matter for your argument, but she was amazingly helpful and patient.

    Also, I’m glad for some anti-preregistration polemics. How could you know you’re doing the right thing if nobody bothers to call you names over it?

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Well you know the old saying: first they ignore you, then they ridicule you, then you ridicule them and the Internet wins.

  • Wouter

    “but equally important, it [pre-registration of studies] will facilitate the publication of negative findings.”

    I wonder, whether that is really the case. As is mentioned in other parts, decisions on publication largely lie with the publisher. Unfortunately, editors of high-impact journals prefer studies that present positive findings. I sincerely doubt that this will change, if a study had been pre-registered. And if this doesn’t change, and the field of neuroscience remains competitive, we’re bound to see a lot of pre-registration bullocks and fraud.

    • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

      The Chambers proposal at Cortex (go here: http://neurochambers.blogspot.de/2012/10/changing-culture-of-scientific.html ), which everyone interested in the topic should know, works this way: authors submit background and methods for a proposed study. These methods are peer reviewed, and if they are accepted, data collection and analysis can commence based on the accepted, pre-registered proposal.
      If your hypothesis is interesting and the reviewers think the methods appropriate, one will gain “In Principle Acceptance”. Now, publication itself depends not at all on the final results that then emerge; if the pre-data collection pre-registration is accepted, publication of the results, whatever they may turn out to be, is guaranteed. Chris Chambers hopes that this will facilitate the publication of negative findings, since authors will get published regardless of their results.

      • Wouter

        Hi Jona,
        Thanks for the link; a very interesting read! However, there still remain many problems with this design. Regarding the design proposed for Cortex by Chris Chambers, I’m not going to discuss all problems here (wouldn’t be appropriate), but I will say this in general: One journal taking on a different approach isn’t going to change the scientific community and how the scientific and general community looks at positive/negative results.
        If Cortex would go through with this idea (and I actually hope they do), I hope they join forces with other influential parties/journals. Otherwise, being a lone sheriff trying to enforce new rules, this idea could die out faster than a fart in the wind.

        • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

          I don’t see Cortex as a sheriff trying to force anybody. Nor can one journal (or any single attempt) “fix” science.
          But Cortex is giving researchers the option to do a new, and potentially, better, way of doing research.

          • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

            As Jona says, Cortex is only one journal but they’re paving the way. Hopefully others will follow and indeed some already are (e.g. Frontiers)…

            The culture of undervaluing positive results is also already beginning to shift, as far as I can see; certainly compared to 5 years ago there’s more being said about it, and organizations are making statements and declarations about how negative result publishing should be encouraged (although to my taste these are often tame and conservative.) It’s a start…

      • http://twitter.com/jscottwagner Scott Wagner

        This version of pre-registration is a lot of work and rigid structure at the beginning to shove on somebody who wants to pursue a good idea. Depending on funding constraints, many studies might want to change approaches midstream, or alter the scope somehow: the system described makes that look suspect, when it might actually be the coolest segue to ever hit the field. I’d think a more modest approach to pre-registration would invite more participation, and participation is huge. Make detailed information voluntary, at least for now. Even basic information about a study can be immensely helpful. The registration data is mostly for replicators, publishers and others, so it’s value to the initial team is not necessarily enough to justify them making a large, revealing commitment (obligation).

        • http://www.facebook.com/jona.sassenhagen Jona Sassenhagen

          As someone who has recently handed in one article each based on both a looser (voluntary, unilateral pre-registration; http://bit.ly/10fVPxJ) and a more rigid (Chamber’s Cortex RR) form of pre-registration, I strongly prefer the tighter format.

          I agree with you that most likely, every bit will help, but I see the strict pre-registration of methods as a necessary step, mostly because loose pre-registration leaves open a lot of possibilities for fishing still.
          Strict pre-registration (“we will investigate gamma oscillations at electrode Cz, comparing condition A vs B, time window 200 to 500ms”) is a major game changer for the researcher. Loose pre-registration (“we will investigate gamma oscillations”) isn’t.
          Well, there’s probably some researchers who’re already doing everything by the book to whom even strict pre-registration wouldn’t change much; to you, I bow.

          Again, note that the Chambers format explicitly allows exploratory analyses. They need to go into a different section though. And fundamentally, if your explorations lead you to a cool finding and new, sound hypothesis, that should be the subject of a new confirmatory, pre-registered study.

          I really like the oil field metaphor by the way.

    • http://twitter.com/jscottwagner Scott Wagner

      Thanks for this. I wonder with you.

      I think there is a hopeful offset to our concern, though, in the form of our worst bugaboo. With average power levels of neurological studies arguably in the 20-40% range today, we need an infrastructure and formal approach that delivers high-quality, replicated science. We ain’t got that yet. Part of what will likely change is our emphasis on “high-impact journals” as the primary distributors of our best science. Our evolution toward a better approach toward science may also take other more unintuitive turns. I’d contend that research will be democratized aggressively in the medium-term, especially government-funded research, so that open source journals and quasi-open journals of various forms will deliver broader and broader sources of quality science to more effective ends. Peer-review will undergo a reformation of sorts, with (very) good and (very) bad iterations of the concept leading to a more heterogenous research environment, both in study design and publication. Broad access to studies and, eventually, data will become much more common soon, as will a broader recognition of the challenges that result in the low power of our studies. This will lead to a higher quality and mix of studies.

      There will also likely be a natural tiering of research, like in oilfield development, where some are wildcatters (initial exploration), some are development (refining and building on successful exploratory research), and some are maintenance. These stages of research are all important. Formally democratizing the process will increase both the use and the interrelationships of these types of research in the future. I think we will eventually have a much more open look into existing research as consumers and ‘wholesalers’ of science, so that, whether wildcatting or replicating, the work will find a market. Replicating research (usually with adjustments) is much the different design and cost challenge.

      Pre-registration may be bullocks-ridden at the start, but it’s a key lever toward any effective, mature vision of what our international research portfolio should look like. Without it, we’ll be missing a basic requirement of high-quality mind and brain research. Even modest pre-registration information will be a huge boon when widely adopted.

  • Pingback: How the Scientific Sausage Gets Made: Preregistration Escrow for Basic Science? | Nucleus Ambiguous

  • Pingback: Wolność czy zniewolenie? | Neuro Bigos

  • Pingback: The big pre-registration debate › Counterbalanced

  • Pingback: Preregistration For Data Science? - Neuroskeptic | DiscoverMagazine.com

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »