What Is Preregistration For?

By Neuroskeptic | July 3, 2018 4:12 pm

fix_sci_small

A paper in Psychological Science was taking a beating on Twitter last month.

In this post, I’m not going to talk about the paper itself but rather, about how it came to be published and how preregistration – a concept I have long advocated – may be being misused.

The paper reports on five studies which all address the same general question. Of these, Study #3 was preregistered and the authors write that it was performed after the other four had been completed. It was also larger than the others. The results of Study #3 closely matched the other studies’.  So far, so good.

However, according to Daniël Lakens on Twitter (I’m not sure how he knows this), Study #3 was conducted on the instruction of the editors (during peer review):

Now, this is where alarm bells started ringing for me. If Psychological Science asked the authors of this paper to carry out Study #3, the reason, presumably, is that they weren’t fully convinced by the other studies. The journal wanted more evidence for the hypothesis that ‘participants in a dimly lit room or wearing sunglasses tended to estimate a lower risk of catching contagious diseases.’ That’s understandable, but what would the editors have done if the results of Study #3 had come back negative?

This is a crucial question because if Psychological Science (or any journal) requires authors to provide more positive results in order to get published, they are putting those authors in a very difficult position. A paper in Psychological Science can make someone’s whole career, and to tie such a publication to the result of a particular study incentivizes (at best) questionable research practices.

To be fair, maybe maybe Psychological Science told the authors ‘you must do Study #3, but we commit to publish your paper whatever the results are.’ That wouldn’t be as bad, but the question would then arise: would this journal have considered the paper at all if the original four studies were not uniformly positive…?

The whole beauty of preregistration is that it would have allowed these authors to get their studies reviewed and accepted for publication before any of the results were in. There would then have been no pressure for Study #3 or any of the other results to be positive – no pressure from the journal, anyway. The hunger for positive results that underlies so much publication bias and p-hacking would never arise.

In other words, asking authors to conduct preregistered replication studies to confirm their non-preregistered findings might be harmful or it might be useful, but it certainly isn’t the best way to do preregistration.

ADVERTISEMENT
  • http://www.mazepath.com/uncleal/EquivPrinFail.pdf Uncle Al

    Psychology is shuck and jive swindle. The mumble factory writes the paper then cherry picks the evidence. It is a balloon whose skin is everted.

    If psychology had any empirical validity at all – any validity at all – schools would work, prisons would empty, homeless would heal, drugs would not cripple good people…and the seditious vicious idiots that seek ruin for this country would be brought to justice by common sense and law.

    #WalkAway

  • http://www.pbase.com/davidjl David Littleboy

    “the hypothesis that ‘participants in a dimly lit room or wearing sunglasses tended to estimate a lower risk of catching contagious diseases.’ ”
    Hmm. Isn’t this an embarrassingly stupid hypothesis? When will journals start using a “you can’t be that inane, can you?” test to reject articles?
    Not for a long time, it seems. A sister publication of yours (by the BPS) has an article favorably discussing psi research. No one asked him that question, apparently…

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+