A New Kind of Peer Review?

By Neuroskeptic | July 13, 2013 2:56 pm

Writing in the Journal of Clinical Epidemiology, a Dr Yvo Smulders of the Netherlands makes a proposal: A two-step manuscript submission process can reduce publication bias

Smulder’s point is that scientific manuscripts should be submitted for peer review with the results and discussion omitted. The reviewers would judge the submission on the strength of the methods and the introduction alone. If they recommended publication, the authors would then send them the full paper.

The reviewers would then have a chance to change their mind and reject it, or ask for further experiments to be carried out, but the ‘bar’ for this to happen would be high.

Hence the scope for reviewer-based publication bias, the tendency to favour ‘positive’ results, would be reduced. Reviewers would have to make a decision on the basis of the experiment itself, regardless of whether the results were positive or not. Smulders says that it would also ease the burden on reviewers in terms of the volume of material they’d need to digest.

It’s a clever notion (and, as Smulders points out, not a new one; it dates to the 1970s, but has never taken off.)

The proposal is reminiscent of the preregistration with peer pre-review model which I’ve advocated. The difference is that in the latter case, the authors submit the introduction and methods before the study has been conducted while in ‘two-step’ submission, the results are already there, just not revealed until later in the process.

The difference is that unlike preregistration, two-step review would not prevent publication bias (or other questionable practices) on the author’s side. Two-step would, however, reduce the incentive for such bias – why fish so hard for a positive result if you know your study would make it into a good journal on the strengths of its methods?

But the proposal would certainly be a step in the right direction and, in fact, could form a natural stepping-stone to a preregistration system.

ResearchBlogging.orgSmulders YM (2013). A two-step manuscript submission process can reduce publication bias. Journal of Clinical Epidemiology PMID: 23845183

  • Wouter

    I’ve been giving pre-registration a lot of thought recently (mainly because of Neuroskeptic’s blog and others, such as neurochambers.blogspot.com). What I find interesting is the following sentence from this particular post:

    “The difference is that unlike preregistration, two-step review would not prevent publication bias (or other questionable practices) on the author’s side.”

    Now my question is, is that really true? From my experiences, it is my opinion that total control is an illusion, a utopia (regardless of the subject). Hence, I believe that scientists who want to cheat, or commit fraud, will be able to do so regardless of the rules infringed upon them. In other words, it is not the “bad scientists” that are targeted by a pre-registration review model, but the scientists that want to conduct proper science.

    Now, do scientists with good intentions always conduct proper science? No, they don’t. But I strongly believe that that is either due to the “publish or perish” attitude, or ignorance. Ignorance is a tough problem, for it not only troubles science but society as a whole, and I don’t think new review models are going to solve ignorance. Then, the “publish or perish” attitude is mainly initiated from the publisher’s side: high-impact journals prefer positive results, which tempts scientists to resort to questionable practices in order to get their work published in the highest ranked journal as possible. Now, this latter issue, as the current blog post agrees with, could theoretically be solved by a two-step review model.

    Thus, I think that there’s no real distinction between the two review models, for they both will only target publication bias effectively from the publisher’s side. Furthermore, I think publication bias from the author’s side will decrease, once the publisher’s side is unbiased towards results. And i don’t think that the part of preregistration, acting upon authors alone, will be do so too, for bad scientists will be bad scientists.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Thanks for the comment.

      First off, preregistration wouldn’t stop fraud – but it’s not intended to. I am doubtful that any system could ever stop outright fraud because no matter how good the rules are, you are still free to break them. But fraud is a small issue compared to questionable practices like p-fishing.

      As you say, a two-step review process could reduce the incentive to fish for positive p-values, etc. because scientists would find it easier to publish negative results.

      My concern however is that a) just getting a publication is not the only incentive for p-fishing – there’s also the incentive to confirm your own theories, to find results that match with what is ‘hot’ in the field, that will get heavily cited, etc. b) as soon as commerical interests enter the equation that adds another huge incentive and c) there may be a fundmental ‘cognitive bias’ in favor of positive results, such that even a scientist on a desert island all alone, would be biased.

      Given which I think two-step review would be a big step forward but preregistration is two big steps.

      I would rather take a drug if I knew it had been shown to work in all of the preregistered trials, compared to all of the trials published (even with two-step review).

  • Vibra Trainer Lloyd Shaw

    Would this model include a revoking system , if major flaws in the study was proven. Even long after the peer review process was passed and the study was published ?

    I am in an industry where literally hundreds of papers managed to get published with fake vibration training machines. Some by “top” sports academics.

    But they still get credited as “experts”.

  • dodgy

    Um…

    The fundamental problem is NOT that there is a failure in the concept of peer review. The problem is that, so long as you make payment and job security dependent on publication and acclaim, then humans will form into groups to provide a self-support process of faking reviews to help their friends.

    We’re social animals. It’s what we do.

    Huge sections of science have now turned into tax-payer-funded jobs for life based on this development. What should happen is that much of these jobs should be reverted to amateur status. Cutting back funding would be painful, but it would ensure that better science is done…

    • Wouter

      I actually agree with the first part of your post. However, cutting back funding would also mean that less science will be done. Science does not merely thrive on the best scientific studies, but also on a wide variety of studies, and the repetition of studies. This is especially true for relatively young scientific subjects, like neuroscience. If society wants to know how the brain works, how the mind works, you’re going to need more than a handful of brainiacs to solve it all.

      • dodgy

        …cutting back funding would also mean that less science will be done….

        Indeed. have you seen the sort of thing that passes for ‘science’ nowadays, especially in the environmental arena? The field is woefully oversubscribed, and people are queuing up to do pointless studies to get their PhDs, and then maintain tenure with pointless publications.

        There MUST be a limit to the numbers of scientists that society is prepared to support, and I think we are well past that number now.

  • Pingback: A New Kind of Peer Review? - Neuroskeptic | Ope...()

  • kndiaye

    Hi Neuroskeptic

    Are you aware of anyone discussing the issue from the editor’s perspective: What should be the editorial strategy for a so-called high-impact journals in such two-step review?

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      That’s a good point. Actually I don’t know of much stuff looking at it from the editor’s POV – it’s a gap in the market, I think.

  • andrewkewley

    In general, scientists should be judged on the quality of their methods, not their results. Unfortunately, scientists are judged on their ‘success’, which is pretty much translated into ‘positive results’.

    Many young scientists have done high quality science, but failed to get the recognition and career advancement they deserve due to not getting exciting/positive results.

  • Emmanuelle Baudry

    But why would a bias for positive results be such a bad thing?

    While I of course agree that fishing for significance is unacceptable, I do think that legit positive results are intrinsincally more interesting that negative ones. In my field (ecology) the goal of experiments is to understand patterns, to determine which relationships exist between variables. There are much more possible relationships than actual one, so showing that a relationship exist, a positive result, is much more informative than showing that it does not.

  • Pingback: A New Kind of Peer Review? - Neuroskeptic | Box...()

  • Pingback: A New Kind of Peer Review? - Neuroskeptic | Eve...()

  • Pingback: A New Kind of Peer Review? - Neuroskeptic | Pub...()

  • Erick Turner

    The Smulders piece cites 4 references in which similar proposals were made. Here are 4 additional references made more recently:

    1- Mirkin JN, Bach PB (2011) Outcome-blinded peer review.
    Arch Intern Med 171: 1213–4–authorreply1214.

    2- Colom F, Vieta E (2011) The need for publishing the
    silent evidence from negative trials. Acta Psychiatr Scand 123: 91–94.

    3- Sridharan L, Greenland P (2009) Editorial policies and
    publication bias: the importance of negative studies. Arch Intern Med 169: 1022–1023.

    4- Turner EH (2013) Publication bias, with a focus on
    psychiatry: causes and solutions. CNS Drugs 27: 457–468.

    The last of these is my own review article. My suggestion differs in the material upon which the preliminary editorial decision is made. Rather than that its being based on an introduction and methods section written *after* the study results are known, it should be based on the the study protocol, which is written *before* the researchers have been informed, and thus biased, by the results.

    The fact that these suggestions have been made by so many parties, especially recently, provides encouragement that this may be a suggestion whose time is coming.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Thanks for the comment Erick. Your proposal sounds very similar to the preregistrationism that I and others are advocating – excellent.

  • Pingback: A New Kind of Peer Review? - Neuroskeptic | Bra...()

  • Ben0000o

    Looks like the journal on Attention, Perception, & Psychophysics is implementing a two step process similar to this. http://link.springer.com/article/10.3758%2Fs13414-013-0502-5

  • Pingback: A New Kind of Peer Review? | Science 2.0 news |...()

  • Pingback: A New Kind of Peer Review? | Higher Education a...()

  • Pingback: A New Kind of Peer Review? | The daily digest |...()

  • Pingback: A New Kind of Peer Review? | Personalized Profe...()

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »