The Peer Reviewer’s Dilemma

By Neuroskeptic | December 11, 2013 1:32 pm

While peer-reviewing a paper this week, I found myself in a quandry.

peer_review_dilemmaThe study in question is an open-label, observational trial of a psychoactive medication. There’s no control group and no randomization.

Personally, I think such studies are of little scientific value, that they more often mislead than enlighten, and that they should only be published when exceptionally informative.

But these studies do get published – in fact, rather similar articles have been published recently in the (mid-ranked) journal I’m reviewing for.

Apart from having no comparison group, there is nothing wrong with this study – the sample size is decent, the paper is well-written, etc. In other words, by the standards of its genre, this manuscript is perfectly good. But I don’t like the genre.

So what should I do? Recommend acceptance, against my convictions? Wouldn’t that be betraying my scientific integrity?

Or do I recommend rejection, and be unfair to the authors? To reject would be unfair because many other reviewers, for this journal, evidently do accept manuscripts like this.

If I rejected it, the authors might curse the capricious nature of peer review, and complain about being unlucky enough to get a reviewer with an axe to grind. They’d probably be right.

I’m inclined to compromise, by accepting the paper, but only after demanding changes, to make the manuscript more open about the limitations of the design. But I have previously written that

The reviewers are often the ones who write the Limitations Section, but I think this is often a way of passing the buck…

rather than bite the bullet and reject a paper, you may feel – well, being realistic, everyone’s in the same boat, who hasn’t cut corners, and we’ve all got to earn a living… etc… – so you recommend the paper is accepted, but in order to avoid feeling like a walkover, you make the authors include your concerns as Limitations.

Which is a stinging indictment of exactly what I’m proposing to do. Ooops.

I wonder how other peer-reviewers resolve this dilemma?

  • Mags A

    That is all you can do but reject the paper. To accept because everyone else does may be fairer to the author but as a reviewer you need to follow your particular beliefs, not those of others.

    Offering a compromise that you feel comfortable publishing is the only way forward if you would rather not reject the paper.

    • Pedro Oliveira

      but as a reviewer you need to follow your particular beliefs, not those of others

      Even if this belief is a bias of a particular reviewer. I agree in the substance with the Neuroskeptic and I don’t like this kind of studies, but allowing such biased behavior is very dangerous.

      Walter White usually says “apples and oranges” and extrapolating the bias argument, and maybe entering in the realm of apples and oranges, I’d like to give an example:
      I’m morally against the Cuban regime and I have a feeling that all scientific research in that country violates the human rights. I can state that to the editor as a concern and recuse myself. But I cannot find a shortcut to punish the Cuban author of a scientific paper.

      • Glenn Jones

        Agreed. I couldn’t have put it better!

    • Glenn Jones

      If you don’t agree with the policies of the journal, then why do you continue to do reviews for it. Peer reviewing should be about objective standards, not subjective beliefs. It the group that manages the journal has determined (rightly or wrongly) that such papers are acceptable, then you have no right to impose your views

      • Neuroskeptic

        OK. But this journal asked me to give them a review of this paper. It’s not like I’m sending them reviews unsolicited. Given that they wanted my opinion, shouldn’t I give it to them?

  • AndreK

    If you want to change something you should reject the paper. In my opinion it’s the only way to improve future studies. Its like a domino effect … hopefully.

  • Diane C

    I would suggest that you also refer to the publications guidelines and checklists available for your resource and use. I would encourage the peer-reviewer to refer to the guidelines available for their resource and use:CONSORT (even though these apply to randomized, controlled trials, many things on the checklist can be applied here)
    GPP2 (there are definitely checklist items for Limitations, Reporting the detail required to ensure unbiased presentation, and Clear description of research hypotheses), and even the ICJME guidelines might be helpful.

    If the items on these checklists are clearly all included in the paper, there is probably no good reason to deny publication, independent of scientific merit.

  • Pedro Oliveira

    If you feel you are biased you should recuse yourself. You said it yourself: “I don’t like the genre”

    The fair thing to do is step out.

    • Neuroskeptic

      I didn’t say I was biased – I have my views on the genre and I think they’re correct, not a product of bias. I’m just aware that I might be wrong and that other people honestly disagree with me.

      • Pedro Oliveira

        I don’t like the genre either. But as a reviewer you are supposed to act according to the rules of the Journal. I think If the Journal accepts this kind of research I’d either leave the board or play by the rules.

  • djlewis

    The key information is this… “…rather similar articles have
    been published recently in the (mid-ranked) journal I’m reviewing for.”

    Apparently the editorial policies of this journal are not up to your standards. So unless you think you are in a position to alter the policies by quiet means, the course I recommend is…

    Return the paper unreviewed. Simultaneously, resign from the review board and give what you’ve said here as a reason. You might even point to this post, as the editor may well see it on her or his own and figure out that you are talking about them (unless you are sufficiently disguising the situation to protect your blogger’s anonymity).

    Of course, you need to ask yourself why you are even still on that board. But perhaps you only recently became aware of the disconnect between them and your own standards.

    • Neuroskeptic

      I see your point, but isn’t that a cop-out? Shouldn’t I stay, and try to argue my corner, rather than quit the field?

      • djlewis

        Sure, if you think you can accomplish something and are willing to spend the “capital”, which might include your anonymity. My answer was based on the assumption that you and the journal were pretty much irreconcilable, ready for divorce.

        But does such a “mid-ranked” journal publishing less than first-class research serve some socially-redeeming function? Or is it just supporting a too-large community of researchers who need outlets to earn their various points and other rewards but aren’t contributing to “science” at their weight?

        • Lou Rooney

          haha ouch

  • Dwayne J. Stephenson

    I don’t understand how review assignment works, but I would seriously consider, I don’t know, working for a different journal with different admissions standards? The question of what constitutes a legitimate scientific paper is a good one, but this is the sort of thing that requires the implementation of a policy (presumably set by whoever heads the journal). You shouldn’t feel like you have to decide on your own what papers are publishable in a peer-review environment: standards like these are set by groups, not individuals.

    • Neuroskeptic

      That’s a good answer. But it reduces peer review to a purely editorial exercise (“is this submission right for this journal?”) rather than a scientific one (“is this true?”)

      • Dwayne J. Stephenson

        I am not sure there’s actually a clear demarcation between those two things. Even stringent scientific standards are no guarantee of truth.

      • Dr Pérez-González

        First of all, I must tell you I have enjoyed reading this post because I have felt the same a couple of times. You are totally right regarding reviewers’ authorship of the Limitation section. On the other hand, I agree with djlewis about the key information. The pr process is conditionated by the Journal standandards. Depending of the scenario I am able to play or not in a jam sesion.

  • Shawnna

    I think you can recommend to reject the paper and give exactly these comments. There is likely another reviewer, which means that the final decision falls to the editor. The editor for the journal would probably be interested to hear what
    reviewers are thinking about the quality of the work they are
    publishing. However, they will probably not reject the paper based on your comments if they are already publishing this type of study.

  • Timothy James Rogers

    Maybe try the psychoactive medication and see for yourself, perhaps? If you need professional help and I mean a person who has used psycho-active medication on a professional level and not a psychiatrist, I’ll help you out and take the journey, regardless of non-peer reviewed article maybe some things just so happen to be truth, like how effective psychoactive medication is.

    • Timothy James Rogers

      The control group was the issue not that it wasn’t peer reviewed*. Would the control group just be everyday normal people who never had psychoactive medicine in which you compare between the two? And then you would need to change certain environmental stimuli and see it’s effects on the different kinds of people that to me would be how you would set up and see the results I feel.

  • practiCalfMRI

    Been lots of discussion of late on the subjects of post-pub peer review, published peer reviews, anon peer reviews and so forth. These are all essentially about trying to improve the quality of scientific publications, although there are tangential debates about credit, impact factors and what have you.

    In any event, let’s do a thought experiment. Imagine first that you know your review will be published anonymously along with the paper (if it’s published). What would you decide? Next, imagine that your review is going to be published but now with your name (or pseudonym) appended to it. Any change in your decision?

  • Pingback: The Peer Reviewer's Dilemma - Discover Magazine...()

  • Franck Ramus

    This is a dilemma for editors, not reviewers to solve. For one thing as a reviewer you never reject a paper, you can only recommend rejection.
    So here you recommend rejection on the grounds that such studies are of little scientific value. You may add a personal note to the editor that you disapprove of this journal’s publishing such studies, and you suggest that they improve their standards. If they don’t, then you will not be reviewing for them anymore, cause what’s the point?
    And then the editors read you and the other reviewers, rethink about their editorial policy, and take their responsibilities. If they publish the paper against your recommendation, you hold your promise and do not accept to review for them again.

  • Cellbio

    Having been in a similar situation of reviewing a paper whose methods were very much the rage but offered little to the scientific community, I offered my clearly stated reasons for rejection and made clear the basis was not derived from the quality of the work or manuscript. I was right, and that style of work is no longer accepted at quality journals, but I also do not get asked to review additional papers and the submitting author has been nothing but raging in his contempt, despite the journal accepting the manuscript. So, you might indeed be resigning de facto, but I encourage you to stick to your principles as the publishing game has turned into a racket (read Randy Sheckman’s recent comments) and the community can’t afford further erosion of scientific integrity.

  • Thom Baguley

    I agree that this is a tricky one. My take is as an editor. First, the editor decides, not the reviewer – so you are partly off the hook. A good review informs both editor and authors (and ideally says the same things to both). In this case I would want a reviewer to articulate their position clearly and explain how to improve the manuscript or the research. This position can be quite nuanced. The editor then has to make a decision that takes those views on board.

    I personally would not normally over-rule specialist reviewers unless I was an expert on the topic in question. In practice you are likely to end up with a split vote if you recommend rejection – on the basis that the regular reviewers tend to accept this stuff. This makes the editor’s job harder and a good editor will take time to think over the best response – both for the paper and the journal. These difficult decisions are precisely those that lead journals to shift policy and practice – in the type of papers and quality of work they accept. These shifts can be explicit (e.g., the journal decides that it will no longer accept a certain type of paper) or more subtle (e.g., the journal will use more stringent criteria for acceptance).

    The advice to not review the paper is not helpful in my view. You are reducing the pool of reviewers to those with less rigorous standards. That can’t be a good thing.

  • Tim Vines

    I think you could actually achieve some change here. I recommend writing to the Editor and cc the Editor in Chief, laying out your reasons why these papers should not be published in their journal. At the very least they should promise to discuss your comments at an upcoming board meeting. I’d also put your reasoning in your ‘confidential comments to the editor’. The review itself can be neutral on whether it should be accepted/rejected (i.e. encourage resubmission), but detail the control group and randomisation issues.

    You’d be amazed at how often changes at a journal take place because someone takes the time to write and express their concerns. It may be that most of the editorial board agrees with you but the issue has never been formally discussed.

    Lastly, don’t forget that the journal may have changed its standards since the previous paper got in, and that sometimes even bad papers end up getting accepted.

  • Marscrumbs

    Science always leads to more science. So even if these observations aren’t conclusive they are still useful.

  • ohwilleke

    I would suggest focusing on how the paper is described. Terms like “field study”, “ethnography”, or “survey results of psychoactive medication users . . .” would clearly convey the nature of the study and not lend it undue authority. Large, carefully gathered data sets, even if not in controlled or randomized have value for some purposes. They simply don’t distinguish placebo effects and cultural views about medicine effectiveness. People do research of this kind all the time in the social sciences and some knowledge is better than none.



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar