How to Hold Scientific Journals Accountable?

By Neuroskeptic | September 15, 2016 3:41 am

Writing in PLoS Biology, neurobiologist Thomas C. Südhof discusses Truth in Science Publishing: A Personal Perspective. Südhof is a Professor in the Department of Molecular and Cellular Physiology at Stanford. A veteran scientist, he’s been publishing since 1982.fixing_science

So what’s the state of science publishing as Südhof sees it?

He first notes that “scientists, public servants, and patient advocates alike increasingly question the validity of published scientific results, endangering the public’s acceptance of science.” This wave of skepticism is understandable, he says, because today’s scientists are often “failing to ensure that studies report facts and not fantasy”.

The problem, Südhof explains, is that the systems for ensuring quality control in science – such as peer-review – have become dysfunctional. Peer-reviewed journals, for instance, chase “trendy” topics, while reviewers often favor their own friends and allies. Yet scientific journals face few consequences when things go wrong:

Journals should be held accountable not only by their owners for the money they make but also by the public for the value they provide – just as a drug company cannot simply sell any drug but has to show that the drug is safe and effective, a journal should not be allowed to “sell” [i.e. publish] its products without being accountable for its content…

If a journal repeatedly publishes papers that draw untenable conclusions, eventually the authors of the papers may be blamed, but editors and reviewers who are arguably responsible for gross negligence are not held responsible. There are insufficient checks and balances in the publishing system; when high-ranked journals repeatedly publish papers that are later considered unreliable or even retracted, the journals seem to face no consequences – their premier status remains untouched.

This is true, but I imagine that high-ranking journals would counter this charge by saying something like this: it’s the scientific community – people who read and cite papers – that decides a journal’s prestige and status. So if we top journals seem to ‘get away with’ publishing bad papers, this can only mean that the scientific community feels that our journals are essentially high quality, the unfortunate problem papers notwithstanding. The community are the ones to blame if you feel that we lack accountability.

Südhof goes on to list some rules that he thinks journals should adhere to:

Reviews should be published, not hidden. Editors should be named as part of the published reviews and should be held accountable if papers fail to meet basic quality and reproducibility standards… Submitted papers should be assessed by a checklist that ensures that proper controls and reagent validations are present, and such validations should be required for the supplementary materials…

How can we get these rules in place? Südhof says that “as ‘voluntary’ action” by the journals seems unlikely, “we should demand rules that inject accountability into the system.”

I agree that if we want scientific journals to be more accountable, we (the scientific community) need to drive this change. But I’m not sure that demanding rules will be enough. Maybe something more akin to ‘direct action’ will be required. Put simply, we could just start holding journals accountable ourselves.

Suppose, for instance, that you as a scientist are unhappy with the quality or policies of a particular journal. Sure, you could complain and demand improvement. But if that doesn’t work, you could put your money where your mouth is and stop submitting papers there, and maybe even stop reading (and citing) papers published there.

Now, depending on the status of the journal, such a boycott might be easier said than done – boycotting Science and Nature might not be great for your career – but if enough people join your boycott it would really pressure that journal to change its ways.

Without at least the credible threat of this kind of community action, journals will never be incentivized to improve, and any amount of new rules and recommendations would lack teeth. Essentially, if we as the scientific community want change, we have to actually change our behaviour, not just ask for it.

ResearchBlogging.orgSüdhof TC (2016). Truth in Science Publishing: A Personal Perspective. PLoS Biology, 14 (8) PMID: 27564858

ADVERTISEMENT
  • Defenestrator

    Boycotts only work when they are really large, and I suspect that not enough critical mass can be built unless a given journal has a really consistent, ongoing pattern. I think a lot of time though, it really is some specific paper that gets uncritically reviewed in an otherwise-good journal.

    For example the PACE trial study of chronic fatigue syndrome was a genuinely bad study that was published in a good journal (The Lancet). That paper has done a lot of harm because it was clearly reviewed by overly-sympathetic readers – the Lancet failed to do its due diligence and broke its own rules about what makes for good science.

    But it’s pretty unlikely that enough scientists are going to learn about that bad study getting published despite fatal flaws and feel outraged enough to say, “I’m going to boycott The Lancet.” Most would still love to publish there because most of what they publish is good.

  • Iftikhar Qayum

    Anybody who criticizes a system should first present an alternative that is tested by reliable methods and found to be better than the existing one. Sudhof should take a does of his own medicine and show that he is advocating good practice.

    • Andrew Neff

      If we waited to criticize a system until we had an alternative ourself, wouldn’t we potentially miss out on the ideas of other people that are stimulated by our criticism?

      • Iftikhar Qayum

        I did not say not to critcize a system. My view is that criticism by itself is of little value and only provokes heartburns. It should provide the background upon which an alternate and better system should be built to replace the one being criticized.

  • Hilda Bastian

    I agree with you and Südhof that we need to do more and it’s an important discussion to have. Boycotting is difficult to achieve on a scale that would make a difference, though. But I’ve come to think we should take more direct action about specific articles than we do.

    For me, there’s a case in point at the moment with PNAS, which I’ve discussed on my blog as well as commenting on PubMed and corresponding with the journal: http://blogs.plos.org/absolutely-maybe/2016/09/02/gender-bias-bias-part-2-unpicking-cherry-picking/

    We need to go an extra few miles than we tend to, I think. Too often, people reserve their critiques for the background and discussion sections of their next paper, and don’t tackle the publication directly. Or we don’t push back at all and grumble and growl in private.

    There are powerful incentives for editors to publish “hot” papers that just end up burning science. Perhaps the disincentive of even stronger, more persistent, more common, and shared criticism might encourage an editorial environment awareness of the consequences:”publish in haste, repent at leisure”.

    I think refusing to cite could distort science in harmful ways, though, so I wince when people argue for that – people say that about open access too. But we collectively really do need to get away from favoring citing articles because of the prestige of the journal they are in.

  • Pingback: Lecturas hackers de la semana – club de ciencia hacker()

  • Peder Isager

    Like with any boycott, I think close to no one is going to do this on their own. This is especially true when there is some personal risk to one’s own career involved. If we in the scientific community want’s to drive the change in our own work environment, I suspect we’ll have to get better organized first.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+