Science Needs Vigilantes

By Neuroskeptic | December 31, 2013 11:36 am

Lately, there have been increasing numbers of online, unofficial – what might be called vigilante – investigations into published scientific work.


The blog Retraction Watch and its comment section are a good example of this. Commenters, often anonymous, will get onto the trail of a certain researcher (generally following a retraction) and scrutinize their publications (e.g. here) looking for plagiarism, image manipulation, statistically improbable data, or other evidence of bad practice.

Other venues for this scientific vigilantism include PubPeer, dedicated blogs such as Science Fraud (now on hiatus after legal threats). On occasion I’ve turned my hand to this.

Now, when one of these investigations gets underway, a funny thing often happens: someone will ask about the “motives” of the investigators.

Sometimes it’s the targets of the investigation who ask this; sometimes it’s the institution to whom the findings are forwarded; sometimes it’s an anonymous comment. Either way, sooner or later someone will question why someone would want to dig into this issue.

But are the vigilantes really doing something that requires explanation?

All they’re doing is reading papers – carefully. In an ideal world, this is what all readers would be doing – paying close attention, not taking anything on faith, checking the sources. And all writers dream of readers giving their work their full and undivided attention.

What needs to be explained is why most of us don’t do this most of the time. And that’s just when it comes to everyday readers. When it comes to publishers, editors, and peer-reviewers, shouldn’t we be questioning their motives in allowing these problematic papers to be accepted?

The most charitable interpretation is that they just  ‘don’t have the time’ to look carefully.

In the case of readers, this is understandable. It would be impossible to perform due diligence on everything one reads, there’s just too much out there. Nonetheless, those readers who aren’t so diligent should be grateful to those who are, and support them. They’re doing a service for the rest of us, at least insofar as they’re getting things right.

It’s not so easy to excuse the lack of vigilance among many peer reviewers and editors. Their job is to scrutinize submitted papers. Every paper accepted despite containing evidence of mischief is a testament to an editor and peer-reviewer(s) who were asleep on watch.

Their duties are hardly onerous – it doesn’t take much effort to run papers through a plagiarism checker, take a good look at the figures for evidence of manipulation, and so forth. It would take a matter of minutes to detect such misconduct. After all, it generally takes vigilantes mere minutes to uncover them, once they’re on the case.

I recently uncovered plagiarized article by taking the first sentence of a paper, Googling it, and finding it identical to another source. From downloading the PDF to smoking gun within one minute! If I did that, anyone could have – editors included.

What motivated me to check that paper? Who cares?

What motivates someone to publish that paper without checking it? Laziness? Naivety? Greed? Now that’s one to ponder.

  • Maurizio Morabito

    When this happened years ago in climate science, we ‘vigilantes’ were all labelled as evil deniers on the pay of evil corporations trolling to destroy the natural world, the planet and civilization itself.

    NIce to see the correct, skeptical attitude to science is instead spreading to other fields. As it should.

    • Neuroskeptic

      That’s a ridiculous comparison. I’m talking about spotting formal scientific misconduct, not criticising scientific theories. But it doesn’t matter really because, since the BEST study, even evil deniers on the pay of evil corporations have concluded that global temperatures are rising.

  • Maurizio Morabito

    Not sure if it’s more ridiculous for you to mention BEST, or the warming everybody has agreed upon since 2007, or not understanding that climate science has more than its share of dodgy scientific papers.

    Go on now, tell your readers that scientific “vigilantism” is only good outside of climatology.

  • Rolf Degen

    Even under the best circumstances – if everybody does the right thing and bothers to work hard to accumulate knowledge – it is very hard to do great, innovative science that delivers a substantial progress in our understanding. Most studies offer only little increases in insight. There are only few Einsteins or Darwins, but there are more frauds and cheaters who make the whole endeavor even more difficult. These people have to know in advance that if they intend to cheat they will be shamed like criminals, like people who pour toxic waste into a lake. I think that right now for a would-be-fraud, the prospect of getting caught is not as deterrent as the prospect of being exposed by Retraction Watch, be it by a vigilante. In earlier times, these kind of things went by mostly unnoticed. A field like psychology can not bear another Stapel.

    • Sean Lamb

      Retraction Watch doesn’t expose anyone, it simply reports the work other people have done.

  • Dave Fernig

    Vigilantes is perhaps a bad choice of wording. Critical reading and the evaluation of evidence are at the heart of not just science, but also democracy. So all scientists should be reading critically, not just a few. How can you have a PhD and not read critically? Publicising one’s critiques is also at the heart of science and democracy, though fear drives many to be silent; the anonymity of PubPeer helps a lot here.

    As for climate science, yes, absolutely, the drive for glamour publishing pushed some to manipulate their data.
    This could turn out to be the worst case of data manipulation humanity has every encountered, because climate change will change your world and for H sapiens, this will be for the worse – life in general will adapt and go on, but maybe not us.

    • Neuroskeptic

      I agree. “Vigilante” is not an ideal term, but I used it because the whole process of investigating and exposing a bad paper does feel like vigilantism. Even though, as you say, it is just reading critically which all scientists should do.

  • TKList

    Free market starting to work in Science.

    • Neuroskeptic

      On the contrary – the free market is often at the root of the problem (predatory open access publishers with a financial interest in publishing as much ‘science’ as possible, the quality being irrelevant.)

      • TKList

        So your problem is free speech? You might want to rethink that. The solution is in the free market. Exposing fake science will be an industry of sorts. Enjoy, you might be leading it someday.

        • Neuroskeptic

          Free speech is not the problem, no. Everyone has always had the right to say what they will and call it ‘science’. This has given the world lots of good works and also a few cranks.

          What is new is people trying to monetize putting the label of ‘peer reviewed science’ onto all and sundry. This is a big part of the problem (although sadly there are other publishers who have no financial motive and yet still publish rubbish.)

          Vigilantes on the other hand work for free and I can’t see how you’d monetize it. Well, except that I get paid by Discover Magazine per pageview, so I suppose that I do monetize my posts – but my vigilante-style posts tend to get fewer hits than my other posts so if anything I’m paying the price of doing it.

          • TKList

            Then you just need to brainstorm on how to monetize it. Philanthropy is one avenue. Start ranking the sources of these shabby peer reviewed science published papers for maximum effect and increased viewing. You can make it like the top ten of science trash. Have peers review and rank. That is a start.

          • Kieron Seymour-Howell

            Perhaps, if anyhow who was proven to have been deliberately misleading, manipulating, or stealing from others were to be dragged out in the street and shot, then left there to rot as a warning, the world might become more careful?

          • TKList

            Please make an appointment with a mental health professional as soon as possible. You sound just like the Taliban.

          • Kieron Seymour-Howell

            *laughs* I wonder at the reaction you attain with your personal and sweeping assessments of people you meet in person?

            My comments are crafted to make people think. The so-called justice systems and governments appear have little or no control over providing a deterrent to would-be criminals at the corporate level. Entities that appear to operate above the law. I should have made it clear I was talking about those people, or groups, who have a large region of responsibility and power over wide areas of the general population, not individuals who act at the citizen, or local level.

          • feloniousgrammar

            Think about eugenics and arbitrary state violence?

            Believe it or not, people can think just fine with your hyperbole.

          • Kieron Seymour-Howell

            My point is that the apathetic ways of “just letting someone else fix the problem” are clearly not working. Most people seem to be entirely preoccupied with various pop-culture, sports, or gossip. The system makes it very “easy” for the culprits to manipulate and corrupt, because no one seems to care anymore.

  • templeruins

    Thing is, non of these studies are truly objective anyway. Even in the hard sciences subjectivity is inescapable at some level, but for the soft sciences it is truly par for course. It’s intrinsic in their beginnings. People and universities also seem to be becoming more polemical as the system dictates.

  • Pingback: Science Needs Vigilantes - US Message Board - Political Discussion Forum()

  • Pingback: Science Needs Vigilantes - Research, computers, social science, chemistry, biology, astronomy, physics, mathematics, AI - City-Data Forum()

  • Bernard Carroll

    These days, I operate as an equal opportunity critic, and I
    have experienced the less than honorable reply tactics you describe. First comes the back of the hand intellectually – oh, you just don’t understand our profound work. Then comes the ad hominem charge of mean motives, sometimes
    orchestrated by the cronies of those I have critiqued. I have also been told to buzz off by at least 5 journal editors because my critiques inevitably made them and their journals look bad for having published the dodgy work in the first place. A standard tactic has been to deep-six my critique, while requiring the authors to publish an apparently spontaneous Erratum or Correction Notice concerning the matters I raised. These Notices aim to put the authors and the journals in the best possible light by being economical with the truth.

    As for the editors, few seem to really do their job nowadays. With the rise of Manuscript Central and the like, authors get to
    nominate who will review the submission (and who won’t), and the editors operate like postmen, as Nature once put it, dispatching manuscripts to reviewers and off to the printer.

    • Neuroskeptic

      I am inclined to give editors and journals the benefit of the doubt – the first time. Everyone makes mistakes. But when you get someone who’s publishing rubbish paper after rubbish paper, the only question that remains is: is this mere incompetence or a deliberate policy?

      • Kieron Seymour-Howell

        It is not about the content, it is about making money, generating clicks and keeping up the constant flow.

        If you think that accuracy and truth are priorities, then you have a lot to study about human psychology.

        Medicine is not about making people healthy, most doctors don’t even like people, people are a resource for them, to mine and farm. The same goes for the legal system, and government, as it has been proven that the exact type of person who wants to be in those positions is usually the least desirable or moral choice to have there.

        Sooner or later you will have to face reality, and that is you are a farm animal. You are a source of revenue, and anything else is annoying. Anything you do that does not involve making someone else money, means you are becoming a problem.

  • gil greening

    Behind this problem is poor reviewing and editing. Who let these manuscripts through in the first place? As well as self-plagiarism, I have detected sneaky practises – repeating data without explaining why it was repeated – a duplicated image, for example. Publication of poor data and incorrect conclusions from these data is almost as bad in my book. And also, when trying to track back when something was said (because someone claiming “I said this first”), found that the original quoted paper, often rather inaccessible, doesn’t say what the author claimed at all.

    This in the top journals as well as those with lower impact factor. My main gripe is not with the weak people who did this, but that their published data then form the stepping stones for new work. Much time can be wasted when the published data turn out to be either wrong or incorrectly interpreted or both.

    Even though it takes time to properly review a manuscript, better to do this rather than let precious time and resources be wasted in future by someone trying to follow up the spurious work.

    • Kieron Seymour-Howell

      To me, that is the rightful road down which incompetence travels after setting its foot. Much of what most people call science is theory and assumption. People are simply too lazy to challenge what they are fed. There is a massive amount of sensationalism and flagrant assumption. I have watched so-called science shows and the amount of pure assumption is astounding.

      If someone subsequently comes along and does REAL science, it will not matter one iota what was done before, because the facts and studies will speak for themselves. Many times the quacks and science religionists, merely get the ball rolling. Real critical science research stem off from that point when a topic becomes popular enough. These subsequent studies spawned by conspiracy theories and other forms of religion science have real value. New discoveries are made because these topics were thrown out there. Merit for injecting new ideas and perspectives into the world is thus achieved.

      A important part of human society is allowing the stupid ones to kill themselves through incompetence and thus weed them out of the gene pool. Medicine has done a lot to interfere with natural selection these days. We need the stupididty and warnings from the misourtunes of others to educate and steer the rest of the population towards a better world. The destiny of some, is merely to be an example to others.

  • Pingback: Science Needs Vigilantes - Neuroskeptic | Disco...()

  • Kieron Seymour-Howell

    One of the best articles on this Website.

  • Pingback: Weekend reads: Most scientific fraudsters keep their jobs, random acts of academic kindness, and more | Retraction Watch()

  • Brendan O’Brien

    Personally I think the focus on literary plagiarism is a distraction from the critical issue which is data fabrication Clearly astute peer review is a good start however major fraud is not detectable via peer review and requires whistle blowers. Furthermore there is the issue of accountability and plausible deniability, in my opinion any author who signs their name to a paper ought to be held responsible if the data is subsequently found to be problematic. In the end the book should stops with the PI, it’s not enough to make an example of a wayward postdoc or student, when all too often malpractice is a consequence of a culture that fosters it.

    • Matt Chew

      Evidence of plagiarism in a paper suggests that the associated data need a harder look as well.

  • Pingback: Leggende – Ocasapiens - Blog -

  • Matt Chew

    Questioning the motives of an accuser is a venerable method of discrediting witnesses, deflecting attention and muddying waters. Every critic or whistle-blower faces ad hominem criticism from those whose misconduct has been daylighted.

  • Pingback: Morsels For The Mind – 03/01/2014 › Six Incredible Things Before Breakfast()

  • Pingback: the Revision Division()

  • Pingback: The rise of vigilante science » Mano Singham()

  • Pingback: Who's Afraid of Negative Psychology? - Neuroskeptic |

  • Zl Kai Burington

    What you call vigilante seems to be right and standard scientific practice. A researcher should always approach published works critically. It is best if these criticisms are made public, whether as part of a published article or as an essay on the Internet. Peer review is at best a weeding process, and journals are businesses. Neither can enaure perfection of what is published. In general, peer review offers a false sense of security or authority to scientific works, while in the early 20th century, before peer review, works were read with more criticism.

  • Pingback: What Is "Open Debate" In Science? - Neuroskeptic()

  • Pingback: Null trial of cognitive behavioral therapy for psychosis given positive spin in British Journal of Psychiatry | Quick Thoughts()



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar