The Trouble With “Limitations” In Science

By Neuroskeptic | May 16, 2013 4:09 pm

Is it always good thing to know your limitations?

Over at Scientific American, Samuel McNerney writes about the dangers of learning about common human cognitive biases. The problem is that it’s easy to find out about, say, confirmation bias, and think “Well, it affects other people, but now I know about it, I am immune to it” – and then proceed exactly as you did before, suffering the bias but now with misplaced confidence in your abilities.

I fear that a similar thing is at work in science, in the form of the Limitations Section.

It’s become fashionable for scientific papers to end with a few paragraphs about “the limitations of the work”. Why is this a problem?

Well, just as a morally confused Catholic might prefer to confess his sins regularly rather than change his sinful ways, the Limitations Section can serve as a kind of ritual cleansing which makes poor scientific practice seem acceptable. There’s a sense, usually unspoken, but I have heard it said, that “We acknowledge this limitation, so don’t criticize it.”

Worse still, if papers are getting published despite the fact that even the authors admit to a certain limitation, it would be easy to conclude that the limitation isn’t all that bad – so why should anyone bother fixing it?

Still, at least acknowledged limitations provide the reader with something to go on. Worse are the trivial ‘problems’ that often fill the Limitations Section, taking up room that ought to be spent addressing the real flaws. This kind of thing is actively misleading. To say that a handful of minor issues are the limitations is tantamount to saying there are no major ones.

When you think about it, it’s strange that scientists are expected to be their own critics in this way. Surely the flaws are for the reader to judge? Or the peer reviewers?

In fact, the reviewers are often the ones who write the Limitations Section, but I think this is often a way of passing the buck. When reviewing a manuscript, one sometimes feels that something about the study is flawed, not spectacularly so but seriously, and that in an ideal world it would prevent the paper from being published.

But rather than bite the bullet and reject the paper, you may feel – well, being realistic, everyone’s in the same boat, who hasn’t cut corners, and we’ve all got to earn a living… etc… – so you recommend the paper is accepted, but in order to avoid feeling like a walkover, you make the authors include your concerns as Limitations.

You act as the confessor for their scientific sins, in other words.

CATEGORIZED UNDER: papers, select, Top Posts, Uncategorized
  • DS

    Confessant: Forgive me father for I have no idea what these preprocessing steps do to my data.

    Priest: God only knows.

  • http://www.wolfsvalley.com/ Une Ulv

    Just another useless thing (worst thing for me are talks with several minutes about what is going to be presented, but nobody is forced to actually read these statements). Who began with it, which was the first paper featuring this?

    • JonFrum

      I’ve seen papers in speech pathology that start out with an apparently required ‘what you are going to learn’ section. Huh? If people can’t learn from just reading the paper, what the hell are they doing reading the paper? It’s like they’re writing for grade school kids.

  • JonFrum

    This fits in with one of my pet peeves – the ‘yes, I know, but’ card. You call out someone for having a too small sample size (or some similar fundamental flaw) in their work, and they say ‘yes, I know about small sample sizes, but…’ and then tell you all about the logistical difficulties in such studies, and if we don’t work with what we have, we’ll never learn anything, blah blah blah…’ Well no – if you sample size is too small, then your study is garbage.

  • toostoned tocare

    ‘Well, just as a morally confused Catholic might prefer to confess his
    sins regularly rather than change his sinful ways, the Limitations
    Section can serve as a kind of ritual cleansing which makes poor
    scientific practice seem acceptable.’

    ‘You act as the confessor for their scientific sins, in other words.’

    You are sounding more like Thomas Szasz in every new blog you post! Well done!

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      If using Catholicism as an analogy makes me Szasz, then you can call me Szasz. But unlike him I’m not going to make a career out of taking that one idea too far.

  • http://twitter.com/Keith_Laws Keith R Laws

    I have regularly expressed my disdain of this ‘technique’- it is absolutely rife in some areas of research (especially clinical psych studies – my bugbear CBT for psychosis) – Despite admitting flaws, a) the ‘seriousness’ of the flaws is never acknowledged b) the studies become uninterpretable clutter flying around in science(http://keithsneuroblog.blogspot.co.uk/2013/02/satellites-of-love-uninterpretable.html) c) authors say- “shouldnt we publish this then? are you saying we shoudl hide it? put in file drawer?” NO!- run it properly!! and d) worse still these pointless studies are used to make claims for cash to run the kind of study they should have run in first place – editors please get a grip!

  • http://twitter.com/infinidiv infinidiv

    But once again it is not the general idea of “confessing” to limitations, it is how it is done. If major limitations are mentioned to explain why a certain conclusion isn’t drawn from the study, then that makes sense. If minor limitations are mentioned to explain what future studies should pay attention to (especially for limitations that become clear during analysis), then that makes sense as well. If the limitations section is randomly written with serious and minor issues and no real conclusions are drawn from these (i.e. a catholic confession), then we have a problem.

    It is not if or what religion you are a part of, it is your actions that matter in the end. And since we all do it we have to be aware of that fact so we can reduce its influence, instead of saying this is all bullshit, because then you are just as likely to confess without realizing you are doing it.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Certainly, Limitations can be done well and ought to be. But I would rather have no Limitations Section and judge the paper for myself, than have a bad one. And I feel that science as a whole might be better without them.

      • Wouter

        Under the same line of reasoning, we might be better off when we quit writing scientific papers altogether.

  • http://www.facebook.com/franck.ramus Franck Ramus

    As your post admits, Limitation sections are bad only to the extent that reviewers and editors don’t do their job properly. But if there is a major flaw in a study, just reject the paper! And if there are more minor but important limitations that the authors fail to acknowledge, well have them do it! It’s your responsibility as a reviewer/editor.
    On top of the already mentioned positive aspects of Limitation sections, as an author I like to use them to reply in advance to predictable (though misguided) criticism. That helps convince readers and spares some arguments after publication.

  • Pingback: DISQUS-ed May 17, 2013 ✪ uneulv

  • DS

    Mentioning limitations and why you think the limitations do not undermine the conclusions of the work is important. But if author can’t do this then the work should not be published. Ideally the author would halt the publication for fear of looking foolish.

    Also I prefer an assumptions section rather than a limitations section.

  • Pingback: The Trouble With “Limitations” In S...

  • Pingback: maillot france

  • Pingback: The Peer Reviewer's Dilemma - Neuroskeptic | DiscoverMagazine.com

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »