Beware Small Positive Studies

By Neuroskeptic | November 12, 2012 9:14 pm

A Letter in the prestigious American Journal of Psychiatry offers a skeptical response to a paper published there recently.

The original article claimed amazing benefits of a safe and cheap brain stimulation technique in treating schizophrenia. But Dutch letter-writers Sommer et al aren’t convinced.

It’s a short piece and worth quoting:

We read with interest the article by Brunelin et al. in the July issue, which described the application of transcranial direct-current stimulation (tDCS) in the treatment of both auditory hallucinations and negative symptoms simultaneously… An effect size of 1.58 was reported for refractory hallucinations, which is remarkably large when compared with the effect sizes of antipsychotic medication (0.4–0.6).

Clinical trials involving nonconvulsive brain stimulation in schizophrenia were first introduced in 1999. Initial effect sizes were very large while samples were small. Some years later, large negative studies were published. To date, 17 placebo controlled transcranial magnetic stimulation (TMS) studies on hallucinations have been published. The mean weighted effect size is now around 0.3. Yet, the negative correlation between effect size and year of publication suggests that over time, the mean effect size may become smaller.

When selective serotonin reuptake inhibitors (SSRIs) were introduced for depression, effect sizes greater than 1.0 were reported, which created their legacy as a wonder drug. Over the course of 20 years, the mean effect size of SSRIs decreased to around 0.3. A similar trend was demonstrated for cognitive-behavioral therapy.

This trend likely results from publication bias. A remarkably high effect size suggests the discovery of a new wonder treatment. Studies with such findings are therefore easily published in high-impact journals. In contrast, studies of similar sample size with marginally or nonsignificant findings are less likely to be accepted for publication. Usually, after some years, negative studies with large sample sizes become available. This is when meta-analyses start to detect a decrease in efficacy.

In this view, the Brunelin et al. study is exemplary of an initial placebo-controlled study applying a new technique: it included a small sample, found remarkably large effects, and is published in a high-impact journal.

We sincerely hope that tDCS is the exception to the rule— as a cheap, safe, and highly effective method to treat both refractory hallucinations and negative symptoms is most welcome. However, given the previous observations for other new treatments, it is realistic to expect that 10 years from now the mean weighted effect size of tDCS will be around 0.3.

Sommer et al are talking about the famous ‘decline effect’, in which the size of an effect mysteriously shrinks the more people study it, which I’ve written about before.

I suspect the authors are right in this case; playing science devil’s advocate though, it’s unfair and unscientific to assume that a new treatment that looks promising will eventually turn out to be mediocre, just because that’s happened to other treatments before.

After all, some things look awesome because they are – penicillin, for example, was heralded as a new wonder drug… and it was.

I very much doubt that tDCS is the new penicillin. But I do think that this kind of speculation is ultimately not very useful. Rather than bemoaning the errors of the past and wondering whether they’ll be repeated, we should reform science to make sure they don’t.

ResearchBlogging.orgSommer IE, Aleman A, Slotema CM, and Schutter DJ (2012). Transcranial stimulation for psychosis: the relationship between effect size and published findings. The American Journal of Psychiatry, 169 (11) PMID: 23128925

  • Anonymous

    But even penicillin ceased to be a wonder drug after a few decades.


    And whilst penicillin was a very simple molecule for a very simple problem, finding a wonder drug for things as complicated as the aberrations of what is to all intents and purposes a virtual persona (consciousness) is so improbable that one can actually state it's impossible.

  • ALRawi

    There is no problem of publishing such findings, but, not in a 12 IF journal. Above all, tDCS has been around for ages, I wonder how it got published.



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar