fMRI and False Positives: A Basic Flaw?

By Neuroskeptic | May 5, 2016 1:11 pm

Over the past few years I’ve covered (1,2,3) the work of Anders Eklund, a Swedish researcher who has discovered a potentially serious flaw in software commonly used to analyse fMRI data.

Eklund has shown that popular parametric statistical analysis tools for fMRI are prone to false positives – they often ‘find’ brain activation even where it doesn’t exist. The issue affects the leading software packages such as FSL and SPM.

One main root of the problem is spatial autocorrelation – the fact that the fMRI signal tends to be similar (correlated) across nearby regions. Spatial autocorrelation is a well known phenomenon and fMRI software tools have systems for dealing with it, but Eklund and his colleagues says that these fixes don’t work properly.

Specifically, the problem is that the software assumes that the spatial autocorrelation function has a Gaussian shape but in fact it has ‘long tails’, with more long-range correlations than expected. Ultimately this leads to false positives.

longtail

Now, Eklund has written to me to say that he’s just learned that the long tail problem was actually noticed nearly a decade ago, back before he began to work on the issue. In 2008, neuroscientists Kriegeskorte et al. published a paper showing long-tailed autocorrelations in fMRI from the human brain and – remarkably – also in fMRI data recorded from a dummy brain, called a phantom.

Kriegeskorte et al.’s phantom was simply a sphere, the size of a human head, full of water. The fact that the phantom data was long-tailed – albeit less so than real brain data – proves that the phenomenon is not driven by neural activity or any biological process. As Kriegeskorte et al. pointed out, it must be a fundamental property of the fMRI scans. However, they didn’t discuss the implications of their findings for false positives in fMRI studies.

untitled

Unfortunately, the 2008 paper appeared in a fairly obscure journal, and it’s fair to say it was ignored by the neuroscience community – it has been cited just 4 times. But it looks like we should have paid more attention.

What can we do to overcome this problem? Eklund and colleagues have argued that one solution would be to move towards non-parametric statistical analysis of fMRI data. Software to implement this kind of analysis has been available for a while, but to date it has not been widely adopted.

ResearchBlogging.orgKriegeskorte, N., Bodurka, J., & Bandettini, P. (2008). Artifactual time-course correlations in echo-planar fMRI with implications for studies of brain function International Journal of Imaging Systems and Technology, 18 (5-6), 345-349 DOI: 10.1002/ima.20166

ADVERTISEMENT
  • Pingback: Missed MRI Critique – Rainy Streets()

  • David Burton

    In normal NMR there are two types of line shapes, Gaussian and Lorentzian. The Gaussian shape only appears in solid state spectra, whereas the Lorentzian shape, which has longer tails, appears in liquid state spectra. Seems somewhat simplistic, but couldn’t the software packages simply change the shape to Lorentzian in order to account for the tails?

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Thanks!

      The shape of the spatial autocorrelation function is not (directly) related to the line shape of the actual NMR spectra.

      So it might be that a Lorentzian would be a better approximation than a Gaussian (and eyeballing it, it might well be), but this would be more or less a coincidence

      • Jeffrey Weimer

        A Lorentzian is narrower at the half-width and broader in the tails. The spectrum shown is insufficient to decide for or against this because it does not show the symmetry of the peak on the left side. In any case, my eyeballing of the peaks suggests, it is a Gaussian with a tail not a Lorentzian (or even a Voigt which is a Gauss + Lorentz convolution).

        As a spectroscopist, I must say, I am surprised by what I interpret as the “fixes” being proposed here–keep the Gaussian peak shape but change the statistical correlation methods. I would suggest, first modify the peak shape function in the code to include a tail component. This will provide one additional parameter as an output from the fit of the model function to the true data. Then, consider what physical insight the new “tail height” parameter brings to the statistics.

      • David Burton

        Thanks for the reply. I would be interested to hear if it turns out to make any difference.

  • Matt

    I have never been a fan of these studies, but the head of NIDA is a huge fan so they get tons of funding. Do you have any thoughts on this study?

    http://archneur.jamanetwork.com/article.aspx?articleid=2516854

  • Johan Carlin

    Just to be clear – this issue in this ms concerns false positives arising from inadequate cluster-based multiple comparisons correction. You’ve written the piece as though it concerns fMRI ‘false positives’ generally. Many fMRI studies would not be affected by this issue, eg, ones that use region of interest-based analyses (it might however have the effect of making regions seem too similar – but this is the opposite of what we’re usually interested in showing in region of interest studies), or ones that used voxel-based multiple comparisons correction.

    By contrast, the first eklundh paper is about false positives in single subject analysis arising from inadequate correction for -temporal- correlation in the data. But an important caveat here is that group analyses would not be sensitive to this issue (provided that a different trial sequence is used in each volunteer).

    There are real and important issues here but it’s good to be precise about the scope of the problem.

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      Thanks for the comment. Yes, that’s right. But note that in Eklund et al.’s most recent work (preprint), they looked at higher level (group) analyses and found that the false positive inflation affects them too: which they concluded “calls into question the validity of countless published fMRI studies based on parametric cluster-wise inference.”

      • Johan Carlin

        That’s right. Issue 2 (spatial autocorrelation) affects any study that uses parametric cluster correction methods, whether single subject or group. Issue 1 (inadequate correction for temporal autocorrelation with AR1 models) is probably more of an issue for single subject analyses in most cases.

        A final observation is that the errors in multiple comparisons correction resulting from Gaussian assumptions do not always lead to overly liberal results. In the recent paper Eklund confirms previous findings (see e.g. work by Nichols and colleagues) that -voxelwise- multiple comparisons correction based on gaussian random fields produces overly -conservative- results. See their Fig 3.

        So fMRI researchers are stuck between a rock and a hard place when it comes to doing multiple comparisons correction of brain maps. You can go voxel-wise and be overly conservative, or cluster-wise and be overly liberal. Permutation tests seem to get closer to the intended false positive rate (and if configured properly can solve issue 1 above as well), but most mainstream analysis packages do not provide easy access to such tests at present.

  • adc50

    This is all horribly reminiscent of the “famous” dead salmon study . . .

    http://blogs.scientificamerican.com/scicurious-brain/ignobel-prize-in-neuroscience-the-dead-salmon-study/

    The authors almost won a Nobel prize for it . . . !!

    See, also chapter 4 of Reinhart A (2015) Statistics Done Wrong. San Francisco, CA: No Starch Press.

    And then there are arguments such as found in . . .

    Why We (Usually) Don’t Have to Worry About Multiple Comparisons J Res Ed Effect, 5: 189–211, 2012.

    Not to mention those which are even stronger . . .

    And, by the way, little if anything is actually Gaussian in biology . . .

    The preferential distribution is typically Gamma . . .

  • MoreInput

    There is some work lately on statistical models like Non-parametric temporal modeling of the hemodynamic response function via a liquid state machine: http://www.sciencedirect.com/science/article/pii/S0893608015000891

    I wander if this is the future of general statistics as well.

  • Pingback: IFM 5/6 – Disruptive Paradigm()

  • Jonathan

    FDR control with BH has no such problem.

  • Pingback: Morsels For The Mind – 06/05/2016 › Six Incredible Things Before Breakfast()

  • Pingback: “fMRI and False Positives: A Basic Flaw?” - Mad In America()

  • Pingback: False-Positive fMRI Hits The Mainstream - Neuroskeptic()

  • Pingback: A deep flaw has been discovered in thousands of neuroscience studies — Quartz()

  • Joe Masters

    Are there any implications of this for studies of ‘functional connectivity’ which use fMRI data?

  • Pingback: Homepage()

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+