fMRI Gets Slap in the Face with a Dead Fish

By Neuroskeptic | September 16, 2009 2:10 pm

A reader drew my attention to this gem from Craig Bennett, who blogs at prefrontal.org:

Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction

This is a poster presented by Bennett and colleagues at this year’s Human Brain Mapping conference. It’s about fMRI scanning on a dead fish, specifically a salmon. They put the salmon in an MRI scanner and “the salmon was shown a series of photographs depicting human individuals in social situations. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.”

I’d say that this research was justified on comedic grounds alone, but they were also making an important scientific point. The (fish-)bone of contention here is multiple comparisons correction. The “multiple comparisons problem” is simply the fact that if you do a lot of different statistical tests, some of them will, just by chance, give interesting results.

In fMRI, the problem is particularly severe. An MRI scan divides the brain up into cubic units called voxels. There are over 40,000 in a typical scan. Most fMRI analysis treats every voxel independently, and tests to see if each voxel is “activated” by a certain stimulus or task. So that’s at least 40,000 separate comparisons going on – potentially many more, depending upon the details of the experiment.

Luckily, during the 1990s, fMRI pioneers developed techniques for dealing with the problem: multiple comparisons correction. The most popular method uses Gaussian Random Field Theory to calculate the probability of falsely “finding” activated areas just by chance, and to keep this acceptably low (details), although there are other alternatives.

But not everyone uses multiple comparisons correction. This is where the fish comes in – Bennett et al show that if you don’t use it, you can find “neural activation” even in the tiny brain of dead fish. Of course, with the appropriate correction, you don’t. There’s nothing original about this, except the colourful nature of the example – but many fMRI publications still report “uncorrected” results (here’s just the last one I read).

Bennett concludes that “the vast majority of fMRI studies should be utilizing multiple comparisons correction as standard practice”. But he says on his blog that he’s encountered some difficulty getting the results published as a paper, because not everyone agrees. Some say that multiple comparisons correction is too conservative, and could lead to genuine activations being overlooked – throwing the baby salmon out with the bathwater, as it were. This is a legitimate point, but as Bennett says, in this case we should report both corrected and uncorrected results, to make it clear to the readers what is going on.

CATEGORIZED UNDER: bad neuroscience, fMRI, funny, science, voodoo
  • dearieme

    Pretty good: I suppose that tests with a defunct salmon are more conclusive than tests with the proverbial stunned mullet.

  • http://www.blogger.com/profile/01612983399048948138 cheryl chow

    So what does the dead salmon say about human emotions? Was it able to guess correctly?

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Well, unfortunately the behavioural data are not reported, possibly because the Salmon was not fluent in English, but the fMRI data suggest it could tell the difference.

    As we neuroimagers like to say “the neural data are more sensitive”

  • http://www.blogger.com/profile/17955468379668664875 prefrontal

    This is a great summary of our research – thank you for the kind words. If anyone has any questions that I can answer just post them below. I will keep an eye on the comments and try to answer any that come up.

    Best ~ Craig Bennett [Prefrontal]

  • http://www.blogger.com/profile/07936109007395575446 Matt

    Will there be a followup with a group analysis? Most fMRI papers report group results.

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    Matt – That was one criticism of the salmon poster, that single subject data may not be evidence enough to justify statistical correction in group fMRI data. It's a valid point. However, the magnitude of the multiple comparisons problem is the same in both individual and group fMRI data – they each have the same number of voxels under investigation. So, each has the same number of chances to find significant results out of simple noise. In this light the results of the salmon can speak to both individual and group fMRI results.

    Scanning a group of salmon is not a bad idea though! I'll have to chat about it with my co-authors. Have a catchy title in mind?

  • http://www.blogger.com/profile/07936109007395575446 Matt

    Voxels that school together, pool together? I don't know…

    As to your point re noise, I'm on board in principle, but I'm not sure the data would comply. For you to get the result in one fish, it suffices that a few voxels reasonably near each other in the brain-pan have time courses correlated with the stimulation. For you to get it in a school of fish, that has to happen in each fish, *and* in a location that's very similar across fish. Definitely not impossible; intuitively a lot less likely. (Also hard to investigate since, to my knowledge, the MNI hasn't gotten around to creating a standard salmon brain atlas yet. Maybe you know an eager marine biologist who'd like to go down in imaging history.)

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Hi Craig, thanks for dropping by! I feel a bit bad because everyone seems to be linking to my post rather than your original poster. But you'll be glad to know it's getting a lot of traffic (2,500 views so far…)

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    Matt – You are definitely correct, it's possible to achieve the same result with a group of fish but perhaps less likely. That is actually one of the issues I am investigating here in my postdoc. We are examining how much variability there is between the individual results that underlie group fMRI results. It turns out that there are large, stable differences between individuals on almost any task. Further, most thresholded individual results may not look similar to the group map at all. You can check out some of the lab's work at: http://tinyurl.com/lh5lql

    Also, if you know any marine biologists looking to create the first MRI stereotaxic space for the salmon do give them my email address. Depending on what the journal reviewers say we might have to go ahead with the group analysis!

    N'Skeptic – No worries at all. Looking at my weblog stats I think people are finding the poster just fine through your site. The whole point of the poster (and the forthcoming paper) is to raise awareness of the multiple comparisons issue. Any method by which that can take place is more than fine with me.

  • http://www.blogger.com/profile/09277940177708996676 Hob

    Neuroskeptic: I linked to this post instead of to the original poster just because the poster is a plain image file with no context. If Craig would embed it in a web page, it'd probably get more links. I really liked your commentary too, though.

  • http://www.blogger.com/profile/16848066611904465368 drbuzz0

    That's pretty funny, in a way. I would have loved to see a video of the experiment.

    “the salmon was shown a series of photographs depicting human individuals in social situations. The salmon was asked to determine what emotion the individual in the photo must have been experiencing.”

    I'm imagining a dead salmon in an MRI machine and some grad student or research assistant standing in front with a clip board and saying “Okay, now what I have here are a series of pictures and I want you to look at each one carefully and then describe in your own words what kind of emotion you think the person or persons in the image are experiencing. I realize you may have some difficulty relating to some, as these are humans and you are… a dead fish…. If you're comfortable, then we'll begin the experiment now”

  • http://www.blogger.com/profile/06915155210764783727 G

    I'm wondering if the issue could be resolved if the salmon was smoked and served with cream cheese…

  • Anonymous

    A few tweeks and this would have been published. For example it is unusual to get images from a dead subject. Also with a live salmon you could get more sophisticated behavioral data (e.g. tail swipe etc). Keep up the good work, the world needs more fMRI studies like this one to cool down the rhetoric.
    See:
    Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The Seductive Allure of Neuroscience Explanations. Journal of Cognitive neuroscience, 20(3), 470-477.

  • Anonymous

    A few tweeks and this would have been published. For example it is unusual to get images from a dead subject. Also with a live salmon you could get more sophisticated behavioral data (e.g. tail swipe etc). Keep up the good work, the world needs more fMRI studies like this one to cool down the rhetoric.
    See:
    Weisberg, D. S., Keil, F. C., Goodstein, J., Rawson, E., & Gray, J. R. (2008). The Seductive Allure of Neuroscience Explanations. Journal of Cognitive neuroscience, 20(3), 470-477.

  • Anonymous

    I am wondering if drawing attention to potentially serious issues like the problem of multiple corrections in fmri in such a way has any merit? After all these issues are been raised and addressed extensively; at the very least our current knowledge is years ahead of the issues raised by this “study”.

    I don't think scientist should take themselves too seriously and being critical is important, but criticism should also be done responsibly, no? Who is the target audience for this?

    I guess one can simply chose farce and ridicule to RE-make points, which can be quite entertaining, but lets not pretend it's something else…

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    Anonymous – You raise a good point. Our target audience for the Salmon is the set of researchers who currently publish neuroimaging results without using multiple comparisons correction. The percentage is higher than you might think. For some journals the percentage of articles with uncorrected results approaches 50%. The more major neuroimaging journals like NeuroImage and HBM are quite a bit better about requiring correction, but even they are not 100%. That is the gap that we wish to address with the Salmon data. It is not meant to be a technical discussion regarding why correction is necessary, but a salient commentary on why all neuroimagers should be using correction with their results.

    In short, the poster/paper is farce, but it has a serious point that speaks to a legitimate statistical issue in neuroimaging publications. We did seek to keep it as a discussion among neuroimagers, but the conference poster was picked up by weblogs around the internet and has now become a point of public debate. That is part of the reason I am continuing to address comments on the work across two dozen weblogs.

  • Anonymous

    So then is this the salmon of doubt?

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    I agree with Craig, the point here is not that multiple comparisons correction is a new idea – Friston's papers on GRFT came out in the early 1990s – rather it's an old idea which is still not as widely adopted as it should be.

    In which case I think a stunt like this is probably the best approach. The arguments for using MCC (at the very least alongside uncorrected analysis) have been around for 15 years, but still not everyone uses it, so it's good to bring the matter to everyone's attention. And what better way to do that than with something like this?

    As for the fact that it's been picked up by blogs, I think this is no bad thing either. So far the coverage has been fairly measured and has mostly avoided the kind of blanket anti-fMRI stuff that you sometimes get.

    On a more general level though I do think it's important that people are exposed to the debates occurring within the fMRI field. A large and growing proportion of neuroscience uses fMRI and this is research of great interest to the general public (it's paid for with their taxes, usually, and it's directly relevant to their lives in terms of mental illness, etc.) So I think they need to know that all is not 100% well in the fMRI world.

  • Anonymous

    Hello Neuroskeptic,

    I guess that is exactly my concern, why should a “stunt” change behavior more than informed discussions of the alternatives?

    From what I can understand (since you admit there is no scientific value to the “study”), the idea is to “shame” people in doing what you think they should do?

    On the other hand, I agree that fMRI research could improve on many levels and that activations in a fish are quite funny. It's also a flashy thing to show on a blog…

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    Anonymous – The goal of our commentary was not to 'shame' people into using multiple comparisons correction, but to educate them. There are a huge number of people conducting neuroimaging research and not all are aware that multiple comparisons correction should be standard practice. While I accept that the salmon commentary has all the trappings of a stunt, it does serve as a salient, memorable reminder of why correction needs to be used. If the salmon causes even a handful of researchers to begin using correction in their fMRI investigations then it has served its purpose well.

  • http://www.blogger.com/profile/08911259276999062492 Michael

    Are we getting closer to the time when a sufficiently large room with enough monkeys and enough typewriters will transcribe “Romeo and Juliet”?

  • http://www.blogger.com/profile/07537969766155042555 Michael Karnerfors

    Asking a dead fish to rate emotions?

    I smell a certain scent of dead fish emanating from the upcoming Ig Nobel prize ceremony. The quesion is if the salmon will be doing the acceptance speech. :D

    /Michael

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Michael – I think the fish fully deserves a reward. It managed to stay perfectly still inside the fMRI scanner, which is more than can be said for many human volunteers.

    Craig – I think this is the key issue, whether it will change behaviour. And I think it has a good chance of doing so. As you say, a lot of people are just unaware of the issue and this has certainly brought it to a few people's attention. If even one paper is improved by the fish, you'll have helped the field.

  • http://www.blogger.com/profile/07537969766155042555 Michael Karnerfors

    Holy coincidence Batman, the Ig Nobel prize ceremony was less than 24 hours ago!

    …and the fish didn't make it. Darn…

    /Michael

  • http://www.blogger.com/profile/08691592398578153731 bbf

    I can't resist. How about “Does going to school increase brain activation in Atlantic Salmon” for the group analysis?

    Blaise

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    It's a good thing the salmon was wild rather than someone's personal domestic salmon. If it had been you'd have needed to use a PET scan.

  • http://www.sports-picks.biz/ Sportspicks

    This is a great summary of our research – thank you for the kind words. If anyone has any questions that I can answer just post them below. I will keep an eye on the comments and try to answer any that come up. sports picks

  • Anonymous

    Nice – and is that an old IRC Reference?

  • Anonymous

    The only aspect that argues against attributing this result to multiple comparisons is the clustering of active voxels. The activation pattern should be random if its simply the result of multiple comparisons coming up randomly. I think the data were manipulated in some systematic way to get this result. Since there are no correction algorithms or models for a salmon object, this could have easily happened. I can't imagine how the investigators registered all the scans for such a relatively small, flat object. The study should be repeated with a cadaver. Since we have methods to analyze a brain shape, the results should essentially be randomly distributed across the dead brain, just as they are when we study a living brain. After studying fMRI and conducting my own studies, the defect is not in the method, its in brains interpreting the results. The great, overriding error is when people uninformed about the method believe that positive fMRI findings are evidence supporting what is essentially a structural model of a disorder or behavior. The fMRI method has more in common with EEG than it does with MRI.

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    Anonymous: The clustering could be an artefact of the smoothing applied to the data. This will tend to turn random noise into patches of noise.

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    @Anonymous – You seem to have the assumption that if the results were 'random noise' then false positives would be evenly distributed throughout the volume. This is never really the case, as a wide array of factors are driving signal variation across the entire set of voxels. False positives can cluster either due to random measurement error, technical (gradients, head coil, etc), or methodological (interpolation, smoothing, etc) reasons. As for correction algorithms, none of them require knowledge of the object/animal being scanned to properly operate. With regard to registration, only spatial normalization requires an organism-specific atlas space. Simple realignment of all functional scans can be done in the original, native acquisition space.

  • Anonymous

    This is a brilliant study. It is not a surprise that it was not published because it showed the massive flaws in so much trash research published based on expensive fMRI machines.

  • http://www.blogger.com/profile/17955468379668664875 Prefrontal

    Actually, just a few points of clarification. The study has been published now, technically twice. Further, I strongly feel that fMRI is an incredibly useful tool that we use to understand the brain. Our hope is that the salmon discussion makes the science better in the end.

  • http://www.blogger.com/profile/08151742418242360421 JadedIdealist

    Yep.
    Multiple comparisons are fine..
    IN A PILOT STUDY.
    Then you go and get new data for the positives and see which ones are real.
    What you don't do – Daryl Bem this means you – is publish your pilot work as though it was final work without the follow ups.

  • http://www.sessionaltherapy.com London Counselling

    Pretty scary that you can get posisive results from a dead fish. Makes you wonder about how valid hi-tech medical tests are and the qualifications of those who calibrate, give and read these types of tests. Good reasons to always ask for a second apinion.

    • http://www.twitter.com/lhyzz Elhyzzabeth

      You have misunderstood the article.

  • Anonymous

    So what happened to the fish after the experiment? This could be the start of an entirely new field of study: The effect of fMRI on the Cooking Characteristics of Atlantic Salmon. The possibilities are endless: baked vs grilled, pan fried vs poached, etc. Not to mention side dishes and condiments. Who says science isn't rewarding?

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »