Brain Voodoo Goes Electric

By Neuroskeptic | May 3, 2013 4:41 pm

Four years ago, neuroscientists became aware of an ominous-sounding manuscript entitled “Voodoo Correlations In Social Neuroscience”. This piece was eventually published under a more prosaic name but it still hit home, with nearly 500 citations so far.

To me, this paper marked the start of a new era of ‘critical’ (in the proper sense of thoughtful discussion and reflection) neuroscience, with fMRI researchers becoming more aware that fundamental statistics are as important as ever, despite the amazing technical advances and novel techniques of the 1990s and 2000s.

Now London neuroscientist James Kilner has reminded us that the ‘voodoo problem’ applies not only to fMRI but also to EEG and MEG, methods for measuring brain electro-magnetic activity: Bias in a common EEG and MEG statistical analysis and how to avoid it

The problem, in essence, is about selecting values out of a random population. If you apply a selection criteria to lots of random variables, and pick out only the highest (or lowest) values, then any statistical tests you run on those picked values will probably be biased because they’re selected. It sounds simple, but in a complex data analysis, it’s surprising how easy it is to select and test without realizing it.

In the case of EEG and MEG, the recorded data consists of anywhere from 20 to 250 sensors or electrodes placed around the head. It’s not clear however which sensors are most ‘interesting’ in any given experiment.

A common practice is to focus on the electrode at which the largest electrical or magnetic response (ERP) is seen to a given stimulus, but Kilner shows that this is dangerous unless care is taken to make the selection criteria independent of the subsequent analysis. Selection itself is fine, but ‘double dipping’ or ‘circular’ testing of the same things that were used as selection criteria (e.g. testing the size of the ERP at the electrode where that ERP is largest) is problematic.

ResearchBlogging.orgKilner, J. (2013). Bias in a common EEG and MEG statistical analysis and how to avoid it Clinical Neurophysiology DOI: 10.1016/j.clinph.2013.03.024

  • Annonymous

    Off Topic. You once replied in a comment: “I didn’t include all of the links to interesting 1boringoldman posts because that would have been endless!”

    I hope you build a post around this:

    In regards Dr. Insel of NIMH and his recent announcements about the RDoC:

    “Do they really think we believe that the NIMH shift to the RDoC was
    independent of the DSM-5 Task Force’s failure to live up to those
    unrealistic goals set back in 2002? All they did was get off the clock
    so they can be grandiose without accountability again.”


    “Once again, Dr. Insel is driving the NIMH like it’s his personal vehicle
    rather than supporting our best and brightest in their own scientific
    directions. I suppose that would be acceptable if he knew where we were
    going, but he clearly doesn’t…”

    This analysis of the RDoC deserves a broader audience.

    • Neuroskeptic

      I’m working on a post about the NIMH stuff and 1bom will be linked all over that baby.

  • petrossa

    Again my thoughts which i expressed to you Neuroskeptic often over the last years. So where is my pat on the shoulder finally?

  • Y.


    • Neuroskeptic

      Hey, I promised I wouldn’t write about that for a while.

      But now you have, yes, it would help. There are other, even easier solutions though (see the discussion in the voodoo/double dipping papers from a couple of years back.)

  • Pingback: 2013-05-03 Spike activity | Connecticut Neurofeedback()



    I absolutely agree with the above point, and it is clear that circular analyses can be used in many different fields.

    I would however add that a big difference between fMRI and EEG is data redundancy: many EEG sensors actually carry very similar information. In other words, EEG is probably less affected by (although not immune to) double dipping issues.


    • James Kilner

      Hi although EEG and MEG has time as well as electrodes space so the effective search space could well be larger that that of fMRI. The redundancy in fMRI is the temporal correlation.

      • Neuroskeptic

        True. I think the problem is less severe than in fMRI, but that’s not saying much, because in fMRI it’s huge. And given that it’s so easy to avoid the issue, there’s no excuse for not doing so.

        • James Kilner

          Agreed. There are no excuses. Just to say if you do time-frequency analysis on multiple electrodes then the search space is huge too.

          • Jona Sassenhagen

            Frequency band x electrode x time. Phase or amplitude. Condition differences (which opens up the choice of what conditions to compare) or difference to baseline (which opens up the choice of baseline). Not to speak of the usual EEG problems (filtering, referencing …).
            Time-frequency analysis: the p-hacking potential of fMRI, all from a cheap EEG package!

  • Marc Lustig

    Please enlighten me on the following:

    Say I am doing visual ERPs and I am interested in an early component like P1, that is well investigated and known to occur between x and y ms post stimulus. However, it is also known that P1 latency is largely dependent on stimulus contrast, with variance of up to 30ms between studies cq. stimulus-configurations (e.g., Luck, 2005).

    Now, suppose further that I have used faint stimuli, thus I expect an early P1, with an onset of, say 60ms. I collect my data and see, to my surprise, a quite late P1 onset at 90ms in the data when averaging across all conditions.

    To a less extreme degree, this is a common scenario in EEG research, but is it also what the authors are talking about? Would it count as ‘selecting the largest effect’ to shift the time-window to the actual onset, or still as ‘chosen a priori’ since I am still in the P1 time-range predicted by the literature?

    I guess the answer lies somewhere in the middle, but I would be happy for confirmation by others (and potential remedies?)

    The same holds also for the choice of electrode of course. Imagine I use a 256 electrode setup. Then I have several electrodes to choose from in the same area that is described as typical for this component in the classical literature using (often using 64 electrodes at best).

    On a different note, I have been suggested another possibility to reduce the dimensionality of the data (and bias), by using global field power instead of focussing on a single electrode ( This approach seems, however, to be primarily used in Swiss institutes. Does anyone know the reasons for that?

    • Jona Sassenhagen

      You can use a method that’s independent of your main outcome to determine parameters for your main outcome. For example, you chose the P1 peak based on a secondary task with a similar latency; or you choose the first 50% of your data sets to find the peak of the P1, and the rest to test it; or, and I think this would be the easiest way, you simply measure base-to-peak amplitude within a certain time window, in which case you don’t need to select anything too precise.
      Alternatively, I think you can usually chose the P1 window based on the mean peak latency over all conditions, and then compare conditions in that time window. That would work if the effect is a modulation of the peak, such as in P1 research. Might not work so well if you’re analysing the P3.

      Fundamentally, you need to keep separating parameters for a test and the outcomes of the test separate. The circularity happens if you use the outcome of a test to select the parameters of the test.

      Mean global field power probably isn’t used as much because it integrates a bunch of electrodes you might not be interested in and where a bunch of stuff might be happening that is, for the purpose of your experiment, noise.



No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.


See More

@Neuro_Skeptic on Twitter


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar