Help I’m Being Regressed To The Mean

By Neuroskeptic | August 24, 2010 5:30 pm

“Regression to the mean” was the bane of my undergraduate statistics class. We knew that it was out there, and that the final exam would have a question about it, but no-one understood it or had ever seen it. A bit like unicorns or fairies.

The lecture notes were unhelpful. They told us what it did – make things wrongly appear to change over time when actually stuff stayed the same – but not what it was. Some people claimed to get it, but they couldn’t explain it to others.

I now see that our mistake was in thinking that there’s some thing called “regression to the mean”. There isn’t. It’s just a rather unhelpful term for what happens in a certain kind of situation, and once you understand those situations, there’s nothing more to learn.

Suppose there’s a number, which varies over time, and at least some of this variation is random. It could be anything from the number of sunspots to rates of cancer. You get interested in this number whenever it gets very high (or very low). Whenever it does, you start tracking the number for a while. Maybe you even try to change it. You notice that the number always seems to be falling (or rising). Why?

Because you only get interested in the number when it’s, by chance, unusually high. The chances are, the next time you look at it, it will be lower: not for any interesting reason, or because “what goes up must come down”, but just because if you take an unusually high number and then generate a new number at random, it’ll probably be lower. That’s why the first number was “unusually high”.

Suppose that you take some people and give them an IQ test twice, a week apart. Call the first test “X” and the second test “Y”. Suppose it’s a crap test that gives entirely random results. Here’s what might happen if you gave the test to 100 people, with each dot a person:
There’s no correlation, because X and Y are both random junk. Nothing to see, move along. But wait a second…
Here’s X, first test score, plotted vs Y-X i.e. the change in score between the first test and the second. There’s a strong negative correlation: people who did well on the first test tended to get worse, and people who did badly, tended to improve. Wow? No. This is a purely statistical effect. It’s meaningless: the “correlation” exists only because we’re correlating X with itself (in the form of Y-X).

It’s a fundamental mistake, and it’s obvious when you look at it like this, yet it’s a surprisingly easy one to make without noticing. Imagine that you’d invented a pill that you think can make people smarter. You decide to test it on “stupid people”, because they’re the ones who need it most. So you give lots of people an IQ test (X), select the worst 10%, and give them the drug. Then you re-test them afterwards (Y). Whoa! They’ve improved! The drug works!

There’s only one stupid person involved in this experiment.

This remains true, even if the IQ tests aren’t entirely random. A test that measures real intelligence will also have an element of luck. By selecting the bottom 10% of scores, you’re selecting people who are both unintelligent and unlucky when they took the test. They’d have scored 11% if they were lucky. So the same problem applies, albeit to a lesser degree.

That’s really all there is to “regression to the mean”. The regression of high or low scores towards the mean score is inevitable, given our definition of “high” and “low” scores, to the extent that scores are random. This is why I said it’s unhelpful to think of it as a thing. The trick is being able to spot it when it happens, and to avoid being mislead by it. If you’re not careful, it can happen anywhere.

Interestingly, the reason why it’s thought of in this unhelpful way is probably because the “discoverer” of regression-to-the-mean, Francis Galton, misunderstood it. He observed this “effect” in some data he’d collected about human height, and he wrongly interpreted it as a real biological fact about genetics. Eventually, people noticed the statistical mistake, but the idea of “regression to the mean” stuck, to the dismay of undergraduates everywhere.

Link: This was inspired by a post on Dorothy Bishop’s blog, Three ways to improve cognitive test scores without intervention.

CATEGORIZED UNDER: methods, statistics, woo
ADVERTISEMENT
  • http://www.franchise-info.ca michael webster

    This is a very helpful explanation, and the example is useful.

    Your example of the “illusionary correlation” Y-X with X is very good.

  • http://www.blogger.com/profile/06135295618666454002 saunterer

    Wonderful post. I still can't believe one can write a blog and keep great posting almost on daily basis ;-). I find your blog fascinating and your writing style impressing.

  • http://racehist.blogspot.com n/a

    It's both (biological and statistical).

    Regression to the mean in height is not due to random errors in height measurements. It occurs because unusual heights result from unusual combinations of genes (+environments) and the children of outlier parents will tend to have more average genes/environments.

    Measurement error (and therefore test-retest regression to the mean) is potentially more of an issue with IQ than height, but there's still “real” regression to the mean for the same reason there is with height.

  • http://hierosolyma.wordpress.com veri

    OMG jibber jabber. That stupid one will probably be me in the future. Lucky I read your post.. aggression and mean is stupid, scribbling it on my arm.

  • Tony

    People really have trouble with this content in college? I remember when I was a kid I was very into baseball and very into stats because of baseball. I remember learning this at age 16-17. You would normally think of it like this: If someone say, hits, .500, with the league average being .200, chances are, he will regress to the mean next season.

  • Tony

    Regression to the mean in height is not due to random errors in height measurements. It occurs because unusual heights result from unusual combinations of genes (+environments) and the children of outlier parents will tend to have more average genes/environments.

    Yes, but the means by which these genes combinations occurred was random within their population

  • Dan K

    The most intuitive way I've found to explain RTTM is this: When you collect a noisy measurement, your measurement is either an overestimate or an underestimate of the true value. The larger (or smaller) the value you observe the more likely it is to be an overestimate (or underestimate).

  • http://www.blogger.com/profile/07645993508473592961 Kaj Sotala (Xuenay)

    This is the clearest explanation of the concept that I've seen so far. Thank you.

  • http://people.umass.edu/jhenning/Justin_Henningsen Justin

    I've seen this in my own repeated-measures data sets. Does anyone have a nice reference (esp. in the biology literature) that I could cite?

  • Douglas Knight

    Does “regression to the mean” need a name? How about “a stupid prediction,” as in “Because the parents' height is a stupid prediction for the children, the children of tall parents are not as tall as their parents”?

    Would something like this, emphasizing prediction, solve the problem?

    I got here from less wrong

  • http://www.blogger.com/profile/06647064768789308157 Neuroskeptic

    It's not just a bad prediction, it's a particular kind of bad prediction. Parent's blood type is a bad predictor of child height too, but for a different reason.

    However I don't think “RTTM” should be named, I think the situation in which it occurs should be named. Something like “Overpredicting from Random Values”. I hope that would make it less confusing.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+