Complex science is very hard

By Razib Khan | February 11, 2013 8:24 pm

Three articles which illustrate the difficulty of the sort of science which tackles what Jim Manzi would term phenomena characterized by high causal density. First, the simplest one is the report that extrapolating from some mouse models to human biological systems may be problematic. Anyone who has talked to human geneticists who use mouse models is aware that these inbred lineages can be somewhat particular and specific. Order the wrong mice, and all of your experimental designs might be for naught. So the result is not surprising, but it seems useful to have it documented in such a concrete fashion (though this has been reported in the media before).

Second, a long piece in The Chronicle of Higher Education on the problems in replicating ground breaking research in the area of priming. This may be a case of a robust result which turns out to fade into irrelevance as time passes, and illustrates the fundamental problems of attempting to do sciences on humans; we’re diverse and protean. I think the jury’s out on this, and we’ll wait and see. Fortunately this probably won’t be an issue we’ll be debating in 10 years, as replications will start to occur, or, they won’t.

Finally, a moderately scathing review in The Wall Street Journal  of the book Blindspot: Hidden Biases of Good People. Here’s the final paragraph:

There is far from a consensus about the IAT—a meta-analysis, you might say, is overdue. It turns out that the authors themselves published one in 2009, reviewing 184 independent samples and nearly 15,000 experimental subjects. The result: The IAT was very weakly correlated with other measures, failing to account for more than 93% of the data. Interestingly, Ms. Banaji and Mr. Greenwald don’t report this in their book [the authors of Blindspot]. Perhaps a blind spot?

You surely know about IAT, the Implicit Association Test. You’ve probably even taken a test online, purporting to measure your bias against particular groups (I have). But here is where “inside knowledge” counts. Years ago a friend who was a cognitive psychologist told me privately that he and many others within the field were very skeptical of the utility of these tests to predict anything of substance, even though they were media friendly. This individual has a good track record, as he was the one who alerted me to the serious problems with Jonah Lehrer’s work as far back as 2006.

Does this mean that you should ignore all science which derives from attempting to infer associations in domains where complexity is the rule? Not at all. But caution is warranted. The reality is that these are the areas where we as humans need to go to discover novel and powerful patterns. But because these are often social or medical domains which have immediate real world consequences we need to be methodologically sound, and not jump the gun. And, unfortunately, excessive immediate and early attention in the media is probably a very bad, perhaps negatively correlated, proxy for how solid a given result will be in the long term.

MORE ABOUT: Complexity
  • Ron Strong

    Reasoned arguments probably have little impact on how people evaluate the validity of tests such as IAT. Political viewpoint on issues such as race and affirmative action almost certainly correlate more closely with acceptance or rejection of such tests than the degree to which one has a solid grasp of the science, statistics and practice surrounding the development and administration of the IAT.

    There’s also likely to be a very strong inverse relationship between attitudes on IAT and IQ tests, in spite of the fact that each is an attempt to measure a complicated aspect of cognition.

    People who accept the IAT – a recently developed test by a small number of researchers that tries to ferret out attitudes the subjects are trying to hide – will tend to reject IQ tests – tests that have been worked on by thousands of scientists over more than a century and that attempt to elucidate something (intelligence) the subjects want to display.

    • razibkhan

      please don’t use these forums to just editorialize. it’s pretty boring, and i might just ban you :-)

      • Riordan

        If there are any changes in the comments policy, then I think its advisable if they are posted on the main page in big, red, capital letters, stating your expectations and what’s allowed and not allowed. Even if there are no changes, currently there’s no disclaimer of that kind anywhere on this page, while the old blog at least had it on the margins. Otherwise it’ll be a waste of time for people to write and post their opinions and for you to read them and then proactively ban them.

        • razibkhan

          long time readers know the standards. i usually warn people if they are being stupid, grinding axes, etc. and in general it is usually easy to figure out who to ban. so it is the commenter who is possibly wasting time. but if you leave a good comment, that won’t be an issue.

  • Robert Ford

    “Is it peer reviewed?” and “cite sources” have become the new cool thing to say in internet comment debates but that’s, unfortunately, become almost meaningless to me. I don’t care if its been reviewed if it was reviewed by people who already agree with the conclusion or inferences of the paper. You’re left to your own good or bad judgment when trying to sift through a site like sciencedaily because *everything* gets published now. There’s some AMAZING stuff and then there’s stuff published by Dacher Keltner:)


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar