The power of false positives in behavior genetics

By Razib Khan | November 23, 2011 9:57 am

Genomes Unzipped, Size matters, and other lessons from medical genetics:

Smaller studies, which had no power to detect these small effects, were essentially random p-value generators. Sometimes the p-values were “significant” and sometimes not, without any correlation to whether a variant was truly associated. Additionally, since investigators were often looking at only a few variants (often just one!) in a single gene that they strongly believed to be involved in the disease, they were often able to subset the data (splitting males and females, for example) to find “significant” results in some subgroup. This, combined with a tendency to publish positive results and leave negative results in a desk drawer, resulted in a conflicted and confusing body of literature which actively retarded medical genetics progress.

An easy thing to pick on is the reliance on “p-values,” thresholds of statistical significance. Just because something is statistically significant doesn’t mean that it is substance. Statistical significance is just a number, and blindly adhering to a numerical standard in most human endeavors often results in a creeping bias and “gaming” of the measurement. There’s going to be a random distribution of p-values, and for publication you just need to fish in the pool below the 0.05 threshold. It just goes to show that you can’t beat taking a step back, and actually thinking about what your results mean and how you came to them.

(as indicated in the post, this is a problem in many domains, probably most worryingly in medical and pharmaceutical studies)

CATEGORIZED UNDER: Science
MORE ABOUT: Statistics
  • Justin Loe

    My perhaps excessively lengthy comment:
    I had one of my own project’s results fall apart when we extended the sample after the pilot study. The problem at that time with MRI studies was that the cost was too large to get samples larger than an n=30. Unless the research group has an exceptional amount of resources, I believe that’s still an issue. I also recall a conference in 2002 in which an NIH statistician gave a very critical presentation on the statistical methods (or lack thereof) of the MRI community in general. Ideally, the sample sizes would be much larger and a PhD statistician would be a co-author who would have reviewed the study design thoroughly. In reality, in many studies with human subjects, recruitment issues and high costs result in low sample sizes. But the pressure to publish, and the science public relations machine pulls out the headline result without the caveats that the results have (1) not been confirmed and (2) there are other contradictory results out there. Obviously, there isn’t a reward in the tenure process for publishing negative results. My personal opinion is that there should a requirement that the data used in the analyses be available once the paper is published, as well as access to the lab in question to discuss the results. The reality is that the journals have frequently failed to uphold rigorous statistical methodology, particularly in epidemiology. These factors, among other reasons, are why many people when they see a new paper associating gene x with disease or phenotype y regard the result as provisional until perhaps 5 or 6 confirmatory papers or published (or more).

  • http://theunsilencedscience.blogspot.com/ nooffensebut

    “Statistical significance is just a number”

    Yes, and that works both ways, and liberal academics have been applying a double standard based on which studies they want to believe. Political correctness is abandoning the argument that race is just a social construct in order to argue that genes behave differently in different “races” because evidence is mounting in behavioral genetics of racial disparities in allele frequencies of potent behavioral variants. Therefore, studies like Widom and Brzustowicz are heavily cited by liberals like Steven Pinker for showing that genes like MAOA affect white violence, but not “non-white” violence, even though MAOA is on the X chromosome, and the study heavily weighted the racial groups’ gender ratios. Ed Yong has attacked the role of OXTR in the behavior of Asians because a study in Koreans seemed to show that culture could override one such behavior, but the link between OXTR and the more extreme case of autism was established in Chinese people. So, one study in Koreans generalizes to “Asians” but not a study of a more obvious outcome in Chinese people?

    There is also a new bias towards assuming that a “gene-environment interaction” affects a behavior rather than accepting the possibility that the main gene effect is real but too weak to achieve significance without the additive effect of a second variable. Avshalom Caspi has argued that studying such interactions is responsible science that disproves racism because it shows that considering environment is always a necessary ingredient. However, the same results have been achieved by considering gene-hormone interactions or using a genetic index of multiple genes. When some still doubt that the gene does anything, why should people who critique or report on these studies so readily accept that the effect requires an “environmental trigger”?

    Plus, there is a drug bias. In the case of MAO, inhibitors were the first antidepressants, which were in heavy use for two decades. They have fallen into disuse due to side effects. If there is such a need for skepticism about whether the gene has an effect, how is it that psychotropic drugs are so widely used? Drugs like tobacco also affect MAO levels and, therefore, psychiatric outcomes, which is a finding not met with skepticism. The answer is not to take away useful drugs but to stop being self-righteous about psychiatric genetics.

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    Ed Yong has attacked the role of OXTR in the behavior of Asians because a study in Koreans seemed to show that culture could override one such behavior, but the link between OXTR and the more extreme case of autism was established in Chinese people. So, one study in Koreans generalizes to “Asians” but not a study of a more obvious outcome in Chinese people?

    where? don’t be lengthy or rambling, just point to the post and section. i am curious.

  • Justin Loe

    Briefly, another example:
    a review of 1374 studies in schizophrenia
    ” These genes [732 genes] had poor statistical power to detect genetic effects typical for human diseases, assessed only 3.7% of genes in the genome, and had low marker densities per gene. Most genes were assessed once or twice (76.9%), providing minimal ability to evaluate consensus across studies. ” http://www.ncbi.nlm.nih.gov/pubmed/21854684

    It’s not too surprising of course that scientists are skeptical and that this problem is very widespread.

  • http://theunsilencedscience.blogspot.com/ nooffensebut

    Here he covered the study Kim et al, highlighting the non-significant finding that the gene had the opposite effect in Koreans compared to whites. We can quibble about whether he really generalized to all Asians, but the two studies he cited by Kim’s team did.

  • razib

    Ok. I read that post when it came out. A normal person wouldn’t have characterized that as an attack. Be more careful with idiosyncratic usage of description. I wasted some time looking for a genuine attack. (this comment is not an opening for discussion)

  • http://theunsilencedscience.blogspot.com/ nooffensebut

    How about Pinker’s statement?

    “For now, the Warrior Gene theory is staggering around with possibly fatal wounds.”

    Attack or no? Context matters. Ideas have consequences. The genetics commentariat could use some intellectual diversity.

    How’s this for moderate consideration:

    “The story of MAOA is the perfect case study for how gradual revelations about the tango between genes and environment can be translated into unconvincing applications and overplayed interpretations. There is no better example of the dangerous state of modern behavioural genetics, no better poster child for how to miscommunicate, misinterpret and misuse genetic discoveries.”

    – Ed Yong

  • Dan

    The GxE literature is a great example of the selective p-value phenomenon. My colleagues provided a good review:

    http://ajp.psychiatryonline.org/article.aspx?volume=168&page=1041

  • http://theunsilencedscience.blogspot.com/ nooffensebut

    “My colleagues provided a good review”

    Speaking of selection bias, this study singles out 5-HTTLPR but makes no mention of MAOA. Could this be the reason?

    “Phenotype in cGxE had to be DSM-IV diagnoses or closely related constructs”

    Was violence a “closely related construct”? Did they substitute antisocial personality disorder for violence? Based on my review of the literature, MAOA seems to affect aggression, but perhaps not enough of the other APD criteria. We should not blindly accept the DSM in studying phenotypes. Why should dissociative identity disorder be in the DSM, but not impulsive violence? What is the point of having intermittent explosive disorder in the DSM, when it receives so little research and clinical attention? Am I focusing too much on the MAOA example? It has been studied now for over 20 years (10 for its cGxE), and Duncan and Keller suggest that “most or even all positive cGxE findings in psychiatry discovered to date represent type I errors.”

    Here is another potential selection bias in Duncan and Keller’s offerings of evidence:

    “genome-wide association studies have largely failed to replicate reported associations from the candidate gene literature”

    So what proportion of the candidate gene literature studies SNPs and CNVs compared to VNTRs, which GWAS are currently incapable of addressing?

  • http://blogs.discovermagazine.com/gnxp razib

    “The story of MAOA is the perfect case study for how gradual revelations about the tango between genes and environment can be translated into unconvincing applications and overplayed interpretations. There is no better example of the dangerous state of modern behavioural genetics, no better poster child for how to miscommunicate, misinterpret and misuse genetic discoveries.”

    better candidate. but it’s not what you were pointing to, so that’s why i questioned you since i was pretty sure i’d read ed’s OXTR stuff. don’t test my patience. you consistently ramble on and on about things which i’m aware of as if it should be a revelation (i assume you know that i know james lee and steve hsu as correspondents and acquaintances, and was aware that they were collaborators when you were pointing steve to james’ paper).

  • http://theunsilencedscience.blogspot.com/ nooffensebut

    I do not believe in the primacy of language over ideas. “Evil communications corrupt good manners.” Though without invective, the substance of Yong’s argument about OXTR (and that of Kim et al), which he repeated three other times, was very much in keeping with the Widom/Brzustowicz strategy to parry genetic evidence based on a poorly supported gene-race/ethnicity interactions. If we accept the Yong-Kim point of view, and a drug comes on the market relating to OXTR, doctors might feel persuaded not to prescribe to Asians. Political correctness can be reverse-reverse discrimination, but you need not call the strategy an “attack,” if you prefer.

    Honestly, I did not know who knew whom, and I think all sorts of smart people read your blog, so they deserve to know what I mean, even if it is obvious to some, but I shall strive for brevity (too late?). I am not interested in personally slighting anyone.

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    but I shall strive for brevity

    if you’re going to repeat yourself, that’s probably best.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com

ADVERTISEMENT

See More

ADVERTISEMENT

RSS Razib’s Pinboard

Edifying books

Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »