Secondly, if you’re familiar with the field of GWA studies, you’ll realize their p-values are ridiculously and suspiciously low (~10^-6 – when in biology do you ever see that?). The reason for this is that GWA studies are built around predictability. A typical p-value for prediction, not training, lies in the 10^-3 to 0.05 range. Ignoring this fact, the authors of the study base their idea that random gene signatures are associated with prognosis on the fact that they can train data sets to cluster without testing them. Those familiar with machine learning know that you can train anything out of any data set you want – the real challenge is finding significance in a separate data set. The over-sensationalized nature of the paper is simply demonstrating this fact. They don’t use any sort of validation or testing sets. The p-values they present are training p-values, not testing p-values. All of their nice, pretty p-values will probably disappear if you applied their trained signatures on a second data set. This was really the main flaw of the paper.

All in all, their main contribution is in the idea that one should use a more rigorous hypothesis testing on gene signatures in that one should show that one’s signature should outperform 95% of random signatures – but in separate testing sets, not in the training set.

http://www.biomedcentral.com/1471-2407/9/214

And the point about dogmas of academic publishing highlights the importance of the open science movement. Looking forward to reading more of your posts on F1000!