Edge question 2011

By Razib Khan | May 11, 2011 11:37 pm

It is the annual Edge “question.” This year, “What Scientific Concept Would Improve Everybody’s Cognitive Toolkit”? It looks like Edge finally updated their design a touch! (happy Chad?) Nothing too fancy, which is probably a good thing. I haven’t read all the answers, but I would submit that everyone needs to digest John Ioannidis’ insight, Why Most Published Research Findings Are False:

There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.

This general issue has long been obvious to many researchers, but Ioannidis’ brought a greater statistical clarity to the problem.


Comments (7)

  1. dave chamberlin

    “What Scientific Concept Would Improve Everybody”s Cognitive ToolKit?”

    1) Making humble pie taste good
    2) Magic bricks that gravitate towards TV’s
    3) Invisible carnivores that only eat stoopid people.

    I tried to read the real answers but everybody was trying to act profound and clever. I gave up when one blue ribbon intellectual started his soliloquy with “watching professional wrestling”

  2. juan

    At business school (at a well-respected, major research university) one of my profs prided herself on being the most “scientific” of our professors. That was her reputation and she liked to brag about how rigorous and “scientific” her classes were. What her approach consisted of was to demand that any claim made by a student be backed with a citation in the literature. That was it. You can imagine what kind of science gets produced in MBA literature and the quality of studies used to prove or disprove various management fads.

    (She was also a huge proponent of using Myers-Briggs test to make hiring and staffing decisions. I consider Myers-Briggs to be one step up from astrology personally.)

    Anyway, it was scientism at it’s best. And, given the non-science backgrounds of a typical MBA class, maybe that’s all that can be hoped for — to get them to look past just the management fad books at Borders. I don’t know. Still, it was an odd style. Any analysis without a citation was dismissed as just personal opinion, while any citation, regardless of the quality of the paper, carried gravitas. An in-class discussion was like a debate on scripture — except each side was quoting scripture from holy books that nobody else in the room had ever heard of before that moment.

  3. John Emerson

    If this is true in medicine, just think how true it must be in economics and other social sciences. In academics the rewards can be enormous.

  4. Matt B.

    My answer: Statistics. Especially the fact that you can’t generalize from one datum.

  5. Chris T

    I wish many scientists would internalize these:

    “The easiest person to fool is yourself.”
    “Everyone is biased, especially you.”


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com


See More


RSS Razib’s Pinboard

Edifying books

Collapse bottom bar