It is the annual Edge “question.” This year, “What Scientific Concept Would Improve Everybody’s Cognitive Toolkit”? It looks like Edge finally updated their design a touch! (happy Chad?) Nothing too fancy, which is probably a good thing. I haven’t read all the answers, but I would submit that everyone needs to digest John Ioannidis’ insight, Why Most Published Research Findings Are False:
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
This general issue has long been obvious to many researchers, but Ioannidis’ brought a greater statistical clarity to the problem.
Razib Khan’s degrees are in biochemistry and biology. He has blogged about genetics since 2002, previously worked in software development, is an Unz Foundation Junior Fellow and lives in the western US. He loves habaneros.
May 12th, 2011 at 6:07 am
“What Scientific Concept Would Improve Everybody”s Cognitive ToolKit?”
1) Making humble pie taste good
2) Magic bricks that gravitate towards TV’s
3) Invisible carnivores that only eat stoopid people.
I tried to read the real answers but everybody was trying to act profound and clever. I gave up when one blue ribbon intellectual started his soliloquy with “watching professional wrestling”
May 12th, 2011 at 8:18 am
At business school (at a well-respected, major research university) one of my profs prided herself on being the most “scientific” of our professors. That was her reputation and she liked to brag about how rigorous and “scientific” her classes were. What her approach consisted of was to demand that any claim made by a student be backed with a citation in the literature. That was it. You can imagine what kind of science gets produced in MBA literature and the quality of studies used to prove or disprove various management fads.
(She was also a huge proponent of using Myers-Briggs test to make hiring and staffing decisions. I consider Myers-Briggs to be one step up from astrology personally.)
Anyway, it was scientism at it’s best. And, given the non-science backgrounds of a typical MBA class, maybe that’s all that can be hoped for — to get them to look past just the management fad books at Borders. I don’t know. Still, it was an odd style. Any analysis without a citation was dismissed as just personal opinion, while any citation, regardless of the quality of the paper, carried gravitas. An in-class discussion was like a debate on scripture — except each side was quoting scripture from holy books that nobody else in the room had ever heard of before that moment.
May 12th, 2011 at 9:41 am
If this is true in medicine, just think how true it must be in economics and other social sciences. In academics the rewards can be enormous.
May 13th, 2011 at 12:45 pm
My answer: Statistics. Especially the fact that you can’t generalize from one datum.
May 13th, 2011 at 4:09 pm
I wish many scientists would internalize these:
“The easiest person to fool is yourself.”
or
“Everyone is biased, especially you.”
May 13th, 2011 at 6:04 pm
[...] Comment of the week, in response to “Edge question 2011″: At business school (at a well-respected, major research university) one of my profs prided herself [...]
May 15th, 2011 at 11:40 pm
[...] (from genomics to biomarker self-monitoring) may finally allow us to push past the problem of false positives which John Ioannidis [...]