The F Problem With The P-Value Sciences

By Neuroskeptic | October 16, 2013 5:00 pm

There is a problem in science today. I’ve written a lot about how to cure it, but in this post I want to outline the nature of the disease as I see it.

The problem goes by many names:

So I’m going to call it the f problem for short.

I like to visualize f as a forking path. Given any particular set of raw data, a researcher faces a series of choices about how to turn it into a ‘result’.

There are choices over which statistical tests to run, on which variables, after excluding which outliers, and applying which preprocessing… and so on:

f_neuroskeptic_science

The f problem is that researchers can try multiple approaches in private, and select for publication the most desirable ones.

Most often, it’s statistically significant effects, that match with prior hypotheses, that are desired. Even if there are no real effects of interest in the data, some comparisons will be ‘positive’ just by chance.

Researchers today face pressure to publish ‘good results’ and are rewarded for doing so – this is what turns f from a theoretical concern into a real one.

Yet f is not a problem for all of science. Broadly, f only affects research in which the results take the form of p-values. Thus fields like mathematics, where results are proofs, are immune.

But even some p-value sciences manage to escape f. This is because in these fields, the nature of the enterprise means that ‘everyone knows’ what experiments are being carried out, and how the data ought to be analyzed – in advance.

In particle physics, for example, it was public knowledge that CERN was looking for the Higgs Boson, and it was also known exactly how they planned to look for it in the data. Openness meant that was no room for f, because f is all about the scope for hidden flexibility.

CATEGORIZED UNDER: FixingScience, media, select, Top Posts
ADVERTISEMENT
NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+