Two Manifestos for Better Science

By Neuroskeptic | January 11, 2017 4:46 am

fixing_science

Two new papers urge scientists to make research more reproducible.

First off, Russ Poldrack and colleagues writing in Nature Reviews Neuroscience discuss how to achieve transparent and reproducible neuroimaging research. Neuroimaging techniques, such as fMRI, are enormously powerful tools for neuroscientists but, Poldrack et al. say, they are at risk of “a ‘perfect storm’ of irreproducible results”, driven by the “high dimensionality of fMRI data, the relatively low power of most fMRI studies and the great amount of flexibility in data analysis.”

Regarding sample sizes and statistical power, for instance, Poldrack et al. warn that despite a trend for increasing sample sizes in fMRI studies in recent years, “in 2015 the median study was only sufficiently powered detect relatively large effects” as their graph shows.

poldrack

The average modern fMRI study has 80% statistical power to detect an effect with Cohen’s d effect size of about 0.75. Poldrack et al. show that a typical task-evoked fMRI effect size is smaller than this, suggesting that “the average fMRI study remains poorly powered for capturing realistic effects.” Which suggests that many of the positive findings reported may not be genuine ones.

hcp

Potentially an even bigger problem is the undisclosed flexibility in fMRI data analysis, which creates the potential for p-hacking, as first highlighted by Joshua Carp in 2012 (I also raised the issue myself.) Poldrack et al. say that the solution to this problem is to adopt the “pre­registration of methods and analysis plans” so that readers can know which analyses were only conceived of once the data had been collected.

Neuroskeptic readers will know that I’ve long been an advocate of preregistration in neuroscience and elsewhere.

Poldrack et al. discuss several other issues facing neuroscience, many of which I’ve blogged about down the years, such as the flaws in certain fMRI analysis tools and researchers failing to use multiple-comparison correction. They make many sensible recommendations as to how to fix these problems.

They conclude that

It is likely that the reproducibility of neuroimaging research is no better than that of many other fields in which it has been shown to be surprisingly low. Given the substantial amount of research funds that are currently invested in neuroimaging research, we believe that it is essential that the field address the issues raised here

Meanwhile, in a paper published yesterday in Nature Human Behaviour, Marcus R. Munafò and colleagues present A manifesto for reproducible science. Munafò was also an author on the Poldrack et al. paper.

Munafò et al. address many of the same issues as Poldrack et al., such as flexible methods, p-hacking, and publication bias in favor of positive results, although they take a broader perspective, considering the problems with science as a whole rather than specifically neuroimaging research. Like Poldrack et al. they recommend preregistration as a solution to many of these problems. Munafò et al. also discuss open access, data sharing and adherence to reporting guidelines such as the TOP guidelines.

munafoMunafò et al. conclude that

Reproducible research practices are at the heart of sound research and integral to the scientific method. How best to achieve rigorous and effient knowledge accumulation is a scientific question; the most effctive solutions will be identified by a combination of brilliant hypothesizing and blind luck, by iterative examination of the effectiveness of each change, and by a winnowing of many possibilities to the broadly enacted few.

Both of these papers offer a comprehensive guide to the problems plaguing the modern scientific process. Ten years ago, hardly anyone was talking about these issues, and five years ago, they were only beginning to be discussed. It’s fantastic that so much attention is now being given to these problems and to how to solve them.

Yet it remains to be seen whether it will be possible to actually implement the necessary reforms on a large scale. There are many very exciting practical initiatives such as the Open Science Framework (OSF) and Registered Reports, which show that a better scientific process is possible, but to date only a minority of scientists have participated in these programs.

Also, it’s notable that both of these pro-reproducibility papers appeared in journals owned by the prestigious Nature Publishing Group (NPG). Does this signal that NPG is going to lend its weight to the cause?

I hope it does, but some scientists are skeptical of this kind of thing. In response to Nature‘s recent promises e.g. to promote replication studies, one PubPeer commenter warned that the journal “cannot and will not keep those promises, because of editorial and corporate conflicts of interest.” Another PubPeer-ite asks whether reformist editorials from journals such as Nature are “a disingenuous marketing ploy to absolve themselves of any responsibility” for causing the problems in the first place.

I am not so cynical, but it’s true that it will be easier said than done to implement these reforms.

ResearchBlogging.orgPoldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM, Munafò MR, Nichols TE, Poline JB, Vul E, & Yarkoni T (2017). Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature reviews. Neuroscience PMID: 28053326

Marcus R. Munafò, Brian A. Nosek, Dorothy V. M. Bishop, Katherine S. Button,, Christopher D. Chambers, Nathalie Percie du Sert, Uri Simonsohn, Eric-Jan Wagenmakers,, & Jennifer J. Ware and John P. A. Ioannidis (2017). A manifesto for reproducible science Nat Hum Behav

CATEGORIZED UNDER: FixingScience, fMRI, papers, select, Top Posts
ADVERTISEMENT
  • Chris Chambers

    Nice post, and thanks for the coverage of our article.

    In response to the understandable skepticism about NPG walking the reproducibility walk rather than just talking the talk, it’s worth noting that Nature Human Behaviour also launched Registered Reports yesterday — see http://www.nature.com/articles/s41562-016-0034 — and the editorial by Stavroula Kousta is explicit in welcoming replication studies, noting that for RRs, as with other article types, “novel studies and high-value replications have equal priority for publication”.

    To the best of my knowledge, this is the first concrete commitment (and mechanism) introduced by an NPG journal to explicitly support replications, and Nature Human Behaviour is also the first NPG journal to accept studies for publication before results are known. This is pretty clear evidence of walking the walking in my view!

    • http://blogs.discovermagazine.com/neuroskeptic/ Neuroskeptic

      That’s awesome about Registered Reports in NHB! Maybe one day soon we’ll even see them in Nature itself!

  • James C. Coyne

    I share your skepticism about the commitment of Nature Communications to open science and reproducibility. However, very shortly Nature’s Science Reports will exceed PLOS One as the largest outlet for open access articles. PLOS has experienced a number of administrative difficulties and now they’ve hired away the senior editor of nature communications. Stay tuned.

  • http://www.mazepath.com/uncleal/qz4.htm Uncle Al

    make research more reproducible” You just killed macroeconomics, psychology, climatology, sociology, penology, political science, religion…government studies, advertising…and 50 years of quantum gravitation.

    The Protestant Reformation ruined humanity by making it fact-based measurable by money, causing affluence and recursive improvement. Put sand in the gears not lube – leviathan bureaucracies of the coercive State.

    • ben

      Of course. Humanity was going along swimmingly before the Protestant Reformation, and now the world has gone to hell…

      • http://www.mazepath.com/uncleal/qz4.htm Uncle Al

        The Protestant Reformation brought forth desirable Hells: exceptionally better for contributors and pretty good for parasites – clean hot and cold tap water (except for Flint, MI), flush toilets, hygiene, electricity, communication, transportation, air conditioning, entertainment; dentistry.

        Catholicism delivered 2000 years of promises and profane images. Eastern religions did longer but not better. “Make research more reproducible” lest we rot within leviathan bureaucracies of a coercive state imposing dogma as tests of faith:
        The irreversible momentum of clean energy
        DOI: 10.1126/science.aam6284 PubMed: 28069665
        The author is a gibbering simpleton.

        • jrkrideau

          Ah yes, Galileo was a Lutheran was he not?

  • Pingback: 01/11/17 – Wednesday’s Interest-ing Reads | Compound Interest-ing!()

  • https://osf.io/qthsf/ David Mellor

    Thanks for continuing to these topics and issues. In regards to “…but to date only a minority of scientists have participated in these programs” we are actively working to spread awareness through education campaigns such as the Preregistration Challenge ( https://cos.io/prereg ) and Open Practice Badges ( https://cos.io/badges ). I just wanted to make sure readers see the initiatives that are taking place to change the current practices.

  • Pingback: Lectuur op zaterdag: 2 x google, 2 x manifesten en te veel testen (en meer) | X, Y of Einstein?()

  • http://www.mazepath.com/uncleal/qz4.htm Uncle Al

    arXiv:1701.01219 By empirical measure, “neuroscience” largely overlaps with happenstance rather than prediction. This places it much closer to religion and macroeconomics than something predictive and useful.

  • djlewis

    The deeper problem, of course, is that efforts toward better processes and studies will simply eliminate a lot of the weaker ones — there just won’t be as many positive results. And in the neuroscience of higher order mental phenomena — cognition, affect, etc — there may be very little left; the whole field could evaporate, or at least shrink back to a small fraction of its current weed-like growth.

    Poldrack would survive that because he is at the top of the heap. He understands the issues — for example invalid reverse inference, explicit or insinuated — and has made a specialty to see wha can be done to overcome them. But most researchers in this area are thriving on the bad science and would be out of jobs.

  • LVS

    I always enjoy reading your posts – especially around replication and the options open to science. Would you be available to give a webinar on the topic? We’d preserve your anonymity… Thank you for considering!

  • http://www.appropedia.org/Category:MOST Joshua Pearce

    In addition it would help if all scientific hardware used was open source hardware itself – ideally able to be replicated with direct digital manufacturing. We have started on this route – but still have a long way to go. http://www.appropedia.org/Open-source_Lab#Examples

  • Pingback: The existential quest of psychological science for its soul | Pearltrees()

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+