Independent Discussion Sections?

By Neuroskeptic | March 20, 2019 8:19 am

Scientific papers should have two Discussion sections – one written by the authors, and the other by an independent researcher.

According to a new paper from Michael S. Avidan, John P. A. Ioannidis and George A. Mashour, this “second discussant” system could help ensure more balanced and objective inference in science.

science_warning1The authors begin by noting that while the reproducibility crisis has focussed attention on the Methods and Results sections of papers, Discussion sections are not free of problems of their own:

Some of the ongoing deficiencies of Discussion sections in medical journals include lack of contextualisation of results using systematic review, selective citation of prior studies that agree with the results, overinterpretation or misinterpretation of findings, and incomplete acknowledgement, analysis, and assessment of study limitations.

To address this issue, Avidan et al. suggest the concept of an additional expert who would write their own discussion, to be published alongside the paper, without seeing the authors’ discussion. The second discussant (I would prefer the term ‘discussionist’) would be chosen by the journal editors, and they would need to be familiar with the general field of research, but wouldn’t necessarily require detailed topic-specific expertise.

Avidan et al. then give an example of independent discussion writing in action. Avidan read the Introduction, Methods, and Results of this paper, about a recent clinical trial of the drug haloperidol. He then wrote a Discussion of the trial results, without having seen the author’s version.

It turned out that, in this case, both Discussions were pretty similar, but there were a couple of differences. In particular, Avidan’s discussion noted that the whole trial was based on a seemingly shaky assumption, one which the authors never questioned.

To me, independent discussion sections is a great idea… but I do have doubts that it would work in practice. Were every paper to need a second discussion section, scientists would be even more over-worked than they already are. Writing, revising and reviewing papers is already a huge amount of work.

Avidan et al. suggest that researchers would be motivated to contribute discussions because they would get a publication out of it:

Much like an editorialist, an independent discussant would fulfill the requirements for authorship as stipulated by the International Committee of Medical Journal Editors, so there should be also an academic incentive for becoming a good discussant and writing balanced, carefully thought Discussions.

(It’s not quite clear if they envisage the discussant being added as an author on the paper, or getting a separate publication for the discussion, but I assume the latter.)

But even if the second discussant was motivated to help, this doesn’t change the fact that there’s only so much time in the day, and that the proposed system would increase the amount of work that goes into each paper.

Perhaps the system could work if it were reserved for especially important papers, but these already tend to get independent discussions (of a kind) anyway, in the form of editorials and (ahem) blog posts…

ADVERTISEMENT
  • http://www.mazepath.com/uncleal/EquivPrinFail.pdf Uncle Al

    Civilization is smothered by carnivorous Mrs. Grundy hordes. Kill every tenth ethicist, then get on with achievement.

  • OWilson

    But where to find the “independent” researcher :)

    I think that any scientific study that impacts public policy in any way, should require the researchers to show all their data and methodology.

    That way there would not be an ongoing dispute over Mann’s infamous “hockey stick” graph, and such.

    Perhaps when a major ‘public policy affecting’ study is published they should invite comments from serious dissenters who may not agree with findings,

    A panel, like a jury, could be enpaneled from other professional walks of life, to hear the discussion and see the evidence, and rule in favor of one side or the other, like in the U.S. S. C., or shelve the matter for further study.

    I have seen this approach work well in land compensation disputes, where lawyers argue the case and relative lay panelists make the decisions.

    Not perfect, but would prevent a lot of the studies from being widely accepted, by politicians, then found to be problematic later!

    See the recent study on sea level rise, November 2018, by published in Nature, by researchers from the University of California San Diego and Princeton

    In spite of the fact that “The study was undertaken by some of the world’s most pre-eminent climate scientists, using state-of-the-art modelling systems reviewed by their peers, and appeared in one of the most prestigious academic journals”

    “There was just one problem: it was wrong” – PhysdotOrg.

    After a skeptic pointed out their errors, they admiited “We quickly realised that our calculations incorrectly treated systematic errors in the O2 measurements as if they were random errors in the error propagation,” author Ralph Keeling wrote on climatehome.org.

    “We really muffed the error margins,” he told the San Diego Tribune.

    Viva skeptics! :)

  • Pingback: [BLOG] Some Thursday links | A Bit More Detail()

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Neuroskeptic

No brain. No gain.

About Neuroskeptic

Neuroskeptic is a British neuroscientist who takes a skeptical look at his own field, and beyond. His blog offers a look at the latest developments in neuroscience, psychiatry and psychology through a critical lens.

ADVERTISEMENT

See More

@Neuro_Skeptic on Twitter

ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Collapse bottom bar
+