No comments please, we've made it through peer review (?)

By Razib Khan | March 25, 2012 12:58 pm

Recently Daniel MacArthur pointed to the vibrant discussion over at Genomes Unzipped on a moderately infamous paper from Science last year, Widespread RNA and DNA Sequence Differences in the Human Transcriptome, asserting that it is “exactly what open peer review should be like.”  This made me wonder, it’s been over five years since Chris Surridge asked why there was so much more commentary on a PLoS ONE paper, By Hook or by Crook? Morphometry, Competition and Cooperation in Rodent Sperm, on blogs than on the paper itself. Has anything changed? The most viewed paper on PLoS Biology, How Many Species Are There on Earth and in the Ocean?, has 9 comments for 45,000 article views. In contrast, Genomes Unzipped has 14 comments for likely far fewer page views. Additionally, if you find the post on the weblog the comments automatically load. Not so with the PLoS Biology paper, you have to click through (yes, I see how this can be a feature, not a bug, but in that case why even bother with comments if you provide an email address for correspondence?)


I’m not going to rehash Joseph Pickrell’s argument in Why publish science in peer-reviewed journals? Rather, I want to reiterate that the ultimate goal here is to figure out how the world works. To understand that you need to get a sense of the science, and if you aren’t a specialist simply consulting peer reviewed literature is not always sufficient or practicable. Recently a researcher was discussing the Science paper at issue above to some graduate students, reiterating Joseph Pickrell’s points. When someone was curious about further details the individual explaining the controversy specifically cited Pickrell’s earlier post on the topic in Genomes Unzipped to get an understanding of the broader issues of the critique. From what I know this person is not a major reader of science blogs. Rather, Pickrell’s post likely came up high on a search engine query, and to his mind was an appropriate reflection of the scientific consensus as he understood it. The question is why does the discussion not begin in the journals themselves? Why do curious people have to scour the internet? When a controversy or “big paper” emerges in a specific area outside of my detailed focus I know what I do: I simply load the appropriate science blogs which cover that topic. There I hope to get a sense of what the people in the field think. Peer review is not sufficient in many cases. Science is a production of humans, and so is fraught with politics, and reputations are thrown about to get a paper in “high impact” journals.

But is anything broken in the first place? Is this how it’s supposed to work, and how it’s going to work? Journals make a huge argument for their value. They charge academic libraries quite a mint (with honorable exceptions made for open access journals!). It seems that they should make at least a good faith effort at post-publication review and extension. Did you know that Nature has a comments feature? Have you ever seen a comment posted via this feature?

CATEGORIZED UNDER: Science
MORE ABOUT: Science
  • http://www.genomesunzipped.org J Pickrell

    Genomes Unzipped has 14 comments for likely far fewer page views

    ~3,500 page views. To be fair about the number of comments, several of them are from me! The other comments are of very high quality and from individuals with extensive knowledge of the (somewhat esoteric) issues. This is probably the most important thing.

  • http://www.genomesunzipped.org J Pickrell

    Also, was just looking through Nature and noticed they’ve gotten rid of comments on articles. Shows how relevant they were if no one noticed when the feature appeared or disappeared.

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    To be fair about the number of comments, several of them are from me!

    well, the comparison to PLoS ONE would have been less favorable; more page views, fewer comments. also, several of the 9 were jonathan eisen doing manual pingbacks. so i think that balances out. and i do want to emphasize that i respect PLoS for actually trying. i recall a major part of bora zivkovic’s job at PLoS was to try and foster engagement.

    individuals with extensive knowledge of the (somewhat esoteric) issues.

    this is an interesting point. the paper was not obscure, it made a big media splash. a few people actually asked me about it when it came out! (i just referred them to your blog actually) the technical details are abstruse, but at least outsiders can get a flavor for reasons to be skeptical. additionally, this may be esoteric compared to whether kim k. has butt implants, but RNA omics is the “next big thing” (e.g., RNA seq), and certainly not esoteric within biology. so i wonder if your the usage of the term ‘esoteric’ undersells the ‘impact’ of ‘insider’ conversations.

    on a different note, i’ve long wished that there was a more open and transparent discussion of multi-level selection. you can talk to researchers at the same university in adjacent fields and get a totally different sense of the ‘orthodoxy’ of the field.

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    #2, but the links are there are old papers, right? i just assumed i hadn’t mastered their awesome functionality ;-)

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    also, the irony here is that ‘web 2.0′ firms are babbling all about the ‘death of comments’ and the rise of the ‘social web,’ and the formal scientific discussion still hasn’t come to grip with the rise of comments!

  • http://johnhawks.net/weblog John Hawks

    Eric Michael Johnson and I wrote a letter on last year’s “penis spines regulatory DNA” paper in Nature (10.1038/nature09774). We went through the editorial process, got a formal reply from the authors, and then the editors decided not to publish them.

    So we posted it as a comment on the article, and the authors posted their reply. Now, when you load the article, both comment and reply are visible in abstract view, without needing a subscription. The comment comes up right with the article. I think it’s much *better* than the “formal” letter to the editor process. The only current downside is that the comments have no DOI, making them hard to cite.

  • http://blogs.discovermagazine.com/gnxp Razib Khan

    #6, well done! wish there was more of that.

  • Anthony

    I’m a regular reader and occasional commenter on your blog, and I would probably never comment directly on a journal paper outside my field. After a blogger like you has digested the paper, I can look at that, and maybe the paper, and ask an interesting question or make a comment, without being to worried that my question or comment was stupid or obvious to someone who knew the field well enough to publish in it. However, if I were to comment directly on a paper, I’d want to be sure that my comment was very directly relevant to the paper, and that I knew the field well enough that my comment wasn’t a worse-than-undergrad question for the field.

  • http://www.genomesunzipped.org J Pickrell

    i wonder if your the usage of the term ‘esoteric’ undersells the ‘impact’ of ‘insider’ conversations.

    Yeah, this is a good point. I was surprised to learn today (on looking at the Genomes Unzipped pageview stats) that my previous post on the Li et al. paper is one of the most viewed posts ever on the site. Admittedly we’re not pulling in millions of visitors a day, but still, this does indicate that “insider” conversations can be relevant and useful for people.

  • http://deevybee.blogspot.com/ Dorothy Bishop

    Thanks for raising this interesting topic.
    In the field of neuroimaging, this blogpost by Daniel Bor stimulated a great debate about methods: http://www.danielbor.com/dilemma-weak-neuroimaging/
    Lots of those commenting were top names in the field.
    One reason may be that the people who blog about articles are not normally the authors of the article, so comments seem less personal. Given that comments will usually raise some problems about a paper (after all, who wants to read comments that just say everything is great!) I suspect many people just don’t want to look like they are trying to start a fight – though they are more comfortable doing so if a 3rd party has raised issues.
    Daniel’s blogpost was precipipated by a post I did on a particular paper, but it raised a lot of more general issues, and I think this was another reason why so many people engaged.
    Comments on a blog also seem less of a big deal; commenting via a journal site somehow seems more formal and ‘official’ – my guess is people are happier about making a rapid, off-the-cuff response to a blogpost because, while it would not be good to be seen to make a mistake, it somehow seems less serious. Perhaps this is because you tend to get correctly pretty rapidly if you make a factual error.

  • http://neuroskeptic.blogspot.com Neuroskeptic

    Maybe it’s because most blogs allow comments without registration, while most journal sites do make you register.

    Even though it only takes 30 seconds to register, it raises the bar for how much effort is required.

  • Nicolas Fanget (@nfanget)

    @J Pickrell and all here, comments are still available on Nature papers, both news and research. It is little used (except for some news stories), but it is there.

    Disclaimer: I work for NPG.

  • http://noble.gs.washington.edu/~mmh1/ Michael Hoffman (@michaelhoffman)

    In my experience it takes a lot longer than 30 sec to register for a comment account on a journal site.
    Whenever I do it, I am usually sorry.

  • http://www.genomesunzipped.org J Pickrell

    Nicolas,

    Thanks for the clarification. Here’s a random article from this week’s Nature:
    http://www.nature.com/nature/journal/v483/n7390/full/nature10866.html

    I can’t find the comment section. Am I just overlooking a link somewhere?

  • Nicolas Fanget (@nfanget)

    Joseph, it is right at the bottom of the page. You can follow this link: http://www.nature.com/nature/journal/v483/n7390/full/nature10866.html#/comments

  • http://www.genomesunzipped.org J Pickrell

    Hm, ok, maybe it’s because I’m logging in through a proxy server, but I don’t get the comment page. Typing that link in directly (after going through the proxy) just gets me to the top of the page.

  • Nicolas Fanget (@nfanget)

    I’m guessing you might have to create a nature.com account first. I’m not on the web team so can’t help much further I’m afraid.

  • http://www.genomesunzipped.org J Pickrell

    Ah! Got it, thanks. You have to be logged in to see the comment section.

  • RF

    Another perspective: other people working in the field may wish to keep their views and criticisms and publish them themselves, rather than ‘giving them away’ on a public site…

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com

ADVERTISEMENT

See More

ADVERTISEMENT

RSS Razib’s Pinboard

Edifying books

Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »