Peer of myself

By Razib Khan | August 19, 2012 4:40 pm

Dr. Joe Pickrell has a follow up to his widely discussed post on updating scientific publication for the 21st century. One section jumped out at me, not because it was revolutionary, but because it made explicit a complaint that I had often heard:

The solution to this problem relies on a simple observation–in my field, I am completely indifferent to whether a paper has been “peer-reviewed” for the basic reason that I consider myself a “peer”. I do not think it extremely hubristic to say that I am reasonably capable of evaluating whether a paper in my field is worth reading, and then if so, of judging its merits. The opinions of other people in the field are of course important, but in no way does the fact that two or three nameless people thought a paper worth publishing influence my opinion of it. This immediately suggests a system in which papers are posted online as soon as the authors think they are ready (on so-called pre-print servers). This system is the default in many physics, math, and economics communities, among others, and as far as I can tell it’s been quite successful.

The reality is that often the “peers” are not peers. How else to explain the publication of the longevity study in Science, now retracted? Or the non-canonical RNA editing? (presumably this is less common of a problem in specialized journals). And sometimes the feedback of peers can indicate that they don’t really know what they’re talking about. For example, I was once told that the authors of a phylogenetics paper which used Bayesian methods were asked to reanalyze their data with a max likelihood framework (jump to the last sentence of this section to see why this is peculiar).

The theory of classical peer review made sense in the pre-internet age. But now there are a plenty of reasons why we might need to revisit this.*

* Not to mention that “peer review” is a somewhat subjective concept. Richard A. Muller has gotten into a back & forth on this issue whether his latest work has undergone peer review. He claims it has, others claim not. I suspect most traditional biologists would be skeptical of Muller’s claim, but physicists would accept it.

CATEGORIZED UNDER: Science
MORE ABOUT: Science
  • DK

    presumably this is less common of a problem in specialized journals

    Percentage-wise, there is less bullshit in specialized journals. But there is still a lot of it there. Notably, the percent of bullshit that draws attention and is dealt with in decisive fashion is definitely higher in glamour mags.

  • Emma

    It depends on what you call “my field” and why you are reading a paper for. Personnally I also think that I am able to evaluate the quality of a paper that deals with my very specialized subfield. But as soon as a paper is even slightly outside my specialty, I would be unable to detect anything else than very obvious errors. For example, I read many ecological papers that use (relatively!) sophisticated statistical analyses, and I usually have not the slightest idea whether they are appropriate given the data set and the question asked.

    And also, peer review helps determine the importance of the science. Given the number of published papers, it is nice to have some filtering.

  • http://www.genomesunzipped.org J Pickrell

    Hi Emma,

    No serious scientists has ever thought about a paper in their field, “Oh, it’s been peer-reviewed, so it must be right”. We all read papers carefully if the results are important to us.

    Yes, filtering is important. Posting preprints happens usually at the same time as submission to a journal. If you don’t feel comfortable evaluating preprints, only read journals (which are filtered).

  • http://www.genomesunzipped.org J Pickrell

    One additional comment on the “filtering” performed by journals. In population genetics, I have to skim the titles of papers in Nature, Science, PNAS, Cell, Current Biology, PLoS Genetics, PLoS Biology, etc., etc., etc. on the off chance that a population genetics paper might be published there. Most issues there’s not, so I skim over titles like “Estimating the Microtubule GTP Cap Size In Vivo” when these papers are not anything like what I’m looking for. How is this even remotely a useful filter?

    I would much prefer a “population genetics” feed of all popgen papers, so even if I’m skimming over the titles of paper that don’t seen interesting, at least they’re vaguely relevant.

  • Emma

    Again, I think that you are right for papers that are in our core area of expertise. But most papers I read are outside this small area and I am not able to really estimate their scientific merit, though I sometime want to use their results. So I really need a system to told me “yes, this paper is (probably!) correct”. It might not be Peer review, perhaps a “like” button or comments would work, but I need some kind of evaluation.

    I think Mendeley group feature might be efficient for “tagging” new papers in a given field like population genetics

  • zendokan

    arxiv works in physics, because there is much stronger mechanism than peer-review:

    Industry

    You cannot publish results on new technical concepts or effects without someone else reproducing this results. Look at the case of Jan Hendrik Schön. Scientific misconduct in laboratory physics is impossible without ruining your repitition in the long term. It will come out, because industry and universities are hunting for patents and any new important scientific improvement will be hardly tested.

    In humanities and social sciences this mechanism, which is much stronger than peer-rewiew, doesnt exist at all. Psychological Study discussed by neuroskeptic recently showed this again.

    http://neuroskeptic.blogspot.co.uk/2012/08/is-poker-game-of-skill-or-luck.html

    You can basically publish what you want. See Sokal-affair. The peer-rewiew doesnt guarantee more than the question is related to current research in a field, the methodology or assumptions might be completely flawed.

    If peer-review is not really able to identify flawed papers written by researchers employed by a university, than you have to ask, whats the merit at all and why not publish on pre-print servers. The plagiarism cases in Germany showed that in humanities peer-review is anyway a fata morgana.

  • kirk

    It seems like the subject here is the mechanics of peer review. What is the accepted measure of the status of a finding via references to published prior art? In other words additional work that builds on earlier work is stronger than editorial review.

  • Jason Malloy

    I would much prefer a “population genetics” feed of all popgen papers, so even if I’m skimming over the titles of paper that don’t seen interesting, at least they’re vaguely relevant.

    Someone should start a website that does this automatically. Papers could be mechanically tagged and sorted through keywords. I already have Google Scholar send me email notices when certain papers get cited, or when key terms are used in new papers (e.g. “Flynn Effect”).

  • Chris_T_T

    This is a problem in computational biology where you often have physiologists without any kind of math background reviewing complex mathematical models.

  • Chad

    Save Pubmed searches and have them send you results on a regular basis (daily, weekly, etc). I do this on many subjects of interest.

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Gene Expression

This blog is about evolution, genetics, genomics and their interstices. Please beware that comments are aggressively moderated. Uncivil or churlish comments will likely get you banned immediately, so make any contribution count!

About Razib Khan

I have degrees in biology and biochemistry, a passion for genetics, history, and philosophy, and shrimp is my favorite food. In relation to nationality I'm a American Northwesterner, in politics I'm a reactionary, and as for religion I have none (I'm an atheist). If you want to know more, see the links at http://www.razib.com

ADVERTISEMENT

See More

ADVERTISEMENT

RSS Razib’s Pinboard

Edifying books

Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »