Zombie Papers of the Undead

By Julianne Dalcanton | August 12, 2008 7:23 am

Questions for the Day:

  1. Why is this paper to referee in my inbox?
  2. Why can’t I safely assume that if I’ve refereed a paper, and haven’t seen a revised version in more than year, that the paper is actually dead?
  3. Why, in spite of my polite-but-scathing review in 2007, have they revised nothing but the discussion?!?!?!?

The paper. It should be dead, but it lives. It walks among us. It’s trying to eat my brain. And if it makes it to the literature, it will try to eat yours.

CATEGORIZED UNDER: Academia, Words
  • ST

    I have been guilty of being on the other side of question #2. When the referee asks for an annoying but inessential revision, usually I just ignore it for O(years) and then get back to it when I have the time, patience, inclination, etc. The more you procrastinate, the more you forget the stuff you did in the paper, which adds to the motivation to procrastinate even further.

  • Mike B

    What sort of thing merits a scathing review? I’m curious, in case I make similar mistakes, but have just gotten “lucky” with non-confrontational referees.

  • http://lablemminglounge.blogspot.com/ Lab Lemming

    Perhaps the other reviewer only just sent back his scintillating “It’s perfect, publish as is” review last week, after sitting on the paper for a year.

  • mph

    Well, the lack of revisions should make it easy to re-review.

  • http://backreaction.blogspot.com/ B

    I recently refereed a paper and when I submitted the report checked a box ‘please don’t send the revised version back to me but to alternative referee enternamehere’. So I was hoping to never see that thing again, but of course it came back two months later. Makes one wonder what that option is good for to begin with.

    The most annoying thing about refereeing though I find if I get back rejected papers from other journals, typically without any change whatsoever. I mean, I usually do at least consider that the referee might have raised a valid point before submitting my paper elsewhere. (I usually decline to review in such cases, I feel like the author should get at least a second opinion.)

  • squawky

    I’m guilty of #2 as well – currently, anyway. This includes begging for extra time from the journal editor when the inevitable ‘send your revisions by X or we’ll withdraw the paper’ email — in my case, new job took up 150% of my time, and the revisions require some serious work. (The editor will be getting an email note from me saying thank you for the extra time when I finish the revisions – I’d put a note in the acknowledgments, but I’m not sure that’s appropriate.)

    You never know what the other reviewer(s) said, but it seems to me that a rebuttal to your comments is required if the revisions aren’t present. Hopefully this makes your job easier (‘authors didn’t respond to any of my comments. major revisions required.)

  • Even Stephan

    A heads up: In the September 2008 issue of Discover magazine, stem cell researcher Robert Lanza expounds on the topic of quantum physics. This blog will be expected to comment.

  • Elliot

    I’m sorry can you send your exact mailing address again? I must have misplaced it.

    e.

  • http://blogs.discovermagazine.com/cosmicvariance/julianne Julianne

    Polite-but-scathing is equivalent to “I see that in Step 2 you have neglected important-well-understood-physical-effect. Unfortunately, without considering this effect, none of the subsequent conclusions are valid.” Most rational scientists would slap their head, say “oh crap!”, and then consider the important-well-understood-physical-effect, rather than choosing to put in a sentence saying “Important-well-understood-physical-effect may also impact our results”, and then return to ignoring it for the rest of the paper.

    I would argue that taking this approach should not take a year.

  • Haelfix

    The peer review process is quasi broken as it stands, particularly when theres violent clashes of reasoning. I’ve had the displeasure to ref twol shoddy, clearly incorrect papers, and had the same thing happen..

    Nearly always, the physicist in question simply ignores the review (and probably grumbles about it to his colleagues about how unfair and incompetent reviewers are) and pawns the paper off to another publication in hopes of landing a sympathetic or less critical ear. Almost like the whole thing is some sort of chore to be dealt with.

    At best you can sometimes get the person to write down physicist jargon for mathematical fiction eg ‘we formally proceed as follows’ or something like that.

    Other typical problem. The person in question sits there and focuses his efforts on one particular puzzle ot solve, and ignores every other constraint from outside his little niche. So when you tell him that subfield x has ruled his result out long ago, they get upset. Even attempts to simply include those relevant results can prove to be difficult.

  • ST

    As an author, it is impossible to publish anything even mildly original and against the grain. As a referee, it is impossible to stop something from getting published. Thats peer-review in a nutshell.

    I should say this is the situation faced by young people in academia. As I have gotten older, publishing has become significantly easier, but stopping something that I think is crap from getting published is still impossible.

  • http://backreaction.blogspot.com/ B

    ST, I agree on the latter. It is very frustrating. I wouldn’t quite say that the peer review process is broken – at least for my own papers I’ve found it often very helpful (seriously) – but it doesn’t quite work as it should. The biggest problem I think is simply we publish way too much, which is due to pressure exerted by inappropriate measures for our ‘success’ as researcher.

  • Tyler

    I will be publishing this thread verbatim as part of my paper on the sociology of the peer review system. No edits will be considered.

  • a CMT

    I think there is an art to rejecting a paper in such a way that it stays rejected. I once had the opportunity to co-review with a bigwig in my field which resulting in a friendly, but utterly devastating review. As far as I know, the paper was never heard from again. I haven’t had much luck in my own rejections.

    I’ve also noticed that some subfields I’ve worked in result in more helpful reviews than others, both in terms of proposals and papers. The cultural differences between subfields suggest that at least part of the problem with peer review is “us” We could all endeavor to be more honest* when reviewing and when responding to reviews.

    * I’m not claiming any malicious intent. An example of what I mean by ‘honest': I’m sure everyone has had the experience of a reviewer pronouncing judgement without having carefully read the paper to see that many of their objections are already answered in the text.

  • brad

    Regarding point #2, you’ll be glad to note that the ApJ now has a new policy. Below
    is an excerpt from the standard response to submissions.

    “The Astrophysical Journal has adopted a new policy that manuscript files become
    inactive, and are considered to have been withdrawn, six months after the most
    recent referee’s report goes to the authors, provided a revised version has not
    been received by that time.”

    Baby steps….

  • Brad Holden

    Sometimes I wonder why we bother (I hope no editors are reading this….)

    I have, on my todo list, a terrible paper. However, it has been on arxiv.org for 4-5 months now. Most of my colleagues and coauthors just send the paper to arxiv.org first, and then submit to the journal. The reader has become, in effect, the referee.

  • Count Iblis

    ST:

    As an author, it is impossible to publish anything even mildly original and against the grain.

    Huh! The opposite is true as far as publishing in physics journals is concerned. The Referees will usually write a very positive report if you are original. They don’t really check the details in the article. So, if you have some nice original idea, your article will be accepted even if your detailed mathematical arguments are very sloppy :)

    In mathematics journals they also welcome original ideas, but there you can expect a Referee report in which every misplaced comma will be mentioned :)

  • ts

    In science, there’s too much pressure to put something out, even just for publicity. Sure, it’s important to produce constantly and constructively to establish your career, but it’s clearly not the case that every single paper in your field is worthy of your time perusing these days — too many papers and authors for that.

    What’s the merit of the traditional peer review process now? With the Internet, it is so easy to let people in the world see whatever you want to put in public. Only people in the know read your papers anyways, so why not let them just decide which papers are worthy of better recognition?

  • Brad Holden

    I do like (the other) brad’s new policy even though I have a coauthor who would be driven crazy by that rule….

  • brad

    Dude, I wasn’t joking. That was in the editor’s preamble to the latest referee report I got.
    I suppose he could have been gently pressuring me in particular, but I think it’s a general
    policy. Start stocking up on the prozac…

  • http://mingus.as.arizona.edu/~bjw/ Ben

    What

  • Count Iblis

    One way to make sure your preprint gets cited a lot is to submit it a few seconds after 8 pm GMT, see here :)

  • http://lablemminglounge.blogspot.com Lab Lemming

    “Other typical problem. The person in question sits there and focuses his efforts on one particular puzzle ot solve, and ignores every other constraint from outside his little niche. So when you tell him that subfield x has ruled his result out long ago, they get upset. Even attempts to simply include those relevant results can prove to be difficult.”

    Why does that matter? Multiple contradictory hypotheses are part of science. As long as every one knows that subfields X and Y produce mutually incompatible models of a particular system that are incapable of explaining each other’s data, it would be silly to reject papers that try to refine either model, as progress would never be made.

    Incidentally, this was the case for the Earth’s mantle (the solid rock portion of the planet between 35 and 2900 km depth, for non geologists here) for much of the late 20th century. Geophysical models could not reproduce the chemical constraints, and geochemical models could not explain the physics.

    But if every paper on the topic had been rejected because they didn’t explain everything, then the problem never would have been sorted out.

  • BRB

    I’m more than a little surprised by this discussion. I would have thought that for all of us who are not Ed Witten, bad refereeing is a *much* more serious problem than the fact that wrong papers get published every now and then. The bad guys in this game are not the poor authors trying to survive, but rather the retards who glance at a paper, realise that it is about some problem on which they have passed their final judgement long ago, and then write a dismissive rejection.

    Most papers on the arxiv are stupefyingly, paralyzingly boring. That, again, is much more of a problem than the appearance of wrong papers.

    Look at it this way: I see here people whining about all the bad papers they have to referee. But what does that cost you, compared with what a rejection costs the victims of incompetent referees? And who forced you to be a referee in the first place?

    When I agree to be a referee, my attitude is that I will not reject unless [a] it is very obvious that the author has nothing to say and is just padding his publication list — this is very rare in fact, or [b] I can show mathematically that there is a fatal error. I will certainly *not* reject a paper just because it conflicts with folklore about the subject, which seems to be the commonest ground for rejection, no doubt because it requires so little work.

  • http://Capitalistimperialistpig.blogspot.com capitalistimperialistpig

    Dear Sean,

    I changed the discussion because your absurd objections proved that the more slow witted readers wouldn’t see the brilliance and transcendant truthiness of my results.

    I waited a year to re-submit because I wanted to give you time to adjust to the more intellectual air of Caltech and absorb some wisdom from your desk.
    A vain effort, I see.
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    .
    A joke, of course. I have never written anything, even a parking ticket, likely to end up in Sean’s inbox.

  • http://Capitalistimperialistpig.blogspot.com capitalistimperialistpig

    Oops,

    I guess Julianne got the paper. Do you have a desk that used to belong to somebody famous?

  • ts

    anticipating a critical reading makes authors more careful

    But in principle authors do that not to satisfy the referee only, but for their work to be accepted by the community at large, right? Scientific process naturally calls for peer reviews all the time, so a paper comes under real scrutiny after publication anyways, no? What is the real advantage of centralizing the refereeing process, when information is so abundant, and most people don’t get info from books (= traditional journals) any more?

    Maybe people really only read abstract and summary these days, so at least one person in the field needs to pay attention…

  • Supernova

    @brad #15: Wow, I got lucky on that one: just resubmitted a paper to ApJ 8 months after the referee report, and it was accepted. Guess I got grandfathered in under the old policy. Now I’ll have to start timing my submissions more carefully, six months before the end of the summer. :)

  • http://mingus.as.arizona.edu/ Ben

    But in principle authors do that not to satisfy the referee only, but for their work to be accepted by the community at large, right? Scientific process naturally calls for peer reviews all the time, so a paper comes under real scrutiny after publication anyways, no? What is the real advantage of centralizing the refereeing process, when information is so abundant, and most people don

  • Fermi-Walker Public Transport

    Weird coincidence that this discussion appears just after I found the CD containing a paper I wrote several years ago and thought lost after a interstate move. A few update and another zombie paper is about to break free.

  • ts

    Maybe we could have a version of the arxiv in which people respond to papers with comments

    About a couple years ago I remember seeing arXiv implementing a comment feature of some sort, or maybe just a way to send a trackback to an article, I don’t remember exactly. It were there for maybe a few weeks and then disappeared…

    I constantly hear people complaining that anything gets published these days (despite that most probably do benefit from such lax standards); papers being rejected due to referee bias is a different problem from low-quality/unworthy papers not being rejected. That implies the traditional peer review process through journal publication is not working the way it should be, no? Yet in the end bad papers won’t survive the scrutiny after becoming public for long, or at least we hope that scientists weed them out, so why put too much emphasis on things getting “published”?

    I guess we already do that to some extend by taking into account citations.

    But If we look at the wild variation of reviews we receive from TAC/reviewers for mostly similar research proposals submitted in different years, it’s pretty clear that there really are no true standards as if science is in the eyes of beholders. Why put so much emphasis on a single person’s review then?

  • Count Iblis

    Would it be a good idea if authors included the referee report in the arXiv preprint, like in this case?

  • bane

    I work in engineering rather than science, so one difference is that there are more “constructing mathematical ideas and algorithms” that need to be motivated by what one can/should eventually be able to apply them to.

    I find the most frustrating thing is papers that aren’t truly padding but also clearly contain just enough that’s new to qualify. The end result is that you get maybe an 8 page paper where 1 page is yet another motivation for a specific task, 1.5 pages is yet another review of how previous people have tackled this/similar problems, 1.5 pages is yet another description of the well known basic mathematical setup (so that the paper is reasonably “self contained”), .5 pages is yet another description of how the experiments were evaluated and why this is appropriate, 1 page is obvious conclusions and yet another listing of references most of which are the ones everyone uses. So 5.5 pages are “duplicating existing stuff”, leaving 2.5 pages of new stuff and experimental validation. Apart from anything else, it takes time and concentration to read those 5.5 pages to be sure they are “yet another …” and there’s not something new hidden within there.

    I do wonder if there’s a better way these days to structure scientific/engineering literature these days than requiring every paper to be reasonably self-contained.

  • rusell

    bad refereeing is a *much* more serious problem than the fact that wrong papers get published every now and then. The bad guys in this game are not the poor authors trying to survive, but rather the retards who glance at a paper, realise that it is about some problem on which they have passed their final judgement long ago, and then write a dismissive rejection.

    I’m glad somebody mentioned this. There’s another side to the refereeing coin, populated by people who have families to feed…

NEW ON DISCOVER
OPEN
CITIZEN SCIENCE
ADVERTISEMENT

Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Cosmic Variance

Random samplings from a universe of ideas.
ADVERTISEMENT

See More

ADVERTISEMENT
Collapse bottom bar
+

Login to your Account

X
E-mail address:
Password:
Remember me
Forgot your password?
No problem. Click here to have it e-mailed to you.

Not Registered Yet?

Register now for FREE. Registration only takes a few minutes to complete. Register now »