The New York Times has an article that reviews the problems with peer review. I don’t know what to think, as it has something of the “and the other side says” air to it, never really coming to any conclusion. There’s a lot of shoddy crap being published in stuff like the Tuvan Journal of Entomology, and always has been, but the big issue is when crap gets into Science and Nature. Ultimately science is a social enterprise based on trust and long term self-correction. There’s a lot of noise in the system, but I don’t see any other alternative out there.
One issue that I’ve been wondering about is that in many of the biological and social sciences many new mathematical and computational techniques are being pushed really quickly, and I’m not totally sure if there’s enough comfort with these methods for the scientists to utilize them without error and the peers who check up on them to catch those errors. Sometimes it can be pretty embarrassing. A precursor to what I’m alluding to was the whole Mitochondrial Eve fiasco, where the original team wasn’t totally comfortable with the new phylogenetic techniques and made some serious methodological errors. A phylogeneticist even told me that in the original paper there was a mistake where the length of a particular branch of the tree was conflated with genetic distance when in reality the tree only reflected the bifurcation into clades, the “long branches” being an artifact of presentation by the program.
Update: Brown gaucho has a lot more commentary.
Update II: Janet has more.
Razib Khan’s degrees are in biochemistry and biology. He has blogged about genetics since 2002, previously worked in software development, is an Unz Foundation Junior Fellow and lives in the western US. He loves habaneros.
May 1st, 2006 at 10:33 pm
To put a quantitative face on how unreliable peer review is, see this book by Dean Simonton, searchable at Amazon. Search for “peer review,” pp 84-91 I think.
But like you said: what else is there? It’s like artistic production — there may be an initial unfounded buzz about a bogus product, but over time the duds die and the “sleeper” hits become well known.
May 1st, 2006 at 10:47 pm
It’s the same in every kind of creative field, whether science, the arts, music, literature – 95% of everything produced is “crap” (unoriginal, obvious, wrong, whatever). The problem is of course knowing what is crap and what isn’t. The obviously bad stuff is, well, obvious, but a lot of it isn’t, and it takes time for our collective filters to sort it all out. At the time, tight-fitting clothes in orange polyester was not obviously a lousy idea, after all, and neither was the concept of an aether, or of Lamarckian inheritance.
Peer review is like fashion magazines or book reviewers – the obviously good and the ridiculously bad will get processed correctly; for the important in-between stuff, what counts is not review but replication and longevity. If the book is still being read twenty years on, filmed, televised and being emulated by newer authors it probably really was good no matter what the review said. And if a piece of research has been replicated numerous times and is still being cited, directly or indirectly, a generation later, it was good.
May 1st, 2006 at 10:50 pm
one unfortunate aspect of the “95% of stuff is crap” is that the lay press tends to publicize the grand positive claims, but not express interest in the later retraction or lack of follow up….
May 1st, 2006 at 11:06 pm
The lay press also publicizes (and publishes) much of that 95% in writing, music and so on as well. It’s not something that anyone can do anything about – the press itself least of all – since, after all, you don’t know if it is crap or not at the time.
To the extent that this is a problem particular to science (and I honestly don’t think it is), the only possible solution is to route around it; a direct solution does not exist. If we as scientists feel unconfortable with one of our little ideas possibly ending up on a newspaper or tabloid page, then perhaps we should cool down and discuss it further informally with our colleagues before putting pen to paper.
In fact, thinking about it probably the big contributing cause has nothing to do with the media – it’s the rush to publish at any cost. Is it good? Is it finished? Is it fully checked? Who cares – publish! We may get scooped, we need to show project progress and we certainly need those extra entries in our CV a lot more than we need to actually be correct. Ok, I’m exaggerating a bit, but I think it is a contributing factor.
May 1st, 2006 at 11:26 pm
While editors and reviewers may ask authors for more information, journals and their invited experts examine raw data only under the most unusual circumstances.
i can’t even imagine trying to look back at someone’s raw data during peer review. otoh, it would be prudent for journals to require disclosure of raw data at publication so that these things can be examined by everyone. having biomedicine switch to a public pre-publication system like http://xxx.lanl.gov/ would probably help also. it works for physics.
May 2nd, 2006 at 12:54 am
My own experience with peer review has been that it is all but worthless, except in one respect (and this may actually be why it’s worthless): it causes researchers to self-edit, and it’s not uncommon for researchers to anticipate reviewers’ objections and run additional studies. In some cases, that might mean superfluous studies that waste everyone’s time, but in many cases, it means crucial pieces of empirical arguments are added.
May 2nd, 2006 at 2:15 am
One thing about peer review is — it’s not great, but it would probably be worse without it.
Crap gets through, and sometimes good stuff has trouble getting through. However, without peer review, much more, much wackier crap would get through. In my field, there are always wacky, out-there things posted to arxiv.org that would never pass peer review. Peer review sets at least some modicum of a filter on things.
There might be a better way to do it, but until we’ve established what that is, I’d hesitate to throw out the baby with the bathwater.
-Rob
May 2nd, 2006 at 3:07 am
yikes, I do not want to hear peer review horror stories right now.
Personally, I feel that Science and Nature are the most likely to let crap through in some ways. Crap does occur in the results section, but far more often it occurs in the discussion section. The FoxP2 paper is probably the most glaring example of this I can think of where they talk about how the mutation allowed the brain to get bigger.
Part of the problem is that as our understanding of science deepens, researchers have to superspecialize. This means that no matter how close the peer relationship is, reviewers may not know what’s going on.
For instance, the fact that I’m using mathematical modelling methods in primatology means that there’ve only been about 5-10 papers before me. So there’s little chance of getting a reviewer who’s familiar with my methods or mathematical modelling as a whole.
When it comes to the Nature and Science stuff, you often have really significant discoveries. However, often enough the implications of the find lie outside the domain of what the researchers themselves are trained in (as in the FoxP2 stuff). They may come to seemingly logical conclusions about such things that aren’t, in fact, valid. And because the peer reviewers are likely from the same discipline as they are, theyr’e unlikely to catch the mistakes.
May 2nd, 2006 at 4:57 am
Didn’t see a trackback so I figured I’d do it the old fashion way
The Problem with Peer Review
Your post inspired a long as hell extended post in the same vein as my comment above.
May 2nd, 2006 at 6:14 am
The basic problem is that many new quantitative methods are inaccessible to the average person – even the average highly intelligent non-specialist. It means people have to have “faith” in the math, because they can’t do a quick check themselves when they see the data.
Mathematicians compound this, by not trying harder to make their techniques more accessible. Also, the types of math taught in high schools are non-optimal. A basic course in Stat would be *much* more useful now than Calculus. As IndianCowboy says, we’ve overspecialized without making more efforts to bridge the gaps, at least not in basic ways.
Part of the problem is Ivory Tower defensiveness: specialists don’t want “their” methods to be accessible to the hoi polloi (including specialists in other fields).
May 2nd, 2006 at 6:16 am
And what about non-peer review? Instead of geneticists flipping through a book on archaeology or history here and there and offering up an explanation for their observations in a halphhazard way, why not submit their findings to specialists in history, archaeology, etc, for comment?
May 2nd, 2006 at 6:26 am
Re interpreting gene trees, I wrote this in a review of Spencer Wells’ book:
“The ages of these various mutations can be estimated, so these markers allow us to read not only the geography of ancient migrations but also their timing. There may be some truth to all of this or it may all be a complete fantasy, like reading patterns in tea leaves, and no one has any idea which perspective on the method is right.”
Things may be getting better: I have complained for years about gene tree papers, last summer at a conference in Cambridge we had a shootout in which I came up with some plausible human histories, generated simulated DNA sequences from the histories, and the participants were to reconstruct the history. It was an utter disaster, so humiliating to everyone that plans to publish it all in the conference proceedings were immediately scuttled in favor of a short summary by me that mentions no names.
May 2nd, 2006 at 7:55 am
argh, I was living in England last summer That sounds like that conference was a lot of fun.
Nusapiens, honestly, I’d say the exact reverse of what you said. What could most benefit researchers are courses in applied calculus. Not ‘pure’ calculus, but more conceptual stuff in how calculus applies to the real world.
The most problematic areas seem not to be in the statistical methods, but in understanding of mathematical reconstruction and modelling. Personally, I feel calculus makes understanding that much easier for me than stats does.
May 2nd, 2006 at 9:04 am
Science and Nature publish articles with a non-normal distribution.
Over a given year, several have long half lives and get cited a lot for quite some time.
Most have short half lives and are only cited by the authors.
The problem is whether a good or bad paper is published in a high impact journal, but in giving undeserved weight to a low impact paper in a high impact journal, e.g., good grant score because, “Look, s/he published in Nature!”
May 2nd, 2006 at 9:45 am
Sorry, but that’s just silly. Math is hard, and there’s not a lot we mathematicians can do to change that.
I’d strongly disagree. For example, you can’t do physics in any meaningful way without calculus. And the entire field of differential equations depends on calculus, and there’s not a single field of science that doesn’t use those.
May 2nd, 2006 at 10:58 am
It’s pretty much an impossible job. People use the reputation of journals to limit the amount of reading they have to do — no one has time to read everything. Peer review is just a convenient time-saving screen and not to much could be expected.
People here have been talking mostly about false positives, but false negatives are just as bad (rejecting a good paper). If you take any chances you’ll publish some crap, but if you don’t you run the risk of publishing mediocre stuff. (Gtanted that a few things are obviously good).
Once you have an old-boy network, or factional control of a journal, you can have a serious narrowing of range. This is a massive factor in the humanities, social sciences and history, and I can imagine that it can be a factor in the sciences too.
May 2nd, 2006 at 12:00 pm
Once you have an old-boy network, or factional control of a journal, you can have a serious narrowing of range. This is a massive factor in the humanities, social sciences and history, and I can imagine that it can be a factor in the sciences too.
Esp in emerging sciences like cog sci or linguistics — and the citation ettiquette is even worse (i.e., completely refusing to in any way acknowledge rival theories, even to argue why the author’s view is superior to them).
May 2nd, 2006 at 12:09 pm
john, it’s a pretty important factor in human behavioral ecology/evolutionary psychology as well. Both camps claim to study the same things but go about it in different ways. (And honestly HBE is better grounded). And they’ve each got their own journals.
The problem comes in that on top of the difference in perspective, there’s also a difference in ‘acceptable material’. As a gross over-exaggeration, HBE come off as barely disguised feminists, while EP researchers are prone to making statements only a couple steps away from the pseudo-evolutionary garbage of eugenetics etc.
May 2nd, 2006 at 4:56 pm
A basic course in Stat would be *much* more useful now than Calculus.
You can’t understand statistics without calculus.
Re: peer review, I need to see alternatives to take criticisms seriously. The fact that there are some failures (even some high profile failures) in preventing bad results from getting published is not a cogent argument. in the end the bad results will fail to replicate as they did with hendrik schon.
give me a good alternative to peer review and we’ll talk.
May 2nd, 2006 at 4:57 pm
EP researchers are prone to making statements only a couple steps away from the pseudo-evolutionary garbage of eugenetics etc.
Heh…like?
May 2nd, 2006 at 7:37 pm
yipes, that should say eugenics not eugenetics. I’ll forget my own head next.
gc, probably the most current example is David P. Barash’s postulation that killing a cheating mate is ‘adaptive’.
Or any of the other heinous and species-specific behaviors we engage in that some EP researchers will try to brush off as ‘adaptive’. Rape for one.
Sometimes it just seems that they go out of their way to find despicable behavior and then insist that we’ve evolved to act that way.
May 2nd, 2006 at 7:41 pm
Sometimes it just seems that they go out of their way to find despicable behavior and then insist that we’ve evolved to act that way.
just because behavior is despicable doesn’t mean it isn’t adaptive. i think some of the accusations of adaptationism out-of-control are probably on the mark, but overall i would assume that considerations of morality are irrelevant when comes to assessing whether there is a biologial bias in a behavior. after all, male lions who take over a pack will kill all the cubs which were fathered by the previous alpha.
May 2nd, 2006 at 10:23 pm
Oh I agree with the above, as a behavioral ecologist I’d be an idiot if I said otherwise. I guess I wasn’t articulate enough.
What I was trying to say was that the HBE camp seems inherently biased against positive male contributions to social groups (the bias of 60′s and 70′s anthropologists covering primitive tribals is almost famous…in one case, a researcher cherry-picked bad hunting days/periods to use for male contribution while actually driving females out to the best spots to gather berries and nuts and crap for his data on female contribution).
And EP goes out of their way to focus on heinous behavior. The reason I used Barash’s example is that while there’s theoretical and empirical justification for punishment of infidelity, there’s almost none for homicidal retaliation. Those alpha male lions don’t actually kill the females that copulated wtih the previous male.
You’re absolutely right that morality need have nothing to do with adaptiveness, but the fact that they seem to pick behaviors to talk about based on their immorality is what I find amusing.
Personally, I’m of the opinion that both are shortsighted. HBE rarely if ever makes any evolutionary claims. While EP frequently makes evolutionary claims with little theoretical justification. What’s needed is to combine the methods of HBE with a phylogenetic approach.
May 2nd, 2006 at 10:38 pm
and a genomic one
i think that focusing on ‘immoral’ behaviors going be a good publishing and publicity route.
May 3rd, 2006 at 4:11 pm
Or any of the other heinous and species-specific behaviors we engage in that some EP researchers will try to brush off as ‘adaptive’. Rape for one.
Rape is not species specific.
Moreover, obviously a male who can get away with a lot of rapes (without getting caught) will have a lot of descendants. Genghis Khan and his immediate male descendants spring to mind.
re: homicidal retaliation, yeah, but as you agree the punishment of infidelity is adaptive. Homicidal retaliation is not itself adaptive, but the *threat* would be. Death is more of a deterrent to a behavior than injury. And Barash presents data to show that spousal abuse actually does affect infidelity rates.
I agree, though, that EP often makes evolutionary claims without boiling things down to the genetic level. That has been a necessary consequence of our current technology. But the kind of molecular detail discussed in Sperm Wars is going to be applied to our genomes soon enough. Note for example the extreme degree of selective pressure on reproductive machinery detected in the recent Moyzis and Wang papers.