A Good Housekeeping Seal of Approval for science? My new piece in Slate

By Carl Zimmer | August 14, 2012 11:26 am

Like a number of other science writers, I’ve become increasingly interested (and concerned) about science’s ability to correct itself. (See my recent pieces about arsenic life, de-discovery, and dysfunctional science.) So I was intrigued by a new project launching today to encourage scientists to embrace the spirit of replication. I write about it at Slate. Check it out.


Comments (4)

  1. interesting… through hard to break through the barrier that doing replication studies simply carries less cachet than original work in a publish-or-perish world (in fact it may add pressure to “fail” to replicate, which is at least a tad more interesting than actually replicating). But something needs to be done, so I wish Dr. Iorns success with the project, especially if it can be spread to other areas.

  2. Martha Bays

    I really hope you go back and look at Brian Krueger’s comments under Arsenic Life. They are exactly in accord with a study in the sociology of science I did in the 1980’s while I was in grad school. Instead of analyzing published papers, I spent several months observing in a chemistry lab. Replicating– running the same experiment over and over while changing one parameter at a time, until the results were consistent– was the backbone of the research.

    During that study I observed 3 failures to replicate. In one, a very junior researcher couldn’t make his results conform to those of a more senior scientist at the same lab. Instead of discounting the junior colleague, the lab personnel investigated further. It turned out the junior person was right, and the whole project had to be scrapped. Of course, that never made the published literature.

    In two other cases, lab personnel found errors in the published literature while trying to replicate it (in order to form a baseline so they could build on the work.) In both cases they published what amounted to a failure-to-replicate. However, since they did a lot of work testing out the issue, their papers were not presented as failures to replicate. They simply presented their own, well-checked results and pointed out that they disagreed with someone else’s publication. I interviewed one of the people whose results were questioned in this way. He straightforwardly admitted they had unknowingly published results that were based on contaminated chemicals from a commercial supplier.

    The conclusion I came to was that studies of science which rely on looking at the literature without looking at what actually is going on in the field or the lab inevitably underestimate just how much replication actually goes on in science.

  3. Martha [2]: Thanks for your comment.

    I’m not clear if you’re saying that your experiences contradict my piece. If that’s the case, I would disagree. I never said that scientists never repeat their own experiments. I’m heartened to read your story of a senior scientist who listened to a junior scientist announce he or she couldn’t replicate some results.

    But you seem to be claiming that repeating an experiment over and over while changing parameters is replication. It’s not, since those parameters can dramatically change the outcome of the experiment. Also, the fact that scientists can replicate their own results is less compelling than other scientists being able to do so independently.

    The example you do give of a published failure to replicate another scientist’s results demonstrates exactly the problem science faces. The failure to replicate a previous study was buried in the published paper–not in the title or in the abstract, I’m assuming. So it would be easy for a scientist interested in taking up a new line of research to overlook this important fact.

    And while the original scientist may have freely admitted that his results were due to contamination, did he retract his paper? Unfortunately, many papers that have been soundly repudiated keep getting cited.

    Finally, I’d ask how representative the experiences you describe are–either of science as a whole, or of a particular discipline. I think the cancer review I describe makes it amply clear that results are getting published that cannot get replicated–at least in biomedical journals. You were observing what’s going on in chemistry labs. Maybe the problem is milder in chemistry?

  4. Martha Bays

    I am sure the problem is milder in chemistry; in chemistry, the goal is to find “the mechanism of action.” In medicine, the goal is to cure people, whether one understands the mechanism of action of a particular drug or not. That is an entirely worthy goal, but it does leave medical research somewhat more vulnerable to error than chemistry.

    I was not intending to criticize your work. I haven’t seen enough that you’ve written on the particular subject of fraud in science to judge your work on that topic, and I have been very impressed with your work on other topics.

    What I wrote was more in the nature of a caution. There was, at least 30 years ago, a lot written in the sociology and philosophy of science based solely on studies of published articles. Most of it was pretty awful, because the writers didn’t seem to understand how the scientists themselves saw the work.

    There were two things I saw by actually hanging out in a lab and talking to the researchers that the work just reading papers missed. One was the amount of repetition that went into the research. The other was the importance of scientific communities. Almost all the people in a particular, small area of study knew each other, at least by reputation. Researchers who got a reputation for shoddy work could have their careers hurt, whether they were actually faced with a “failure to replicate” paper or not. I don’t remember if the paper I referred to was formally retracted. (I have a vague impression it was.) But I know a lot of people working in that small area of study knew about the mistake.

    Again, I’m not trying to criticize what you have done. I’m just trying to point to some issues I think are helpful to address.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

The Loom

A blog about life, past and future. Written by DISCOVER contributing editor and columnist Carl Zimmer.

About Carl Zimmer

Carl Zimmer writes about science regularly for The New York Times and magazines such as DISCOVER, which also hosts his blog, The LoomHe is the author of 12 books, the most recent of which is Science Ink: Tattoos of the Science Obsessed.


See More

Collapse bottom bar