Publishing in Large Collaborations

By John Conway | November 14, 2009 11:19 pm

I had that somewhat rare experience two days ago, getting the message from Physical Review Letters that our paper had finally been published online. In our field it can take quite a long time to get a paper all the way to publication; this one took longer than usual…

The paper describes the results of our search for Higgs bosons predicted in supersymmetric theories, in the CDF experiment at Fermilab. Alas, we didn’t see any evidence for Higgs boson production, despite earlier hints that something might lurk in our sample, and so we were able to rule out some regions of supersymmetric parameter space. We first obtained a preliminary result from this analysis in late 2007. A group of us from Rutgers University, University of Valencia, and UC Davis all worked directly on it, within our several-hundred-member collaboration, using Tevatron data recorded up through mid-2007. Since then we’ve more than tripled our data sample, but this result stuck with the data sample on which it was based and has finally reached publication.

The analysis was the topic of the Ph.D. thesis of my student Cris Cuenca, who was formally a student of my former postdoc Juan Valls at University of Valencia. Cris was a visitor in our group at UC Davis, and I was effectively his thesis advisor. After we got the preliminary results, Cris focused on writing his thesis, eventually defending it in Valencia in April 2008. One nice effect of that was that I got to visit Valencia for the first time: what a fantastic city!

Once the thesis was done, it was clear we needed to publish the result formally. In fact, we should have been already writing the paper but as usual it’s hard to find time to get started on writing projects. Here is a place we could have saved some time, though…

Anyway, I wrote a draft after the birth of our son Ian in June 2008, while helping with baby care at home. In our collaboration there is a very formal review process before submitting a paper for publication. The spokespeople of the collaboration “godparents” who perform a full internal review of the analysis and the draft of the paper. Without naming names, we got some very knowledgeable godparents who asked us hard questions. Some of these questions took weeks to answer, because more data analysis had to be performed. And some of them led to even more questions. This review process is a good thing — it ensures that the quality of the final paper is very high, and that the result is correct to the best of our ability. In fact, in the course of the review process we found that there was a minor software bug, and we repeated the full end stage of the analysis. As it turned out, the bug had very little effect on the final result but we needed to be sure. (At present there are only unknown bugs in the analysis software!)

Whenever there is a change to an analysis like this, it needs to be re-approved in our physics analysis group meetings, with two presentations: a “pre-blessing”, and then a “blessing” two weeks later. (Hey, I’m not responsible for the pseudo-religious jargon used in this process…) This eventually happened in March 2008.

With the result final and the godparents happy with the paper draft, it was time for the general collaboration review. The collaboration gets two weeks to comment on each draft. Then the authors go through comment by comment and reply to the commenters, modifying the draft as needed. The godparents reviewed our replies and then we arrive at the next draft. This part of the process can take many weeks depending on how much time the authors have two devote to the paper. Once the final draft stage is reached, a “paper reading” is scheduled at the weekly general collaboration meeting. Following the presentation of the result, the collaboration has 48 hours before the paper is submitted to the journal. For us this happened, finally, in June of this year.

We heard back from PRL in late July, with blind referee comments to address. There then ensued a back-and-forth between us and the referees, answering questions, making changes to our submission, and eventually reaching agreement that the paper would be published in PRL. This happened a few weeks ago, and our paper has now appeared in what I think is still considered to be the most prestigious journal in our field, though Nature possibly tops it. (That might inspire a comment flame war but I hope not…)

Maybe this is an extreme example, and I certainly will endeavor to bring results to publication much more quickly in the future. (I always say that.) Certain results, if they are “hot”, can be published on a fast track in CDF, within weeks, but that is quite rare.

Many will, no doubt, argue that print media of almost every form is on the way out. Will this happen to print science journals? I do think there is a strong need for blindly refereed publication of scientific results, even though many scientists have reviewed these papers by the time they are submitted.

And clearly once the LHC experiments have physics results to publish, we will need a very rapid means of getting them into print. For any striking new discovery we’ll want to have a paper submitted for publication when we announce the result…this is in contrast to the case of not-very-striking results, where we announce the results at conferences first and publish later. The reason is that the experiments will try to establish scientific priority by publishing striking results before the other one does, but I have to wonder, in the modern age of electronic media and collaborations with thousands of members, whether simply announcing or presenting the the result in public doesn’t accomplish that anyway. If ATLAS says they see a resonance in muon pairs at 1.5 TeV mass and so does CMS, the same week, will we really say “ATLAS found it first” or “ATLAS was the one to discover it and CDF confirmed it?” I hope the science mainstream media don’t present such a thing that way…but more than that I just hope this is a problem we will actually face!

  • Luis

    Sean, out of curiosity, what are the standard time frames in the publication process in cosmology and physics? I’m asking because my own field (theoretical linguistics) is desperatingly slow. If I submit to a major journal, I usually have to wait between 4 and 6 months for it to come back from blind refereeing, even if it is a short one (a couple of times I’ve had to wait 10 months). This bugs me because, when I’m asked to serve as a referee, I’m always given a 5-week deadline. If the referees think the manuscript has to be resubmitted, add another 4 to 6 months for the second round of reviewing. Assuming it finally gets accepted, there is the issue of getting it into paper, which can take upwards of a year.

    For illustration: in May I received the journal issue containing a paper I had first submitted in late 2005. This month, a proceedings volume is supposed to come out with a joint paper we submitted in March 2007 (the conference itself happened in Dec. 2006). One of my colleagues is organizing a workshop, for which she wanted to invite a very distinguished prof. who has just published a wonderful paper in the top journal of the field. The distinguished prof.’s response was: “sorry, but I can’t really contribute to this workshop — I haven’t worked on this topic since 2005, when I submitted the paper”.

    If it wasn’t because I need the publications to get tenure, I’d be happy just uploading my papers to an online repository and then let interested people contact me with comments.

  • Pieter Kok

    John, I think the question whether print media will survive in the sciences has been answered already by the community. The preprint server is the go-to place for most of us first thing in the morning, and the final published paper has the official seal of approval of the journal (after the refereeing process). Libraries will want to keep paper copies because of durability issues (e.g. try getting your data off an old 5 1/4″ disk these days, good luck!), but the rest of us print out individual papers when we need them.

    Luis, it can take a very long time in physics as well (Physical Review Letters has gained some notoriety over the years, especially for a “letters” journal). However, all journals I know of allow you to put a copy on the preprint server (Nature only after publication, but they tend to be quick anyway). If there is a preprint server for your field, you should check which journals allow you to use it before publication. The culture of publishing preprints must grow organically, and the community has to force the hnd of the journals (especially the for-profit ones).

  • Cartesian

    About publications and their delay, I am currently looking for somebody to endorse me for a publication on arXiv in the field of the Atomic Physics. So if anybody can do it and has some time…This is not a bad system but I did remark that some on arXiv do try to avoid the fact to be responsible of endorsement.

  • Sili

    Come to think of it: How does one go about blinding a paper for the referees?

    I mean, the introduction for instance is bound to refer to “our previous work[ref]”, so it’s not enough to just remove the names from the manuscript. And that doesn’t even touch upon how some subjects are so specialised that anyone qualified to review it, will know immediately who the author is.

  • Typo Guy

    Because this is an interesting post of lasting value, and because it’s amusing to “review” a post on a reviewing process, here are my copy edits, with questionable words highlighted in bold and placed in context:

    … that are paper had finally been published online.

    In fact, in the course of the review process we found that there was a minor software bug found, and we repeated the full end stage of the analysis.

    As it turned out, the boat had very little effect on the final result but we needed to be sure.

    The godparents read view our replies and then we arrive at the next draft. This part of the process can take many weeks depending on how much time the authors have two devote to the paper.

    Following the presentation of the result, the collaboration has 48 hours before the papers [are] submitted to the journal.

    (In the last paragraph above, it appears that “are” needs to be inserted.)

  • Charon

    In astronomy, arXiv is also the place to go. I try to skim journals too, but only read arXiv regularly. The review process is nonetheless quite important, to ensure quality. This leads to some inconsistency, because some people publish to arXiv right when their paper is submitted, some when their paper is accepted, and a few when their paper is rejected. I try to only put papers up after acceptance, because it seems often people read the arXiv versions, and change their citations when the journal edition comes out, but don’t read the new journal version.

    I also have only a few first-author papers under my belt, and have been very lucky about speed (last one was only two months from being submitted to being published in the journal online, three months to published on paper). This is somewhat unusual for astronomy (my results were observations that were interesting but uncontroversial), but a few months to at most a year seems typical. The linguistics example above is freaky. It would be very demotivating – no one would write a paper on anything that a new observatory might look at in the next couple years, because those new observations would be more interesting than whatever you wrote years ago that was finally being published…

  • John

    Thanks, Typo Guy! The reason there were so many typos is that I used a dictation program to write this, and I did not scan carefully enough for typos, especially the ones that are homonyms…

    The blinding process is one sided – the reviewers know who the authors are, but the authors don’t know who the reviewers are.

  • Sean

    I honestly don’t know how you guys do it. I find five-person collaborations to be getting quite a bit unwieldy.

  • Phillip Helbig

    Priority? The way things are going, the first person to tweet something will get the
    scientific priority. What is this world coming to? (I am far from being a Luddite, but
    espousing Twitter demonstrates a disability to separate the wheat from the chaff.)

  • Cartesian

    The problem with endorsing and arXiv as well is that the culture about censorship is not the same in every country (Russia, China…), so there are some persons to who it seems less good to ask for an endorsement than some others.

  • Phillip Helbig

    It is really stretching it to compare the ArXiv “censorship” to that in Russia or China.

  • Pingback: Internet Cafe Solution » Blog Archive » Publishing in Large Collaborations | Cosmic Variance | Discover …()

  • Trupti

    Pieter Kok,

    Nature does allow you to put your paper on arXiV prior to it’s publication/acceptance. Read Nature’s embargo policy page: where it says, “Neither conference presentations nor posting on recognized preprint servers constitute prior publication.”

  • Thad

    “…I hope the science mainstream media don’t present such a thing that way…but more than that I just hope this is a problem we will actually face!”

    Then the mainstream media need to see your process, including the debates and deliberations. What about a public blog during the data processing stage? It would be fascinating to see the collaborative process, even if in general terms to hide the behind-the-scenes details. A record of meetings, who was in attendance, who gave reports, who raised issues, about what. Even if the public didn’t have access to the data, the software, etc., are there enough crumbs to show how diligently so many bright people are scrutinizing the problem?

  • John

    Thad, part of the point of my post is just how carefully our results do get scrutinized internally. I don’t foresee discussing except in general terms how the process unfolds at internal meetings and emails… Ultimately the fact that CMS And ATLAS ought to agree on what they see or don’t see, working more or less independently, will serve as the best possible scientific cross check.

  • Cartesian

    To Phillip Helbig 11 : The history of science shows that there are some scientific ideas which embarrass as with Galileo Galilei (even if good and truths), and if there is not a culture of tolerance this kind if idea can trigger some scientific censorship. So if you contact some persons of China or Russia in order to endorse a work there is the risk that they do not have the same moderation as some others.

  • Cristina

    Congratulations on your paper, John! Your initial result kept me up quite a few nights, since I was working on a similar search for the “competition”. By the way, I was probably the only person in DZero who thought our “rebuttal” made public the same day Anton gave his Wine & Cheese talk was not exactly worth making public yet. Our internal review is just as tedious as yours, more often than not. In the end it’s a very good thing, but leads to quite a few grey hairs in the process.


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Cosmic Variance

Random samplings from a universe of ideas.

See More

Collapse bottom bar