Nathaniel goes on to say that “… there’s a simple solution that should satisfy you theorists nicely: JOIN THE EXPERIMENT! …”.

A flaw in Nathaniel’s solution is that not every theorist/analyst will get to be affiliated with the experiment collaboration.

It seems to me that a more comprehensive, even simpler, solution would be to make the data public, in a format that is the work-product of Nathaniel and his fellow experimenters, by a paper authored by Nathaniel and his fellow experimenters.
Then, any theorist/analyst (whether or not affiliated) should cite that paper, so that Nathaniel et al would have a very high citation rating.

Further, if any theorist/analyst might ask Nathaniel et al for help in understanding the data, Nathaniel et al should be listed as coauthors for providing such help.

I have tried to follow that spirit in stuff that I have written. For example, in my writings about Fermilab T-quark data, I give explicit credit to Erich Ward Varnes whose 1997 UC Berkely PhD thesis contained data that I found very useful.

Tony Smith
http://www.valdostamuseum.org/hamsmith/

After working for six years on MINOS, I will get ONE (count ‘em) paper. OK, I’ll be fair. Three papers. Out of two hundred authors. If the data were made available publicly, then this paper wouldn’t even get cited… some theorist would come along, do a slightly more sophisticated analysis, and I the paper wouldn’t even get cited.

Even worse, to make the data public we now have to publish the methods and documentation how to use the data (which will NEVER just be a list of 4-vectors; there are correlations and resolution functions on every experiment) and that will take the experimentalists even more work.

Don’t get me wrong.. I love what I do. But I slave over computer code, measure crosstalk, invent calibration sources, crawl under dusty machines, travel, travel, travel, sit on interminably phone calls (every day) so that I can get those few weeks of analysing the data before everyone else. Now I can’t even do that?

Happily, there’s a simple solution that should satisfy you theorists nicely: JOIN THE EXPERIMENT! I need a three more people in my calibration group to measure attenuation curves. I need two more to get automated processing running and document things. We need people to think deeply about statistics, and to make sure our MC models are good. We need people who understand the theory well to suggest what fits to make and the best way of presenting the data. But, of course, that’s a lot of work, so not many of you take us up on the offer.

—Nathaniel

I think the biggest problem with releasing data from the LHC experiments would be the shear volume. How much would CMS or ATLAS record in a day? What kind of background reduction are people expecting here?

let me comment this phrase of yours:

“In principle I’m in favor of releasing the data, in practice I doubt that it would work. Without an intimate knowledge of the idiosyncrasies of the detector, too many spurious results would be hard to resist.”

My idea is that the release of data is really similar to release of software code in open source projects (my professional field).

For example, for software products like Netscape/Mozilla/Firefox, the code was originally proprietary and secret; then, there was the decision to release the code of the product itself. The idea was to create a large community of developers, able to contribute to the improvement of the product.

However, for more than one year after the release of the code, the contribution from the developers outside the original development team was minimal. There was a lot of interest from other developers, but they were not able to provide any significant change to the code.

The reason was understood shortly after. The code itself was only part of the knowledge that had built the product. Each line of code was the result of several decisions made by the developers, and contained assumptions there were not easy to make explicit.

In short, the code was the result of a long and complex process, but in order to contribute to the code, you had first to became part of the process. Only after the assumptions became clearer, it was possible for other people to make significant contributions.

Which is the relation I see with LHC data ?

Data are the result of complex processes, where there is a lot of hidden knowledge that is necessary in order to understand what a number really mean in a certain context. People outside the process cannot understand what the raw data can really mean, without a proper understanding of the process itself.

However, if the parallel I have made is anything significat, IT IS useful to make data available, as far as you understand that you have to make clear which is process through which they are produced and elaborated.

Then, other people can make useful proposal on how to improve the understanding of data. In fact, making the process public has significatively improved the process itself.

Again turning to the analogy with cosmology/astrophysics, I suspect many theorists in cosmology (myself included) are learning more about Bayesian statistics, priors, Markov Chains and all the rest of it than we would have ever dreamed. And I can certainly point to some deeply flawed papers in the literature that might never have seen the light if the raw data was not freely available. However, theorists *can* learn this stuff, and don’t like to look silly in public, so they have plenty of motivation for doing so.

In the end the theorists will either learn enough of the subtleties to do it themselves, or work with experimentalists who know how to perform the relevant analyses. Many (most?) papers are flawed in some way, and the community would react to the flurry of theorist-written data-driven papers by hiking its overall level of skepticism a notch or two. Just as it did with the arrival of the Arxiv, which does an end-run around peer review (for what that is worth, but don’t get me started)

As Sean pointed out above, one side-effect of the present system is that it is very hard for particle theorists to collaborate with experimentalists, if the whole collaboration needs to sign off on papers that any single member writes (and this is *after* the data is in the public domain). Again speaking from my own experience, my foray into the world of data-analysis has largely been conducted in collaboration with someone who understood the issues involved at the outset (although not an “experimentalist” in the strict sense of the term), and it is a singularly productive mode of collaboration. To the extent that the “rules” of experimental particle physics discourage this sort of collaboration they are clearly ounter-productive.

Secondly, the cosmological community has benefitted greatly from the development of the Cosmomc package which greatly simplifies the Monte Carlo Markov Chain analyses of cosmological data. (It is not theorist-proof however, as I have seen several publicly displayed figures that showed chains which, to my now practiced eye, were clearly unconverged). My guess is that if more experimental particle physicss data was made publicly available it would seed a small industry in the development of software tools that facilitated its analysis.

The data belongs to taxpayers, so far as I’m concerned.

In my experience experimentalists are extremely suspicious of results, since they know what actually goes into making them — hence the tradition of requiring confirmation from an independent experiment for discovery claims. Without some kind of (at least short-term) data encapsulation, problems will arise: imagine experiments looking at each other’s data! (Related reasons are behind the rise of “blind analyses,” where a collaboration hides its data from itself, for fear that it will find what it wants to find.) Even if only a small fraction of a collaboration reads a paper thoroughly, that’s still an awful lot of experience-years.

And finally, a point that vaguely amuses me: we are used to a feedback system where either (a) theorists predict something, experiment finds it, theorists claim vindication because the prediction was ante hoc instead of post hoc, or (b) experiment finds something unexpected, everyone scrambles to see how models could accomodate this result, some things can’t and are excluded. What happens to this waltz if the theorists get to look at the data at the same time the experimentalists do?

[Note: none of this implies that I don't think processed HEP data should be released after a (longish) while, or that short-term data release might not be a good thing if we find ourselves with a one-detector ILC.]