Bump Hunting (Part 2)

By John Conway | January 26, 2007 2:11 pm

Was it real?

I stared up at the atrium of Building 40, two things becoming apparent. First, with a result with a little bump like this, there was going to be a lot more work to do. Second, I had better get myself back to Fermilab soon rather than remain at CERN.

I changed my ticket to the next Monday as there was nothing available the next day. This meant I had the weekend at CERN to work. I sent email to my still-asleep colleagues in the US, and got busy.

When you see a spectrum like the one we had, with a few bins having more than normal rate, it could be a number of things. It’s always possible that you forgot some background, that the simulation of the detector and the analysis is not perfect, or simply a statistical fluctuation.

High energy particle collisions are governed by quantum mechanics, and therefore intrinsically random processes. If you expect to see a certain number of events with certain characteristics after running for some time, you don’t get exactly that number when you run the experiment. If you expect 10, you might observe 8, or 12, or, more rarely, 20. That was the case here: in the spectrum of what we called “visible mass,” a few of the bins had more events than expected. So was this a statistical fluctuation, a foul-up in the simulation, or the beginning of a real signal?

As the apocryphal student lab report said “our experiment didn’t have enough data so we had to use statistics”. In our field we use statistics all the time. We want to make quantitative statements about the things that make the hair on the back of your neck stand up, or perhaps yawn. Being quantitative means that we can be as objective as possible, and offer to our colleagues and to the world a rational interpretation of the data. Doing these sorts of analyses is my particular specialty in particle physics. Now it was time to do it here.

As a common language in our field we try to relate all our statistical questions back to the good old “normal” distribution, the familiar bell-shaped gaussian. A gaussian is peaked in the middle with long tails on the high and low side. It is quite proper to think of it as an indication of how often you expect some measurement to give you a certain result: most often in the middle, and less often out in the tails. It’s characterized by a “standard deviation” or, since we use the Greek letter sigma to represent that, “1 sigma.” If your measurement is described by a Gaussian, about 2/3 of the time it will lie within plus or minus one sigma of the peak. In 1/6 if the cases, then, it will be more than one sigma on the high side. The more sigma away from the peak, the less likely it is that it happens.

In particle physics we see statistical fluctuations all the time, and so to make a discovery of something really new, we have to demand that the probability of a random fluctuation causing the excess is very, very small. Usually we demand five sigma, that is five standard deviations from the peak, to claim “discovery.” With less statistical significance, we hedge and use words like “evidence” or “indication.”

So how significant was this? I quickly fired up my program to fit the spectrum including a Higgs signal, and, sure enough, the data preferred a SUSY Higgs with a mass of about 160 GeV, or about 170 times the mass of the proton. The peak rate was quite consistent with our sensitivity. And, to boot, it appeared that the statistical significance for that mass was about 2.5 standard deviations. Below that level people are not very interested, but at this level it starts to get intriguing.

The problem is that this is a bump hunt, and we didn’t know where the bump might show up in terms of the mass of the Higgs, since we don’t know what the Higgs mass is! That means that the real question is this: could a random fluctuation give us a bump like this anywhere in the spectrum, not just at 160 GeV.

This calculation was going to take a while. I needed to run a random simulation that generated lots of possible experimental outcomes with no Higgs signal present, and then fit the spectrum for all possible Higgs masses, and see how often I got a 2.5-sigma result. I set about coding this up.

Fortunately I had all the pieces I needed to perform the calculation, and I stitched them together, testing as I went. Then it was time to launch the job. I moved the code to our computer cluster back in California and set up scripts to split the calculation across multiple computers. With a single command, I set 20 CPUs going, generating random spectra and looking for the Higgs where there was no Higgs. It looked like it would be a number of days to get the answer.

It is a source of continual amazement to me that a mere 15 years ago the state of the art in computing at CERN was our big VAX 9000, really the last of the mainframes. Windows desktop PCs were useless to us since we couldn’t program them, and Unix workstations were coming into vogue but still not that fast. So now I had just launched a job which was utilizing about 2000 times more computing power than the old VAX. And this was only 20 CPUs! We now have a global grid with tens of thousands of nodes accessible, and more every day.

So in the mean time, it was time to assess the damage the bump had done to our limits. When I had opened the box, I had expected that we’d see no sign of a Higgs signal, and then proceed to rule out certain regions of supersymmetric parameter space where, if nature had chosen to live there, we would have seen a Higgs signal. Our usual statistical standard is somewhat loose in this regard: we require “95% confidence” to rule out a certain parameter set. In fact that means that there is less than a 5% chance that, if that parameter set is true, we’d have seen what we saw or an even larger excess of Higgs. You kind of have to think about that for a good while to see why such inside-out logic is needed. The problem is that there is a crucial thing we don’t know: the probability that the Higgs exists or not!

With this excess, we’d be able to rule out a lot less territory than we had hoped. I ran the short jobs to calculate the rate limits, and sent them off to Amit to convert into the final SUSY parameter limits, and started writing it all up.

In our collaboration we have a very formal internal review process for getting out results. We need to document everything in advance of two presentations to the appropriate physics analysis meeting. The first presentation is called a “pre-blessing” and is where the real knives come out. The presenter is peppered with deep, probing questions about every aspect of the analysis, usually for an hour or more. Though it can seem like a blood sport at times, this is an absolutely essential part of the scientific process: if we aren’t our own worst skeptics then someone else will do it for us.

Anton was awake now, got the message about the excess, and started asking all the right questions. Amit chimed in and we quickly formed a plan: we had until Sunday at midnight to get the note posted, and we divided up the work. He shot back parameter limits to me and I turned it into a nice plot with Illustrator:


The dark purple region in the plot shows the paramter values we exclude with 95% confidence. The region labeled “expected” is what we thought we’d be able to exclude. Clearly we had not reached our expected sensitivity due to the fluctuation. As we get more and more data, we expect the excluded region to cover more and more of the plot, to lower and lower “tan beta” which is one of the main SUSY parameters. We illustrated the somewhat mild dependence of our result on input assumptions, and compared with the limits obtained years ago at the LEP 2 accelerator at CERN. We’d made progress, but the little bump was hurting us.

I returned to Fermilab on Monday, and met with Anton and Cris to brainstorm about possible problems, and questions that might arise. Anton was going to make the presentation on Thursday, and we needed to armor-plate the result, which was sure to elicit lots of tough questions. He had done an amazing job at documenting everything – our note ran to over 70 pages, much of it plot after plot comparing the data and simulation in our control samples and in other subsamples of the data. Everything checked out – there was no reason to believe that the little bump was anything other than either a statistical fluctuation or something new.

My statistical significance jobs came back with an answer: there was about a 2% chance that a random fluctuation could have given us a bump of the magnitude we saw. That’s about a 2 sigma effect, and those come and go every day in our field. Still, we are all human and have to wonder whether this is real or not.

The presentation went fine – we were “preblessed” and scheduled for final blessing the first week of the new year, the week before my talk at Aspen. We really did not receive any difficult questions, as we’d done our homework well. Over the holidays we could expect questions and comments from our collaborators, and had some more time to do cross checks.

But by the blessing talk three weeks later, a number of people who had not been at the preblessing presentation caught wind of the excess, and showed up at the meeting to ask tough questions. In then end, though, we were blessed and I had a green light to show the result at Aspen the following Tuesday. I worked all weekend on the talk, in which I now focused a lot of attention on the new result, which would be the thing of most interest to the crowd.

It’s pretty rare these days to have a high profile talk like one at Aspen and also your own brand new result to show – sort of a perfect storm. My attitude was to show the facts and draw the obvious conclusion: we need more data! As anticipated, the part of my talk that drew the most interest and questions was our new result. It was about as fun as it gets in this field!

Afterward though, on the ski slopes, in fact, a colleague from our competitor experiment Dzero at the Tevatron, Greg Landsberg, a professor at Brown, intimated to me that their experiment had a result nearing completion. This occupied my thoughts the rest of the afternoon. What had they seen? Did they have an excess too? Last year there had been a small hint of one, but nothing exciting.

At the evening session Greg let the cat out of the bag: Dzero had a deficit where we had an excess! I’d have rather he waited a day or two to let me know, but, eventually the world would find out. Their method was quite similar to ours, so it was hard to escape the conclusion that if they really had a deficit, our excess is more likely to be a statistical fluctuation. We’re still waiting to see the answer from them, which should happen soon. The nice thing is that we have another factor of two more data to analyze soon. By early summer we should have a much better idea what it is we are seeing.

A week from today Anton will present the result at a Friday Fermilab “Wine and Cheese” seminar which, despite the genteel name, is one of the toughest audiences I know of. A single question there, with 150 people in the audience, can quickly erupt into a barrage, with the speaker barely able to catch his or her breath. But Anton is a real pro, and it will be very interesting to be there and see how it all goes.

We’ve caused quite a stir considering how insignificant the bump really is. I’ve gotten requests from all over to see my Aspen slides, to answer questions. When I was back at CERN last week to carry new pixel detectors to CERN, I got stopped by lots of people who had heard about the result and wanted to know more. To me it shows that there is intense interest in finally nailing the Higgs, whatever its nature, and also it shows that the Tevatron has a real shot at making big discoveries before the LHC turns on. Real physics data will come from the LHC by the middle of next year!

In the end, some day we are going to have something new right there in our data, and we cannot shrink from it. We’ve gone a very long time with no truly new discovery in particle physics, no observation that truly changes the paradigm. We’ve gotten used to fluctuations coming and going, and are justly skeptical of any new ones that come along. But I think I got a glimpse that Saturday morning of what it will feel like when we do have something new, and real, and it’s a sensation that I hope I’ll have again some day soon.

  • Coin


    Could there be something wrong with the HTML in this post? For me, when I look at this post or the front page currently containing this post, the white background (that the other posts on this site have) is gone and the sidebars are all weird and misplaced.

  • http://www.physics.rutgers.edu/~lath/ Amit Lath

    One personal item: John mentions the “preblessing” talk, which happened on December 14. That was the day my brother got married, in Pittsburgh!! (yes, on a thursday…)

    Needless to say, I could not attend the preblessing.
    And I had to work hard to pay attention at my own brother’s wedding. Anton did a great job presenting the analysis.

    I was there at the blessing two weeks later (and John connected electronically). As John says, tough questions about the excess.

  • Josh

    I also am having the same problem viewing it (using Firefox as my browser).

  • http://www.astro.utoronto.ca/~zqhuang zq

    For me there is nothing weird on the page, except for that it takes a few minutes to show my post. I guess it’s due to the spam filter.

  • Josh

    Problem’s gone now.

  • Coin

    Fixed, hooray. Except now the title appears to be “Bump Huning” :)

  • smm

    Best first post(s) ever! Thanks for the great story.

  • http://www.pha.jhu.edu/~npa/ N. Peter Armitage

    Nice story! How exciting.

    I was a cub undergrad with a little job doing some programming for Tom Devlin and CDF on the Rutgers VAX/VMS systems when John Conway was (I think) an asst. professor there. Even then (early 90’s) when I had zero appreciation for large scale computing, those systems struck me as terribly clunky and something my father might have used. My foray into HE Exp stopped around then, but no doubt there is a better way to do this stuff now! Anyway, I look forward to reading Part 3. Sounds like there may be a little delay though before that gets written!

  • Herm

    I have absolutely no idea what you’re talking about, but I’m strangely fascinated by the story. Good luck on your quest — it’s Ahab-like.

  • Hiranya

    Wow, this is the most fascinating post I have seen on this blog! Really conveys the excitement of doing science. Please write more!

  • http://theeternaluniverse.blogspot.com Joseph Smidt

    Thanks again for the great posts. You kept me checking out this blog all day to see post two. Hopefully soon it can be officially announced we’ve found the Higgs. That will be a great day. Supersymmetry, here we come! :)

  • http://www.astro.utoronto.ca/~zqhuang zq

    Higgs with mass of ~170GeV does not imply SUSY, doesn’t it? Personally I like the idea of Higgs, but has no interests in SUSY (or more properly, know almost nothing about SUSY), maybe I should learn more about it.

  • http://blogs.discovermagazine.com/cosmicvariance/john John

    Would the observation of a 170 GeV Higgs imply supersymmetry? Not necessarily. In the case we are looking for, the Higgs has greatly enhanced coupling to the down-type quarks (like the b) and leptons (like the tau). If
    we do see such a signal, at the Tevatron or later at the LHC, it’s likely because of this enhancement. That is a hallmark of SUSY, but not the only possibility within the full range of SUSY scenarios. And SUSY is not the only thing that could give us this sort of signal – alternatives such as “technicolor” or one of its variants might also give an enhanced rate for resonant tau pair production at masses like these. But now I am getting into territory better covered by JoAnne I think.

  • Paul Valletta

    Hi John, great introduction to the trials and tribulations of what must be a really facinating environment, the experimentalist.

    Just forgive me for including this link:

    The Standard Model has really gone places since ancient greek times!..it has an enormous number of paramiters, actually with every new discovery, there has been a plethora of fine tuning “follow_ups”, which under the circumstances is quite obvious, if something new appears, then although a gap has been filled, it leads to more questions as understanding progreses.

    After the initial big splash that comes with best_fit experiments, normally there is a period where the wave receeds, and one can take stock of other incoming waves over the horizon, if one can spot the “next wave” sooner, then hopefully your teams hard work will enable you to position yourself WRT, having a great “ride of your life” !
    Best paul.

  • Chris W.

    In the context of these two admirable posts, some readers might also enjoy this little article by David Mermin, What’s wrong with those talks?

    (The concluding author credit notes that “The first physics colloquium he ever gave was so dreadful he was not invited back for 22 years.”)

  • Pingback: Brit royals in town & Higgs « skies of blue()

  • Michael Saelim

    Awesome. Just awesome. I can’t wait until I get to do all of that stuff. At least I can go to CERN – I’ve been accepted to the University of Michigan/CERN REU program. I hope I get to work on supporting the completion of the LHC. :)

  • citrine

    Great post JoAnne!

    You convey the excitement of possibly making a breakthrough discovery while explaining the process in everyday terms.

    On what basis are the different cutoff sigmas decided? Why not 4 sigma instead of 5?

  • http://quasar9.blogspot.com/ Quasar9

    Hi John, String or M-Theory aside.
    What would you consider to be the most significant implication, or rather application (possible practical use) for a Higgs with a mass around 160 Gev or ~170 times the mass of a proton

  • http://blogs.discovermagazine.com/cosmicvariance/john John

    Citrine, it’s rather arbitrary whether we choose 4 sigma or 5 sigma as our standard for “discovery”. In fact we discovered the top quark in 1995 with about 4.9 sigma, and no one doubted it. If we had a four sigma effect now, the excitement would be very great, for sure! At that point, of course, you need independent confirmation from the other experiments and subsequent data.

    Quasar9: After the discovery of the electron in 1897, J.J. Thomson expressed that he doubted the electron would ever be of practical use. (I’ve searched for the quote but cannot find it…) In the case of the Higgs there is no immediate practical use. But it could lead us to new theoretical and experimental breakthroughs that could transform technology in ways we cannot imagine, any more than Thomson imagined our own modern electronic age.

  • citrine


    Thank you! Great to see you join the CV bloggers.

  • Scott

    I feel a lot of excitement reading this knowing that we’ve got intelligent people working on the cutting edge.

  • Eric

    For pedagogical reasons, perhaps it should be remarked that statistical fluctuations in data do not have to be quantum mechanical in nature–they occur in classical measurements as well. For example, if you try to measure the length of a table with a ruler your data will follow a Gaussian. Its spread is associated with the statistical uncertainty of the measurement, while shifts from the true length are associated with systematic errors corresponding to the ruler ticks, misalignment of ruler and table, etc.

  • Eric

    Forgot to add that I liked the article. :) It’s nice to see the excitement of experimental work conveyed to a larger audience…

  • Ed Matheson


    Could this possibly be the quote that you mentioned?

    “Could anything at first sight seem more impractical than a body which is so small that its mass is an insignificant fraction of the mass of an atom of hydrogen?”
    — J.J. Thomson.

    If not, just ignore this comment.

  • beezle

    I wonder if you could comment (help?) on three things:

    1) The LEP data which excludes SM mH 2 vs. the new CDF mW which suggests mH 80 +36-26 (at 68% CL) With indications that mH is considerably (my words) greater than 80 GeV/c2, why are plots such as this one from CDF used? Wouldn’t finding Higgs far away from that central value indicate a problem with our understanding?

    2) Is it possible to mistake a SM Higgs for one of the MSSM Higgs?

    3) What of the heavy SM Higgs ~ 160 GeV/c2 ? And why are tau not considered in the WW dilepton channel?

    Apologies if I am asking terribly silly things – I only have an undergrad physics background and Perkins was long ago and well below the level of the easist of the papers at the CDF (and other) site.

  • beezle

    sorry line 1) should read: ..excludes SM mH

  • beezle

    arrrg! im sure you know what I meant :) I guess its a html tag thing.. excludes SM mH less than 114.4 sorry for the extra comment(s)

  • Pingback: Chrononautic Log 改 » Blog Archive » Sine ira et studio()

  • http://quantumnonsense.blogspot.com/ Qubit

    Didn’t the Titanic have one of those domes as well? At least your not in the Ocean, not yet anyway! Lets hope the higgs is no iceburg. We all know what happend to Titanic her on her maiden voyage. The best advice; stay close to Alice there’s almost one in every life boat and I guess at some point we have to see how far the rabbit hole goes!


  • Rich Wilbourn

    Thanks for the great post and all of the behind-the-scenes details. It’s fun to live vicariously through someone is in the hunt for the Higgs!

  • Pingback: mis 3 quarks » Breves: 30/01/07()

  • Pingback: Not Even Wrong » Blog Archive » Short Items()

  • http://www.math.uni-hamburg.de/home/schreiber urs

    Dzero had a deficit where we had an excess! […] Their method was quite similar to ours, so it was hard to escape the conclusion that if they really had a deficit, our excess is more likely to be a statistical fluctuation.

    That’s really weird, it seems.

    Given that apparently any bump of the size we are talking about is something, if not statistically significant, still noteworthy enough to warrant all this attention, isn’t it an all the more remarkable coincidence that two groups see a fluctuation (of whatever sign and magnitude) at the same position?

  • Chromodynamix

    ‘God’, this particle hunt is getting exciting. Like the emergence of a long lost animal from the Berea tar pits.
    First forming a ‘bump’.
    Two sigma is a bit ‘iffy’, but I think the LHC will 4 and 5 sigma the data after it fires up in August 2007.
    Good luck!

  • Pingback: Untitled « parasite blog()

  • Pingback: appletree » Blog Archive » Carnivalia()

  • Pingback: Rumors about the Higgs « A Quantum Diaries Survivor()

  • Pingback: The Atom Smashers « Peculiar Velocity()

  • Pingback: 18 months after the Higgs affair… « A Quantum Diaries Survivor()


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Cosmic Variance

Random samplings from a universe of ideas.

See More

Collapse bottom bar