However I think we need all the tools we can get, to fight fraud. Stats, whistle-blowers, data sleuths and just plain nosy people. All have a role to play.

Then there’s the issue of misleading science that’s not fraud. There is a lot of that and I think it’s more harmful than fraud in the aggregate. But it’s a separate issue.

I suspect Jones was quietly asked to leave. MD heads of departments don’t discuss that sort of thing with mere MS employees. Within months he was working at another prestigious research hospital, but the rumor mill (which I had nothing to do with) was passing the fraud information around.

I don’t know if his fraud would have showed up in a single statistical analysis, if he was consistently adding, say 40 grams, to each animal’s weight. The data distribution would still be the same. One would need to compare his results to others in the field to see that he was getting much better results. Now the faked longer survival rates might raise a red flag. But the data could just mean he was a much better surgeon than the other researchers. Had Dr. Smith not by chance gotten curious and looked at his partner’s rats, a paper might have been published that would have had Dr. Jones looking like a genius.

One observation directly about the blog essay: They are using methods other than just testing randomness of data. Confronting the researcher before you make the information public is an ethical requirement. As another reader pointed out, on rare occasions, the data will show little variation and still be legitimate data.

Another problem in research is that MDs and PhDs sometimes think that they can just go to a computer and do a statistical test. If it comes out with a good p value, they think they have a clinically significant finding and publish it. You at least need to plot individual data points to see the pattern and consult a statistician to evaluate your data. One PhD I worked with thought a t test was all he needed and got a grant to continue his research. The people giving grants need to check raw data, not just p values, before giving out money. Upon in-depth analysis of his data, the main contribution to the statistical difference was a small number of outliers, with a very tiny trend for the two groups to differ. The paper was never published, but could have been published as a “no the theory didn’t pan out” paper. Researchers don’t like to publish papers that show a failed hypothesis and many journals don’t either.

In my 15 years of research, those are the only two cases I knew about at my institutions. One was fraud and the other probably ignorance maybe combined with ego. I say “ego” as some researchers may think they knew enough that they didn’t need a statistician.

There is a lot of pressure to publish or perish. Sometimes the bad data are fraudulent and sometimes the researcher just doesn’t know statistics well enough. So when you read a paper and no statistician is listed as author, or thanked in the acknowledgements, have a healthy skepticism about it. If you are on a committee awarding grants, insist on seeing the report of a PhD statistician.

Presenting preliminary data at meetings may not always help reveal mistakes, although an MD I once worked for did point out a major flaw in someone’s research at a meeting and the researcher admitted he didn’t know that his method was seriously flawed. My boss said he could hear groans all over the room, so the researcher was easily convinced.

I saw a graph by a well-respected scientist at an international meeting. Since part of my job was to consult with our institution’s PhD statistician about our department’s data, I could see that there was one data point that pushed this researcher’s data to statistical significance. He had 5 data points to calculate an R value. Four formed an almost perfect square and the fifth was far out, but in line with two points, so that the R looked great. To those of you who don’t have a background in correlations and R values, data that forms a square means you have no correlation at all. One grad student had the courage to politely point that out. He was shot down. This was a prominent researcher presenting his pet theory and no one else in a very large auditorium had the courage to support the grad student. No one at the meeting knew me, so I figured if I spoke, it would be a waste of time. Even respected scientists can want so much for their theory to be true that they won’t see the data doesn’t support it. He was doing research way out of his field, but it was in my field. To be fair, I expected that if he got enough data he would eventually support his theory, as similar research had already done. I think he was on the right track, but he claimed he had made it all the way to Central Station, while he was actually still in the boondocks.

So there are many ways to get bad data, but in my opinion, the biggest “sin” is to be human and want so badly to have good data, that you don’t evaluate your own work sufficiently.

One of these days I’ll learn to type.

In one case I remember data on hundreds of explosives being timed by hand with a stop watch and then being presented to an accuracy of a hundreth of a second. Now I suspect that you might be able to time the start of hundreds of explosions outdoors by hand to an accuracy of a hundreth of a second if you had come to earth as a child in a rocket from the planet Krypton. But I doubt a mere mortal could do it repeatedly. Bad quality data. But they got away with it because almost no one questioned how the data was timed and because they argued that the data might have problems but gee whiz they had so many data points.

I suspect the land based temperature measurements being used by our good friends in the well respected field of climatogy is mostly garbage data. I’m not an expert in the area. But if you look at the pictures of some of the temperature measurement sites the sites don’t look too Kosher. And the secrecy surrounding the computer codes that manipulate the data is a red flag.

I suspect we only seeing a tiny bit of the fraus that happens in science.

Of course the more organized, crafty and devious fraudsters will eventually use Monte Carlo simulations to generate fake data. Assuming they have not done so already!