Comments on: Deceiving with data

By: Divalent

Divalent — Wed, 08 Aug 2012 19:12:26 +0000

“… about 1 in 20 of which will be completely spurious …”

Er, not exactly. All of them could be spurious. ~1 out 20 things you test for at the 0.05 significance level will be “spurious”. What fraction those are the pool of things you find to be statistically significant depends on how many things you test for actually are real correlations.

By: Siod

Siod — Wed, 08 Aug 2012 19:08:57 +0000

#3, I don’t think there’s anything necessarily wrong with interpretation. What I meant is that people will look at a body of supporting evidence in support of their intuitions and then draw a conclusion from that evidence. This is interpretation. And then other people (experts typically) will have reviewed *all or most* the literature or evidence and draw a sound conclusion (that is still interpretation, but it’s interpretation given all the evidence).

The problem comes in when 1) the people that are drawing conclusions don’t realize they’re interpreting a body of evidence and 2) the audience doesn’t realize how the person drawing conclusions interpreted the body of evidence or what evidence they decided to include in that body.

Of course, there are some conclusions that are just painfully obvious and undeniable, but these are typically in the harder sciences like physics.

By: Razib Khan

Razib Khan — Wed, 08 Aug 2012 18:18:39 +0000

#3, perhaps the example is not valid, but i often put up my GSS variables, etc. but very few commenters actually extend/critique my analysis through replication. that’s one reason i get lazy and just put up the link. i’ll provide variables if asked, but that rarely happens.

By: James Winters

James Winters — Wed, 08 Aug 2012 18:00:30 +0000

Me and Sean recently published a paper on a related problem in linguistics Constructing Knowledge: Nomothetic Approaches to Language Evolution. We argue that these sorts of statistical studies should be confined to hypothesis-generating, with other methods (e.g. experiments) then being used to test these assumptions and provide greater explanatory power.

By: Chad

Chad — Wed, 08 Aug 2012 17:50:52 +0000

I have absolutely no problem with drawing conclusions. That’s what you do. You get data, you analyze, and you try and draw conclusions. The way you overcome cherry picking is to expose it. The way you expose it is by demanding absolute clarity in methods.

By: Siod

Siod — Wed, 08 Aug 2012 16:50:24 +0000

This drives me insane. People don’t realize it’s intellectually dishonest to cherry pick, and they don’t realize drawing conclusions is interpretation. We have make this behavior as taboo as just making shit up.

By: Chad

Chad — Wed, 08 Aug 2012 15:51:17 +0000

The solution to this is two fold:

1) All raw datasets (regardless of field) once published should be made open access and deposited in a databank somewhere.

2) Journals must demand more stringent and detailed methodology reports. Far far far too often journals allow people to get away with saying “program X was used to analyze data” and that is it. The authors should be explicit in all aspects of the analysis, the settings/arguments, the code, everything. You give me a set of data and I can create all kinds of spurious results and if all I write in the methods section is “program X was used” then its very easy for me to pull a fast one on people.