a thoughtful web.
Good ideas and conversation. No ads, no tracking.   Login or Take a Tour!
comment by b_b
b_b  ·  1486 days ago  ·  link  ·    ·  parent  ·  post: Pubski: October 21, 2020

Public university. Lots of NIH money. I think I can describe in better detail what happened without crossing a line. Basically the guy published the results of an experiment (a drug trial of an animal model of a disease), which he reported to be a several day experiment, and was described as such in the methods. When I got access to the full data, the experiment was actually a several week experiment. The "statistically significant" effect he observed was an obvious random occurrence, as there was no indication that there were any between group differences at any other time point. So what happened, obviously, is that they just decided to publish the experiment as if it were always supposed to end at the time point that he published it to be.

That might seem not so bad to people outside the field, but in the business that is considered straight up lying, no wiggle room, no grey area--just plain lying. When you do drug trials you have to always describe what you are going to measure before the experiment takes place. The reason for that is that you can calculate, based on some assumptions about the data, what the a priori odds of a "significant" finding are. The trouble with not doing that is that you can measure a whole bunch of stuff and then look for things that appear different between groups, which there are bound to be if you do enough measurements. Then you can calculate what the a priori odds would have been had you not run the trial, and say, "Great! There's only a 1% chance of making this finding by chance, so it must be true." But it isn't. It's like throwing 5 heads in a row and then convincing yourself that the coin is loaded, because there's only a 3% chance of that happening, so it meets the p value requirement. But there's a 3% chance of any given series of 5 coin flips, and one of them has to happen.

It's offensive to me that the dude is out there getting grant money based on this horseshit. But the grant game is fucked anyway, so whatever. What's really annoying to me is that he's out there raising drug money for this. Like you want to waste millions of dollars and thousands of hours of time just to boost your ego a little? It doesn't make any sense to me. What kind of an ego gets off on an obvious lie? Where is the accomplishment to be proud of? I guess the real problem is with science. The incentives are set up to make people lie, so I suppose we're all irresistible to that force at some point.

Edit: I should point out that I emailed the potential financier about this, which he just straight up forwarded to the scientist in question while doing nothing except redacting my name and editing out the part where I called it "fraud." So if this were to come up from his research office, it would be very obvious where it came from.