__Login__or

__Take a Tour__!

I'm not normally a stay-in-your-lane kind of guy. I went to grad school for physics and now I run the pharmacology department at a small drug company. Never cared much for staying in my lane. That said, this dude should stay in his lane.

There's a big difference between P-value manipulation and a small effect size. He is confusing the two things pretty starkly. When you design a study, you take everything you know about the data *a priori*, which is usually something about the delta between, say a pristine sample and your perturbed cohort, and also something about the variance of either or both cohorts. Then you make an assumption about what effect size ** you** would count as significant, and you calculate how many subjects you need to study to observe the effect with, say, an 80% chance of success

*if the effect you assumed is real*. That is how good stats is done. And the assumed effect size was chosen precisely because you chose it to be meaningful.

Finding a true mean difference that is so small as to be meaningless often requires more subjects that is feasible to study. So I think in the case when stats are done correctly (which is to say prospectively not retrospectively), effect size and statistical significance should be simpatico.

That said, there is a who other topic of relative vs. absolute risk, and of course there are policy tradeoffs that can't be settled by stats. All one can say is here's what I set out to measure and here's what I actually measured. Then it's up to society to figure out what to do about it. If that's, e.g., a new cancer screening tool and it reduces deaths from some specific cancer by 70% because it catches it early, you'd say great, why not mandate it. But then you find out that only 1/10,000 people develop that type of cancer, so that "70%" actually means less than 0.0001% of the population. Well now you have to think about (a) how much does the test cost; (b) how invasive is it; and (c) what is the false positive rate?

I guess my point is that there's no simple way to judge what the tradeoff between effect size and statistical significance is, because we live in a world where nothing exists in a vacuum. Each new study of each new intervention needs to be evaluated on its own terms in its own reality. Making blanket statements that A matters more than B is plainly wrong.