2.3 years ago by
Given a test of some effect where p = 0.05 according to some
test-statistic t, you expect to see a test-statistic
greater than t (or equivalently, a p-value < 0.05) on a test
of random data 5% of the time.
By accepting p = 0.05 for a single test, you're
accepting that there's a 5% chance that effect or difference
may be due to random variation--and that there may not be
an actual "effect" at all.
A consequence of this is that, if the magic number for publication
is p <= 0.05, then, the expected number of publications which have
erroneously rejected their null hypothesis is:
p_average * N_publications
so, given a fixed p-value cutoff, we can expect the number of falsely
rejected null hypothesis to increase as more papers are published.