P-Values Corrected For Study Power More Accurately Represent Statistical Significance
The traditional definition of statistical significance is a p-value of less than or equal to 0.05. The p-value is the probability that the research data did not find anything significantly new.
When the p-value is 0.05, there is a 1 in 20 chance the research findings are due to normal variation. The p-value only says how likely the research findings are due to normal variation. It does not say how likely the research findings represent something new and statistically significant.
The scientific method compares two hypotheses: first is the null hypothesis, which is that nothing is going on; second is the alternative hypothesis, that something new and significant is happening. For example, imagine that a new medicine is developed that the researcher thinks may lower blood pressure. The null hypothesis is that it DOES NOT lower blood pressure, and the alternative hypothesis is that it DOES lower blood pressure.
The p-value indicates the probability that the null hypothesis is true, the probability that the new medicine does not lower blood pressure. It does not indicate the probability that the new medicine does lower blood pressure.
This find distinction leads us to a new definition of statistical significance. Instead of relying on p-values, which essentially represent the specificity of the test statistic, we find it more useful to know the probability that a positive test statistic indicates that the alternative hypothesis is true, not the probability that the null hypothesis is true.
From the above we can derive the following: a p-value alone does not indicate the likelihood of the alternative hypothesis being true. The positive predictive value of a test statistic requires us to know both study power and the p-value. The positive predictive value is the equal to: [power/(power + p-value)]. To achieve 95% confidence that a test statistic represents statistical significance, the cutoff p-value (i.e. maximum p-value) needs to be adjusted by study power.
Using standard definitions for 2 x 2 contingency tables, statistical significance only occurs when: [ a / (a+b) >= 0.95 ], or in other words, that the positive predictive value is 95% or higher.
REFERENCE: Statistical Significance: a New Definition by Tom Heston, MD
Originally published: Jan 15, 2013 on the now defunct Yahoo! Voices http://voices.yahoo.com/a-definition-statistical-signifance-11963283.html http://goo.gl/Vz04UJ
