What is "statistical power"

n. (context statistics English) The probability that a statistical test will reject a false null hypothesis, that is, that it will not make a type II error, producing a false negative.

The power or ''' sensitivity ''' of a binary hypothesis test is the probability that the test correctly rejects the null hypothesis (H) when the alternative hypothesis (H) is true. It can be equivalently thought of as the probability of accepting the alternative hypothesis (H) when it is true—that is, the ability of a test to detect an effect, if the effect actually exists. That is,

$$\mbox{power} = \mathbb P\big( \mbox{reject } H_0 \big| H_1 \mbox{ is true} \big)$$

The power of a test sometimes, less formally, refers to the probability of rejecting the null when it is not correct, though this is not the formal definition stated above. The power is in general a function of the possible distributions, often determined by a parameter, under the alternative hypothesis. As the power increases, there are decreasing chances of a Type II error (false negative), which are also referred to as the false negative rate (β) since the power is equal to 1−β, again, under the alternative hypothesis. A similar concept is Type I error, also referred to as the “false positive rate” or the level of a test under the null hypothesis.

Power analysis can be used to calculate the minimum sample size required so that one can be reasonably likely to detect an effect of a given size. For example: “how many times do I need to toss a coin to conclude it is rigged?” Power analysis can also be used to calculate the minimum effect size that is likely to be detected in a study using a given sample size. In addition, the concept of power is used to make comparisons between different statistical testing procedures: for example, between a parametric and a nonparametric test of the same hypothesis.