Minimum Detectable Effect vs. Statistical Power in A/B Tests

The minimum detectable effect (MDE) is the smallest difference an experiment is designed to detect under chosen assumptions. Statistical power is the probability of detecting an effect of a specified size if that effect is real. MDE describes the design target; power describes the design’s ability to find it.

MDE is a planning threshold, not a promise

A smaller MDE means the experiment is intended to distinguish subtler differences from random variation. Detecting a smaller change generally requires more observations, a longer test, or a more precise measurement. The MDE should reflect a difference that would matter to the decision, not an arbitrary preference for a small percentage.

An experiment can produce a result smaller than its MDE, but the study may not have enough evidence to distinguish that result from noise. Conversely, an observed difference larger than the MDE still needs uncertainty and implementation checks.

• Choose the outcome metric before calculating sample size.
• Use a realistic baseline rate and allocation ratio.
• Set an MDE based on practical value and feasible traffic.

Power describes detection probability

Power is commonly chosen during experiment planning alongside a false-positive threshold. Higher power reduces the chance of missing an effect of the chosen size, but it usually requires more sample. It does not mean the test has that probability of proving the treatment works after any particular result.

Low power can make a real but modest effect difficult to detect. A non-significant result from a small test is not strong evidence that the variants are identical; it may simply be inconclusive.

• Plan sample size before observing the outcome.
• Avoid stopping early just because a dashboard looks favorable.
• Report confidence intervals and practical impact with the test result.

Plan the experiment as a decision

A useful design connects traffic, baseline performance, MDE, power, and the cost of waiting. If the necessary sample is unrealistic, consider a higher-impact change, a more frequent outcome, or a different research method rather than lowering the evidence standard after launch.

Keep the test’s assignment, exposure, and outcome definitions stable. Check sample-ratio imbalance and instrumentation before interpreting a result. Power calculations cannot repair broken event collection or a test that exposes different audiences to each variant.