What is statistical significance?

Statistical significance is a measure used to determine whether an observed difference or relationship in data is likely to be due to a real effect rather than random variation.

In CRO and A/B testing, statistical significance helps evaluate whether the difference between a control version and a test version provides enough statistical evidence to conclude that the observed result is unlikely to have occurred by chance alone.

For example, suppose:

• Original product page: 4.0% conversion rate
• New product page: 4.5% conversion rate

Statistical testing can help determine whether the 0.5-percentage-point difference is supported by enough data to be considered statistically significant.

Common concepts involved include:

• P-value: Measures how compatible the observed result is with a specified null hypothesis.
• Significance level (α): A threshold chosen before testing, commonly 0.05.
• Confidence interval: Shows a range of plausible values for the estimated effect.
• Sample size: Larger, well-designed samples can provide more reliable estimates.

Statistical significance does not mean that a result is necessarily large, important, or guaranteed to repeat. CRO teams should also consider effect size, practical business impact, test quality, and confidence intervals when interpreting results.