What the p-value means, and what it does not
The p-value answers one narrow question: if A and B really converted the same, how often would you see a difference at least this large by chance? A p-value of 0.03 means about 3 times in 100.
It does not tell you:
- the chance that B is better (a p-value of 0.03 is not "97% sure B wins");
- how big the improvement is (use the confidence interval for that);
- that the result will hold next month, on other traffic, or on another page.
Worked example
Version A had 12,000 visitors and 360 orders. Version B had 12,000 visitors and 420 orders.
- Rates: A 3.0%, B 3.5%. Relative uplift: +16.7%
- z = 2.18, two-sided p-value = 0.029
- 95% confidence interval for the difference: +0.05 to +0.95 percentage points
The result is significant at 95% (0.029 is below 0.05), and the interval does not include zero. But look at the range: the true lift could be almost nothing (+0.05 points) or close to a full point. You have a winner, and you do not know its size well.
90%, 95% or 99%?
Now a closer test: A has 10,000 visitors and 300 orders (3.0%), B has 10,000 visitors and 345 orders (3.45%, +15%). The p-value is 0.072.
| Confidence level | Significant when p is below | This test |
|---|---|---|
| 90% | 0.10 | Significant |
| 95% | 0.05 | Not significant |
| 99% | 0.01 | Not significant |
Pick the level before the test, not after you see the p-value. Use 95% by default. 90% is reasonable for a cheap change that is easy to undo. Use 99% when the change is expensive to build or hard to roll back.
One-sided or two-sided
This calculator uses a two-sided test: it asks if B is different from A in either direction. A one-sided test only asks if B is better, which halves the p-value and makes winners easier to declare. It also hides the case where B hurts sales. Some testing tools use one-sided tests by default, so their p-values can look better than ours on the same data.
The peeking problem
If you check the result every day and stop the first time p drops below 0.05, your real false-winner rate is far above 5%. Evan Miller's article "How Not To Run an A/B Test" explains why. Fix the sample size in advance with the A/B test sample size calculator, and read the result once you reach it.
Statistical vs practical significance
With enough traffic, a 0.02-point difference becomes significant. That does not make it worth the cost of the change. Ask two questions: is it real (the p-value), and is it big enough to matter (the interval, multiplied by your traffic and order value)?
Check the split first: sample ratio mismatch
If you planned a 50/50 split, the visitor counts should be close. 10,000 against 9,400 is far too uneven to be chance. That usually means a bug: a redirect that loses visitors in one version, or a bot filter that treats the versions differently. When the split is wrong, the p-value is not reliable, whatever it says. Fix the setup and run the test again.
And if your store does not have the traffic for clean tests, start with the problems that need no test. Our CRO audit guide shows how to find them.
Method: two-sided two-proportion z-test with a pooled standard error; the confidence interval uses the unpooled standard error.