The four inputs, in plain words
- Baseline conversion rate. How often the page converts today, for the goal you test (usually orders). Take it from the last 4 to 8 weeks of the same page, not from the whole store.
- Minimum detectable effect (MDE), relative. The smallest change you want the test to catch. 10% on a 3% baseline means 3.3%. If the real change is smaller, the test will probably miss it.
- Significance (confidence), usually 95%. Controls false winners. At 95%, a test of two identical versions shows a "winner" about 1 time in 20.
- Power, usually 80%. Controls missed winners. At 80%, a real change of your MDE size shows up 4 times in 5. The other time, the test says "no difference".
Worked examples
All rows use 95% significance, 80% power and two variants. The duration column assumes 10,000 visitors a week into the test, split evenly.
| Baseline | MDE (relative) | Variant must reach | Visitors per variant | Total | Weeks at 10,000/week |
|---|---|---|---|---|---|
| 3% | 10% | 3.3% | 53,211 | 106,422 | 11 |
| 3% | 20% | 3.6% | 13,914 | 27,828 | 3 |
| 3% | 30% | 3.9% | 6,455 | 12,910 | 2 |
| 2% | 20% | 2.4% | 21,109 | 42,218 | 5 |
| 5% | 10% | 5.5% | 31,234 | 62,468 | 7 |
Two things stand out. Halving the MDE from 20% to 10% nearly quadruples the traffic you need (13,914 to 53,211 per variant). And a lower baseline needs more traffic for the same relative change: 2% needs 21,109 per variant where 3% needs 13,914.
Stricter settings cost traffic too. For the 3% / 20% row, 90% power needs 18,626 per variant, and 99% significance needs 20,704.
How long to run the test
Divide the total by your weekly traffic into the tested page and round up to whole weeks. Buyers on a Sunday evening do not behave like buyers on a Tuesday morning. A test that stops after 10 days counts some weekdays twice and others once, so the result reflects a week that never happens.
Two more rules:
- Run at least one full week, even when the calculator says you need fewer visitors.
- Decide the sample size before you start and do not stop early because the result looks good. Use the A/B test significance calculator once you reach it.
Low traffic? Fix, do not test
A store with 2,000 visitors a week and a 2% rate needs 22 weeks to detect a 20% lift (42,218 visitors in total). Most stores cannot hold a page still for five months.
At that traffic, testing small changes is the wrong tool. Fix the problems that are clearly broken: a hidden add-to-cart button on mobile, a shipping cost that appears only at checkout, product photos that load after 5 seconds. You do not need a test to know those cost sales. Save testing for questions with no obvious answer, such as which of two headlines works better. Our CRO audit guide shows how to find the clear problems first.
Method: two-sided two-proportion z-test with a pooled p̄; sample sizes match Evan Miller's calculator.