A/B test significance calculator
Most tests get called too early. This tells you whether the winner is real or whether you are looking at noise.
Your numbers
Enter the totals for each version. Do not stop the test just because it looks good today.
Result
What confidence actually means
Confidence does not mean "B is 95% better." It means: if the two versions were really identical, you would see a gap this large less than 5 times in 100 by pure chance.
That is a much weaker claim than most people think, and it is why the number matters. Flip a coin ten times and you might get seven heads. That does not make the coin biased.
The mistake almost everyone makes
Watching a test daily and stopping the moment it crosses 95% is called peeking, and it badly inflates your false-positive rate. If you check every day and stop at the first significant moment, you will "find" winners that do not exist a large share of the time.
Decide your sample size before you start. Then look once.
Two things to check before you believe a result
- At least 25 conversions per version, and preferably 100. Percentages built on tiny numbers move enormously when one more person converts.
- Run for whole weeks. Traffic behaves differently at weekends. A test that runs Tuesday to Friday is measuring the working week, not your audience.
A worked example
1,000 visitors each. A converts 50 (5%), B converts 65 (6.5%). That is a 30% relative improvement, which sounds decisive — but it only reaches about 90% confidence. Not enough. Double the traffic with the same rates and it passes comfortably.
This is why small sites struggle to test small changes: you simply cannot gather enough data to detect a 10% improvement in a reasonable time. Test bigger changes instead.
Common questions
Is 95% confidence a rule?
It is a convention, not a law. For a low-risk change, 90% may be enough. For something expensive or hard to reverse, wait for 99%. What matters is choosing the bar before you look at the result, not after.
How long should I run a test?
Until you hit your planned sample size, and always in whole weeks. Two full weeks is a sensible minimum for most sites, because it covers both weekends.
Can I test more than two versions?
You can, but the maths changes: testing many versions at once raises the chance that one looks like a winner by luck alone. With two versions this calculator is exactly right.
Want this checked on your real account?
A tool can only work with the numbers you give it. If you want someone to look at what is actually happening in your ad account, the first conversation is free.
Book a free consultation