Free A/B Test Significance Calculator
Find out if your A/B test winner is real or just chance. Enter visitors and conversions for each version, or plan your next test with the sample size and duration calculator.
A/B Test Significance Calculator: Find out if your A/B test winner is real or just chance. Enter visitors and conversions for each version, or plan your next test with the sample size and duration calculator. It is free, needs no sign-up and runs in your browser. How to use it: 1) enter visitors and conversions for version A and version B, and add C if you tested three versions; 2) choose the confidence level and a one-sided or two-sided test, then press Check significance; 3) read the verdict, uplift and p-value, or switch to Sample size and duration to plan the next test. The calculator compares conversion rates with a two-proportion z-test, the standard method for website and ad tests. It tells you whether the difference between versions is larger than random noise would usually produce.
How the A/B test calculator works
The calculator compares conversion rates with a two-proportion z-test, the standard method for website and ad tests. It tells you whether the difference between versions is larger than random noise would usually produce. Everything runs in your browser.
Formulas
- Conversion rate p = Conversions / Visitors
- Pooled rate = (cA + cB) / (nA + nB)
- z = (pB – pA) / sqrt(pooled x (1 – pooled) x (1/nA + 1/nB))
- p-value from the normal distribution, two-sided or one-sided
- Relative uplift = (pB – pA) / pA
- Sample size per variant = (z(1-a/2) x sqrt(2 x pbar x (1 – pbar)) + z(1-b) x sqrt(p1(1-p1) + p2(1-p2)))^2 / (p2 – p1)^2
Each conversion rate gets a Wilson confidence interval, and the difference gets an unpooled interval. Example: 100 of 1,000 visitors on A and 130 of 1,000 on B is a 30% uplift with a two-sided p-value of about 0.036, significant at 95%.
Three variants
With a C version, each challenger is compared with A using a Bonferroni corrected threshold so the overall false positive rate stays at your chosen level. The tool also warns about very small samples and uneven traffic splits.
Limitations
The test assumes independent visitors and a fixed sample size. Checking results every day and stopping as soon as they look significant inflates false positives, so set the sample size first. It does not handle revenue per visitor or Bayesian analysis.
A/B Test Significance Calculator FAQs
What does statistically significant mean in an A/B test?
It means the difference between versions is unlikely to be caused by random chance alone. At 95% confidence, a difference this large would appear less than 5% of the time if the versions truly performed the same. It does not guarantee the uplift size, which is why the tool also shows a confidence interval.
What is a good p-value for an A/B test?
Most teams use 0.05, which matches 95% confidence. A p-value below 0.05 is treated as significant. For risky or costly changes you can use 0.01, meaning 99% confidence. For quick, low risk tests some teams accept 0.10. Choose the threshold before the test starts, not after.
How many visitors do I need for an A/B test?
It depends on your current conversion rate and the smallest uplift you want to detect. With a 10% baseline and a 20% relative uplift, you need about 3,841 visitors per version at 95% confidence and 80% power. Smaller effects or lower baselines need far more traffic.
How long should I run an A/B test?
Run it until you reach the planned sample size and for at least one full week, ideally two, so weekday and weekend behaviour are both included. Enter your daily visitors in the sample size tab to get the number of days. Avoid tests longer than about eight weeks.
Want our experts to implement these improvements?
Get professional support from Hayyat Networks. Request a free consultation.