A/B Test Calculator (significance & sample size)
Compare two conversion rates: p-value, lift, Wilson and Newcombe intervals, a sample-ratio-mismatch check, and the sample size the next test needs.
p = 0.01232 is below α = 0.05: a difference this large would be unusual if the variants converted equally.
Intervals under each figure are 95% two-sided; the one for the difference is Newcombe's, built from the two Wilson intervals, which stays sensible near 0% and 100% where the textbook ± formula does not.
Sample size for the next test
Detects a change from 5.00% to 6.00%. Uses the confidence level and hypothesis above.
Calculated in the browser. The inputs are kept in local storage.
How to use
- Enter visitors and conversions for the control and the variant; the example compares 500 and 580 conversions out of 10,000 each.
- Choose the confidence level and whether you are testing for any difference or only for improvement.
- Read the rates with their intervals, the difference with Newcombe's interval, the lift and the p-value.
- If a sample-ratio warning appears, stop and find the cause before acting on the result.
- To plan the next test, give the baseline rate, the smallest effect worth detecting and the power, and read the visitors needed per variant.
About this tool
ShowHide
Enter visitors and conversions for two variants and get each conversion rate with a confidence interval, the difference and relative lift, the z statistic and p-value of the two-proportion test, and — below — the number of visitors a future test needs to detect a given effect. Three details separate this from a quick online calculator. First, the intervals. The ± 1.96 standard error formula taught for a proportion misbehaves for small counts and rates near zero, which is where conversion rates live; each rate here gets a Wilson score interval or, optionally, the exact Clopper–Pearson interval, and the difference gets Newcombe's hybrid interval built from the two Wilson intervals, the method his 1998 comparison of eleven alternatives recommended. Second, the sample-ratio check. If traffic was meant to be split evenly and the visitor counts are too far apart to be chance — a chi-square test below p = 0.001 — the tool says so above the result, because a mismatched split means users were assigned or counted wrongly and the comparison is no longer fair; this is among the most common reasons A/B results are wrong, and the least often checked. Third, the planner is honest about what it assumes: it uses the standard normal-approximation formula at your confidence level and power, and the sizes it gives were checked by simulating hundreds of thousands of experiments at those sizes, which rejected at the intended rate to within half a percentage point. What no calculator can fix is peeking: stopping a test the first time p dips under 0.05 inflates the false-positive rate far beyond 5%, so fix the sample size first and read the result once.
Frequently asked questions
Why not use the ± 1.96 standard error interval?
Because it is unreliable for proportions near 0% or 100% and for small samples: it can extend below zero, and its real coverage often falls well short of the stated 95%. The Wilson interval fixes this at no cost in complexity, and Newcombe showed in 1998 that combining two Wilson intervals gives a far better interval for a difference than the simple formula. Clopper–Pearson is exact in the sense of never under-covering, at the price of being wider.
What is a sample ratio mismatch?
It is when the number of users in each variant differs from the planned split by more than chance allows. It usually means a bug — a redirect that loses some users, a bot filter applied to one side, tracking that fires differently — and it biases the comparison in unknown ways. The tool tests the visitor counts against an even split and warns below p = 0.001, the threshold commonly used by experimentation platforms.
Can I stop the test as soon as it becomes significant?
Not if you want the p-value to mean what it says. Checking repeatedly and stopping at the first p below 0.05 gives many more chances for noise to cross the line, and the true false-positive rate can end up several times higher. Decide the sample size in advance with the planner and evaluate once, or use a method designed for sequential monitoring.
How is the sample size calculated?
With the usual formula for comparing two proportions: the sum of the critical value for your confidence level times the standard deviation under no difference and the critical value for your power times the standard deviation under the effect, squared, divided by the squared difference. It is per variant, so the total is twice the figure. It was checked against simulation: at the sizes it returns, simulated experiments reached the intended power.
Why is relative lift shown without an interval?
Because the interval for a ratio of two rates is noticeably asymmetric and needs a different method from the one for a difference; a symmetric ± around the lift would be misleading. The difference in percentage points, with Newcombe's interval, carries the same information without that trap, and a lift is easy to read off it: divide by the control rate.
Related tools
t-Test Calculator (paired, Welch, one-sample)
One-sample, paired and two-sample t-tests from raw data or summary statistics: t, degrees of freedom, p-value, confidence interval and Cohen's d.
Math & Statistics
What Is My Screen Resolution? (Hz and Scaling)
See your screen resolution, physical pixels, scaling (devicePixelRatio), browser window size, refresh rate, colour depth and HDR support at a glance.
Everyday
Bloom Filter Calculator: Size, Hashes & Error Rate
Size a Bloom filter for n items at a target false-positive rate, or get a filter's error rate or capacity, with the best number of hash functions.
Developer
Rule of Three Calculator
Solve a proportion a/b = c/d — enter any three values, leave one field empty, and the fourth is found by cross-multiplication.
Math & Statistics
Fibonacci Sequence Generator
Generate the first N Fibonacci numbers with exact big-integer precision — with the last term, the running sum and the golden-ratio approximation.
Math & Statistics
Mann–Whitney U & Wilcoxon Signed-Rank Test
Rank-based tests for two independent samples or paired data: W or V, an exact or corrected p-value, rank-biserial r and the Hodges–Lehmann estimate.
Math & Statistics