A/B Test Significance Calculator
Objective: Evaluate A/B test results to determine whether a conversion-rate difference is statistically significant.
Results Summary
The test is significant.
Variant B (8.87%)
converted at +16.75%
(+1.27pp)
vs Variant A (7.60%).
Key Takeaway: The observed difference is large enough relative to sample noise to indicate a winner with 99.96% confidence.
Show Your Work
1. Conversion Rates
CVR = conversions / visitors
A: 912 / 12,000 = 0.0760 = 7.60%.
B: 1,047 / 11,800 = 0.0887 = 8.87%.
The conversion rate is the share of visitors who completed the desired action.
2. Relative Lift
Lift = (CVR B - CVR A) / CVR A
(8.87% - 7.60%) / 7.60% = +16.75%.
Lift measures how much Variant B changed performance relative to Variant A.
3. Pooled Rate
p = (conversions A + conversions B) / (visitors A + visitors B)
(912 + 1,047) / (12,000 + 11,800) = 0.0823 = 8.23%.
The z-test assumes no difference between variants first, so it uses one combined conversion rate as the baseline.
4. Standard Error
SE = sqrt(p * (1 - p) * (1/nA + 1/nB))
sqrt(0.0823 * 0.9177 * (1/12,000 + 1/11,800)) = 0.003563.
Standard error estimates how much random variation to expect between the two sampled conversion rates.
5. Z-Score
z = (CVR B - CVR A) / SE
(0.0887 - 0.0760) / 0.003563 = 3.5724.
The z-score shows how many standard errors separate Variant B from Variant A.
6. P-Value
p-value = two-tailed probability from |z|
z = 3.5724 gives p-value = 0.000354.
The p-value estimates how likely this difference would be if the variants truly had the same conversion rate.
7. Statistical Significance
Significance = (1 - p-value) * 100
(1 - 0.000354) * 100 = 99.96%.
This app uses the significance percentage as a readable confidence-style score for the test result.
8. Wilson Confidence Intervals
95% CI = Wilson interval for each conversion rate
A: 7.14% to 8.09%. B: 8.37% to 9.40%.
These ranges show plausible values for each variant's true conversion rate, accounting for sample size.
Variant Data
| Metric | Variant A | Variant B |
|---|---|---|
| Visitors | 12,000 | 11,800 |
| Conversions | 912 | 1,047 |
| Non-conversions | 11,088 | 10,753 |
| Conversion rate | 0.0760 (7.60%) | 0.0887 (8.87%) |
| 95% confidence interval | 7.14% to 8.09% | 8.37% to 9.40% |
Test Statistics
| Absolute difference: Variant B vs Variant A | +1.27 percentage points |
|---|---|
| Relative lift: Variant B vs Variant A | +16.75% |
| Z-score | 3.5724 |
| Two-tailed p-value | 0.000354 |
| Statistical significance | 99.96% |
Methodology
Significance uses a two-proportion, two-tailed z-test. Error bars and shaded chart bands use 95% Wilson confidence intervals. The curves use beta distributions to visualize plausible conversion-rate ranges for each variant.
v0.1.23 · Updated Apr 18, 2026