Free AB Test Calculator

Understanding the A/B Test

The A/B test — also called a split test or two-proportion Z-test — is a statistical technique used to compare two groups and assess whether their difference on a specific outcome is statistically meaningful. Marketers, product developers, and researchers rely on it to decide, for example, whether a new website layout yields a higher conversion rate or whether a medical treatment outperforms a control. The A/B test belongs to the family of Z-tests and is designed to evaluate whether any observed gap between two proportions is real or simply due to random chance.

Every A/B test starts with a null hypothesis, which assumes that the two populations have the same true proportion. The goal is to generate enough evidence to reject that null hypothesis — i.e., to prove that the groups truly differ. The tool presented here automates the entire calculation, requiring only the sample sizes and the number of positive outcomes (or conversion rates) for each group, along with your preferred confidence level.

What Does “Statistical Significance” Mean?

Statistical significance describes an experimental outcome so extreme that it would be very unlikely to occur by random chance alone. Suppose you flip a coin one hundred times and observe sixty heads. Is the coin biased, or did you simply get lucky? A significance test helps you make that call. If the result is statistically significant, you can reasonably attribute it to an underlying factor (the coin is unfair) rather than to sampling error.

The Mechanics of a Two-Proportion Z-Test

To conduct an A/B test, you need data from two groups:

• Sample sizes: $n_{1}$ and $n_{2}$
• Number of positive outcomes: $t_{1}$ and $t_{2}$

From these, calculate each group’s proportion:

$$p_{1} = \frac{t_{1}}{n_{1}}, \qquad p_{2} = \frac{t_{2}}{n_{2}}$$

Then compute the overall pooled proportion — the combined positive rate across both groups:

$$p = \frac{t_{1} + t_{2}}{n_{1} + n_{2}}$$

The test statistic is the Z-score, which measures how far apart the two proportions are relative to the expected variation:

$$Z = \frac{p_{1} - p_{2}}{\sqrt{p(1 - p)\left(\dfrac{1}{n_{1}} + \dfrac{1}{n_{2}}\right)}}$$

Because this is a two-tailed test (we are looking for any difference, not a specific direction), we split the significance level (alpha) in half. The decision rule compares the absolute Z-score to a critical value $Z_{\alpha/2}$, which depends on the chosen confidence level:

Confidence Level	Alpha Level ($\alpha$)	Critical Z-score ($Z_{\alpha/2}$)
90%	10%	1.645
95%	5%	1.960
98%	2%	2.326
99%	1%	2.576

If $|Z| > Z_{\alpha/2}$, the result lies in the rejection region, and you reject the null hypothesis — the difference is statistically significant. If $|Z| < Z_{\alpha/2}$, you fail to reject the null hypothesis; the observed gap could reasonably be due to chance.

Step-by-Step Procedure

1. Choose a confidence level (commonly 90%, 95%, 98%, or 99%).
2. Determine the critical Z-score from the table above.
3. Calculate $p_{1}$, $p_{2}$, the pooled proportion $p$, and finally the Z-score using the formulas.
4. Compare $|Z|$ with $Z_{\alpha/2}$.
5. Conclude: reject the null hypothesis if $|Z| > Z_{\alpha/2}$; otherwise, accept it.

A Worked Example

Consider two website pages tested for conversion:

• Group A: $n_{1} = 40$, $t_{1} = 2$ → conversion rate $p_{1} = 5\%$
• Group B: $n_{2} = 12$, $t_{2} = 5$ → conversion rate $p_{2} \approx 41.7\%$

Pooled proportion:

$$p = \frac{2 + 5}{40 + 12} = \frac{7}{52} \approx 0.1346$$

Z-score calculation:

$$Z = \frac{0.05 - 0.4167}{\sqrt{0.1346 \times 0.8654 \times \left(\frac{1}{40} + \frac{1}{12}\right)}} \approx \frac{-0.3667}{0.1123} \approx -3.27$$

For a 90% confidence level, $Z_{\alpha/2} = 1.645$. Since $|Z| = 3.27 > 1.645$, the difference is statistically significant. Group B’s higher conversion rate is unlikely to be a fluke.

You can replicate this example using the A/B test calculator on this page — just enter the sample sizes, positive outcomes, and confidence level.

Conditions for a Reliable Split Test

The two-proportion Z-test relies on several assumptions:

• Sample size: Each group should have at least 30 observations; 50 or more per group is ideal. Smaller samples violate the normality assumption required by the Z-test.
• Balance: The sample sizes should be approximately equal. While not strictly required, severe imbalance reduces the test’s power and validity.
• Random sampling: Data must be collected randomly from the target populations. Non‑representative samples lead to biased, unreliable results.

Using the A/B Test Calculator

This tool simplifies the entire procedure:

1. Enter the sample size and number (or percentage) of positive results for each group.
2. Select your desired confidence level from the dropdown (90%, 95%, 98%, or 99%).
3. Click “Calculate” — the calculator runs the two-proportion Z-test and instantly tells you whether the difference is statistically significant.

No manual formula work is needed. The result includes the Z-score and a clear statement about significance, allowing you to make data‑driven decisions quickly.