Significance test
Two-proportion z-test calculator
The two-proportion z-test asks whether two percentages from independent samples — two survey subgroups, or the control and variant of an A/B test — differ by more than sampling error. It divides the difference by its standard error under the null hypothesis, z = (p_B − p_A) / √(p̄(1 − p̄)(1/n_A + 1/n_B)), and is the same test as a 2×2 chi-square without continuity correction (z² = χ²).
How results are checked
Named statistical review pending. Selected automated reference cases are published on the validation page.Run this on your own data — free, no signup
Upload a CSV, Excel, SPSS, Stata, SAS, R, Parquet or JSON file. Everything runs in your browser; your file never leaves your device.
Open the workspace →Test the difference
Enter the number of successes and the sample size for each group — or a percentage and n — and choose a two-sided or one-sided test. The calculator reports the pooled z-test, confidence intervals for the difference, the relative lift, Cohen's h and Fisher's exact p-value, and sizes the next test for a difference you care about. You can also open a survey file and compare two groups of respondents; it never leaves your browser.
Significance level α = 0.05. Intervals are two-sided at this level.
Or compare two groups from a survey file
Pick the question, the answers that count, and the column that splits the two groups. Unweighted counts; the file stays in this tab.
CSV, Excel, SPSS, Stata, SAS, R, Parquet, SQLite or JSON. The file is read in this browser tab and never uploaded. For Triple-S, select the .sss or .xml metadata file and its data file together.
Difference B − A
+3.0 pts
Statistically significant at 95% confidence (p = .050): the gap is larger than sampling error alone would usually produce.
A = 12.0% · B = 15.0%
z = (p_B − p_A) / √(p̄(1 − p̄)(1/n_A + 1/n_B)) = 0.0300 / 0.0153 = 1.963
Test results
| Statistic | Value |
|---|---|
| Proportion A | 12.00% (120/1000) |
| Proportion B | 15.00% (150/1000) |
| Difference B − A | +3.00 pts |
| Pooled proportion | 13.50% |
| z (pooled standard error) | 1.963 |
| p-value (z-test) · Two-sided: B ≠ A | p = .050 |
| χ² = z² (df = 1, no continuity correction) | 3.854 · p = .050 |
| Difference CI — unpooled Wald (95%) | +0.01 pts … +5.99 pts |
| Difference CI — Newcombe hybrid score (Wilson) (95%) | +0.00 pts … +6.00 pts |
| Relative lift (B ÷ A − 1) | +25.0% |
| Cohen's h | 0.088 (negligible) |
| Fisher's exact test p-value · Two-sided: B ≠ A | p = .058 |
The pooled z-test and Pearson's chi-square test on the 2×2 table are the same test: z² = χ² without continuity correction, with the same two-sided p-value.
Power and sample size for a target difference
Plan the next wave, or check whether this test could have detected the difference you care about. Uses the confidence level and alternative above.
From 12.0% to 15.0%.
| Required n per group — Normal approximation (Fleiss), as the sample size calculator | 2,036 |
|---|---|
| Required n per group — With continuity correction | 2,102 |
| Required n per group — Arcsine / Cohen's h (statsmodels NormalIndPower) | 2,031 |
| Cohen's h | 0.088 |
| Power of the current sample (n = 1000 and 1000) | 50.2% |
Unequal groups or several comparisons? Use the sample size calculator.
Pooled test, unpooled interval
The test uses the pooled proportion because, under the null hypothesis, both groups share one rate. The confidence interval describes the difference you actually observed, so its simplest form uses the unpooled standard error √(p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B). That Wald interval misbehaves for small groups and extreme rates; Newcombe's hybrid score interval, built from two Wilson intervals, is the better default and is shown alongside.
With small expected counts (below about 5 in any cell) the normal approximation is unreliable: report Fisher's exact test instead — it is computed here for every input.
The method
Formula
z = (p_B − p_A) / √(p̄(1 − p̄)(1/n_A + 1/n_B)), p̄ = (x_A + x_B)/(n_A + n_B)
where
- x, n
- successes and sample size in each group; p = x / n
- p̄
- pooled proportion, the common rate under the null hypothesis
- z²
- Pearson's χ² for the 2×2 table, without continuity correction (df = 1)
- CI
- (p_B − p_A) ± z_{1−α/2} · √(p_A(1 − p_A)/n_A + p_B(1 − p_B)/n_B), or Newcombe's hybrid score interval
- h
- Cohen's h = 2·asin√p_B − 2·asin√p_A (0.2 small, 0.5 medium, 0.8 large)
- lift
- relative change p_B / p_A − 1
Worked example
In our synthetic snack-brand tracker (generated sample data, not real respondents), top-2-box purchase interest in a chili-lime flavour is 115 of 203 = 56.7% among 18–34-year-olds (B) and 51 of 179 = 28.5% among those aged 55+ (A).
The difference is 28.2 points; with the pooled proportion 43.5%, z = 5.54, p < .001 (χ² = 30.70). The 95% Newcombe interval for the difference runs from 18.3 to 37.2 points, the relative lift is +99% and Cohen's h is 0.58 (medium by Cohen's benchmarks).
To detect a smaller 5-point gap from the same 28.5% baseline with 80% power at α = 0.05 (two-sided), you would need about 1,342 respondents per group.
Using it well
When to use it
Use it when
- Use: Comparing a percentage between two independent groups: two regions, two survey waves with fresh samples, men vs women.
- Use: A/B tests of conversion, click-through or sign-up rates with users randomised to one variant each.
- Use: Planning how many respondents or users each group needs to detect a given difference.
Not the right tool when
- Avoid: The same people are in both groups (before/after, two questions to one sample) — use McNemar's test for paired yes/no data.
- Avoid: A subgroup against the total it is part of — the samples overlap; compare the subgroup with everyone else instead.
- Avoid: More than two groups or answer categories — use the chi-square test or column-proportion significance letters.
How to interpret it
Rule of thumb
If p < α the difference is statistically significant: a gap this large would be unusual if the two population rates were equal. Read the confidence interval for its plausible size and Cohen's h for practical importance — with large samples, tiny differences become significant. Decide on one-sided testing before seeing the data, and don't stop an A/B test the moment it crosses p < 0.05: repeated peeking inflates false positives.
Questions
Frequently asked questions
Is the two-proportion z-test the same as a chi-square test?
Yes, for two groups and a yes/no outcome. The pooled z-test and Pearson's chi-square test on the 2×2 table give z² = χ² and the same two-sided p-value, as long as the chi-square is computed without Yates' continuity correction. The z-test adds a direction, so it can also be run one-sided.
When should I use Fisher's exact test instead?
When any expected count in the 2×2 table is below about 5 — small groups or rare outcomes. Fisher's test conditions on the margins and is exact; it is slightly conservative, so its p-value is usually a little larger than the z-test's.
Can I compare two percentages from the same survey question?
Yes, if they come from two separate groups of respondents — say men and women answering the same question. Not if they are two answers from the same people (brand A vs brand B awareness) or a subgroup vs the total: those samples overlap and the standard error is wrong.
What is the difference between the pooled and unpooled standard error?
The pooled standard error assumes the null hypothesis that both groups share one rate, so it is used for the test statistic. The unpooled one uses each group's own rate and describes the observed difference, so it is used for the Wald confidence interval.
How many users do I need per group for an A/B test?
It depends on the baseline rate, the smallest lift worth detecting, α and power. Use the power section of this calculator: for example, moving from 10% to 12% at α = 0.05 and 80% power needs several thousand users per group, while large differences need far fewer.
Sources and next steps
References & further reading
- [1]Newcombe, R. G. (1998). Interval estimation for the difference between independent proportions: comparison of eleven methods. Statistics in Medicine, 17(8), 873–890.
- [2]Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical Methods for Rates and Proportions (3rd ed.). Hoboken, NJ: Wiley.
- [3]Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
- [4]statsmodels — proportions_zteststatsmodels.org
- [5]SciPy — scipy.stats.fisher_exactdocs.scipy.org
Try it on your own data — free, no signup
Upload a CSV, Excel, SPSS, Stata, SAS, R, Parquet or JSON file. Everything runs in your browser; your file never leaves your device.
Open the workspace →