Original research
Chi-square vs Fisher's exact test on 2×2 tables: a 340,000-table simulation study
We simulated 340,0002×2 contingency tables with the exact statistical engine that powers this site and measured how often Pearson's chi-square, Yates' correction, and Fisher's exact test reject at α = 0.05 — under the null hypothesis, where 5% is the target, and under real effects, where higher is better.
seed 20260807 · 20,000 tables per configuration · 17 configurations · study date 2026-08-07
Key findings
- Under the null, Pearson's chi-square stayed closest to the nominal 5% false-positive rate across every configuration (observed range 3.94%–5.92%), edging slightly above 5% once each group had 50+ observations.
- Yates' correction and Fisher's exact test were consistently conservative — rejecting as rarely as 0.48% of null tables at the smallest samples — which buys protection against false positives at the cost of power.
- With a real effect present, Pearson detected it more often at every sample size; the gap over Fisher reached roughly 11.5% of tables at 20 observations per group and narrowed as samples grew.
- Pearson and Fisher disagreed on the 0.05 decision in up to 11.5% of tables — so which test a tool reports materially changes conclusions at small n.
False-positive rate under the null hypothesis
Both groups drawn from the same binomial probability. The nominal target is 5%; rates are over non-degenerate tables.
| Probabilities | n / group | Pearson | Yates | Fisher | Pearson–Fisher disagree | Degenerate |
|---|---|---|---|---|---|---|
| p = 0.5 vs 0.5 | 10 | 3.96% | 1.20% | 1.20% | 2.77% | 0 |
| p = 0.5 vs 0.5 | 20 | 4.33% | 1.77% | 2.07% | 2.27% | 0 |
| p = 0.5 vs 0.5 | 50 | 5.92% | 3.89% | 3.89% | 2.04% | 0 |
| p = 0.5 vs 0.5 | 100 | 5.52% | 3.86% | 3.86% | 1.65% | 0 |
| p = 0.3 vs 0.3 | 10 | 3.94% | 1.24% | 1.24% | 2.70% | 11 |
| p = 0.3 vs 0.3 | 20 | 5.26% | 1.97% | 2.33% | 2.94% | 0 |
| p = 0.1 vs 0.1 | 20 | 4.15% | 0.48% | 1.38% | 2.77% | 348 |
| p = 0.1 vs 0.1 | 50 | 5.09% | 1.82% | 1.82% | 3.27% | 1 |
Power under real effects
Group 2 drawn with a higher success probability than group 1. Higher rejection rates are better.
| Probabilities | n / group | Pearson | Yates | Fisher | Pearson–Fisher disagree |
|---|---|---|---|---|---|
| p = 0.4 vs 0.5 | 20 | 8.6% | 4.3% | 4.9% | 3.7% |
| p = 0.4 vs 0.5 | 50 | 18.1% | 13.2% | 13.2% | 4.9% |
| p = 0.4 vs 0.5 | 100 | 31.2% | 26.1% | 26.1% | 5.1% |
| p = 0.4 vs 0.6 | 20 | 22.1% | 12.9% | 14.1% | 7.9% |
| p = 0.4 vs 0.6 | 50 | 54.2% | 46.1% | 46.1% | 8.1% |
| p = 0.4 vs 0.6 | 100 | 83.1% | 79.3% | 79.3% | 3.8% |
| p = 0.4 vs 0.7 | 20 | 45.9% | 32.2% | 34.4% | 11.5% |
| p = 0.4 vs 0.7 | 50 | 87.1% | 82.5% | 82.5% | 4.6% |
| p = 0.4 vs 0.7 | 100 | 99.3% | 98.9% | 98.9% | 0.4% |
Methods
For each configuration we generated 20,000 independent 2×2 tables. Each table contains two groups of n independent Bernoulli observations with success probabilities p₁ and p₂, sampled by exact inverse-transform from a seeded mulberry32 PRNG (seed 20260807). Tables were evaluated with this repository's own implementations: Pearson's chi-square with p-values from the regularized incomplete gamma function, Yates' continuity-corrected chi-square, and the bounded Fisher-Irwin two-sided exact test. Tables with a zero margin — where no test is defined — are reported separately and excluded from the rates.
The simulation is deterministic and fully reproducible. The published numbers are regenerated by an automated test that re-runs the same code with the same seed:
What this means in practice
At moderate samples (50+ per group), Pearson's chi-square is well calibrated and slightly more powerful — a reasonable default for 2×2 tables, matching the guidance in our comparison guide. At small samples, expect Fisher and Yates to miss real effects more often, and expect Pearson to reject marginally more than 5% of true nulls.
Because the tests disagree on up to 11.5% of tables at small n, the choice of test is a real analytical decision, not a cosmetic one. That is why this site shows all applicable tests side by side in the chi-square calculator and the Fisher's exact test page, and documents the formulas on the methods page.
Limits
- Simulation results describe these data-generating processes (independent binomial sampling, 2×2 tables, α = 0.05 decisions) — they are evidence, not proof, and do not cover larger tables, dependent samples, or weighted data.
- Rates are Monte Carlo estimates over 20,000tables per configuration; the standard error of a 5% rate is about 0.15 percentage points.
- Fisher's exact test here is the two-sided Fisher-Irwin ordering used by this site's engine; other two-sided orderings can differ slightly.