Significance testing
Bonferroni correction for crosstab significance tests
A banner table tests every pair of columns in every row, so false positives pile up: with four columns, six pairs per row tested at 5% give about a 26% chance of at least one spurious letter. The Bonferroni correction multiplies each pairwise p-value by the number of pairs in the row (or tests each at α / pairs, 0.05 / 6 = 0.0083), which keeps that family-wise chance at 5%. Holm's step-down version controls the same error rate and is never less powerful, so it is a safe upgrade.
How results are checked
Named statistical review pending. Selected automated reference cases are published on the validation page.Why a crosstab creates a multiple-comparison problem
Significance letters compare column percentages within a row: is the share among 18–34s higher than among 55+? With k columns there are k(k − 1)/2 such pairs per row: 3 pairs for 3 columns, 6 for 4, 28 for 8. Each test has a 5% false-positive rate on its own, so the chance that at least one pair is flagged by luck grows with every column.
A correction defines a family of tests and controls the family-wise error rate (FWER): the probability of one or more false positives within the family. SPSS Custom Tables, and Crosstabs, define the family as all column pairs within one row.
Bonferroni and Holm, step by step
Bonferroni: with m comparisons, adjusted p = min(1, m × p). Declare a pair significant if the adjusted p is at most α. It is simple and valid for any dependence between the tests, and conservative when m is large.
Holm: sort the m p-values ascending, p₍₁₎ ≤ … ≤ p₍ₘ₎. The adjusted value of the i-th is max over j ≤ i of (m − j + 1) × p₍ⱼ₎, capped at 1. The smallest p-value is treated exactly as in Bonferroni; each later one is multiplied by a smaller factor. Holm rejects everything Bonferroni rejects, and sometimes more.
The method
Formula
Bonferroni: p̃i = min(1, m·pi) · Holm: p̃(i) = min(1, maxj ≤ i (m − j + 1)·p(j))
m = k(k − 1)/2 · FWER without correction ≤ 1 − (1 − α)m (independent tests)
where
- m
- comparisons in the family: column pairs within one row, k testable columns
- p_(i)
- the i-th smallest unadjusted p-value in the family
- p̃
- adjusted p-value; a pair is significant when p̃ ≤ α
- Per pair
- pooled two-proportion z-test of the two column percentages
Worked example
Sample data: the synthetic snack-brand tracker loaded by the free tools' “Use sample data” (600 generated respondents, not real people). The table shows the share of promoters, passives and detractors on the 0–10 recommend question by region: East (A, n = 132), North (B, n = 133), South (C, n = 166), West (D, n = 169). Each row compares 6 pairs of columns with a pooled two-proportion z-test.
| Row | East (A) | North (B) | South (C) | West (D) |
|---|---|---|---|---|
| Promoters (9–10) | 20.5% | 24.1% | 34.3% | 20.1% |
| Passives (7–8) | 35.6% | 29.3% | 33.7% | 42.0% |
| Detractors (0–6) | 43.9% | 46.6% | 31.9% | 37.9% |
Without a correction, 5 of the 18 pairwise tests have p < 0.05. With 6 tests per row and nothing real going on, the chance of at least one false alarm in a row would be about 1 − 0.956 = 26.5% (treating the tests as independent).
| Promoters: pair | Unadjusted p | Bonferroni p | Holm p |
|---|---|---|---|
| South vs West | 0.0034 | 0.0206 | 0.0206 |
| East vs South | 0.0081 | 0.0489 | 0.0407 |
| North vs South | 0.0534 | 0.3206 | 0.2137 |
| North vs West | 0.4106 | 1.000 | 1.000 |
| East vs North | 0.4805 | 1.000 | 1.000 |
| East vs West | 0.9426 | 1.000 | 1.000 |
After either correction, 2 pairs remain significant (Bonferroni) and 2 (Holm): South has more promoters than West and East, so the South cell carries two letters. Holm gives the second-smallest p-value a smaller adjustment (0.0407 instead of 0.0489) because it multiplies by 5 rather than 6. The detractor gap between North and South (unadjusted p = 0.0095) does not survive at 95%: adjusted p = 0.0568. Neither does the passives difference, so those two rows carry no upper-case letters. Because the detractor pair's adjusted p is still below 0.10, a table that also shows a 90% tier marks it with a lower-case letter: worth a look, not a headline.
Using it well
When to use it
Use it when
- Use: Any banner table with significance letters, where every row compares several column pairs.
- Use: Pairwise follow-up comparisons after an overall chi-square test.
- Use: Reports where a single false claim of difference is costly; Holm gives the same protection with more power.
Not the right tool when
- Avoid: Exploratory screening of hundreds of cells, where controlling the false discovery rate (Benjamini–Hochberg) fits better.
- Avoid: A single planned comparison stated before seeing the data; it needs no correction.
- Avoid: Weighted survey tables tested as if the weighted counts were respondents; fix the test base first, then correct.
How to interpret it
Rule of thumb
A letter that survives the correction is a difference you can report for that row. A difference that is significant only before correction is a lead, not a finding, especially in a long tab book where many rows are tested.
Next step: test your own banner table
Crosstabs marks column-proportion differences with letters in the free survey crosstab tool, Bonferroni-corrected across the column pairs of each row by default, at 95% (upper case) and 90% (lower case), with columns under 30 respondents left out of the tests. Saved tab-book projects can switch the correction to Holm, Benjamini–Hochberg or none in the table designer.
To check a single comparison by hand, the two-proportion z-test calculator gives the unadjusted p-value to multiply by the number of comparisons.
Questions
Frequently asked questions
Why do crosstab significance letters need a Bonferroni correction?
Because each row of a banner table runs many pairwise tests at once. With four banner columns there are six pairs per row; testing each at 5% gives roughly a one-in-four chance that at least one pair looks significant when no real difference exists. The correction keeps that family-wise chance at 5%.
How does the Bonferroni correction work?
Multiply each p-value by the number of comparisons in the family (capping at 1), or equivalently test each comparison at α divided by the number of comparisons. With six pairs and α = 0.05, a pair is significant only if its unadjusted p-value is below 0.0083.
What is the difference between Bonferroni and Holm?
Holm is a step-down version of Bonferroni. Sort the p-values from smallest to largest, multiply the smallest by m, the next by m − 1, and so on, keeping the adjusted values in order. It controls the same family-wise error rate but is never less powerful than Bonferroni, so it can find a difference Bonferroni misses.
What counts as the family of comparisons?
In SPSS Custom Tables and most tab software, the family is all column pairs within one row of one table, and that is the rule Crosstabs follows. Comparisons across rows, tables and banner breaks are not corrected together, so a large tab book still produces some false positives; treat isolated letters with caution.
Is Bonferroni too conservative?
It can be when there are many columns, because the per-test threshold shrinks quickly and real differences are missed. Holm is a free improvement. When you care about the share of false discoveries among many flagged cells rather than any single false alarm, the Benjamini–Hochberg false discovery rate is a less strict alternative.
Sources and next steps
References & further reading
- [1]Dunn, O. J. (1961). Multiple comparisons among means. Journal of the American Statistical Association, 56(293), 52–64.
- [2]Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2), 65–70.
- [3]Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57(1), 289–300.
Try it on your own data — free, no signup
Upload a CSV, Excel, SPSS, Stata, SAS, R, Parquet or JSON file. Everything runs in your browser; your file never leaves your device.
Open the workspace →