Survey precision
Design effect and effective sample size
The design effect (deff) says how much a survey's design inflates variance compared with a simple random sample of the same size; the effective sample size is n / deff. For unequal weights, Kish's formula gives neff = (Σw)² / Σw², so deff = 1 + CV² of the weights. Use the effective sample size, not the number of interviews or the weighted total, when you compute margins of error and significance tests on weighted data: the margin of error grows by √deff.
How results are checked
Named statistical review pending. Selected automated reference cases are published on the validation page.Where design effects come from
Weighting. When some respondents count more than others, the estimate leans on fewer people than were interviewed. The more unequal the weights, the larger the loss. Kish's weighting design effect captures this from the weights alone.
Clustering. Interviewing several people per household, school or sampling point adds less information than the same number spread out, because people in a cluster resemble each other. The clustering design effect is roughly 1 + (m − 1)ρ: with clusters of 10 and ρ = 0.05 it is 1.45.
Stratification. Sampling separately within strata that differ on the outcome usually reduces variance, pulling the design effect below what weighting and clustering alone would give.
Using the effective sample size
Replace n with neff wherever a formula assumes a simple random sample: the standard error of a proportion becomes √(p(1 − p) / neff), so the margin of error is multiplied by √deff. Compute it for the base you are testing (a banner column, a segment), because a subgroup's weights can be more or less equal than the whole sample's. Report the unweighted base alongside, so readers know how many interviews stand behind each figure.
The method
Formula
deff = Vardesign(θ̂) / VarSRS(θ̂) · neff = n / deff
Kish (weights): neff = (Σwi)² / Σwi² · deffw = 1 + CVw² · clustering: deffc ≈ 1 + (m̄ − 1)ρ
MOE = z × √deff × √(p(1 − p) / n)
where
- w_i
- the weight of respondent i (scale does not matter)
- CV_w
- standard deviation of the weights divided by their mean
- m̄, ρ
- average cluster size and intra-cluster correlation of the outcome
- z
- 1.96 for a 95% margin of error
Worked example
Sample data: the synthetic shopper survey loaded by the latent class tool's “Use sample data” (900 generated respondents, not real people). Its weight column is a mild post-stratification-style weight by age group: from 0.766 to 1.320.
- Coefficient of variation of the weights: 0.143.
- Design effect from weighting: 1 + CV² = 1.020 (the same as n × Σw² / (Σw)²).
- Effective sample size: 900 / 1.020 = 882.0. The weights cost the equivalent of about 18 interviews.
- Margin of error for one estimate: 50.3% of the 871 respondents who answered (weighted) use coupons. As a simple random sample the 95% margin would be ±3.32 points; with this question's design effect of 1.020 (effective n = 853.7) it is ±3.35 points.
These weights are close to equal, so the loss is small. The design effect grows with the square of the weights' spread: a CV of 0.5 gives 1.25 and a CV of 1.0 gives 2.0, which halves the effective sample. Notice also that within the 18–34 group, where every respondent got a similar weight, the design effect is only 1.004 (n = 272): most of the weighting loss comes from differences between groups, not within them.
Using it well
When to use it
Use it when
- Use: Any weighted survey: report margins of error and run tests on the effective, not the nominal, sample size.
- Use: Planning: inflate the target sample by the expected design effect (n = n_SRS × deff).
- Use: Checking a weighting scheme: a large Kish design effect signals weights that trade too much precision for bias reduction.
Not the right tool when
- Avoid: Treating the Kish design effect as the full design effect for a clustered or stratified sample: estimate those with design-based methods.
- Avoid: Using the weighted total (often scaled to the population) as the base for significance tests.
- Avoid: Applying one overall design effect to every subgroup without checking the subgroup's own weights.
How to interpret it
Rule of thumb
A design effect of 1.2 means your sample is worth 1 / 1.2 ≈ 83% of its nominal size for precision; 2.0 means half. Margins of error grow by √deff: about 10% wider at 1.2, about 41% wider at 2.0.
Next step: put your design effect into the margin of error
The free margin of error calculator takes a design effect, or computes Kish's effective sample size and design effect from weights you paste, and returns the widened margin of error with Wald, Wilson, exact and Agresti–Coull intervals.
In the workspace, tables with a declared survey weight test column letters on each column's Kish effective base (an approximation, as in SPSS Custom Tables and Q), guided raking in saved projects reports the effective sample size of the new weight, and Complex Samples procedures estimate full design effects from strata, clusters and weights. The survey weighting and raking guide explains how those weights are built.
Questions
Frequently asked questions
What is a design effect?
The ratio of an estimate's variance under the actual survey design to its variance under a simple random sample of the same size. A design effect of 1.5 means the estimate is as uncertain as it would be from a simple random sample two thirds the size. Weighting, clustering and stratification all change it.
How do I calculate the effective sample size from weights?
Kish's approximation: n_eff = (Σw)² / Σw², and the weighting design effect is n / n_eff, which equals 1 plus the squared coefficient of variation of the weights. It depends only on how unequal the weights are, not on their scale.
Is the Kish design effect the full design effect?
No. It measures the loss from unequal weights only. Clustered samples add a clustering effect, roughly 1 + (m − 1)ρ for clusters of average size m and intra-cluster correlation ρ, and stratification can reduce variance. The full design effect differs from question to question and is estimated with design-based methods such as Taylor linearisation or replicate weights.
Should I report the weighted or the unweighted base?
Report the unweighted number of interviews, and use the effective sample size, not the weighted total, for margins of error and significance tests. The weighted total is often scaled to the population or to the sample size and says nothing about precision.
How can I keep the design effect down?
Weight on as few variables as the bias requires, collapse sparse weighting cells, and trim extreme weights (accepting a small bias for a large variance reduction). Better still, reduce the need for large weights at the design stage with quotas or better sample sources.
Sources and next steps
References & further reading
- [1]Kish, L. (1965). Survey Sampling. Wiley.
- [2]Kish, L. (1992). Weighting for unequal Pᵢ. Journal of Official Statistics, 8(2), 183–200.
- [3]Lumley, T. (2010). Complex Surveys: A Guide to Analysis Using R. Wiley.
Try it on your own data — free, no signup
Upload a CSV, Excel, SPSS, Stata, SAS, R, Parquet or JSON file. Everything runs in your browser; your file never leaves your device.
Open the workspace →