SPSS command: SPSS FLEISS MULTIRATER KAPPA
Fleiss' kappa for multiple raters in your browser
Fleiss' kappa measures how consistently a group of raters assign items to nominal categories, beyond what chance would produce.
Free · No account · Files stay on your device · Updated 18 October 2026
What it tells you
Crosstabs runs SPSS's Fleiss multirater kappa on your device, with an overall kappa and a kappa for every rating category, each with an asymptotic standard error, z test and confidence interval.
- Overall agreement among all raters across all items, corrected for chance.
- Which categories raters agree on and which they confuse, from the per-category kappas.
- How often each category is used (proportion of ratings), which drives chance agreement.
- Whether agreement is significantly above chance, overall and for each category.
When to use it
Use it when
- Three or more raters classify the same set of items into unordered categories, such as coders labelling open-ended answers.
- Different raters rate different items but every item gets the same number of ratings, which is what Fleiss' kappa assumes.
- Auditing a coding frame: low category kappas point to labels that need clearer definitions.
Use something else when
- Ordered categories with two raters: weighted kappa credits near misses.
- Exactly two raters on nominal categories: Cohen's kappa in the Crosstabs statistics panel is the usual choice.
- Items have different numbers of ratings: rows with a missing rating are dropped, so consider Krippendorff's alpha instead.
How to run it in Crosstabs
- 01
Open your data
Arrange the file with one row per item and one column per rater (or rating slot), then switch to Statistics mode.
- 02
Choose Fleiss' Kappa
In the Agreement and diagnostics group, pick Fleiss' Kappa (FLEISS MULTIRATER KAPPA).
- 03
Select the raters
Tick the rater columns under Raters (one variable per rater); at least two are required and up to 40 are accepted. Non-integer numeric ratings are truncated.
- 04
Set the confidence level and run
Choose the Confidence level (%), declare any weight as a frequency weight, and press Run.
Worked example
Computed by the Crosstabs engine from the synthetic data below when this page was built — not typed in.
Data
30 synthetic scans, each classified by four radiologists as benign, uncertain or malignant. Every rater picks the true class with probability .75 and otherwise one of the other two at random. Fixed seed.
| Item | Kappa | Asymptotic Std. Error | Z | Sig. | 95% CI |
|---|---|---|---|---|---|
| Overall Agreement | 0.294 | 0.053 | 5.52 | <.001 | 0.190 – 0.399 |
Standard errors are asymptotic under the null hypothesis of no agreement beyond chance (Fleiss, Nee and Landis 1979).
| Rating Category | Proportion of ratings | Kappa | Asymptotic Std. Error | Z | Sig. |
|---|---|---|---|---|---|
| 1 Benign | 0.450 | 0.327 | 0.075 | 4.38 | <.001 |
| 2 Uncertain | 0.292 | 0.287 | 0.075 | 3.86 | <.001 |
| 3 Malignant | 0.258 | 0.261 | 0.075 | 3.50 | <.001 |
How to read it
Across 4 raters and 30 scans, Fleiss' κ = 0.294 (95% CI 0.190 – 0.399): observed agreement is 0.544 against 0.354 expected by chance. Agreement is strongest for “1 Benign” (κ = 0.327) and weakest for “3 Malignant” (κ = 0.261), which is where training or clearer criteria would help most.
Output tables explained
- Overall Kappa
- Overall agreement: kappa, asymptotic standard error, Z, significance and confidence interval. The standard error is computed under the null hypothesis of chance agreement (Fleiss, Nee and Landis 1979).
- Kappas for Individual Categories
- For each rating category: the proportion of all ratings it received, its kappa, standard error, Z, significance and confidence interval.
Assumptions and notes
- Every item is rated by the same number of raters; items with any missing rating are excluded.
- Ratings are made independently and categories are nominal (order is ignored).
- Standard errors are valid for testing against zero; the confidence interval built from them is approximate when kappa is far from zero.
How the results are validated
On a reference file of 45 subjects rated by five raters, with frequency weights, the overall kappa matches statsmodels 0.15 fleiss_kappa and R irr 0.85 kappam.fleiss to 1e-12 and the overall z matches irr to 1e-9; category kappas and z values agree with irr's detail table within 6e-4, the precision irr prints. Validation evidence
Questions
Do the same raters need to rate every item?
No. Fleiss' kappa treats raters as interchangeable, so different people may rate different items, as long as each item has the same number of ratings.
Why is one category's kappa much lower than the others?
Raters confuse that category with others, often because its definition overlaps a neighbour ("uncertain" versus "benign"). Clarifying that boundary usually raises the overall kappa most.
Can I use Fleiss' kappa with two raters?
The panel runs with two, but the result is not Cohen's kappa: Fleiss assumes interchangeable raters and pools their category use. For two fixed raters, Cohen's kappa is the usual statistic.
What if each row is a rating pattern with a count?
Set the count as the workspace weight and declare it a frequency weight; every count is then treated as that many identical items.
Related procedures
- Weighted KappaWEIGHTED KAPPA
- Crosstabs StatisticsCROSSTABS /STATISTICS
- Reliability AnalysisRELIABILITY
- Multiple Correspondence AnalysisMULTIPLE CORRESPONDENCE
- Free SPSS alternative: full coverage list
Run Fleiss' Kappa on your own data
Open a .sav, CSV or Excel file. Everything runs in your browser; the file never leaves your device.
SPSS is a trademark of IBM Corp. Crosstabs is not affiliated with or endorsed by IBM.