IBM Support

Weighted Kappa, Kappa for ordered categories



I would like to measure agreement between 2 raters who have rated several objects on an ordinal scale with 5 levels. The kappa measure available in SPSS Crosstabs seems to treat the variables as nominal. Can SPSS calculate kappa for ordered categories?

Resolving The Problem

One way to calculate Cohen's kappa for a pair of ordinal variables is to use a weighted kappa. The idea is that disagreements involving distant values are weighted more heavily than disagreements involving more similar values. So, ratings of 1 and 5 for the same object (on a 5-point scale, for example) would be weighted heavily, whereas ratings of 4 and 5 on the same object - a more minor disagreement on an ordinal scale- would have a low weighting. Weighted kappa is not available from Crosstabs or any other SPSS procedure. The kappa in Crosstabs will treat the scale as nominal. You can use the SPSS MATRIX commands to run a weighted kappa. There is also an SPSS Extension command available to run weighted kappa, as described at the bottom of this technical note..

There is a discussion of weighted kappa in Agresti (1990, 2002, references below). There are multiple possibilities for the weighting scheme in a weighted kappa statistic. The SPSS commands below compute weighted kappa for each of 2 weighting schemes. In each scheme, weights range from 0 to 1, with the weight equal to 1 for cells on the diagonal (where the raters agree exactly) and equal to 0 for cells in the upper right and lower left corners (where disagreement is as large as possible). The first version of weighted kappa (WK1) uses weights that are based on the absolute distance (in number of rows or columns) between categories. The second version (WK2) uses a set of weights that are based on the squared distance between categories. Agresti cites a Fleiss and Cohen (1973) paper for the second method. Both versions attribute greater impact to disagreements that are more rows or columns away from the diagonal than those close to the diagonal (thus some ordinality is assumed for the variables).

In the example below, the commands from DATA LIST to END DATA instruct SPSS to read the data into an active data file. The data are read into SPSS as a matrix of counts of paired ratings. Suppose that the rows correspond to the ratings provided by rater 1. Columns refer to the ratings provided by rater 2. In the example data, there were 25 objects for which both raters gave a rating of 1. There were 12 objects which were rated as 2 by Rater 1 and as 1 by Rater 2. The commands from MATRIX to END MATRIX form the input variables into a matrix structure and then compute kappa. For application to your own data, you will need to edit the command:

GET x /var=x1 to x5.

so that the variables listed after "/VAR = " correspond to the names of the variables in your own data. The phrase "GET x" forms these variables into a matrix which is then called X. You are not expected to have a variable named X in your data set. X is referenced throughout the remaining commands, so if you change the matrix name in this command, you will need to change it throughout the remaining commands. Otherwise, all of the commands after "Get x" can be run without modification for particulars of the data set.

* Read in the IxI matrix of counts for I-level scale.
DATA LIST LIST / x1 to x5.
25 5 3 2 0
12 18 8 3 1
0 3 14 4 2
1 3 3 19 4
2 1 5 9 20

* Commands to compute weighted Kappa .
GET x /var=x1 to x5.
compute wt1=make(nrow(x),ncol(x),0).
compute wt2=make(nrow(x),ncol(x),0).
compute prop=x/msum(x).
loop i=1 to nrow(x).
loop j=1 to ncol(x).
compute wt1(i,j)=1-(abs(i-j)/(nrow(x)-1)).
compute wt2(i,j)=1-((i-j)/(nrow(x)-1))**2.
end loop.
end loop.
compute wk1num=msum(wt1&*prop)-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1den=1-msum(mdiag(rsum(prop))*wt1*mdiag(csum(prop))).
compute wk1=wk1num/wk1den.
print wk1.
compute wk2num=msum(wt2&*prop)-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2den=1-msum(mdiag(rsum(prop))*wt2*mdiag(csum(prop))).
compute wk2=wk2num/wk2den.
print wk2.
end matrix.

Agresti, A (1990). Categorical Data Analysis. New York: Wiley. (p. 367).

Agresti, A (2002). Categorical Data Analysis (2nd Ed.). New York: Wiley. (p. 435).

Fleiss, J.L., & Cohen, J. (1973). The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement, 33, 613-619.

You might also want to look at :
Valiquette, C.A.M., Lesage, A.D., & Mireille, C. (1994), Computing Cohen's Kappa coefficients using SPSS MATRIX. "Behavior Research Methods, Instruments, & Computers", 26(1), 60-61.
Valiquette et al. show SPSS MATRIX code for both weighted and unweighted kappa examples from Cohen's original papers. The weights are included in the DATA LIST-END DATA commands as a matrix that is adjacent to the matrix of counts, rather than being calculated in COMPUTE commands. This allows the user to specify weights for each disagreement cell that are not strictly a function of the distance between the row and column of that cell.

As mentioned above, there is an SPSS Extension command that will estimate a weighted kappa with related statistics. Extension commands employ the Python and/or R Programmability Plugins that are available with SPSS Statistics. Stats Weighted Kappa.spe is available from the Extension Commands page. It requires the Python plugin and at least version 18.0.2 of SPSS Statistics.

[{"Product":{"code":"SSLVMB","label":"SPSS Statistics"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"Not Applicable","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"Not Applicable","Edition":"","Line of Business":{"code":"LOB10","label":"Data and AI"}}]

Historical Number


Document Information

Modified date:
16 April 2020