Overview (FLEISS MULTIRATER KAPPA command)

FLEISS MULTIRATER KAPPA assesses the interrater agreement to determine the reliability among the various raters. A higher agreement provides more confidence in the ratings reflecting the true circumstance, generalized the unweighted kappa statistic to measure the agreement among any constant number of raters while assuming:
  • There is no connection between raters.
  • The number of raters is a constant.
  • Each subject is rated by the same group containing only a single rater.
  • No weights can be assigned to the various disagreements.

FLEISS MULTIRATER KAPPA {variable_list} is a required command that invokes the procedure to estimate the Fleiss' multiple rater kappa statistics. The command names all the variables to be used in the FLEISS MULTIRATER KAPPA procedure.

  • At least two item variables must be specified to run any reliability statistic.
  • At least two ratings variables must be specified.
  • The variables selected as items can also be selected as ratings.

Options

Fleiss' multiple-rater kappa
Enables or disables the Fleiss' multiple-rater kappa statistics.
Display agreement on individual categories
Specifies whether or not to output the agreement on individual categories. By default, the output suppresses the estimation on any individual categories. When enabled, multiple tables display in the output.
Ignore string cases
Controls whether or not the string variables are case sensitive. /CRITERIA IGNORECASE=FALSE is the default setting, which indicates that the strings are case sensitive. When /CRITERIA IGNORECASE=TRUE is specified, the strings are not case sensitive.
String category labels are displayed in uppercase
Controls whether the category labels in the output tables are displayed in uppercase or lowercase. The keyword takes effect only when /CRITERIA IGNORECASE=TRUE. /PRINT [CASE(UPPER)] is the default setting, which displays the string category labels in uppercase. When /PRINT [CASE(LOWER)] is specified, the string category labels are all displayed in lowercase.
Asymptotic significance level (%)
Specifies the significance level for the asymptotic confidence intervals. The value must be a single double value in (0,100) with /CRITERIA ASYMPTOTIC_CILEVEL=95 as the default setting.
Exclude both user-missing and system missing values
Controls the exclusion of user-missing and system-missing values. By default /MISSING CLASSMISSING=EXCLUDE is the default setting, which excludes user-missing and system-missing values.
User-missing values are treated as valid
/MISSING CLASSMISSING=INCLUDE treats user-missing and system-missing values as valid data. The specification is applied to factors and categorical design variables.

Basic specification

FLEISS MULTIRATER KAPPA variable_list: Invokes the Fleiss' Multiple-Rater Kappa procedure.

Subcommand order

The subcommands can be named in any order.

Syntax rules

  • At least two item variables must be selected to run any reliability statistic.
  • When at least two ratings variables are selected, the FLEISS MULTIRATER KAPPA syntax is pasted.
  • There is no connection between raters.
  • The number of raters is a constant.
  • Each subject is rated by the same group containing only a single rater.
  • The variables selected as items can also be selected as ratings.

Limitation

  • No weights can be assigned to the various disagreements.