A Study of Conventional Missing at Random on Kappa-like Statistics for Raters Agreement Measurement

Abstract

Kappa-like statistics such as Cohen Kappa and Intraclass Kappa have been in existence in the last four decades. They are mainly for the measurement of overall level of agreement that may exist between two raters. Subjects are being classified into categories by raters, interviewers or observers in all life or social science researches. In any experiment that involves measurement, counting, treatments or diagnosis, irrespective of the level of reliability, there could be likelihood for some elements of missing observations. In this research work, we examine the effects of these missing observations with different missing patterns from the square table of cross-classified ratings of raters on these two Kappa-like statistics. We assumed a conventional missing at random (CMAR) which is a special case of missing at random (MAR) mechanism criteria. We observed that value from Intraclass Kappa statistic are consistently lower than their respective estimates from Cohen Kappa statistic and missingness improved or worsened the strength of agreement as the missing percentage increases depending on the missing pattern under consideration.