Interrater reliability for incomplete and dependent data

The central question of this dissertation was how to estimate the interrater reliability (IRR) from incomplete and dependent observational data. IRR expresses the degree to which subjects' attribute scores are independent of raters. Various IRR coefficients have been proposed to estimate the IRR from data, of which the intraclass correlation coefficients and Cohen’s Kappa are probably most famous. Most of these IRR coefficients are defined for a two-way observational design in which multiple independent subjects are each rated by multiple independent raters, and these raters are the same for each subject. However, in social and behavioral observational research, the raters are often not the same for each subject, resulting in incomplete data. Also, the rated subjects are often nested within clusters or relationships, resulting in dependent data. Prior to the developments in this dissertation, none of the conceptualizations of IRR could be readily used for both incomplete and dependent data, resulting in less or non-informative IRR coefficients. Using the framework of Generalizability theory (GT), I extended the different definitions of intraclass correlation coefficients for IRR to incomplete and dependent data. Using simulation studies, I tested various estimation methods to estimate these intraclass correlation coefficients from such data. Throughout the different chapters, I provide guidance in planning observational studies. To guide researchers in using these methods, the chapters contain applications of the estimation methods to empirical data from clinical and developmental domains. Ultimately, these methods reach a wide range of researchers, to eventually lead to more reliable observational research.