Search

Unlabelled: CONSTRUCT: The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low., Background: Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear., Approach: Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates., Results: Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats., Conclusions: The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.

This article describes rates of missing item responses in personal digital assistant (PDA) assessments as compared to paper assessments. Data come from the evaluation of a classroom-based leisure, life skills, and sexuality education program delivered to high school students in Cape Town, South Africa. Analyses show that the paper assessments had much higher rates of missing-ness than PDA assessments. This association is moderated by item order. Certain analyses also suggest that paper assessments have higher rates of missingness for items pertaining to participants' sexual behavior. Implications of these results for evaluation research will be discussed.