Sources of Variation That Affect the Reliability of Reading Classroom Observation Measures. Instructional Research Laboratory Technical Series #R83004.

Given the complexity of classroom reading observations, maintenance of measurement reliability is a concern to researchers. Sources of variation contributing to the unreliability of many measurements may be either (1) lasting and specific, (2) lasting and general, (3) temporary and specific, or (4) temporary and general. Lasting-specific sources of variations, such as observers' attitudes toward the coding system and differences between training conditions and actual classroom settings, can be controlled through appropriate criteria for observer selection and training procedures that are content specific to the actual observation tasks. Lasting-general sources of variation include observers' observation skills and the supervision condition, which can be controlled, in part, by researchers and observers for discussing recently completed observations. Although temporary-specific sources of variation, such as observer fatigue, attention span, and memory, are not always under experimenter control, they can be managed with some success by such techniques as regulation of observation periods. Temporary-general sources, including location, surroundings, and physical condition of the observer, are more amenable to researcher control. If an observer is ill, for example, the observation session can be cancelled. The systematic consideration of these sources of variation is critical to understanding classroom interaction and behavior. (MM)