Search

Procedural fidelity is the extent to which independent variables are implemented as designed. Despite 40 years of discussion about the importance of procedural fidelity for behavioral research, reporting of fidelity data remains an uncommon practice in behavior‐analytic journals. Researchers have speculated about reasons for underreporting, but the perspectives of scholars about when reporting is warranted or necessary have not yet been explored. Thus, the purpose of this study was to evaluate possible reasons for infrequent reporting of fidelity data in behavior‐analytic studies. To address this purpose, we conducted focus groups with scholars in applied behavior analysis. Five themes emerged regarding why procedural fidelity data are not typically reported. We provide a discussion about how these themes are interrelated and offer suggestions and recommendations to assist with the collection and reporting of fidelity data. [ABSTRACT FROM AUTHOR]

We examined articles with experiments published in the Journal of Applied Behavior Analysis and in Behavior Analysis in Practice from 2017 through 2021 to determine how frequently procedural fidelity was assessed. When procedural fidelity was assessed, we determined how often a measure of interobserver agreement for those fidelity data was provided. We also determined how often a measure of interobserver agreement for participants' behavior was provided. Across both journals and all years, 54.7% of relevant articles provided a measure of procedural fidelity. Of them, 17.7% provided a measure of interobserver agreement for procedural fidelity. In marked contrast, 96.4% provided interobserver agreement data for participants' behavior. It is unfortunate that applied behavior analysts frequently fail to provide procedural fidelity data and, when they do, often fail to provide interobserver agreement data for the fidelity data. Reviewers for, and editors of, behavior‐analytic journals are encouraged to strongly consider the relative value of procedural fidelity and agreement on procedural fidelity measures when rendering recommendations on the suitability of a given submission. [ABSTRACT FROM AUTHOR]

Interobserver agreement (IOA) is important for research and practice, and supports the consistency of behavioral data (Kahng et al., 2011). Although general parameters for how much IOA is needed have been suggested (Bailey & Burch, 2018), it is unknown if the total number of sessions with IOA might impact the IOA coefficient. In this study, IOA was reanalyzed using functional analysis data at various cutoffs. Obtained IOA from these analyses was then compared to the original IOA. Overall, results suggested that, at least when using highly trained observers in a structured clinical setting, there were no significant differences in IOA across cutoffs. However, IOA was sensitive to overall rate of responding in the functional analysis. These data are encouraging, particularly for practitioners, because they provide preliminary support that the amount of sessions with IOA may not be as important as the consistency of the data. [ABSTRACT FROM AUTHOR]

We examined the effects of several variations in response rate on the calculation of total, interval, exact-agreement, and proportional reliability indices. Trained observers recorded computer-generated data that appeared on a computer screen. In Study 1, target responses occurred at low, moderate, and high rates during separate sessions so that reliability results based on the four calculations could be compared across a range of values. Total reliability was uniformly high, interval reliability was spuriously high for high-rate responding, proportional reliability was somewhat lower for high-rate responding, and exact-agreement reliability was the lowest of the measures, especially for high-rate responding. In Study 2, we examined the separate effects of response rate per se, bursting, and end-of-interval responding. Response rate and bursting had little effect on reliability scores; however, the distribution of some responses at the end of intervals decreased interval reliability somewhat, proportional reliability noticeably, and exact-agreement reliability markedly. [ABSTRACT FROM AUTHOR]

The three algorithms most frequently selected by behavior-analytic researchers to compute interobserver agreement with continuous recording were used to assess the accuracy of data recorded from video samples on handheld computers by 12 observers. Rate and duration of responding were recorded for three samples each. Data files were compared with criterion records to determine observer accuracy. Block-by-block and exact agreement algorithms were susceptible to inflated agreement and accuracy estimates at lower rates and durations. The exact agreement method appeared to be overly stringent for recording responding at higher rates (23.5 responses per minute) and for higher relative duration (72% of session). Time-window analysis appeared to inflate accuracy assessment at relatively high but not at low response rate and duration (4.8 responses per minute and 8% of session, respectively). [ABSTRACT FROM AUTHOR]

We reviewed all research articles in 10 recent volumes of the Journal of Applied Behavior Analysis (JABA): Vol. 28(3), 1995, through Vol. 38(2), 2005. Continuous recording was used in the majority (55%) of the 168 articles reporting data on free-operant human behaviors. Three methods for reporting interobserver agreement (exact agreement, block-by-block agreement, and time-window analysis) were employed in more than 10 of the articles that reported continuous recording. Having identified these currently popular agreement computation algorithms, we explain them to assist researchers, software writers, and other consumers of JABA articles. [ABSTRACT FROM AUTHOR]