Search

An approach commonly used to measure new toxicity test method (NTM) performance in validation studies is to divide toxicity results into positive and negative classifications, and then identify true positive (TP), true negative (TN), false positive (FP) and false negative (FN) results. After this step is completed, the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are calculated. Although these statistics are widely used and often the only statistics used to assess the performance of toxicity test methods, there is little specific guidance in the validation literature on what values for these statistics indicate adequate performance. The purpose of this study was to begin developing data-based answers to this question by characterizing the CPS obtained from an NTM whose data have a completely random association with a reference test method (RTM).Determining the CPS of this worst-case scenario is useful because it provides a lower baseline from which the performance of an NTM can be judged in future validation studies. It also provides an indication of relationships in the CPS that help identify random or near-random relationships in the data. The results from this study of randomly associated tests show that the values obtained for the statistics vary significantly depending on the cut-offs chosen, that high values can be obtained for individual statistics, and that the different measures cannot be considered independently when evaluating the performance of an NTM. When the association between results of an NTM and RTM is random the sum of the complementary pairs of statistics (sensitivity+ specificity, NPV+PPV) is approximately 1, and the prevalence (i.e., the proportion of toxic chemicals in the population of chemicals) and PPV are equal. Given that combinations of high sensitivity-low specificity or low specificity-high sensitivity (i.e., the sum of the sensitivity and specificity equal to approximately 1) indicate lack of predictive capacity, an NTM having these performance characteristics should be considered no better for predicting toxicity than by chance alone. [ABSTRACT FROM AUTHOR]

An area that requires further research is how best to measure test method performance in validation studies and how to set criteria that should be used to judge the adequacy of this performance. The studies reported here were designed to begin an investigation of these questions. Computer simulations were used to generate data sets similar to those that might be obtained from a large validation study. These data were then analysed using three procedures including determination of the 95% prediction interval (PI), calculation of Pearson's correlation coefficient and calculation of the contingent probability statistics (CPS), sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). The results of this work suggest that of the three approaches examined, quantitative measurements with calculation of the 95% PI provide the most information to allow discrimination between the performance of several different NTMs. The results also suggest that dividing data sets into positive and negative toxicity classifications followed by the calculation of CPS leads to considerable information loss. This loss of information may be so significant that it is not possible in certain circumstances to distinguish between NTMs that are adequate and those that are not. [ABSTRACT FROM AUTHOR]

Often, the only measures of toxicity test performance provided in validation studies are the contingent probability statistics (CPS) sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV). Sensitivity and specificity are generally used in preference to NPV and PPV since NPV and PPV are assumed to vary with changes in prevalence while sensitivity and specificity are assumed to be independent of changes in prevalence. The purpose of the studies reported here was to test whether or not sensitivity and specificity are actually independent of changes in prevalence. Results derived from these studies indicate that sensitivity and specificity vary significantly depending on the prevalence of toxic substances in the set of chemicals being tested. This means sensitivity and specificity should not always be considered constant indicators of toxicity test performance. [ABSTRACT FROM AUTHOR]