Improving the Robustness of Diagnostic Accuracy Results By Asking Study Readers to Further Distinguish Subjects Who Appear to be Without the Condition Of Interest

Rationale and Objectives In diagnostic accuracy studies, cases in which a reader does not see the condition of interest are often given the same score for ROC analysis (e.g. confidence score of 0%). However, many of these cases can be further distinguished and doing so may result in more robust ROC results. Materials and Methods We examined two recent, real-world studies which used different procedures to encourage readers to further distinguish subjects who appear to be without the condition of interest. For each study, we calculated the results under two conditions. In the “zeros distinguished” (ZD) condition, we incorporated the confidence scores collected to further distinguish the normal-looking subjects. In the “zeros not distinguished” (ZND) condition, we disregarded these scores and simply gave the unit of analysis a score of zero whenever the reader did not identify the condition of interest in that unit. We compared the two conditions on (1) coverage of the ROC space and (2) discrepancy between parametric and nonparametric results. Results Compared to the ZND condition, coverage of the ROC space was improved in the ZD condition for all ROC curves in both studies. In the first study, there was a significant reduction in the discrepancy between parametric and nonparametric results (median discrepancy in ZND vs ZD condition: 0.033 vs 0.011, p = 0.012). A similar reduction was not seen in the second study, though the discrepancies were very low in both conditions (0.003 vs 0.006, p = 0.313). Conclusion Prompting readers to further distinguish cases in which they do not see the condition of interest may result in more robust ROC results, though further exploration of this topic is warranted.