Search

RATIONALE AND OBJECTIVES: Integral to the mission of the National Institutes of Health-sponsored Lung Imaging Database Consortium is the accurate definition of the spatial location of pulmonary nodules. Because the majority of small lung nodules are not resected, a reference standard from histopathology is generally unavailable. Thus assessing the source of variability in defining the spatial location of lung nodules by expert radiologists using different software tools as an alternative form of truth is necessary.MATERIALS AND METHODS: The relative differences in performance of six radiologists each applying three annotation methods to the task of defining the spatial extent of 23 different lung nodules were evaluated. The variability of radiologists' spatial definitions for a nodule was measured using both volumes and probability maps (p-map). Results were analyzed using a linear mixed-effects model that included nested random effects.RESULTS: Across the combination of all nodules, volume and p-map model parameters were found to be significant at P CONCLUSION: Radiologists represent the major source of variance as compared with drawing tools independent of drawing metric used. Although the random noise component is larger for the p-map analysis than for volume estimation, the p-map analysis appears to have more power to detect differences in radiologist-method combinations. The standard deviation of the volume measurement task appears to be proportional to nodule volume.

RATIONALE AND OBJECTIVES: The Lung Image Database Consortium (LIDC) is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource to promote the development of computer-aided detection or characterization of pulmonary nodules. To obtain the best estimate of the location and spatial extent of lung nodules, expert thoracic radiologists reviewed and annotated each scan. Because a consensus panel approach was neither feasible nor desirable, a unique two-phase, multicenter data collection process was developed to allow multiple radiologists at different centers to asynchronously review and annotate each CT scan. This data collection process was also intended to capture the variability among readers.MATERIALS AND METHODS: Four radiologists reviewed each scan using the following process. In the first or "blinded" phase, each radiologist reviewed the CT scan independently. In the second or "unblinded" review phase, results from all four blinded reviews were compiled and presented to each radiologist for a second review, allowing the radiologists to review their own annotations together with the annotations of the other radiologists. The results of each radiologist's unblinded review were compiled to form the final unblinded review. An XML-based message system was developed to communicate the results of each reading.RESULTS: This two-phase data collection process was designed, tested, and implemented across the LIDC. More than 500 CT scans have been read and annotated using this method by four expert readers; these scans either are currently publicly available at http://ncia.nci.nih.gov or will be in the near future.CONCLUSIONS: A unique data collection process was developed, tested, and implemented that allowed multiple readers at distributed sites to asynchronously review CT scans multiple times. This process captured the opinions of each reader regarding the location and spatial extent of lung nodules.

The LIDC is developing a publicly available database of thoracic computed tomography (CT) scans as a medical imaging research resource. A unique multi-center data collection process and communication system were developed to share image data and to capture the location and spatial extent of lung nodules as marked by expert radiologists. A two-phase data collection process was designed to allow multiple radiologists at different centers to asynchronously review and annotate each CT image series. Four radiologists reviewed each case using this process. In the first or "blinded" phase, each radiologist reviewed the CT series independently. In the second or "unblinded" review phase, the results from all four blinded reviews are compiled and presented to each radiologist for a second review. This allows each radiologist to review their own annotations along with those of the other radiologists. The results from each radiologist's unblinded review were compiled to form the final unblinded review. There is no forced consensus in this process. An XML-based message system was developed to communicate the results of each reading. This two-phase data collection process was designed, tested and implemented across the LIDC. It has been used for more than 130 CT cases that have been read and annotated by four expert readers and are publicly available at (http://ncia.nci.nih.gov). A data collection process was developed, tested and implemented that allowed multiple readers to review each case multiple times and that allowed each reader to observe the annotations of other readers.