Search

Place recognition in colonoscopy is needed for various reasons. 1) If a certain region needs to be rechecked during an endoscopy, the endoscopist needs to re-localize the camera accurately to the region of interest. 2) Place recognition is needed for same-patient follow-up colonoscopy to localize the region where a polyp was cut off. 3) Recent development in colonoscopic 3D reconstruction needs place recognition to establish long-range correspondence, e.g., for loop closure. However, traditional image retrieval techniques do not generalize well in colonic images. Moreover, although place recognition or instance-level image retrieval is a widely researched topic in computer vision and several benchmarks have been published for it, there has been no specific research or benchmarks in endoscopic images, which are significantly different from common images used in traditional computer vision tasks. In this paper we present a testing dataset with manually labeled groundtruth which comprises 10126 images from 20 colonoscopic subsequences. We perform an extensive evaluation on different existing place recognition techniques using different metrics.

High screening coverage during colonoscopy is crucial to effectively prevent colon cancer. Previous work has allowed alerting the doctor to unsurveyed regions by reconstructing the 3D colonoscopic surface from colonoscopy videos in real-time. However, the lighting inconsistency of colonoscopy videos can cause a key component of the colonoscopic reconstruction system, the SLAM optimization, to fail. In this work we focus on the lighting problem in colonoscopy videos. To successfully improve the lighting consistency of colonoscopy videos, we have found necessary a lighting correction that adapts to the intensity distribution of recent video frames. To achieve this in real-time, we have designed and trained an RNN network. This network adapts the gamma value in a gamma-correction process. Applied in the colonoscopic surface reconstruction system, our light-weight model significantly boosts the reconstruction success rate, making a larger proportion of colonoscopy video segments reconstructable and improving the reconstruction quality of the already reconstructed segments.

Colonoscopy is the gold standard for pre-cancerous polyps screening and treatment. The polyp detection rate is highly tied to the percentage of surveyed colonic surface. However, current colonoscopy technique cannot guarantee that all the colonic surface is well examined because of incomplete camera orientations and of occlusions. The missing regions can hardly be noticed in a continuous first-person perspective. Therefore, a useful contribution would be an automatic system that can compute missing regions from an endoscopic video in real-time and alert the endoscopists when a large missing region is detected. We present a novel method that reconstructs dense chunks of a 3D colon in real time, leaving the unsurveyed part unreconstructed. The method combines a standard SLAM system with a depth and pose prediction network to achieve much more robust tracking and less drift. It addresses the difficulties for colonoscopic images of existing simultaneous localization and mapping (SLAM) systems and end-to-end deep learning methods.

Colonoscopy is the most widely used medical technique to screen the human large intestine (colon) for cancer precursors. However, frequently parts of the surface are not visualized, and it is hard for the endoscopist to realize that from the video. Non-visualization derives from lack of orientations of the endoscope to the full circumference of parts of the colon, occlusion from colon structures, and intervening materials inside the colon. Our solution is real-time dense 3D reconstruction of colon chunks with display of the missing regions. We accomplish this by a novel deep-learning-driven dense SLAM (simultaneous localization and mapping) system that can produce a camera trajectory and a dense reconstructed surface for colon chunks (small lengths of colon). Traditional SLAM systems work poorly for the low-textured colonoscopy frames and are subject to severe scale/camera drift. In our method a recurrent neural network (RNN) is used to predict scale-consistent depth maps and camera poses of successive frames. These outputs are incorporated into a standard SLAM pipeline with local windowed optimization. The depth maps are finally fused into a global surface using the optimized camera poses. To the best of our knowledge, we are the first to reconstruct dense colon surface from video in real time and to display missing surface.

Classifying medically imaged objects, e.g., into diseased and normal classes, has been one of the important goals in medical imaging. We propose a novel classification scheme that uses a skeletal representation to provide rich non-Euclidean geometric object properties. Our statistical method combines distance weighted discrimination (DWD) with a carefully chosen Euclideanization which takes full advantage of the geometry of the manifold on which these non-Euclidean geometric object properties (GOPs) live. Our method is evaluated via the task of classifying 3D hippocampi between schizophrenics and healthy controls. We address three central questions. 1) Does adding shape features increase discriminative power over the more standard classification based only on global volume? 2) If so, does our skeletal representation provide greater discriminative power than a conventional boundary point distribution model (PDM)? 3) Especially, is Euclideanization of non-Euclidean shape properties important in achieving high discriminative power? Measuring the capability of a method in terms of area under the receiver operator characteristic (ROC) curve, we show that our proposed method achieves strongly better classification than both the classification method based on global volume alone and the s-rep-based classification method without proper Euclideanization of non-Euclidean GOPs. We show classification using Euclideanized s-reps is also superior to classification using PDMs, whether the PDMs are first Euclideanized or not. We also show improved performance with Euclideanized boundary PDMs over non-linear boundary PDMs. This demonstrates the benefit that proper Euclideanization of non-Euclidean GOPs brings not only to s-rep-based classification but also to PDM-based classification.

This article discusses a novel framework to analyze rotational deformations of real three-dimensional objects. The rotational deformations such as twisting or bending have been observed as the major variation in some medical applications, where the features of the deformed three-dimensional objects are directional data. We propose modeling and estimation of the global deformations in terms of generalized rotations of directions. The proposed method can be cast as a generalized small circle fitting on the unit sphere. We also discuss the estimation of descriptors for more complex deformations composed of two simple deformations. The proposed method can be used for a number of different three-dimensional object models. Two analyses of three-dimensional object data are presented in detail: one using skeletal representations in medical image analysis and the other from biomechanical gait analysis of the knee joint. Supplementary materials for this article are available online.

Purpose: To evaluate the feasibility of patient specific deformation models (PSDM) in the male pelvis for IGRT by limited angular imaging.Methods: In IGRT via limited angular imaging, insufficient angular projections are acquired to uniquely determine a 3D attenuation distribution. For highly limited geometries, image quality may be too poor for successful non‐rigid registration. This can be overcome by restricting the transformation space to one containing only feasible transformations learned from prior 3D images. This has been successfully applied in the lung region where a majority of deformation is due to respiratory motion which can be adequately observed at planning time with RCCT. Typically, the phases of the RCCT are registered together to form an group‐wise mean image and transformations to each training image. PCA is then performed on the transformation displacement vector fields. The transformation is found at treatment time by registration of digitally reconstructedradiographs of the transformed image to the measured projections, optimizing over the parameters of the PCA subspace. In the male pelvis, deformation is much more complicated than respiratory deformation and is largely inter‐fractional due to changes in bladder and rectal contents, articulation, and motion of the bowels. A similar model is developed for the male pelvis which takes into account pelvic anatomical information and handles the more complicated deformation space. Results: Using the leave‐one‐out method, dice similarity coefficients in the prostate compared with manual segmentations are increased over the those obtained by rigid registration and are comparable with those obtained by 3D non‐rigid registration methods. Conclusions: This method produces better results than rigid registration and is comparable with results obtained by 3D/3D registration even though it uses limited angle projections. However, its relies on daily training CTs, so it is not yet a viable clinical method. Funding provided in part by Siemens Medical

Purpose: To study the feasibility of a novel 2D/3D image registration method, called Projection Metric Learning for Shape Kernel Regression (PML‐SKR), in supporting on‐board x‐ray imaging systems to perform real‐time image‐guidedradiation therapy in the lung.Methods: PML‐SKR works in two stages: planning and treatment. At planning stage, firstly it parameterizes the patient's respiratory deformation from the patient's treatment‐planning Respiratory‐Correlated CTs (RCCTs) by doing PCA analysis on the inter‐phase respiratory deformations. Secondly, it simulates a set of training projection images from a set of deformed CTs where their associated deformation parameters are sampled within 3 standard deviations of the parameter's values observed in the RCCTs. Finally, it learns a Riemannian distance metric on projection intensity for each deformation parameter. The learned distance metric forms a Gaussian kernel of a kernel regression that minimizes the leave‐one‐out regression residual of the corresponding deformation parameter. At treatment stage, PML‐SKR interpolates the patient's 3D deformation parameters from the parameter's values in the training cases using the kernel regression with the learned distance metrics. Results: We tested PML‐SKR on the NST (Nanotube Stationary Tomosynthesis) x‐ray imaging system. In each test case, a DRR (dimension: 64×64) of an x‐ray source in the NST was simulated from a target CT for registration. The target CTs were deformed by normally distributed random samples of the first three deformation parameters. We generated 300 synthetic test cases from 3 lung datasets and measured the registration quality by the mTRE (mean Target Registration Error) over all cases and all voxels at tumor sites. With PML‐SKR's registrations, the average mTRE and its standard deviation are down from 10.89±4.44 to 0.67±0.46 mm using 125 training projection images. The computation time for each registration is 12.71±0.70 ms. Conclusion: The synthetic results have shown PML‐SKR's promise in supporting real‐time, accurate, and low‐dose lungIGRT. This work was partially supported by Siemens Medical Solutions.

Shape from shading (SFS) has been studied for decades; nevertheless, its overly simple assumptions and its ill-conditioning have resulted in infrequent use in real applications. Price et al. recently developed an iterative scheme named shape from motion and shading (SFMS) that models both shape and reflectance of an unknown surface simultaneously. SFMS produces a fairly accurate, dense 3D reconstruction from each frame of a pharyngeal endoscopic video, albeit with inconsistency between the 3D reconstructions of different frames. We present a comprehensive study of the SFMS scheme and several improvements to it: (1) We integrate a deformable registration method into the iterative scheme and use the fusion of multiple surfaces as a reference surface to guide the next iteration’s reconstruction. This can be interpreted as incorporating regularity of a frame’s reconstruction with that of temporally nearby frames. (2) We show that the reflectance model estimation is crucial and very sensitive to noise in the data. Moreover, even when the surface reflection is not assumed to be Lambertian, the reflectance model estimation function in SFMS is still overly simple for endoscopy of human tissue. By removing outlier pixels, by preventing unrealistic BRDF estimation, and by reducing the falloff speed of illumination in SFS to account for the effect of multiple bouncing of the light, we improve the reconstruction accuracy.

In computer vision and image analysis, image registration between 2D projections and a 3D image that achieves high accuracy and near real-time computation is challenging. In this paper, we propose a novel method that can rapidly detect an object’s 3D rigid motion or deformation from a 2D projection image or a small set thereof. The method is called CLARET (Correction via Limited-Angle Residues in External Beam Therapy) and consists of two stages: registration preceded by shape space and regression learning. In the registration stage, linear operators are used to iteratively estimate the motion/deformation parameters based on the current intensity residue between the target projec-tion(s) and the digitally reconstructed radiograph(s) (DRRs) of the estimated 3D image. The method determines the linear operators via a two-step learning process. First, it builds a low-order parametric model of the image region’s motion/deformation shape space from its prior 3D images. Second, using learning-time samples produced from the 3D images, it formulates the relationships between the model parameters and the co-varying 2D projection intensity residues by multi-scale linear regressions. The calculated multi-scale regression matrices yield the coarse-to-fine linear operators used in estimating the model parameters from the 2D projection intensity residues in the registration. The method’s application to Image-guided Radiation Therapy (IGRT) requires only a few seconds and yields good results in localizing a tumor under rigid motion in the head and neck and under respiratory deformation in the lung, using one treatment-time imaging 2D projection or a small set thereof.

Endoscopy enables high resolution visualization of tissue texture and is a critical step in many clinical workflows, including diagnosis and radiation therapy treatment planning for cancers in the nasopharynx. However, an endoscopic video does not provide explicit 3D spatial information, making it difficult to use in tumor localization, and it is inefficient to review. We introduce a pipeline for automatically reconstructing a textured 3D surface model, which we call an endoscopogram, from multiple 2D endoscopic video frames. Our pipeline first reconstructs a partial 3D surface model for each input individual 2D frame. In the next step (which is the focus of this paper), we generate a single high-quality 3D surface model using a groupwise registration approach that fuses multiple, partially overlapping, incomplete, and deformed surface models together. We generate endoscopograms from synthetic, phantom, and patient data and show that our registration approach can account for tissue deformations and reconstruction inconsistency across endoscopic video frames.

One goal of statistical shape analysis is the discrimination between two populations of objects. Whereas traditional shape analysis was mostly concerned with single objects, analysis of multi-object complexes presents new challenges related to alignment and pose. In this paper, we present a methodology for discriminant analysis of multiple objects represented by sampled medial manifolds. Non-euclidean metrics that describe geodesic distances between sets of sampled representations are used for alignment and discrimination. Our choice of discriminant method is the distance-weighted discriminant because of its generalization ability in high-dimensional, low sample size settings. Using an unbiased, soft discrimination score, we associate a statistical hypothesis test with the discrimination results. We explore the effectiveness of different choices of features as input to the discriminant analysis, using measures like volume, pose, shape, and the combination of pose and shape. Our method is applied to a longitudinal pediatric autism study with 10 subcortical brain structures in a population of 70 subjects. It is shown that the choices of type of global alignment and of intrinsic versus extrinsic shape features, the latter being sensitive to relative pose, are crucial factors for group discrimination and also for explaining the nature of shape change in terms of the application domain.

We present a scheme that propagates a reference skeletal model (s-rep) into a particular case of an object, thereby propagating the initial shape-related layout of the skeleton-to-boundary vectors, called spokes. The scheme represents the surfaces of the template as well as the target objects by spherical harmonics and computes a warp between these via a thin plate spline. To form the propagated s-rep, it applies the warp to the spokes of the template s-rep and then statistically refines. This automatic approach promises to make s-rep fitting robust for complicated objects, which allows s-rep based statistics to be available to all. The improvement in fitting and statistics is significant compared with the previous methods and in statistics compared with a state-of-the-art boundary based method.

Deformable shape models (DSMs) comprise a general approach that shows great promise for automatic image segmentation. Published studies by others and our own research results strongly suggest that segmentation of a normal or near-normal object from 3D medical images will be most successful when the DSM approach uses (1) knowledge of the geometry of not only the target anatomic object but also the ensemble of objects providing context for the target object and (2) knowledge of the image intensities to be expected relative to the geometry of the target and contextual objects. The segmentation will be most efficient when the deformation operates at multiple object-related scales and uses deformations that include not just local translations but the biologically important transformations of bending and twisting, i.e., local rotation, and local magnification. In computer vision an important class of DSM methods uses explicit geometric models in a Bayesian statistical framework to provide a priori information used in posterior optimization to match the DSM against a target image. In this approach a DSM of the object to be segmented is placed in the target image data and undergoes a series of rigid and nonrigid transformations that deform the model to closely match the target object. The deformation process is driven by optimizing an objective function that has terms for the geometric typicality and model-to-image match for each instance of the deformed model. The success of this approach depends strongly on the object representation, i.e., the structural details and parameter set for the DSM, which in turn determines the analytic form of the objective function. This paper describes a form of DSM called m-reps that has or allows these properties, and a method of segmentation consisting of large to small scale posterior optimization of m-reps. Segmentation by deformable m-reps, together with the appropriate data representations, visualizations, and user interface, has been implemented in software that accomplishes 3D segmentations in a few minutes. Software for building and training models has also been developed. The methods underlying this software and its abilities are the subject of this paper.

Medial representation of a three-dimensional (3-D) object or an ensemble of 3-D objects involves capturing the object interior as a locus of medial atoms, each atom being two vectors of equal length joined at the tail at the medial point. Medial representation has a variety of beneficial properties, among the most important of which are 1) its inherent geometry, provides an object-intrinsic coordinate system and thus provides correspondence between instances of the object in and near the object(s); 2) it captures the object interior and is, thus, very suitable for deformation; and 3) it provides the basis for an intuitive object-based multiscale sequence leading to efficiency of segmentation algorithms and trainability of statistical characterizations with limited training sets. As a result of these properties, medial representation is particularly suitable for the following image analysis tasks; how each operates will be described and will be illustrated by results: segmentation of objects and object complexes via deformable models; segmentation of tubular trees, e.g., of blood vessels, by following height ridges of measures of fit of medial atoms to target images; object-based image registration via medial loci of such blood vessel trees; statistical characterization of shape differences between control and pathological classes of structures. These analysis tasks are made possible by a new form of medial representation called m-reps, which is described.

This paper presents a novel processing scheme for the automatic and robust computation of a medial shape model, which represents an object population with shape variability. The sensitivity of medial descriptions to object variations and small boundary perturbations are fundamental problems of any skeletonization technique. These problems are approached with the computation of a model with common medial branching topology and grid sampling. This model is then used for a medial shape description of individual objects via a constrained model fit. The process starts from parametric 3D boundary representations with existing point-to-point homology between objects. The Voronoi skeleton of each sampled object boundary is partitioned into non-branching medial sheets and simplified by a novel pruning algorithm using a volumetric contribution criterion. Using the surface homology, medial sheets are combined to form a common medial branching topology. Finally, the medial sheets are sampled and represented as meshes of medial primitives. Results on populations of up to 184 biological objects clearly demonstrate that the common medial branching topology can be described by a small number of medial sheets and that even a coarse sampling leads to a close approximation of individual objects.

M-reps (formerly called DSLs) are a multiscale medial means for modeling and rendering 3D solid geometry. They are particularly well suited to model anatomic objects and in particular to capture prior geometric information effectively in deformable models segmentation approaches. The representation is based on figural models, which define objects at coarse scale by a hierarchy of figures—each figure generally a slab representing a solid region and its boundary simultaneously. This paper focuses on the use of single figure models to segment objects of relatively simple structure. A single figure is a sheet of medial atoms, which is interpolated from the model formed by a net, i.e., a mesh or chain, of medial atoms (hence the name m-reps), each atom modeling a solid region via not only a position and a width but also a local figural frame giving figural directions and an object angle between opposing, corresponding positions on the boundary implied by the m-rep. The special capability of an m-rep is to provide spatial and orientational correspondence between an object in two different states of deformation. This ability is central to effective measurement of both geometric typicality and geometry to image match, the two terms of the objective function optimized in segmentation by deformable models. The other ability of m-reps central to effective segmentation is their ability to support segmentation at multiple levels of scale, with successively finer precision. Objects modeled by single figures are segmented first by a similarity transform augmented by object elongation, then by adjustment of each medial atom, and finally by displacing a dense sampling of the m-rep implied boundary. While these models and approaches also exist in 2D, we focus on 3D objects. The segmentation of the kidney from CT and the hippocampus from MRI serve as the major examples in this paper. The accuracy of segmentation as compared to manual, slice-by-slice segmentation is reported.

Object descriptions used for 3D segmentation by deformable models and for statistical characterization of 3D object classes benefit from having intrinsic correspondences over deformation of the objects or multiple instances in the same object class. These correspondences apply over a variety of spatial scale levels and consequently lead to efficient segmentation and probability distributions of geometry that are trainable with an achievable number of training instances. This paper describes a figural coordinate system provided by m-reps models and shows how such coordinates not only provide the required positional correspondences, but also are intuitive and provide orientational and metric correspondences. Examples are given for the segmentation of kidneys from CT and for the statistical characterization of schizophrenia and control classes of cerebral ventricles and of hippocampus pairs.

This paper presents a multiscale framework based on a medial representation for the segmentation and shape characterization of anatomical objects in medical imagery. The segmentation procedure is based on a Bayesian deformable templates methodology in which the prior information about the geometry and shape of anatomical objects is incorporated via the construction of exemplary templates. The anatomical variability is accommodated in the Bayesian framework by defining probabilistic transformations on these templates. The transformations, thus, defined are parameterized directly in terms of natural shape operations, such as growth and bending, and their locations. A preliminary validation study of the segmentation procedure is presented. We also present a novel statistical shape analysis approach based on the medial descriptions that examines shape via separate intuitive categories, such as global variability at the coarse scale and localized variability at the fine scale. We show that the method can be used to statistically describe shape variability in intuitive terms such as growing and bending.

Fusion between an endoscopic movie and a CT can aid specifying the tumor target volume for radiotherapy. That requires a deformable pharyngeal surface registration between a 3D endoscope reconstruction and a CT segmentation. In this paper, we propose to use local geometric features for deriving a set of initial correspondences between two surfaces, with which an association graph can be constructed for registration by spectral graph matching. We also define a new similarity measurement to provide a meaningful way for computing inter-surface affinities in the association graph. Our registration method can deal with large non-rigid anatomical deformation, as well as missing data and topology change. We tested the robustness of our method with synthetic deformations and showed registration results on real data.

Recent 2D/3D deformable registration methods have achieved real-time computation of 3D lung deformations by learning global regressions that map projection intensities to deformation parameters globally in a shape space. If the mapping matrices are specialized to a specific local region of the shape space, the linear mappings will perform better than mappings trained to work on the global shape space. The major contribution of this paper is presenting a novel method that supports shape-space-localized learning for 2D/3D deformable registration and uses regression learning as an example. The method comprises two stages: training and application. In the training stage, it recursively finds normalized graph cuts that best separate the training samples given the number of desired training partitions. Second, in each training partition the projection mapping matrices are learned by linear regressions locally. Third, the method trains a decision forest for deciding into which training partition a target projection image should be classified, on the basis of projection image intensity and gradient values in various image regions. In the application stage, the decision forest classifies a target projection image into a training partition and then the learned linear regressions for that training partition are applied to the target projection image intensities to yield the desired deformation. This local regression learning method is validated on both synthetic and real lung datasets. The results indicate that the forest classification followed by local regressions yields more accurate and yet still real-time 2D/3D deformable registration than global regressions.

In image-guided radiotherapy (IGRT) of disease sites subject to respiratory motion, soft tissue deformations can affect localization accuracy. We describe the application of a method of 2D/3D deformable registration to soft tissue localization in abdomen. The method, called registration efficiency and accuracy through learning a metric on shape (REALMS), is designed to support real-time IGRT. In a previously developed version of REALMS, the method interpolated 3D deformation parameters for any credible deformation in a deformation space using a single globally-trained Riemannian metric for each parameter. We propose a refinement of the method in which the metric is trained over a particular region of the deformation space, such that interpolation accuracy within that region is improved. We report on the application of the proposed algorithm to IGRT in abdominal disease sites, which is more challenging than in lung because of low intensity contrast and nonrespiratory deformation. We introduce a rigid translation vector to compensate for nonrespiratory deformation, and design a special region-of-interest around fiducial markers implanted near the tumor to produce a more reliable registration. Both synthetic data and actual data tests on abdominal datasets show that the localized approach achieves more accurate 2D/3D deformable registration than the global approach.

A model of object shape by nets of medial and boundary primitives is justified as richly capturing multiple aspects of shape and yet requiring representation space and image analysis work proportional to the number of primitives. Metrics are described that compute an object representation's prior probability of local geometry by reflecting variabilities in the net's node and link parameter values, and that compute a likelihood function measuring the degree of match of an image to that object representation. A paradigm for image analysis of deforming such a model to optimize a posteriori probability is described, and this paradigm is shown to be usable as a uniform approach for object definition, object-based registration between images of the same or different imaging modalities, and measurement of shape variation of an abnormal anatomical object, compared with a normal anatomical object. Examples of applications of these methods in radiotherapy, surgery, and psychiatry are given.

Believing that figural zoom invariance and the cross-figural boundary linking implied by medial loci are important aspects of object shape, we present the mathematics of and algorithms for the extraction of medial loci directly from image intensities. The medial loci called cores are defined as generalized maxima in scale space of a form of medial information that is invariant to translation, rotation, and, in particular, zoom. These loci are very insensitive to image disturbances, in strong contrast to previously available medial loci, as demonstrated in a companion paper. Core-related geometric properties and image object representations are laid out which, together with the aforementioned insensitivities, allow the core to be used effectively for a variety of image analysis objectives.

The medial loci called cores are generalized maxima in scale space of a form of medial information that is invariant to translation, rotation, and zoom. S. M. Pizeret al.show that these invariances imply medial loci having the property that boundary sensing is made with apertures proportional to figural width (Comput. Vision Image Understand.69, 1998). This property of cores leads to an insensitivity to image disturbances, in strong contrast to previously available medial loci. In particular, image disturbances whose amplitude under the core's aperture, proportional to the figural width, is small compared to the intensity difference of the figure and its background produce only small changes in the position and width given by the core. These insensitivities to image disturbances, including additive intensity noise, imaging blur, boundary perturbations, and object variations are discussed and demonstrated here.

Transparency can be a useful device for depicting multiple overlapping surfaces in a single image. The challenge is to render the transparent surfaces in such a way that their 3D shape can be readily understood and their depth distance from underlying structures clearly perceived. This paper describes our investigations into the use of sparsely-distributed discrete, opaque texture as an artistic device for more explicitly indicating the relative depth of a transparent surface and for communicating the essential features of its 3D shape in an intuitively meaningful and minimally occluding way. The driving application for this work is the visualization of layered surfaces in radiation therapy treatment planning data, and the technique is illustrated on transparent isointensity surfaces of radiation dose. We describe the perceptual motivation and artistic inspiration for defining a stroke texture that is locally oriented in the direction of greatest normal curvature (and in which individual strokes are of a length proportional to the magnitude of the curvature in the direction they indicate), and we discuss two alternative methods for applying this texture to isointensity surfaces defined in a volume. We propose an experimental paradigm for objectively measuring observers' ability to judge the shape and depth of a layered transparent surface, in the course of a task which is relevant to the needs of radiotherapy treatment planning, and use this paradigm to evaluate the practical effectiveness of our approach through a controlled observer experiment based on images generated from actual clinical data.

4D image-guided radiation therapy (IGRT) for free-breathing lungs is challenging due to the complicated respiratory dynamics. Effective modeling of respiratory motion is crucial to account for the motion affects on the dose to tumors. We propose a shape-correlated statistical model on dense image deformations for patient-specic respiratory motion estimation in 4D lung IGRT. Using the shape deformations of the high-contrast lungs as the surrogate, the statistical model trained from the planning CTs can be used to predict the image deformation during delivery verication time, with the assumption that the respiratory motion at both times are similar for the same patient. Dense image deformation fields obtained by diffeomorphic image registrations characterize the respiratory motion within one breathing cycle. A point-based particle optimization algorithm is used to obtain the shape models of lungs with group-wise surface correspondences. Canonical correlation analysis (CCA) is adopted in training to maximize the linear correlation between the shape variations of the lungs and the corresponding dense image deformations. Both intra- and inter-session CT studies are carried out on a small group of lung cancer patients and evaluated in terms of the tumor location accuracies. The results suggest potential applications using the proposed method.

We present a novel 2D/3D deformable registration method, called Registration Efficiency and Accuracy through Learning Metric on Shape (REALMS), that can support real-time Image-Guided Radiation Therapy (IGRT). The method consists of two stages: planning-time learning and registration. In the planning-time learning, it firstly models the patient's 3D deformation space from the patient's time-varying 3D planning images using a low-dimensional parametrization. Secondly, it samples deformation parameters within the deformation space and generates corresponding simulated projection images from the deformed 3D image. Finally, it learns a Riemannian metric in the projection space for each deformation parameter. The learned distance metric forms a Gaussian kernel of a kernel regression that minimizes the leave-one-out regression residual of the corresponding deformation parameter. In the registration, REALMS interpolates the patient's 3D deformation parameters using the kernel regression with the learned distance metrics. Our test results showed that REALMS can localize the tumor in 10.89 ms (91.82 fps) with 2.56±1.11 mm errors using a single projection image. These promising results show REALMS's high potential to support real-time, accurate, and low-dose IGRT.

The three-dimensional (3D) reconstruction of intracerebral vessels from two-dimensional (2D) projection views is an important clinical problem that, so far, has eluded solution. This report describes a new approach that uses projection images to build arterial trees progressively from an underlying 3D network, using a new method to pair shadow images on widely separated projection views. As a test of our general methodology, we have reconstructed a middle cerebral arterial tree from two projection views of a magnetic resonance dataset and have tested the accuracy of reconstruction against the original 3D dataset. This report describes the general approach to 3D vascular reconstruction and the computer program used to perform the final reconstruction step. The results suggest that accurate, 3D reconstruction of intracranial vessels is indeed possible from as few as two projection views.

The techniques of a posteriori image restoration and iterative image feature extraction are described and compared. Image feature extraction methods known as graduated nonconvexity (GNC); variable conductance diffusion (VCD), anisotropic diffusion, and biased anisotropic diffusion (BAD), which extract edges from noisy images, are compared with a restoration/feature extraction method known as mean field annealing (MFA). All are shown to be performing the same basic operation: image relaxation. This equivalence shows the relationship between energy minimization methods and spatial analysis methods and between their respective parameters of temperature and scale. As a result of the equivalence, VCD is demonstrated to minimize a cost function, and that cost is specified explicitly. Furthermore, operations over scale space are shown to be a method of avoiding local minima. >

Representation of object shape by medial structures has been an important aspect of image analysis. Methods for describing objects in a binary image by medial axes are well understood. Many attempts have been made to construct similar medial structures for objects in gray scale images. In particular, researchers have studied images by analyzing the graphs of the intensity data and identifying ridge and valley structures on those surfaces. In this paper we review many of the definitions for ridges. Computational vision models require that medial structures should remain invariant under certain transformations of the spatial locations and intensities. For each ridge definition we point out which invariances the definition satisfies. We also give extensions of the concepts so that we can located-dimensional ridge structures withinn-dimensional images. A comparison of the ridge structures produced by the different definitions is given both by mathematical examples and by an application to a 2-dimensional MR image of a head.

We present a paradigm for empirical evaluation of digital image enhancement algorithms for mammography that uses psychophysical methods for implementation and analysis of a clinically relevant detection task. In the experiment, the observer is asked to detect and assign to a quadrant, or indicate the absence of, a simulated mammographic structure characteristic of cancer embedded in a background image of normal breast tissue. Responses are indicated interactively on a computer workstation. The parameter values for the enhancement applied to the composite image may be varied on each trial, and structure detection performance is estimated for each enhancement condition. Preliminary investigations have provided insight into an appropriate viewing duration, and furthermore, suggest that nonradiologists may be used under this methodology for the tasks investigated thus far, for predicting parameter values for clinical investigation. We are presently using this method in evaluating several contrast enhancement algorithms of possible benefit in mammography. These methods enable an objective, clinically relevant evaluation, for the purpose of optimal parameter determination or performance assessment, of digital image-processing methods potentially used in mammography.

The Multiscale Medial Axis is a means for detecting and representing object shape at multiple scales simultaneously. One of its key characteristics is that the scale used to measure object properties (and hence to represent the object) is proportional to the local width of the object. This allows it to separate fine-scale detail from larger-scale gross shape properties of the object in a manner dictated by the object itself. It works directly from image intensities and does not require a prior segmentation of the image or explicit determination of object boundaries. Fuzzy (non-binary) boundary measures are used to compute fuzzy medial measures, and axis points are identified as ridges in this fuzzy medial space. This paper presents some of the basic concepts of the Multiscale Medial Axis, describes ils computation, and demonstrates some preliminary results of its application to medical images from variety of imaging modalities.

The multiscale medial axis (MMA) is a principled means of describing both the spatial and width properties of objects in grey-scale images. We describe its computation and provide an example of its use in an image registration task.

Contrast enhancement is a fundamental step in the display of digital images whose goal is to improve human performance of visual tasks. The end result of display is the perceived brightness occurring in the human observer. Methods which incorporate local image structure in a way similar to human vision would therefore seem to have an advantage. A range of locally adaptive contrast enhancement methods are described, including a number of recently developed structure-sensitie techniques. Countering the difficulty that such methods are typically evaluated only subjectively, objective approaches for comparing the effectiveness of contrast enhancement methods are discussed.

Two methods for identifying and analyzing the multiresolution behavior of ridges and valleys in grey-scale images are described. The first method uses the tools of differential geometry to focus on local image behavior. The resulting vertex curves mark the tops of ridges and bottoms of valleys in an image. The second method focuses on the global drainage patterns of rainfall on a terrain map. The resulting watershed boundaries also identify the tops of ridges and bottoms of valleys in an image. By following these two geometric representations through scale space, the authors build resolution hierarchies on ridges and valleys in the image that can be utilized for interactive image segmentation. >

This paper examines a new method of image processing that combines information at multiple scales in order to locate boundaries. The method employs a technique of edge-affected diffusion, where blurring is limited by the presence of edges as measured at the scale of interest. By repeating such processing and measuring gradients at successively smaller scales one is able to trace a "path" through scale space which can preserve accurate information about boundaries of objects, and yet selectively remove objects that fall below a scale of interest. We compared this approach with the anisotropic diffusion technique described by Perona and Malik, which depends only on the local gradient of intensity of the processed image. We show some examples which indicate that this method could be useful for boundary detection in the presence of blurring and noise and which is also capable of performing grouping of distinct objects at various scales. This paper also examines the sensitivity of this process with respect to one′s choice of parameters.

We report on the results a 3-year project which had as its goal the development of methods to enhance radiation portal films to improve their readability. We had previously reported on a portal film enhancement technique, contrast limited adaptive histogram equalization, which could enhance low contrast detail, but degraded sharply contrasted edges. A new method, unsharp masking followed by contrast limited adaptive histogram equalization, now appears to overcome this problem. A clinical trial to test whether enhanced portal films could be read more accurately than standard ones was undertaken. The trial involved 12 readers from two institutions doing 276 readings. In this trial the enhanced films were judged to be of higher quality than the non-enhanced films (p < .001) and were read more accurately (p = .026). The usefulness and difficulties of routinely performing portal film enhancement in a busy radiation therapy department are discussed.

The authors present the intensity axis of symmetry (IAS) method for describing the shape of structures in grey-scale images. They describe the spatial and intensity variations of the image simultaneously rather than by the usual two-step process of using intensity properties of the image to segment an image into regions and describing the spatial shape of these regions. The result is an image shape description that is useful for a number of computer vision applications. The method relies on minimizing an active surface functional that provides coherence in both the spatial and intensity dimensions while deforming into an axis of symmetry. Shape-based image segmentation is possible by identifying image regions associated with individual components of the IAS. The resulting image regions have geometric coherence and correspond well to visually meaningful objects in medical images. >

This paper presents a novel patient repositioning method from limitedangle tomographic projections. It uses a machine learning strategy. Given a single planning CT image (3D) of a patient, one applies patient-specific training. Using the training results, the planning CT image, and the raw image projections collected at the treatment time, our method yields the difference between the patient’s treatmenttime postition and orientation and the planning-time position and orientation. In the training, one simulates credible treatment-time movements for the patient, and by regression it formulates a multiscale model that expresses the relationship giving the patient’s movements as a function of the corresponding changes in the tomographic projections. When the patient’s real-time projection images are acquired at treatment time, their differences from corresponding projections of the planning-time CT followed by applications of the calculated model allows the patient’s movements to be estimated. Using that estimation, the treatment-time 3D image can be estimated by transforming the planning CT image with the estimated movements,and from this, changes in the tomographic projections between those computed from the transformed CT and the real-time projection images can be calculated. The iterative, multiscale application of these steps converges to the repositioning movements. By this means, this method can overcome the deficiencies in limited-angle tomosynthesis and thus assist the clinician performing an image-guided treatment. We demonstrate the method’s success in capturing patients’ rigid motions with subvoxel accuracy with noise-added projection images of head and neck CTs.

Objectives The effect of luminance range compression on the visual threshold of a target in a computed tomographic (CT) scan was investigated in eight experiments. Methods Both visual increment thresholds of a hot target and visual decrement thresholds of a cold target were obtained. Realistic noisy CT images simulating a scan reconstruction and partial volume blurring were tested as representative of complex medical images. Negative versions, high luminance versions, and noise-free versions of these CT images also were tested. Results and conclusions In all the noisy images, as the luminance range of the image was compressed, proportionately smaller physical luminance differences between the target and its local background were needed to reach visual threshold. However, the thresholds were the same in terms of the difference in CT numbers between target and background. Noise in terms of CT numbers sets the threshold for a wide range of display conditions. In the noise-free CTs, as the luminance range was compressed, the luminance differences between the target and background needed to reach threshold also decreased, but only marginally. However, in terms of CT numbers, the thresholds were increased.

A combination of segmentation tools and fast volume renderers that provides an interactive exploration environment for volume visualization is discussed. The tools and renderers include mechanisms that distribute volume data across multiple processors, as well as image compositing techniques and solutions to representation problems in the selection and display of subregions within bounding volumes. A volume visualization technique using the interactive control of images rendered directly from volume data coupled with a user-controlled semantic classification tool is described. The variations of parallel volume rendering being explored on the Pixel-Planes 5 system and the region-of-interest selection methods and the interactive tools used by the system are presented. The flexibility and power of combining volume rendering with region-of-interest selection techniques are demonstrated using examples of medical imaging applications. >

Intensity modulated radiation therapy (IMRT) for cancers in the lung remains challenging due to the complicated respiratory dynamics. We propose a shape-navigated dense image deformation model to estimate the patient-specific breathing motion using 4D respiratory correlated CT (RCCT) images. The idea is to use the shape change of the lungs, the major motion feature in the thorax image, as a surrogate to predict the corresponding dense image deformation from training.To build the statistical model, dense diffeomorphic deformations between images of all other time points to the image at end expiration are calculated, and the shapes of the lungs are automatically extracted. By correlating the shape variation with the temporally corresponding image deformation variation, a linear mapping function that maps a shape change to its corresponding image deformation is calculated from the training sample. Finally, given an extracted shape from the image at an arbitrary time point, its dense image deformation can be predicted from the pre-computed statistics.The method is carried out on two patients and evaluated in terms of the tumor and lung estimation accuracies. The result shows robustness of the model and suggests its potential for 4D lung radiation treatment planning.

A computer algorithm which segments gray-scale images into regions of interest (objects) has been developed. These regions can provide the basis for scene analysis (including shape-parameter calculation) or surface-based, shaded-graphics display. The algorithm creates a tree structure for image description by defining a linking relationship between pixels in successively blurred versions of the initial image. The image is described in terms of nested light and dark regions. This algorithm, successfully implemented in one, two, and three dimensions, can theoretically work with any number of dimensions. The interactive postprocessing developed technique selects regions from the descriptive tree for display in several ways: pointing to a branch of the image description tree, specifying by sliders the range of scale and/or intensity of all regions which should be displayed, and pointing (on the original image) to any pixel in the desired region. The algorithm has been applied to approximately 15 computer tomography (CT) images of the abdomen. >

The advancing technology for automatic segmentation of medical images should be accompanied by techniques to inform the user of the local credibility of results. To the extent that this technology produces clinically acceptable segmentations for a significant fraction of cases, there is a risk that the clinician will assume every result is acceptable. In the less frequent case where segmentation fails, we are concerned that unless the user is alerted by the computer, she would still put the result to clinical use. By alerting the user to the location of a likely segmentation failure, we allow her to apply limited validation and editing resources where they are most needed. We propose an automated method to signal suspected non-credible regions of the segmentation, triggered by statistical outliers of the local image match function. We apply this test to m-rep segmentations of the bladder and prostate in CT images using a local image match computed by PCA on regional intensity quantile functions. We validate these results by correlating the non-credible regions with regions that have surface distance greater than 5.5mm to a reference segmentation for the bladder. A 6mm surface distance was used to validate the prostate results. Varying the outlier threshold level produced a receiver operating characteristic with area under the curve of 0.89 for the bladder and 0.92 for the prostate. Based on this preliminary result, our method has been able to predict local segmentation failures and shows potential for validation in an automatic segmentation pipeline.

A main focus of statistical shape analysis is the description of variability of a population of geometric objects. In this paper, we present work towards modeling the shape and pose variability of sets of multiple objects. Principal geodesic analysis (PGA) is the extension of the standard technique of principal component analysis (PCA) into the nonlinear Riemannian symmetric space of pose and our medial m-rep shape description, a space in which use of PCA would be incorrect. In this paper, we discuss the decoupling of pose and shape in multi-object sets using different normalization settings. Further, we introduce methods of describing the statistics of object pose and object shape, both separately and simultaneously using a novel extension of PGA. We demonstrate our methods in an application to a longitudinal pediatric autism study with object sets of 10 subcortical structures in a population of 47 subjects. The results show that global scale accounts for most of the major mode of variation across time. Furthermore, the PGA components and the corresponding distribution of different subject groups vary significantly depending on the choice of normalization, which illustrates the importance of global and local pose alignment in multi-object shape analysis. Finally, we present results of using distance weighted discrimination analysis (DWD) in an attempt to use pose and shape features to separate subjects according to diagnosis, as well as visualize discriminating differences.

Automated medical image segmentation is a challenging task that benefits from the use of effective image appearance models. In this paper, we compare appearance models at three regional scales for statistically characterizing image intensity near object boundaries in the context of segmentation via deformable models. The three models capture appearance in the form of regional intensity quantile functions. These distribution-based regional image descriptors are amenable to Euclidean methods such as principal component analysis, which we use to build the statistical appearance models. The first model uses two regions, the interior and exterior of the organ of interest. The second model accounts for exterior inhomogeneity by clustering on object-relative local intensity quantile functions to determine tissue-consistent regions relative to the organ boundary. The third model analyzes these image descriptors per geometrically defined local region. To evaluate the three models, we present segmentation results on bladders and prostates in CT in the context of day-to-day adaptive radiotherapy for the treatment of prostate cancer. Results show improved segmentations with more local regions, probably because smaller regions better represent local inhomogeneity in the intensity distribution near the organ boundary.

A crucial problem in statistical shape analysis is establishing the correspondence of shape features across a population. While many solutions are easy to express using boundary representations, this has been a considerable challenge for medial representations. This paper uses a new 3-D medial model that allows continuous interpolation of the medial manifold and provides a map back and forth between it and the boundary. A measure defined on the medial surface then allows one to write integrals over the boundary and the object interior in medial co-ordinates, enabling the expression of important object properties in an object-relative coordinate system. We use these integrals to optimize correspondence during model construction, reducing variability due to the model parameterization that could potentially mask true shape change effects. Discrimination and hypothesis testing of populations of shapes are expected to benefit, potentially resulting in improved significance of shape differences between populations even with a smaller sample size.