Search

Motivation Current spectral unmixing methods for multiplex fluorescence microscopy have a limited ability to cope with high spectral overlap as they only analyze spectral information over individual pixels. Here, we present adaptive Morphologically Constrained Spectral Unmixing (MCSU) algorithms that overcome this limitation by exploiting morphological differences between sub-cellular structures, and their local spatial context. Results The proposed method was effective at improving spectral unmixing performance by exploiting: (i) a set of dictionary-based models for object morphologies learned from the image data; and (ii) models of spatial context learned from the image data using a total variation algorithm. The method was evaluated on multi-spectral images of multiplex-labeled pancreatic ductal adenocarcinoma (PDAC) tissue samples. The former constraint ensures that neighbouring pixels correspond to morphologically similar structures, and the latter constraint ensures that neighbouring pixels have similar spectral signatures. The average Mean Squared Error (MSE) and Signal Reconstruction Error (SRE) ratio of the proposed method was 39.6% less and 8% more, respectively, compared to the best of all other algorithms that do not exploit these spatial constraints. Availability and Implementation Open source software (MATLAB). Supplementary information Supplementary data are available at Bioinformatics online.

Motivation: There is a need for effective automated methods for profiling dynamic cell–cell interactions with single-cell resolution from high-throughput time-lapse imaging data, especially, the interactions between immune effector cells and tumor cells in adoptive immunotherapy. Results: Fluorescently labeled human T cells, natural killer cells (NK), and various target cells (NALM6, K562, EL4) were co-incubated on polydimethylsiloxane arrays of sub-nanoliter wells (nanowells), and imaged using multi-channel time-lapse microscopy. The proposed cell segmentation and tracking algorithms account for cell variability and exploit the nanowell confinement property to increase the yield of correctly analyzed nanowells from 45% (existing algorithms) to 98% for wells containing one effector and a single target, enabling automated quantification of cell locations, morphologies, movements, interactions, and deaths without the need for manual proofreading. Automated analysis of recordings from 12 different experiments demonstrated automated nanowell delineation accuracy >99%, automated cell segmentation accuracy >95%, and automated cell tracking accuracy of 90%, with default parameters, despite variations in illumination, staining, imaging noise, cell morphology, and cell clustering. An example analysis revealed that NK cells efficiently discriminate between live and dead targets by altering the duration of conjugation. The data also demonstrated that cytotoxic cells display higher motility than non-killers, both before and during contact. Contact: broysam@central.uh.edu or nvaradar@central.uh.edu Supplementary information: Supplementary data are available at Bioinformatics online.

Motivation: The arbor morphologies of brain microglia are important indicators of cell activation. This article fills the need for accurate, robust, adaptive and scalable methods for reconstructing 3-D microglial arbors and quantitatively mapping microglia activation states over extended brain tissue regions. Results: Thick rat brain sections (100–300 µm) were multiplex immunolabeled for IBA1 and Hoechst, and imaged by step-and-image confocal microscopy with automated 3-D image mosaicing, producing seamless images of extended brain regions (e.g. 5903 × 9874 × 229 voxels). An over-complete dictionary-based model was learned for the image-specific local structure of microglial processes. The microglial arbors were reconstructed seamlessly using an automated and scalable algorithm that exploits microglia-specific constraints. This method detected 80.1 and 92.8% more centered arbor points, and 53.5 and 55.5% fewer spurious points than existing vesselness and LoG-based methods, respectively, and the traces were 13.1 and 15.5% more accurate based on the DIADEM metric. The arbor morphologies were quantified using Scorcioni’s L-measure. Coifman’s harmonic co-clustering revealed four morphologically distinct classes that concord with known microglia activation patterns. This enabled us to map spatial distributions of microglial activation and cell abundances. Availability and implementation: Experimental protocols, sample datasets, scalable open-source multi-threaded software implementation (C++, MATLAB) in the electronic supplement, and website (www.farsight-toolkit.org). http://www.farsight-toolkit.org/wiki/Population-scale_Three-dimensional_Reconstruction_and_Quanti-tative_Profiling_of_Microglia_Arbors Contact: broysam@central.uh.edu Supplementary information: Supplementary data are available at Bioinformatics online.

Enabled by novel molecular markers, fluorescence microscopy enables the monitoring of multiple cellular functions using live cell assays. Automated image analysis is necessary to monitor such model systems in a high-throughput and high-content environment. Here, we demonstrate the ability to simultaneously track cell cycle phase and cell motion at the single cell level. Using a recently introduced cell cycle marker, we present a set of image analysis tools for automated cell phase analysis of live cells over extended time periods. Our model-based approach enables the characterization of the four phases of the cell cycle G1, S, G2, and M, which enables the study of the effect of inhibitor compounds that are designed to block the replication of cancerous cells in any of the phases. We approach the tracking problem as a spatio-temporal volume segmentation task, where the 2D slices are stacked into a volume with time as the z dimension. The segmentation of the G2 and S phases is accomplished using level sets, and we designed a model-based shape/size constraint to control the evolution of the level set. Our main contribution is the design of a speed function coupled with a fast marching path planning approach for tracking cells across the G1 phase based on the appearance change of the nuclei. The viability of our approach is demonstrated by presenting quantitative results on both controls and cases in which cells are treated with a cell cycle inhibitor.

This protocol and the accompanying software program called LEVER (lineage editing and validation) enable quantitative automated analysis of phase-contrast time-lapse images of cultured neural stem cells. Images are captured at 5-min intervals over a period of 5-15 d as the cells proliferate and differentiate. LEVER automatically segments, tracks and generates lineage trees of the stem cells from the image sequence. In addition to generating lineage trees capturing the population dynamics of clonal development, LEVER extracts quantitative phenotypic measurements of cell location, shape, movement and size. When available, the system can include biomolecular markers imaged using fluorescence. It then displays the results to the user for highly efficient inspection and editing to correct any errors in the segmentation, tracking or lineaging. To enable high-throughput inspection, LEVER incorporates features for rapid identification of errors and for learning from user-supplied corrections to automatically identify and correct related errors.

A growing number of screening applications require the automated monitoring of cell populations in a high-throughput, high-content environment. These applications depend on accurate cell tracking of individual cells that display various behaviors including mitosis, merging, rapid movement, and entering and leaving the field of view. Many approaches to cell tracking have been developed in the past, but most are quite complex, require extensive post-processing, and are parameter intensive. To overcome such issues, we present a general, consistent, and extensible tracking approach that explicitly models cell behaviors in a graph-theoretic framework. We introduce a way of extending the standard minimum-cost flow algorithm to account for mitosis and merging events through a coupling operation on particular edges. We then show how the resulting graph can be efficiently solved using algorithms such as linear programming to choose the edges of the graph that observe the constraints while leading to the lowest overall cost. This tracking algorithm relies on accurate denoising and segmentation steps for which we use a wavelet-based approach that is able to accurately segment cells even in images with very low contrast-to-noise. In addition, the framework is able to measure and correct for microscope defocusing and stage shift. We applied the algorithms on nearly 6000 images of 400,000 cells representing 32,000 tracks taken from five separate datasets, each composed of multiple wells. Our algorithm was able to segment and track cells and detect different cell behaviors with an accuracy of over 99%. This overall framework enables accurate quantitative analysis of cell events and provides a valuable tool for high-throughput biological studies.

Understanding how stem and progenitor cells choose between alternative cell fates is a major challenge in developmental biology. Efforts to tackle this problem have been hampered by the scarcity of markers that can be used to predict cell division outcomes. Here we present a computational method, based on algorithmic information theory, to analyze dynamic features of living cells over time. Using this method, we asked whether rat retinal progenitor cells (RPCs) display characteristic phenotypes before undergoing mitosis that could foretell their fate. We predicted whether RPCs will undergo a self-renewing or terminal division with 99% accuracy, or whether they will produce two photoreceptors or another combination of offspring with 87% accuracy. Our implementation can segment, track and generate predictions for 40 cells simultaneously on a standard computer at 5 min per frame. This method could be used to isolate cell populations with specific developmental potential, enabling previously impossible investigations.

This paper presents unsupervised algorithms for discovering previously unknown subspace trends in high-dimensional data sets without the benefit of prior information. A subspace trend is a sustained pattern of gradual/progressive changes within an unknown subset of feature dimensions. A fundamental challenge to subspace trend discovery is the presence of irrelevant data dimensions, noise, outliers, and confusion from multiple subspace trends driven by independent factors that are mixed in with each other. These factors can obscure the trends in conventional dimension reduction & projection based data visualizations. To overcome these limitations, we propose a novel graph-theoretic neighborhood similarity measure for detecting concordant progressive changes across data dimensions. Using this measure, we present an unsupervised algorithm for trend-relevant feature selection, subspace trend discovery, quantification of trend strength, and validation. Our method successfully identified verifiable subspace trends in diverse synthetic and real-world biomedical datasets. Visualizations derived from the selected trend-relevant features revealed biologically meaningful hidden subspace trend(s) that were obscured by irrelevant features and noise. Although our examples are drawn from the biological domain, the proposed algorithm is broadly applicable to exploratory analysis of high-dimensional data including visualization, hypothesis generation, knowledge discovery, and prediction in diverse other applications.

An automated method is presented for selecting optimal parameter settings for vessel/neurite segmentation algorithms using the minimum description length principle and a recursive random search algorithm. It trades off a probabilistic measure of image-content coverage against its conciseness. It enables nonexpert users to select parameter settings objectively, without knowledge of underlying algorithms, broadening the applicability of the segmentation algorithm, and delivering higher morphometric accuracy. It enables adaptation of parameters across batches of images. It simplifies the user interface to just one optional parameter and reduces the cost of technical support. Finally, the method is modular, extensible, and amenable to parallel computation. The method is applied to 223 images of human retinas and cultured neurons, from four different sources, using a single segmentation algorithm with eight parameters. Improvements in segmentation quality compared to default settings using 1000 iterations ranged from 4.7%-21%. Paired t-tests showed that improvements are statistically significant (p

Detecting regions of change in multiple images of the same scene taken at different times is of widespread interest due to a large number of applications in diverse disciplines, including remote sensing, surveillance, medical diagnosis and treatment, civil infrastructure, and underwater sensing. This paper presents a systematic survey of the common processing steps and core decision rules in modern change detection algorithms, including significance and hypothesis testing, predictive models, the shading model, and background modeling. We also discuss important preprocessing methods, approaches to enforcing the consistency of the change mask, and principles for evaluating and comparing the performance of change detection algorithms. It is hoped that our classification of algorithms into a relatively small number of categories will provide useful guidance to the algorithm designer.

This paper presents a set of algorithms for robust detection of vasculature in noisy retinal video images. Three methods are studied for effective handling of outliers. The first method is based on Huber's censored likelihood ratio test. The second is based on the use of a /spl alpha/ -trimmed test statistic. The third is based on robust model selection algorithms. All of these algorithms rely on a mathematical model for the vasculature that accounts for the expected variations in intensity/texture profile, width, orientation, scale, and imaging noise. These unknown parameters are estimated implicitly within a robust detection and estimation framework. The proposed algorithms are also useful as nonlinear vessel enhancement filters. The proposed algorithms were evaluated over carefully constructed phantom images, where the ground truth is known a priori, as well as clinically recorded images for which the ground truth was manually compiled. A comparative evaluation of the proposed approaches is presented. Collectively, these methods outperformed prior approaches based on Chaudhuri et al. (1989) matched filtering, as well as the verification methods used by prior exploratory tracing algorithms, such as the work of Can et al. (1999). The Huber censored likelihood test yielded the best overall improvement, with a 145.7% improvement over the exploratory tracing algorithm, and a 43.7% improvement in detection rates over the matched filter.

Background Automated segmentation of fluorescently labeled cell nuclei in three-dimensional confocal images is essential for numerous studies, e.g., spatiotemporal fluorescence in situ hybridization quantification of immediate early gene transcription. High accuracy and automation levels are required in high-throughput and large-scale studies. Common sources of segmentation error include tight clustering and fragmentation of nuclei. Previous region-based methods are limited because they perform merging of two nuclear fragments at a time. To achieve higher accuracy without sacrificing scale, more sophisticated yet computationally efficient algorithms are needed. Methods A recursive tree-based algorithm that can consider multiple object fragments simultaneously is described. Starting with oversegmented data, it searches efficiently for the optimal merging pattern guided by a quantitative scoring criterion based on object modeling. Computation is bounded by limiting the depth of the merging tree. Results The proposed method was found to perform consistently better, achieving merging accuracy in the range of 92% to 100% compared with our previous algorithm, which varied in the range of 75% to 97%, even with a modest merging tree depth of 3. The overall average accuracy improved from 90% to 96%, with roughly the same computational cost for a set of representative images drawn from the CA1, CA3, and parietal cortex regions of the rat hippocampus. Conclusion Hierarchical tree model-based algorithms significantly improve the accuracy of automated nuclear segmentation without sacrificing speed. © 2004 Wiley-Liss, Inc.

This paper presents a method to exploit rank statistics to improve fully automatic tracing of neurons from noisy digital confocal microscope images. Previously proposed exploratory tracing (vectorization) algorithms work by recursively following the neuronal topology, guided by responses of multiple directional correlation kernels. These algorithms were found to fail when the data was of lower quality (noisier, less contrast, weak signal, or more discontinuous structures). This type of data is commonly encountered in the study of neuronal growth on microfabricated surfaces. We show that by partitioning the correlation kernels in the tracing algorithm into multiple subkernels, and using the median of their responses as the guiding criterion improves the tracing precision from 41% to 89% for low-quality data, with a 5% improvement in recall. Improved handling was observed for artifacts such as discontinuities and/or hollowness of structures. The new algorithms require slightly higher amounts of computation, but are still acceptably fast, typically consuming less than 2 seconds on a personal computer (Pentium III, 500 MHz, 128 MB). They produce labeling for all somas present in the field, and a graph-theoretic representation of all dendritic/axonal structures that can be edited. Topological and size measurements such as area, length, and tortuosity are derived readily. The efficiency, accuracy, and fully-automated nature of the proposed method makes it attractive for large-scale applications such as high-throughput assays in the pharmaceutical industry, and study of neuron growth on nano/micro-fabricated structures. A careful quantitative validation of the proposed algorithms is provided against manually derived tracing, using a performance measure that combines the precision and recall metrics.

Motivated by the problem of retinal image registration, this paper introduces and analyzes a new registration algorithm called Dual-Bootstrap Iterative Closest Point (Dual-Bootstrap ICP). The approach is to start from one or more initial, low-order estimates that are only accurate in small image regions, called bootstrap regions. In each bootstrap region, the algorithm iteratively: 1) refines the transformation estimate using constraints only from within the bootstrap region; 2) expands the bootstrap region; and 3) tests to see if a higher order transformation model can be used, stopping when the region expands to cover the overlap between images. Steps 1): and 3), the bootstrap steps, are governed by the covariance matrix of the estimated transformation. Estimation refinement [Step 2)] uses a novel robust version of the ICP algorithm. In registering retinal image pairs, Dual-Bootstrap ICP is initialized by automatically matching individual vascular landmarks, and it aligns images based on detected blood vessel centerlines. The resulting quadratic transformations are accurate to less than a pixel. On tests involving approximately 6000 image pairs, it successfully registered 99.5% of the pairs containing at least one common landmark, and 100% of the pairs containing at least one common landmark and at least 35% image overlap.

Background Automated segmentation of fluorescently-labeled cell nuclei in 3D confocal microscope images is essential to many studies involving morphological and functional analysis. A common source of segmentation error is tight clustering of nuclei. There is a compelling need to minimize these errors for constructing highly automated scoring systems. Methods A combination of two approaches is presented. First, an improved distance transform combining intensity gradients and geometric distance is used for the watershed step. Second, an explicit mathematical model for the anatomic characteristics of cell nuclei such as size and shape measures is incorporated. This model is constructed automatically from the data. Deliberate initial over-segmentation of the image data is performed, followed by statistical model-based merging. A confidence score is computed for each detected nucleus, measuring how well the nucleus fits the model. This is used in combination with the intensity gradient to control the merge decisions. Results Experimental validation on a set of rodent brain cell images showed 97% concordance with the human observer and significant improvement over prior methods. Conclusions Combining a gradient-weighted distance transform with a richer morphometric model significantly improves the accuracy of automated segmentation and FISH analysis. Cytometry Part A 56A:23–36, 2003. © 2003 Wiley-Liss, Inc.

This paper presents a robust unsupervised harmonic co-clustering method for profiling arbor morphologies for ensembles of reconstructed brain cells (e.g., neurons, microglia) based on quantitative measurements of the cellular arbors. Specifically, this method can identify groups and sub-groups of cells with similar arbor morphologies, and simultaneously identify the hierarchical grouping patterns among the quantitative arbor measurements. The robustness of the proposed algorithm derives from use of the diffusion distance measure for comparing multivariate data points, harmonic analysis theory, and a Haar-like wavelet basis for multivariate data smoothing. This algorithm is designed to be practically usable, and is embedded into the actively linked three-dimensional (3-D) visualization and analytics system in the free and open source FARSIGHT image analysis toolkit for interactive exploratory population-scale neuroanatomic studies. Studies on synthetic datasets demonstrate its superiority in clustering data matrices compared to recent hierarchical clustering algorithms. Studies on heterogeneous ensembles of real neuronal 3-D reconstructions drawn from the NeuroMorpho database show that the proposed method identifies meaningful grouping patterns among neurons based on arbor morphology, and revealing the underlying morphological differences.

Currently, no available pathological or molecular measures of tumor angiogenesis predict response to antiangiogenic therapies used in clinical practice. Recognizing that tumor endothelial cells (EC) and EC activation and survival signaling are the direct targets of these therapies, we sought to develop an automated platform for quantifying activity of critical signaling pathways and other biological events in EC of patient tumors by histopathology. Computer image analysis of EC in highly heterogeneous human tumors by a statistical classifier trained using examples selected by human experts performed poorly due to subjectivity and selection bias. We hypothesized that the analysis can be optimized by a more active process to aid experts in identifying informative training examples. To test this hypothesis, we incorporated a novel active learning (AL) algorithm into FARSIGHT image analysis software that aids the expert by seeking out informative examples for the operator to label. The resulting FARSIGHT-AL system identified EC with specificity and sensitivity consistently greater than 0.9 and outperformed traditional supervised classification algorithms. The system modeled individual operator preferences and generated reproducible results. Using the results of EC classification, we also quantified proliferation (Ki67) and activity in important signal transduction pathways (MAP kinase, STAT3) in immunostained human clear cell renal cell carcinoma and other tumors. FARSIGHT-AL enables characterization of EC in conventionally preserved human tumors in a more automated process suitable for testing and validating in clinical trials. The results of our study support a unique opportunity for quantifying angiogenesis in a manner that can now be tested for its ability to identify novel predictive and response biomarkers.

Recently, this group published fast algorithms for automatic tracing (vectorization) of the vasculature in live retinal angiograms, and for the extraction of visual landmarks formed by vascular bifurcations and crossings. These landmarks are used for feature-based image matching for controlling a computer-assisted laser retinal surgery instrument currently under development. This paper describes methods to schedule the vascular tracing computations to maximize the rate of growth of quality of the partial tracing results within a frame cycle. There are two main advantages. First, progressive image matching from partially extracted landmark sets can be faster, and provide an earlier indication of matching failure. Second, the likelihood of successful image matching is greatly improved since the extracted landmarks are of the highest quality for the given computational budget. The scheduling method is based on quantitative measures for the computational work and the quality of landmarks. A coarse grid-based analysis of the image is used to generate seed points for the tracing computations, along with estimates of local edge strengths, orientations, and vessel thickness. These estimates are used to define criteria for real-time preemptive scheduling of the tracing computations. It is shown that the optimal schedule can only be achieved in perfect hindsight, and is thus unrealizable. This leads to scheduling heuristics that approximate the behavior of the optimal algorithm. One such approximation produced approximately 400% improvement in the quality of the partial results at a defined milestone, as compared to random scheduling. The resulting algorithm can be readily implemented on conventional and multiple-processor systems, and is being applied to computer-assisted laser retinal surgery.

Although laser retinal surgery is the best available treatment for choroidal neovascularization, the current procedure has a low success rate (50%). Challenges, such as motion-compensated beam steering, ensuring complete coverage and minimizing incidental photodamage, can be overcome with improved instrumentation. This paper presents core image processing algorithms for (1) rapid identification of branching and crossover points of the retinal vasculature; (2) automatic montaging of video retinal angiograms; (3) real-time location determination and tracking using a combination of feature-tagged point-matching and dynamic-pixel templates. These algorithms tradeoff conflicting needs for accuracy, robustness to image variations (due to movements and the difficulty of providing steady illumination) and noise, and operational speed in the context of available hardware. The algorithm for locating vasculature landmarks performed robustly at a speed of 16-30 video image frames/s depending upon the field on a Silicon Graphics workstation. The montaging algorithm performed at a speed of 1.6-4 s for merging 5-12 frames. The tracking algorithm was validated by manually locating six landmark points on an image sequence with 180 frames, demonstrating a mean-squared error of 1.35 pixels. It successfully detected and rejected instances when the image dimmed, faded, lost contrast, or lost focus.

One of the major goals in biomedical image processing is accurate segmentation of networks embedded in volumetric data sets. Biological networks are composed of a meshwork of thin filaments that span large volumes of tissue. Examples of these structures include neurons and microvasculature, which can take the form of both hierarchical trees and fully connected networks, depending on the imaging modality and resolution. Network function depends on both the geometric structure and connectivity. Therefore, there is considerable demand for algorithms that segment biological networks embedded in three-dimensional data. While a large number of tracking and segmentation algorithms have been published, most of these do not generalize well across data sets. One of the major reasons for the lack of general-purpose algorithms is the limited availability of metrics that can be used to quantitatively compare their effectiveness against a pre-constructed ground-truth. In this paper, we propose a robust metric for measuring and visualizing the differences between network models. Our algorithm takes into account both geometry and connectivity to measure network similarity. These metrics are then mapped back onto an explicit model for visualization.

Multitemporal Association Tracking (MAT) is a new graph-based method for multitarget tracking in biological applications that reduces the error rate and implementation complexity compared to approaches based on bipartite matching. The data association problem is solved over a window of future detection data using a graph-based cost function that approximates the Bayesian a posteriori association probability. MAT has been applied to hundreds of image sequences, tracking organelle and vesicles to quantify the deficiencies in axonal transport that can accompany neurodegenerative disorders such as Huntington’s Disease and Multiple Sclerosis and to quantify changes in transport in response to therapeutic interventions.

The accuracy and reliability of automated neurite tracing systems is ultimately limited by image quality as reflected in the signal-to-noise ratio, contrast, and image variability. This paper describes a novel combination of image processing methods that operate on images of neurites captured by confocal and widefield microscopy, and produce synthetic images that are better suited to automated tracing. The algorithms are based on the curvelet transform (for denoising curvilinear structures and local orientation estimation), perceptual grouping by scalar voting (for elimination of non-tubular structures and improvement of neurite continuity while preserving branch points), adaptive focus detection, and depth estimation (for handling widefield images without deconvolution). The proposed methods are fast, and capable of handling large images. Their ability to handle images of unlimited size derives from automated tiling of large images along the lateral dimension, and processing of 3-D images one optical slice at a time. Their speed derives in part from the fact that the core computations are formulated in terms of the Fast Fourier Transform (FFT), and in part from parallel computation on multi-core computers. The methods are simple to apply to new images since they require very few adjustable parameters, all of which are intuitive. Examples of pre-processing DIADEM Challenge images are used to illustrate improved automated tracing resulting from our pre-processing methods.

This paper describes an automated method to profile the velocity patterns of small organelles (BDNF granules) being transported along a selected section of axon of a cultured neuron imaged by time-lapse fluorescence microscopy. Instead of directly detecting the granules as in conventional tracking, the proposed method starts by generating a two-dimensional spatio-temporal map (kymograph) of the granule traffic along an axon segment. Temporal sharpening during the kymograph creation helps to highlight granule movements while suppressing clutter due to stationary granules. A voting algorithm defined over orientation distribution functions is used to refine the locations and velocities of the granules. The refined kymograph is analyzed using an algorithm inspired from the minimum set cover framework to generate multiple motion trajectories of granule transport paths. The proposed method is computationally efficient, robust to significant levels of noise and clutter, and can be used to capture and quantify trends in transport patterns quickly and accurately. When evaluated on a collection of image sequences, the proposed method was found to detect granule movement events with 94% recall rate and 82% precision compared to a time-consuming manual analysis. Further, we present a study to evaluate the efficacy of velocity profiling by analyzing the impact of oxidative stress on granule transport in which the fully automated analysis correctly reproduced the biological conclusion generated by manual analysis.

This paper presents robust 3-D algorithms to segment vasculature that is imaged by labeling laminae, rather than the lumenal volume. The signal is weak, sparse, noisy, nonuniform, low-contrast, and exhibits gaps and spectral artifacts, so adaptive thresholding and Hessian filtering based methods are not effective. The structure deviates from a tubular geometry, so tracing algorithms are not effective. We propose a four step approach. The first step detects candidate voxels using a robust hypothesis test based on a model that assumes Poisson noise and locally planar geometry. The second step performs an adaptive region growth to extract weakly labeled and fine vessels while rejecting spectral artifacts. To enable interactive visualization and estimation of features such as statistical confidence, local curvature, local thickness, and local normal, we perform the third step. In the third step, we construct an accurate mesh representation using marching tetrahedra, volume-preserving smoothing, and adaptive decimation algorithms. To enable topological analysis and efficient validation, we describe a method to estimate vessel centerlines using a ray casting and vote accumulation algorithm which forms the final step of our algorithm. Our algorithm lends itself to parallel processing, and yielded an 8 x speedup on a graphics processor (GPU). On synthetic data, our meshes had average error per face (EPF) values of (0.1-1.6) voxels per mesh face for peak signal-to-noise ratios from (110-28 dB). Separately, the error from decimating the mesh to less than 1% of its original size, the EPF was less than 1 voxel/face. When validated on real datasets, the average recall and precision values were found to be 94.66% and 94.84%, respectively.

The maximum-likelihood (ML) method for the quantitative analysis of electron-microscopic autoradiographs has been shown to be substantially superior to the conventional crossfire (CF) method. It can generate reliable and accurate tracer concentration estimates with far fewer micrographs and produce valid estimates even at counts low enough to preclude the use of the crossfire method while eliminating the need for special ad hoc treatment of narrow membranous structures as well as the secondary verification of the tracer concentration estimates. Despite these significant advantages, the large computational requirements of the ML method has to date hampered its widespread use. In this paper, we present a new line-integration method that allows us to reduce the computational requirements of the ML method to a point where it becomes feasible to implement it on a small computer system of the type typically available to a laboratory user of EM autoradiography. We present the complete line-integration method for the particular case of EM autoradiography with tritium, and show how it can be adapted to other isotopes. We have constructed a software package that implements the complete maximum-likelihood method on the IBM PC class of machines using our line-integration method. Features of this software package which are of particular importance to the research community are device independence, which makes it usable with a large variety of currently available laboratory equipment, and easy portability of the software and data between different computer systems.

This paper presents a method for improved automatic delineation of dendrites and spines from three-dimensional (3-D) images of neurons acquired by confocal or multi-photon fluorescence microscopy. The core advance presented here is a direct grayscale skeletonization algorithm that is constrained by a structural complexity penalty using the minimum description length (MDL) principle, and additional neuroanatomy-specific constraints. The 3-D skeleton is extracted directly from the grayscale image data, avoiding errors introduced by image binarization. The MDL method achieves a practical tradeoff between the complexity of the skeleton and its coverage of the fluorescence signal. Additional advances include the use of 3-D spline smoothing of dendrites to improve spine detection, and graph-theoretic algorithms to explore and extract the dendritic structure from the grayscale skeleton using an intensity-weighted minimum spanning tree (IW-MST) algorithm. This algorithm was evaluated on 30 datasets organized in 8 groups from multiple laboratories. Spines were detected with false negative rates less than 10% on most datasets (the average is 7.1%), and the average false positive rate was 11.8%. The software is available in open source form.

Automatic segmentation of cell nuclei is an essential step in image cytometry and histometry. Despite substantial progress, there is a need to improve accuracy, speed, level of automation, and adaptability to new applications. This paper presents a robust and accurate novel method for segmenting cell nuclei using a combination of ideas. The image foreground is extracted automatically using a graph-cuts-based binarization. Next, nuclear seed points are detected by a novel method combining multiscale Laplacian-of-Gaussian filtering constrained by distance-map-based adaptive scale selection. These points are used to perform an initial segmentation that is refined using a second graph-cuts-based algorithm incorporating the method of alpha expansions and graph coloring to reduce computational complexity. Nuclear segmentation results were manually validated over 25 representative images (15 in vitro images and 10 in vivo images, containing more than 7400 nuclei) drawn from diverse cancer histopathology studies, and four types of segmentation errors were investigated. The overall accuracy of the proposed segmentation algorithm exceeded 86%. The accuracy was found to exceed 94% when only over- and undersegmentation errors were considered. The confounding image characteristics that led to most detection/segmentation errors were high cell density, high degree of clustering, poor image contrast and noisy background, damaged/irregular nuclei, and poor edge information. We present an efficient semiautomated approach to editing automated segmentation results that requires two mouse clicks per operation.

A growing number of screening applications require the automated monitoring of cell populations in a high-throughput, high-content environment. These applications depend on accurate cell tracking of individual cells that display various behaviors including mitosis, occlusion, rapid movement, and entering and leaving the field of view. We present a tracking approach that explicitly models each of these behaviors and represents the association costs in a graph-theoretic minimum-cost flow framework. We show how to extend the minimum-cost flow algorithm to account for mitosis and merging events by coupling particular edges. We applied the algorithm to nearly 6,000 images of 400,000 cells representing 32,000 tracks taken from five separate datasets, each composed of multiple wells. Our algorithm is able to track cells and detect different cell behaviors with an accuracy of over 99%.

Since the introduction by Shepp and Vardi [Shepp, L. A. & Vardi, Y. (1982) IEEE Trans. Med. Imaging 1, 113-121] of the expectation-maximization algorithm for the generation of maximum-likelihood images in emission tomography, a number of investigators have applied the maximum-likelihood method to imaging problems. Though this approach is promising, it is now well known that the unconstrained maximum-likelihood approach has two major drawbacks: (i) the algorithm is computationally demanding, resulting in reconstruction times that are not acceptable for routine clinical application, and (ii) the unconstrained maximum-likelihood estimator has a fundamental noise artifact that worsens as the iterative algorithm climbs the likelihood hill. In this paper the computation issue is addressed by proposing an implementation on the class of massively parallel single-instruction, multiple-data architectures. By restructuring the superposition integrals required for the expectation-maximization algorithm as the solutions of partial differential equations, the local data passage required for efficient computation on this class of machines is satisfied. For dealing with the "noise artifact" a Markov random field prior determined by Good's rotationally invariant roughness penalty is incorporated. These methods are demonstrated on the single-instruction multiple-data class of parallel processors, with the computation times compared with those on conventional and hypercube architectures.

This paper presents automated methods to quantify dynamic phenomena such as cell–cell interactions and cell migration patterns from time-lapse series of multi-channel three-dimensional image stacks of living specimens. Various 5-dimensional ( x , y , z , t , λ ) images containing dendritic cells (DC), and T-cells or thymocytes in the developing mouse thymic cortex and lymph node were acquired by two-photon laser scanning microscopy (TPLSM). The cells were delineated automatically using a mean-shift clustering algorithm. This enables morphological measurements to be computed. A robust multiple-hypothesis tracking algorithm was used to track thymocytes (the DC were stationary). The tracking data enable dynamic measurements to be computed, including migratory patterns of thymocytes, and duration of thymocyte-DC contacts. Software was developed for efficient inspection, corrective editing, and validation of the automated analysis results. Our software-generated results agreed with manually generated measurements to within 8%.

This updates an earlier publication by the authors describing a robust framework for detecting vasculature in noisy retinal fundus images. We improved the handling of the "central reflex" phenomenon in which a vessel has a "hollow" appearance. This is particularly pronounced in dual-wavelength images acquired at 570 and 600 nm for retinal oximetry. It is prominent in the 600 nm images that are sensitive to the blood oxygen content. Improved segmentation of these vessels is needed to improve oximetry. We show that the use of a generalized dual-Gaussian model for the vessel intensity profile instead of the Gaussian yields a significant improvement. Our method can account for variations in the strength of the central reflex, the relative contrast, width, orientation, scale, and imaging noise. It also enables the classification of regular and central reflex vessels. The proposed method yielded a sensitivity of 72% compared to 38% by the algorithm of Can et al., and 60% by the robust detection based on a single-Gaussian model. The specificity for the methods were 95%, 97%, and 98%, respectively.

This paper presents an automated method to identify arteries and veins in dual-wavelength retinal fundus images recorded at 570 and 600 nm. Dual-wavelength imaging provides both structural and functional features that can be exploited for identification. The processing begins with automated tracing of the vessels from the 570-nm image. The 600-nm image is registered to this image, and structural and functional features are computed for each vessel segment. We use the relative strength of the vessel central reflex as the structural feature. The central reflex phenomenon, caused by light reflection from vessel surfaces that are parallel to the incident light, is especially pronounced at longer wavelengths for arteries compared to veins. We use a dual-Gaussian to model the cross-sectional intensity profile of vessels. The model parameters are estimated using a robust M-estimator, and the relative strength of the central reflex is computed from these parameters. The functional feature exploits the fact that arterial blood is more oxygenated relative to that in veins. This motivates use of the ratio of the vessel optical densities (ODs) from images at oxygen-sensitive and oxygen-insensitive wavelengths (ODR = OD600/OD570) as a functional indicator. Finally, the structural and functional features are combined in a classifier to identify the type of the vessel. We experimented with four different classifiers and the best result was given by a support vector machine (SVM) classifier. With the SVM classifier, the proposed algorithm achieved true positive rates of 97% for the arteries and 90% for the veins, when applied to a set of 251 vessel segments obtained from 25 dual wavelength images. The ability to identify the vessel type is useful in applications such as automated retinal vessel oximetry and automated analysis of vascular changes without manual intervention.

Brain structural complexity has confounded prior efforts to extract quantitative image-based measurements. We present a systematic 'divide and conquer' methodology for analyzing three-dimensional (3D) multi-parameter images of brain tissue to delineate and classify key structures, and compute quantitative associations among them. To demonstrate the method, thick ( approximately 100 microm) slices of rat brain tissue were labeled using three to five fluorescent signals, and imaged using spectral confocal microscopy and unmixing algorithms. Automated 3D segmentation and tracing algorithms were used to delineate cell nuclei, vasculature, and cell processes. From these segmentations, a set of 23 intrinsic and 8 associative image-based measurements was computed for each cell. These features were used to classify astrocytes, microglia, neurons, and endothelial cells. Associations among cells and between cells and vasculature were computed and represented as graphical networks to enable further analysis. The automated results were validated using a graphical interface that permits investigator inspection and corrective editing of each cell in 3D. Nuclear counting accuracy was >89%, and cell classification accuracy ranged from 81 to 92% depending on cell type. We present a software system named FARSIGHT implementing our methodology. Its output is a detailed XML file containing measurements that may be used for diverse quantitative hypothesis-driven and exploratory studies of the central nervous system.

This paper presents methods to model complex vasculature in three-dimensional (3-D) images using cylindroidal superellipsoids, along with robust estimation and detection algorithms for automated image analysis. This model offers an explicit, low-order parameterization, enabling joint estimation of boundary, centerlines, and local pose. It provides a geometric framework for directed vessel traversal, and extraction of topological information like branch point locations and connectivity. M-estimators provide robust region-based statistics that are used to drive the superellipsoid toward a vessel boundary. A robust likelihood ratio test is used to differentiate between noise, artifacts, and other complex unmodeled structures, thereby verifying the model estimate. The proposed methodology behaves well across scale-space, shows a high degree of insensitivity to adjacent structures and implicitly handles branching. When evaluated on synthetic imagery mimicking specific structural complexities in tumor microvasculature, it consistently produces ubvoxel accuracy estimates of centerlines and widths in the presence of closely-adjacent vessels, branch points, and noise. An edit-based validation demonstrated a precision level of 96.6% at a recall level of 95.4%. Overall, it is robust enough for large-scale application.

Detection and analysis of changes from retinal images is important in clinical practice, quantitative scoring of clinical trials, computer-assisted reading centers, and in medical research. This paper presents a fully-automated approach for robust detection and classification of changes in longitudinal time-series of fluorescein angiograms (FA). The changes of interest here are related to the development of choroidal neo-vascularization (CNV) in wet macular degeneration. Specifically, the changes in CNV regions as well as the retinal pigment epithelium (RPE) hypertrophic regions are detected and analyzed to study the progression of disease and effect of treatment. Retinal features including the vasculature, vessel branching/crossover locations, optic disk and location of the fovea are first segmented automatically. The images are then registered to sub-pixel accuracy using a 12-dimensional mapping that accounts for the unknown retinal curvature and camera parameters. Spatial variations in illumination are removed using a surface fitting algorithm that exploits the segmentations of the various features. The changes are identified in the regions of interest and a Bayesian classifier is used to classify the changes into clinically significant classes. The automated change analysis algorithms were found to have a success rate of 83%

A fully automated approach is presented for robust detection and classification of changes in longitudinal time-series of color retinal fundus images of diabetic retinopathy. The method is robust to: 1) spatial variations in illumination resulting from instrument limitations and changes both within, and between patient visits; 2) imaging artifacts such as dust particles; 3) outliers in the training data; 4) segmentation and alignment errors. Robustness to illumination variation is achieved by a novel iterative algorithm to estimate the reflectance of the retina exploiting automatically extracted segmentations of the retinal vasculature, optic disk, fovea, and pathologies. Robustness to dust artifacts is achieved by exploiting their spectral characteristics, enabling application to film-based, as well as digital imaging systems. False changes from alignment errors are minimized by subpixel accuracy registration using a 12-parameter transformation that accounts for unknown retinal curvature and camera parameters. Bayesian detection and classification algorithms are used to generate a color-coded output that is readily inspected. A multiobserver validation on 43 image pairs from 22 eyes involving nonproliferative and proliferative diabetic retinopathies, showed a 97% change detection rate, a 3 % miss rate, and a 10% false alarm rate. The performance in correctly classifying the changes was 99.3%. A self-consistency metric, and an error factor were developed to measure performance over more than two periods. The average self consistency was 94% and the error factor was 0.06%. Although this study focuses on diabetic changes, the proposed techniques have broader applicability in ophthalmology.

This paper presents model-based information-theoretic methods to quantify the complexity of tumor microvasculature, taking into account shape, textural, and structural irregularities. The proposed techniques are completely automated, and are applicable to optical slices (3-D) or projection images (2-D). Improvements upon the prior literature include: (i) measuring local (vessel segment) as well as global (entire image) vascular complexity without requiring explicit segmentation or tracing; (ii) focusing on the vessel boundaries in the complexity estimate; and (iii) added robustness to image artifacts common to tumor microvasculature images. Vessels are modeled using a family of super-Gaussian functions that are based on the superquadric modeling primitive common in computer vision. The superquadric generalizes a simple ellipsoid by including shape parameters that allow it to approximate a cylinder with elliptical cross-sections (generalized cylinder). The super-Gaussian is obtained by composing a superquadric with an exponential function giving a form that is similar to a standard Gaussian function but with the ability to produce level sets that approximate generalized cylinders. Importantly, the super-Gaussian is continuous and differentiable so it can be fit to image data using robust non-linear regression. This fitting enables quantification of the intrinsic complexity of vessel data vis-a-vis the super-Gaussian model within a minimum message length (MML) framework. The resulting measures are expressed in units of information (bits). Synthetic and real-data examples are provided to illustrate the proposed measures.

This paper addresses the problem of migrating large and complex computer vision code bases that have been developed off-line, into efficient real-time implementations avoiding the need for rewriting the software, and the associated costs. Creative linking strategies based on Linux loadable kernel modules are presented to create a simultaneous realization of real-time and off-line frame rate computer vision systems from a single code base. In this approach, systemic predictability is achieved by inserting time-critical components of a user-level executable directly into the kernel as a virtual device driver. This effectively emulates a single process space model that is nonpreemptable, nonpageable, and that has direct access to a powerful set of system-level services. This overall approach is shown to provide the basis for building a predictable frame-rate vision system using commercial off-the-shelf hardware and a standard uniprocessor Linux operating system. Experiments on a frame-rate vision system designed for computer-assisted laser retinal surgery show that this method reduces the variance of observed per-frame central processing unit cycle counts by two orders of magnitude. The conclusion is that when predictable application algorithms are used, it is possible to efficiently migrate to a predictable frame-rate computer vision system.

Fluorescence in situ hybridization (FISH) of neural activity-regulated, immediate-early gene (IEG) expression provides a method of functional brain imaging with cellular resolution. This enables the identification, in one brain, of which specific principal neurons were active during each of two distinct behavioral epochs. The unprecedented potential of this differential method for large-scale analysis of functional neural circuits is limited, however, by the time-intensive nature of manual image analysis. A comprehensive software tool for processing three-dimensional, multi-spectral confocal image stacks is described which supports the automation of this analysis. Nuclei counterstained with conventional DNA dyes and FISH signals indicating the sub-cellular distribution of specific, IEG RNA species are imaged using different spectral channels. The DNA channel data are segmented into individual nuclei by a three-dimensional multi-step algorithm that corrects for depth-dependent attenuation, non-isotropic voxels, and imaging noise. Intra-nuclear and cytoplasmic FISH signals are associated spatially with the nuclear segmentation results to generate a detailed tabular/database and graphic representation. Here we present a comprehensive validation of data generated by the automated software against manual quantification by human experts on hippocampal and parietal cortical regions (96.5% concordance with multi-expert consensus). The high degree of reliability and accuracy suggests that the software will generalize well to multiple brain areas and eventually to large-scale brain analysis.

Real-time spatial referencing is an important alternative to tracking for designing spatially aware ophthalmic instrumentation for procedures such as laser photocoagulation and perimetry. It requires independent, fast registration of each image frame from a digital video stream (1024 /spl times/ 1024 pixels) to a spatial map of the retina. Recently, we have introduced a spatial referencing algorithm that works in three primary steps: 1) tracing the retinal vasculature to extract image feature (landmarks); 2) invariant indexing to generate hypothesized landmark correspondences and initial transformations; and 3) alignment and verification steps to robustly estimate a 12-parameter quadratic spatial transformation between the image frame and the map. The goal of this paper is to introduce techniques to minimize the amount of computation for successful spatial referencing. The fundamental driving idea is to make feature extraction subservient to registration and, therefore, only produce the information needed for verified, accurate transformations. To this end, the image is analyzed along one-dimensional, vertical and horizontal grid lines to produce a regular sampling of the vasculature, needed for step 3) and to initiate step 1). Tracing of the vascular is then prioritized hierarchically to quickly extract landmarks and groups (constellations) of landmarks for indexing. Finally, the tracing and spatial referencing computations are integrated so that landmark constellations found by tracing are tested immediately. The resulting implementation is an order-of-magnitude faster with the same success rate. The average total computation time is 31.2 ms per image on a 2.2-GHz Pentium Xeon processor.

Automated methods are described for tracing and analysis of changes in angiogenic vasculature imaged by a multiphoton laser-scanning confocal microscope. Utilizing chronic animal window models, time series of in vivo 3-D images were acquired on approximately the same target volume of the same specimen while undergoing angiogenic change (typically every 24 h for 7 days). Objective, precise, 3-D, rapid, and fully automated vessel morphometry was performed using an adaptive tracing algorithm that is based on a generalized irregular cylinder model of the vasculature. This algorithm was found to be not only adaptive enough for tracing angiogenic vasculature, but also very efficient in its use of computer memory, and fast, taking less than 1 min to trace a 768 × 512 × 32, 8-bit/pixel 3-D image stack on a Dell Pentium III 1-GHz computer. The automatically traced centerlines were manually validated on six image stacks and the average spatial error was measured to be 2 pixels, with an average concordance of 81% between manual and automated traces on a voxel basis. The tracing output includes geometrical statistics of traced vasculature and serves as the basis of statistical change analysis. The computer methods described here are designed to be scalable to much larger hypothesis testing studies involving quantitative measurements of tumor angiogenesis, gene expression relative to known vascular structures, and impact of drug delivery.

Algorithms are presented for fully automatic three-dimensional (3D) tracing of neurons that are imaged by fluorescence confocal microscopy. Unlike previous voxel-based skeletonization methods, the present approach works by recursively following the neuronal topology, using a set of 4 /spl times/ N/sup 2/ directional kernels (e.g., N = 32), guided by a generalized 3D cylinder model. This method extends our prior work on exploratory tracing of retinal vasculature to 3D space. Since the centerlines are of primary interest, the 3D extension can be accomplished by four rather than six sets of kernels. Additional modifications, such as dynamic adaptation of the correlation kernels, and adaptive step size estimation, were introduced for achieving robustness to photon noise, varying contrast, and apparent discontinuity and/or hollowness of structures. The end product is a labeling of all somas present, graph-theoretic representations of all dendritic/axonal structures, and image statistics such as soma volume and centroid, soma interconnectivity, the longest branch, and lengths of all graph branches originating from a soma. This method is able to work directly with unprocessed confocal images, without expensive deconvolution or other preprocessing. It is much faster that skeletonization, typically consuming less than a minute to trace a 70 MB image on a 500 MHz computer. These properties make it attractive for large-scale automated tissue studies that require rapid on-line image analysis, such as high-throughput neurobiology/angiogenesis assays, and initiatives such as the Human Brain Project.

Algorithms are presented for rapid, automatic, robust, adaptive, and accurate tracing of retinal vasculature and analysis of intersections and crossovers. This method improves upon prior work in several ways: 1) automatic adaptation from frame to frame without manual initialization/adjustment, with few tunable parameters; 2) robust operation on image sequences exhibiting natural variability, poor and varying imaging conditions, including over/under-exposure, low contrast, and artifacts such as glare; 3) does not require the vasculature to be connected, so it can handle partial views; and 4) operation is efficient enough for use on unspecialized hardware, and amenable to deadline-driven computing, being able to produce a rapidly and monotonically improving sequence of usable partial results. Increased computation can be traded for superior tracing performance. Its efficiency comes from direct processing on gray-level data without any preprocessing, and from processing only a minimally necessary fraction of pixels in an exploratory manner, avoiding low-level image-wide operations such as thresholding, edge detection, and morphological processing. These properties make the algorithm suited to real-time, on-line (live) processing and is being applied to computer-assisted laser retinal surgery.

Three-dimensional (3-D) image analysis algorithms and experimental results that demonstrate the feasibility of fully automated tracing of neurons from fluorescence confocal microscopy data are presented. The input to the automated analysis is a set of successive optical slices that have been acquired using a confocal scanning laser microscope. The output of the system is a labelled graph representation of the neuronal topology that is spatially aligned with the 3-D image data. A variety of topological and metric analyses can be carried out using this representation. For instance, precise measurements of volumes, lengths, diameters and tortuosities can be made over specific portions of the neuron that are specified in terms of the graph representation. The effectiveness of the method is demonstrated for a set of sample fields featuring selectively stained neurons. Additional work will be needed to refine the method for unsupervised use with complex data involving multiple intertwined neurons and extremely fine dendritic structures.

We present recent developments of a widefield computer/microscope system and image reconstruction algorithm for producing three-dimensional (3D) increased depth of field images in the form of brightfield stereo pairs of thick specimens. The theoretical principle of this image reconstruction technique is based on Weiner-type inverse filtering. A number of extensions and refinements to our previous work have included further testing of the system with a broader class of specimens and the implementation of several pragmatic refinements important for future 3D microscopy systems. These refinements include histogram modification routines for improving visualization, a preprocessing routine to eliminate edge artifacts due to circular convolution and other effects, stereo viewing angle optimization, a rule of thumb estimate for the axial sampling rate, and incorporation of a variation of the Fast Fourier Transform and filtering operations that significantly reduce computational time. Images of spyrogyra, neonatal rat hippocampal neurons, and cervical/vaginal cell smears are presented to show the utility of these methods for 3D visualization. The primary advantages of these methods are that they operate with an ordinary transmitted light microscope and are inexpensively implemented on a personal computer with reasonable computation time.

We present an automated method to perform accurate, rapid, and objective measurement of the blood oxygen saturation over each segment of the retinal vascular hierarchy from dual-wavelength fundus images. Its speed and automation (2 s per entire image versus 20 s per segment for manual methods) enables detailed level-by-level measurements over wider areas. An automated tracing algorithm is used to estimate vessel centerlines, thickness, directions, and locations of landmarks such as bifurcations and crossover points. The hierarchical structure of the vascular network is recovered from the trace fragments and landmarks by a novel algorithm. Optical densities (OD) are measured from vascular segments using the minimum reflected intensities inside and outside the vessel. The OD ratio (ODR=OD 600 /OD 570 ) bears an inverse relationship to systemic HbO 2 saturation (SO 2 ). The sensitivity for detecting saturation change when breathing air versus pure oxygen was calculated from the measurements made on six subjects and was found to be 0.0226 ODR units, which is in good agreement with previous manual measurements by the dual-wavelength technique, indicating the validity of the automation. A fully automated system for retinal vessel oximetry would prove useful to achieve early assessments of risk for progression of disease conditions associated with oxygen utilization.