Search

Understanding the efficacy of digital biomarkers in vision-based human motion analysis is essential, not only for interpreting the computer-aided exam results but also for advancing the next generation of digital health tool solutions. This study extensively analyzes digitized neurological examination (DNE) biomarkers for detecting and documenting exam features of Parkinson's disease (PD) and other neurological disorders (OD). Collected over 113 participants, DNE-113, a multi-test DNE database of finger tapping, finger to finger, forearm roll, stand-up and walk, and facial activation tests, covering a broader range of neurological abnormalities beyond PD is first proposed. Subsequently, DNE-113 is integrated into pyDNE - a convenient open-source toolbox, streamlining the creation and assessment of digital biomarkers. This toolbox empowers us to assess the quality of DNE biomarkers across diverse classification tasks. We showcase the discriminative potency of DNE biomarkers, successfully characterizing abnormal signals in neurological patients. Our findings highlight not only the potential use cases but also the persisting challenges in constructing digital biomarkers for computer-aided movement analysis on PD and OD patients.

Adaptive sampling that exploits the spatiotemporal redundancy in videos is critical for always-on action recognition on wearable devices with limited computing and battery resources. The commonly used fixed sampling strategy is not context-aware and may under-sample the visual content, and thus adversely impacts both computation efficiency and accuracy. Inspired by the concepts of foveal vision and pre-attentive processing from the human visual perception mechanism, we introduce a novel adaptive spatiotemporal sampling scheme for efficient action recognition. Our system pre-scans the global scene context at low-resolution and decides to skip or request high-resolution features at salient regions for further processing. We validate the system on EPIC-KITCHENS and UCF-101 (split-1) datasets for action recognition, and show that our proposed approach can greatly speed up inference with a tolerable loss of accuracy compared with those from state-of-the-art baselines. Source code is available in https://github.com/knmac/adaptive_spatiotemporal.

The ability to use digitally recorded and quantified neurological exam information is important to help healthcare systems deliver better care, in-person and via telehealth, as they compensate for a growing shortage of neurologists. Current neurological digital biomarker pipelines, however, are narrowed down to a specific neurological exam component or applied for assessing specific conditions. In this paper, we propose an accessible vision-based exam and documentation solution called Digitized Neurological Examination (DNE) to expand exam biomarker recording options and clinical applications using a smartphone/tablet. Through our DNE software, healthcare providers in clinical settings and people at home are enabled to video capture an examination while performing instructed neurological tests, including finger tapping, finger to finger, forearm roll, and stand-up and walk. Our modular design of the DNE software supports integrations of additional tests. The DNE extracts from the recorded examinations the 2D/3D human-body pose and quantifies kinematic and spatio-temporal features. The features are clinically relevant and allow clinicians to document and observe the quantified movements and the changes of these metrics over time. A web server and a user interface for recordings viewing and feature visualizations are available. DNE was evaluated on a collected dataset of 21 subjects containing normal and simulated-impaired movements. The overall accuracy of DNE is demonstrated by classifying the recorded movements using various machine learning models. Our tests show an accuracy beyond 90% for upper-limb tests and 80% for the stand-up and walk tests.

Background foreground separation (BFS) is a popular computer vision problem where dynamic foreground objects are separated from the static background of a scene. Typically, this is performed using consumer cameras because of their low cost, human interpretability, and high resolution. Yet, cameras and the BFS algorithms that process their data have common failure modes due to lighting changes, highly reflective surfaces, and occlusion. One solution is to incorporate an additional sensor modality that provides robustness to such failure modes. In this paper, we explore the ability of a cost-effective radar system to augment the popular Robust PCA technique for BFS. We apply the emerging technique of algorithm unrolling to yield real-time computation, feedforward inference, and strong generalization in comparison with traditional deep learning methods. We benchmark on the RaDICaL dataset to demonstrate both quantitative improvements of incorporating radar data and qualitative improvements that confirm robustness to common failure modes of image-based methods.

Quantitative identification of the transitions between anaesthetic states is very essential for optimizing patient safety and quality care during surgery but poses a very challenging task. The state-of-the-art monitors are still not capable of providing their manifest variables, so the practitioners must diagnose them based on their own experience. The present paper proposes a novel real-time method to identify these transitions. Firstly, the Hurst method is used to pre-process the de-noised electro-encephalograph (EEG) signals. The maximum of Hurst's ranges is then accepted as the EEG real-time response, which induces a new real-time feature under moving average framework. Its maximum power spectral density is found to be very differentiated into the distinct transitions of anaesthetic states and thus can be used as the quantitative index for their identification.

Generalized nucleus segmentation techniques can contribute greatly to reducing the time to develop and validate visual biomarkers for new digital pathology datasets. We summarize the results of MoNuSeg 2018 Challenge whose objective was to develop generalizable nuclei segmentation techniques in digital pathology. The challenge was an official satellite event of the MICCAI 2018 conference in which 32 teams with more than 80 participants from geographically diverse institutes participated. Contestants were given a training set with 30 images from seven organs with annotations of 21,623 individual nuclei. A test dataset with 14 images taken from seven organs, including two organs that did not appear in the training set was released without annotations. Entries were evaluated based on average aggregated Jaccard index (AJI) on the test set to prioritize accurate instance segmentation as opposed to mere semantic segmentation. More than half the teams that completed the challenge outperformed a previous baseline. Among the trends observed that contributed to increased accuracy were the use of color normalization as well as heavy data augmentation. Additionally, fully convolutional networks inspired by variants of U-Net, FCN, and Mask-RCNN were popularly used, typically based on ResNet or VGG base architectures. Watershed segmentation on predicted semantic segmentation maps was a popular post-processing strategy. Several of the top techniques compared favorably to an individual human annotator and can be used with confidence for nuclear morphometrics.

The objective of this article is to demonstrate the feasibility of estimating the backscatter coefficient (BSC) using an in situ calibration source. Traditional methods of estimating the BSC in vivo using a reference phantom technique do not account for transmission losses due to intervening layers between the ultrasonic source and the tissue region to be interrogated, leading to increases in bias and variance of BSC-based estimates. To account for transmission losses, an in situ calibration approach is proposed. The in situ calibration technique employs a titanium sphere that is well-characterized ultrasonically, biocompatible, and embedded inside the sample. A set of experiments was conducted to evaluate the embedded titanium spheres as in situ calibration targets for BSC estimation. The first experiment quantified the backscattered signal strength from titanium spheres of three sizes: 0.5, 1, and 2 mm in diameter. The second set of experiments assessed the repeatability of BSC estimates from the titanium spheres and compared these BSCs to theory. The third set of experiments quantified the ability of the titanium bead to provide an in situ reference spectrum in the presence of a lossy layer on top of the sample. The final set of experiments quantified the ability of the bead to provide a calibration spectrum over multiple depths in the sample. All experiments were conducted using an L9-4/38 linear array connected to a SonixOne system. The strongest signal was observed from the 2-mm titanium bead with the signal-to-noise ratio (SNR) of 11.6 dB with respect to the background speckle. Using an analysis bandwidth of 2.5-5.5 MHz, the mean differences between the experimentally derived BSCs and BSCs derived from the Faran theory were 0.54 and 0.76 dB using the array and a single-element transducer, respectively. The BSCs estimated using the in situ calibration approach without the layer and with the layer and using the reference phantom approach with the layer were compared to the reference phantom approach without the layer present. The mean differences in BSCs were 0.15, 0.73, and -9.69 dB, respectively. The mean differences of the BSCs calculated from data blocks located at depths that were either 30 pulse lengths above or below the actual bead depth compared to the BSC calculated at bead depth were -1.55 and -1.48 dB, respectively. The results indicate that an in situ calibration target can account for overlaying tissue losses, thereby improving the robustness of BSC-based estimates.

In an increasing number of applications of focused ultrasound (FUS) therapy, such as opening of the blood-brain barrier or collapsing microbubbles in a tumor, elevation of tissue temperature is not involved. In these cases, real-time visualization of the field distribution of the FUS source would allow localization of the FUS beam within the targeted tissue and allow repositioning of the FUS beam during tissue motion. In this paper, in order to visualize the FUS beam in situ, a 6-MHz single-element transducer ( f /2) was used as the FUS source and aligned perpendicular to a linear array which passively received scattered ultrasound from the sample. An image of the reconstructed intensity field pattern of the FUS source using bistatic beamforming was then superimposed on a registered B-mode image of the sample acquired using the same linear array. The superimposed image is used to provide anatomical context of the FUS beam in the sample being treated. The intensity field pattern reconstructed from a homogeneous scattering phantom was compared with the field characteristics of the FUS source characterized by the wire technique. The beamwidth estimates at the FUS focus using the in situ reconstruction technique and the wire technique were 1.5 and 1.2 mm, respectively. The depth-of-field estimates for the in situ reconstruction technique and the wire technique were 11.8 and 16.8 mm, respectively. The FUS beams were also visualized in a two-layer phantom and a chicken breast. The novel reconstruction technique was able to accurately visualize the field of an FUS source in the context of the interrogated medium.

Most variational formulations for structure-texture image decomposition force structure images to have small norm in some functional spaces, and share a common notion of edges, i.e., large-gradients or -intensity differences. However, such definition makes it difficult to distinguish structure edges from oscillations that have fine spatial scale but high contrast. In this paper, we introduce a new model by learning deep variational prior for structure images without explicit training data. An alternating direction method of multiplier (ADMM) algorithm and its modular structure are adopted to plug deep variational priors into an iterative smoothing process. The central observations are that convolution neural networks (CNNs) can replace the total variation prior, and are indeed powerful to capture the natures of structure and texture. We show that our learned priors using CNNs successfully differentiate highamplitude details from structure edges, and avoid halo artifacts. Different from previous data-driven smoothing schemes, our formulation provides another degree of freedom to produce continuous smoothing effects. Experimental results demonstrate the effectiveness of our approach on various computational photography and image processing applications, including texture removal, detail manipulation, HDR tone-mapping, and nonphotorealistic abstraction.

We propose a novel relighting approach that takes advantage of multiple color plus depth images acquired from a consumer camera. Assuming distant illumination and Lambertian reflectance, we model the reflected light field in terms of spherical harmonic coefficients of the bi-directional reflectance distribution function and lighting. We make use of the noisy depth information together with color images taken under different illumination conditions to refine surface normals inferred from depth. We first perform refinement on the surface normals using the first order spherical harmonics. We initialize this non-linear optimization with a linear approximation to greatly reduce computation time. With surface normals refined, we formulate the recovery of albedo and lighting in a matrix factorization setting, involving second order spherical harmonics. Albedo and lighting coefficients are recovered up to a global scaling ambiguity. We demonstrate our method on both simulated and real data, and show that it can successfully recover both illumination and albedo to produce realistic relighting results.

This paper proposes a novel approach to tensor completion, which recovers missing entries of data represented by tensors. The approach is based on the tensor train (TT) rank, which is able to capture hidden information from tensors thanks to its definition from a well-balanced matricization scheme. Accordingly, new optimization formulations for tensor completion are proposed as well as two new algorithms for their solution. The first one called simple low-rank tensor completion via TT (SiLRTC-TT) is intimately related to minimizing a nuclear norm based on TT rank. The second one is from a multilinear matrix factorization model to approximate the TT rank of a tensor, and is called tensor completion by parallel matrix factorization via TT (TMac-TT). A tensor augmentation scheme of transforming a low-order tensor to higher orders is also proposed to enhance the effectiveness of SiLRTC-TT and TMac-TT. Simulation results for color image and video recovery show the clear advantage of our method over all other methods.

We focus on the problem of still image-based human action recognition, which essentially involves making prediction by analyzing human poses and their interaction with objects in the scene. Besides image-level action labels (e.g., riding, phoning), during both training and testing stages, existing works usually require additional input of human bounding boxes to facilitate the characterization of the underlying human-object interactions. We argue that this additional input requirement might severely discourage potential applications and is not very necessary. To this end, a systematic approach was developed in this paper to address this challenging problem of minimum annotation efforts, i.e., to perform recognition in the presence of only image-level action labels in the training stage. Experimental results on three benchmark data sets demonstrate that compared with the state-of-the-art methods that have privileged access to additional human bounding-box annotations, our approach achieves comparable or even superior recognition accuracy using only action annotations in training. Interestingly, as a by-product in many cases, our approach is able to segment out the precise regions of underlying human-object interactions.

In this paper, we propose a fine-grained image categorization system with easy deployment. We do not use any object/part annotation (weakly supervised) in the training or in the testing stage, but only class labels for training images. Fine-grained image categorization aims to classify objects with only subtle distinctions (e.g., two breeds of dogs that look alike). Most existing works heavily rely on object/part detectors to build the correspondence between object parts, which require accurate object or object part annotations at least for training images. The need for expensive object annotations prevents the wide usage of these methods. Instead, we propose to generate multi-scale part proposals from object proposals, select useful part proposals, and use them to compute a global image representation for categorization. This is specially designed for the weakly supervised fine-grained categorization task, because useful parts have been shown to play a critical role in existing annotation-dependent works, but accurate part detectors are hard to acquire. With the proposed image representation, we can further detect and visualize the key (most discriminative) parts in objects of different classes. In the experiments, the proposed weakly supervised method achieves comparable or better accuracy than the state-of-the-art weakly supervised methods and most existing annotation-dependent methods on three challenging datasets. Its success suggests that it is not always necessary to learn expensive object/part detectors in fine-grained image categorization.

Discovering common visual patterns (CVPs) from two images is a challenging task due to the geometric and photometric deformations as well as noises and clutters. The problem is generally boiled down to recovering correspondences of local invariant features, and the conventionally addressed by graph-based quadratic optimization approaches, which often suffer from high computational cost. In this paper, we propose an efficient approach by viewing the problem from a novel perspective. In particular, we consider each CVP as a common object in two images with a group of coherently deformed local regions. A geometric space with matrix Lie group structure is constructed by stacking up transformations estimated from initially appearance-matched local interest region pairs. This is followed by a mean shift clustering stage to group together those close transformations in the space. Joining regions associated with transformations of the same group together within each input image forms two large regions sharing similar geometric configuration, which naturally leads to a CVP. To account for the non-Euclidean nature of the matrix Lie group, mean shift vectors are derived in the corresponding Lie algebra vector space with a newly provided effective distance measure. Extensive experiments on single and multiple common object discovery tasks as well as near-duplicate image retrieval verify the robustness and efficiency of the proposed approach.

We propose a vector space approach for inverse rendering of a Lambertian convex object with distant light sources. In this problem, the texture of the object and arbitrary lightings are both to be recovered from multiple images of the object and its 3D model. Our work is motivated by the observation that all possible images of a Lambertian object lie around a low-dimensional linear subspace spanned by the first few spherical harmonics. The inverse rendering can therefore be formulated as a matrix factorization, in which the basis of the subspace is encoded in a spherical harmonic matrix S associated with the object’s geometry. A necessary and sufficient condition on S for unique factorization is derived with an introduction to a new notion of matrix rank called nonseparable full rank. A singular value decomposition-based algorithm for exact factorization in the noiseless case is introduced. In the presence of noise, two algorithms, namely, alternating and optimization based are proposed to deal with two different types of noise. A random sample consensus-based algorithm is introduced to reduce the size of the optimization problem, which is equal to the number of pixels in each image. Implementations of the proposed algorithms are done on a real data set.

This paper presents an efficient technique for performing a spatially inhomogeneous edge-preserving image smoothing, called fast global smoother. Focusing on sparse Laplacian matrices consisting of a data term and a prior term (typically defined using four or eight neighbors for 2D image), our approach efficiently solves such global objective functions. In particular, we approximate the solution of the memory-and computation-intensive large linear system, defined over a d-dimensional spatial domain, by solving a sequence of 1D subsystems. Our separable implementation enables applying a linear-time tridiagonal matrix algorithm to solve d three-point Laplacian matrices iteratively. Our approach combines the best of two paradigms, i.e., efficient edge-preserving filters and optimization-based smoothing. Our method has a comparable runtime to the fast edge-preserving filters, but its global optimization formulation overcomes many limitations of the local filtering approaches. Our method also achieves high-quality results as the state-of-the-art optimization-based techniques, but runs ∼10-30 times faster. Besides, considering the flexibility in defining an objective function, we further propose generalized fast algorithms that perform Lγ norm smoothing (0 < γ < 2) and support an aggregated (robust) data term for handling imprecise data constraints. We demonstrate the effectiveness and efficiency of our techniques in a range of image processing and computer graphics applications.

Estimating dense correspondence or depth information from a pair of stereoscopic images is a fundamental problem in computer vision, which finds a range of important applications. Despite intensive past research efforts in this topic, it still remains challenging to recover the depth information both reliably and efficiently, especially when the input images contain weakly textured regions or are captured under uncontrolled, real-life conditions. Striking a desired balance between computational efficiency and estimation quality, a hybrid minimum spanning tree-based stereo matching method is proposed in this paper. Our method performs efficient nonlocal cost aggregation at pixel-level and region-level, and then adaptively fuses the resulting costs together to leverage their respective strength in handling large textureless regions and fine depth discontinuities. Experiments on the standard Middlebury stereo benchmark show that the proposed stereo method outperforms all prior local and nonlocal aggregation-based methods, achieving particularly noticeable improvements for low texture regions. To further demonstrate the effectiveness of the proposed stereo method, also motivated by the increasing desire to generate expressive depth-induced photo effects, this paper is tasked next to address the emerging application of interactive depth-of-field rendering given a real-world stereo image pair. To this end, we propose an accurate thin-lens model for synthetic depth-of-field rendering, which considers the user-stroke placement and camera-specific parameters and performs the pixel-adapted Gaussian blurring in a principled way. Taking ~1.5 s to process a pair of 640×360 images in the off-line step, our system named Scribble2focus allows users to interactively select in-focus regions by simple strokes using the touch screen and returns the synthetically refocused images instantly to the user.

In this paper, a probability-based rendering (PBR) method is described for reconstructing an intermediate view with a steady-state matching probability (SSMP) density function. Conventionally, given multiple reference images, the intermediate view is synthesized via the depth image-based rendering technique in which geometric information (e.g., depth) is explicitly leveraged, thus leading to serious rendering artifacts on the synthesized view even with small depth errors. We address this problem by formulating the rendering process as an image fusion in which the textures of all probable matching points are adaptively blended with the SSMP representing the likelihood that points among the input reference images are matched. The PBR hence becomes more robust against depth estimation errors than existing view synthesis approaches. The MP in the steady-state, SSMP, is inferred for each pixel via the random walk with restart (RWR). The RWR always guarantees visually consistent MP, as opposed to conventional optimization schemes (e.g., diffusion or filtering-based approaches), the accuracy of which heavily depends on parameters used. Experimental results demonstrate the superiority of the PBR over the existing view synthesis approaches both qualitatively and quantitatively. Especially, the PBR is effective in suppressing flicker artifacts of virtual video rendering although no temporal aspect is considered. Moreover, it is shown that the depth map itself calculated from our RWR-based method (by simply choosing the most probable matching point) is also comparable with that of the state-of-the-art local stereo matching methods.

This paper addresses the denoising problem associated with magnetic resonance spectroscopic imaging (MRSI), where signal-to-noise ratio (SNR) has been a critical problem. A new scheme is proposed, which exploits two low-rank structures that exist in MRSI data, one due to partial separability and the other due to linear predictability. Denoising is performed by arranging the measured data in appropriate matrix forms (i.e., Casorati and Hankel) and applying low-rank approximations by singular value decomposition (SVD). The proposed method has been validated using simulated and experimental data, producing encouraging results. Specifically, the method can effectively denoise MRSI data in a wide range of SNR values while preserving spatial-spectral features. The method could prove useful for denoising MRSI data and other spatial-spectral and spatial-temporal imaging data as well.

This paper presents a novel approach for depth video enhancement. Given a high-resolution color video and its corresponding low-quality depth video, we improve the quality of the depth video by increasing its resolution and suppressing noise. For that, a weighted mode filtering method is proposed based on a joint histogram. When the histogram is generated, the weight based on color similarity between reference and neighboring pixels on the color image is computed and then used for counting each bin on the joint histogram of the depth map. A final solution is determined by seeking a global mode on the histogram. We show that the proposed method provides the optimal solution with respect to L(1) norm minimization. For temporally consistent estimate on depth video, we extend this method into temporally neighboring frames. Simple optical flow estimation and patch similarity measure are used for obtaining the high-quality depth video in an efficient manner. Experimental results show that the proposed method has outstanding performance and is very efficient, compared with existing methods. We also show that the temporally consistent enhancement of depth video addresses a flickering problem and improves the accuracy of depth video.

The plenoptic function (POF) provides a powerful conceptual tool for describing a number of problems in image/video processing, vision, and graphics. For example, image-based rendering is shown as sampling and interpolation of the POF. In such applications, it is important to characterize the bandwidth of the POF. We study a simple but representative model of the scene where band-limited signals (e.g., texture images) are "painted" on smooth surfaces (e.g., of objects or walls). We show that, in general, the POF is not band limited unless the surfaces are flat. We then derive simple rules to estimate the essential bandwidth of the POF for this model. Our analysis reveals that, in addition to the maximum and minimum depths and the maximum frequency of painted signals, the bandwidth of the POF also depends on the maximum surface slope. With a unifying formalism based on multidimensional signal processing, we can verify several key results in POF processing, such as induced filtering in space and depth-corrected interpolation, and quantify the necessary sampling rates., (© 2011 IEEE)

Three-dimensional impedance maps (3DZMs) are virtual volumes of acoustic impedance values constructed from histology to represent tissue microstructure acoustically. From the 3DZM, the ultrasonic backscattered power spectrum can be predicted and model based scatterer properties, such as effective scatterer diameter (ESD), can be estimated. Additionally, the 3DZM can be exploited to visualize and identify possible scattering sites, which may aid in the development of more effective scattering models to better represent the ultrasonic interaction with underlying tissue microstructure. In this study, 3DZMs were created from a set of human fibroadenoma samples. ESD estimates were made assuming a fluid-filled sphere form factor model from 3DZMs of volume 300×300×300 μm. For a collection of 33 independent human fibroadenoma tissue samples, the ESD was estimated to be 111±40.7 μm. The 3DZMs were then investigated visually to identify possible scattering sources which conformed to the estimated model scatterer dimensions. This estimation technique allowed a better understanding of the spatial distribution and variability of the estimates throughout the volume.

We study the theory and algorithms of an optimal use of multidimensional signal reconstruction from multichannel acquisition by using a filter bank setup. Suppose that we have an N-channel convolution system, referred to as N analysis filters, in M dimensions. Instead of taking all the data and applying multichannel deconvolution, we first reduce the collected data set by an integer M×M uniform sampling matrix [Formula: see text], and then search for a synthesis polyphase matrix which could perfectly reconstruct any input discrete signal. First, we determine the existence of perfect reconstruction (PR) systems for a given set of finite-impulse response (FIR) analysis filters. Second, we present an efficient algorithm to find a sampling matrix with maximum sampling rate and to find a FIR PR synthesis polyphase matrix for a given set of FIR analysis filters. Finally, once a particular FIR PR synthesis polyphase matrix is found, we can characterize all FIR PR synthesis matrices, and then find an optimal one according to design criteria including robust reconstruction in the presence of noise.

We present a new noniterative approach to synthetic aperture radar (SAR) autofocus, termed the multichannel autofocus (MCA) algorithm. The key in the approach is to exploit the multichannel redundancy of the defocusing operation to create a linear subspace, where the unknown perfectly focused image resides, expressed in terms of a known basis formed from the given defocused image. A unique solution for the perfectly focused image is then directly determined through a linear algebraic formulation by invoking an additional image support condition. The MCA approach is found to be computationally efficient and robust and does not require prior assumptions about the SAR scene used in existing methods. In addition, the vector-space formulation of MCA allows sharpness metric optimization to be easily incorporated within the restoration framework as a regularization term. We present experimental results characterizing the performance of MCA in comparison with conventional autofocus methods and discuss the practical implementation of the technique.

We propose a new approach to quantitatively analyze the rendering quality of image-based rendering (IBR) algorithms with depth information. The resulting error bounds for synthesized views depend on IBR configurations including the depth and intensity estimate errors, the scene geometry and texture, the number of actual cameras, their positions and resolution. Specifically, the IBR error is bounded by the summation of three terms, highlighting the impact of using multiple actual cameras, the impact of the noise level at the actual cameras, and the impact of the depth accuracy. We also quantify the impact of occlusions and intensity discontinuities. The proposed methodology is applicable to a large class of common IBR algorithms and can be applied locally. Experiments with synthetic and real scenes show that the developed error bounds accurately characterize the rendering errors. In particular, the error bounds correctly characterize the decay rates of synthesized views' mean absolute errors as O(lambda(-1)) and O(lambda(-2)), where lambda is the local density of actual samples, for 2-D and 3-D scenes, respectively. Finally, we discuss the implications of the proposed analysis on camera placement, budget allocation, and bit allocation.

Magnetic resonance imaging (MRI) uses applied spatial variations in the magnetic field to encode spatial position. Therefore, nonuniformities in the main magnetic field can cause image distortions. In order to correct the image distortions, it is desirable to simultaneously acquire data with a field map in registration. We propose a joint estimation (JE) framework with a fast, noniterative approach using harmonic retrieval (HR) methods, combined with a multi-echo echo-planar imaging (EPI) acquisition. The connection with HR establishes an elegant framework to solve the JE problem through a sequence of 1-D HR problems in which efficient solutions are available. We also derive the condition on the smoothness of the field map in order for HR techniques to recover the image with high signal-to-noise ratio. Compared to other dynamic field mapping methods, this method is not constrained by the absolute level of the field inhomogeneity over the slice, but relies on a generous pixel-to-pixel smoothness. Moreover, this method can recover image, field map, and T2* map simultaneously.

We propose a wavelet-based codec for the static depth-image-based representation, which allows viewers to freely choose the viewpoint. The proposed codec jointly estimates and encodes the unknown depth map from multiple views using a novel rate-distortion (RD) optimization scheme. The rate constraint reduces the ambiguity of depth estimation by favoring piecewise-smooth depth maps. The optimization is efficiently solved by a novel dynamic programming along trees of integer wavelet coefficients. The codec encodes the image and the depth map jointly to decrease their redundancy and to provide a RD-optimized bitrate allocation between the two. The codec also offers scalability both in resolution and in quality. Experiments on real data show the effectiveness of the proposed codec.

A significant problem in interventional magnetic resonance imaging is limited imaging speed. This paper addresses this problem using a new signal model known as union-of-subspaces. This model enables an effective use of sparse sampling and prior information to significantly improve the imaging speed. The proposed method has been validated using simulations on real interventional imaging data, and is shown to provide high speed and quality imaging. The approach is very flexible and can be applied to other imaging applications as well.

Synthetic aperture radar (SAR) autofocus techniques that optimize sharpness metrics can produce excellent restorations in comparison with conventional autofocus approaches. To help formalize the understanding of metric-based SAR autofocus methods, and to gain more insight into their performance, we present a theoretical analysis of these techniques using simple image models. Specifically, we consider the intensity-squared metric, and a dominant point-targets image model, and derive expressions for the resulting objective function. We examine the conditions under which the perfectly focused image models correspond to stationary points of the objective function. A key contribution is that we demonstrate formally, for the specific case of intensity-squared minimization autofocus, the mechanism by which metric-based methods utilize the multichannel defocusing model of SAR autofocus to enforce the stationary point property for multiple image columns. Furthermore, our analysis shows that the objective function has a special separble property through which it can be well approximated locally by a sum of 1-D functions of each phase error component. This allows fast performance through solving a sequence of 1-D optimization problems for each phase component simultaneously. Simulation results using the proposed models and actual SAR imagery confirm that the analysis extends well to realistic situations.

The contourlet transform was proposed to address the limited directional resolution of the separable wavelet transform. One way to guarantee good approximation behavior is to let the directional filters in the contourlet filter bank have sharp frequency response. This requires filters with large support size. We seek to isolate the key filter property that ensures good approximation. In this direction, we propose filters with directional vanishing moments (DVM). These filters, we show, annihilate information along a given direction. We study two-channel filter banks with DVM filters. We provide conditions under which the design of DVM filter banks is possible. A complete characterization of the product filter is, thus, obtained. We propose a design framework that avoids 2-D factorization using the mapping technique. The filters designed, when used in the contourlet transform, exhibit nonlinear approximation comparable to the conventional filters while being shorter and, therefore, providing better visual quality with less ringing artifacts. Furthermore, experiments show that the proposed filters outperform the conventional ones in image approximation and denoising.

In 1992, Bamberger and Smith proposed the directional filter bank (DFB) for an efficient directional decomposition of 2-D signals. Due to the nonseparable nature of the system, extending the DFB to higher dimensions while still retaining its attractive features is a challenging and previously unsolved problem. We propose a new family of filter banks, named NDFB, that can achieve the directional decomposition of arbitrary N-dimensional (N > or =2) signals with a simple and efficient tree-structured construction. In 3-D, the ideal passbands of the proposed NDFB are rectangular-based pyramids radiating out from the origin at different orientations and tiling the entire frequency space. The proposed NDFB achieves perfect reconstruction via an iterated filter bank with a redundancy factor of N in N-D. The angular resolution of the proposed NDFB can be iteratively refined by invoking more levels of decomposition through a simple expansion rule. By combining the NDFB with a new multiscale pyramid, we propose the surfacelet transform, which can be used to efficiently capture and represent surface-like singularities in multidimensional data.

We present a new method for general multidimensional multichannel deconvolution with finite impulse response (FIR) convolution and deconvolution filters using Gröbner bases. Previous work formulates the problem of multichannel FIR deconvolution as the construction of a left inverse of the convolution matrix, which is solved by numerical linear algebra. However, this approach requires the prior information of the support of deconvolution filters. Using algebraic geometry and Gröbner bases, we find necessary and sufficient conditions for the existence of exact deconvolution FIR filters and propose simple algorithms to find these deconvolution filters. The main contribution of our work is to extend the previous Gröbner basis results on multidimensional multichannel deconvolution for polynomial or causal filters to general FIR filters. The proposed algorithms obtain a set of FIR deconvolution filters with a small number of nonzero coefficients (a desirable feature in the impulsive noise environment) and do not require the prior information of the support. Moreover, we provide a complete characterization of all exact deconvolution FIR filters, from which good FIR deconvolution filters under the additive white noise environment are found. Simulation results show that our approaches achieve good results under different noise settings.

In this paper, we develop the nonsubsampled contourlet transform (NSCT) and study its applications. The construction proposed in this paper is based on a nonsubsampled pyramid structure and nonsubsampled directional filter banks. The result is a flexible multiscale, multidirection, and shift-invariant image decomposition that can be efficiently implemented via the à trous algorithm. At the core of the proposed scheme is the nonseparable two-channel nonsubsampled filter bank (NSFB). We exploit the less stringent design condition of the NSFB to design filters that lead to a NSCT with better frequency selectivity and regularity when compared to the contourlet transform. We propose a design framework based on the mapping approach, that allows for a fast implementation based on a lifting or ladder structure, and only uses one-dimensional filtering in some cases. In addition, our design ensures that the corresponding frame elements are regular, symmetric, and the frame is close to a tight one. We assess the performance of the NSCT in image denoising and enhancement applications. In both applications the NSCT compares favorably to other existing methods in the literature.

We address the best basis problem--or, more generally, the best representation problem: Given a signal, a dictionary of representations, and an additive cost function, the aim is to select the representation from the dictionary which minimizes the cost for the given signal. We develop a new framework of multitree dictionaries, which includes some previously proposed dictionaries as special cases. We show how to efficiently find the best representation in a multitree dictionary using a recursive tree-pruning algorithm. We illustrate our framework through several examples, including a novel block image coder, which significantly outperforms both the standard JPEG and quadtree-based methods and is comparable to embedded coders such as JPEG2000 and SPIHT.

The contourlet transform is a new two-dimensional extension of the wavelet transform using multiscale and directional filter banks. The contourlet expansion is composed of basis images oriented at various directions in multiple scales, with flexible aspect ratios. Given this rich set of basis images, the contourlet transform effectively captures smooth contours that are the dominant feature in natural images. We begin with a detailed study on the statistics of the contourlet coefficients of natural images: using histograms to estimate the marginal and joint distributions and mutual information to measure the dependencies between coefficients. This study reveals the highly non-Gaussian marginal statistics and strong interlocation, interscale, and interdirection dependencies of contourlet coefficients. We also find that conditioned on the magnitudes of their generalized neighborhood coefficients, contourlet coefficients can be approximately modeled as Gaussian random variables. Based on these findings, we model contourlet coefficients using a hidden Markov tree (HMT) model with Gaussian mixtures that can capture all interscale, interdirection, and interlocation dependencies. We present experimental results using this model in image denoising and texture retrieval applications. In denoising, the contourlet HMT outperforms other wavelet methods in terms of visual quality, especially around edges. In texture retrieval, it shows improvements in performance for various oriented textures.

We characterize and design multidimensional (MD) orthogonal filter banks using special paraunitary matrices and the Cayley transform. Orthogonal filter banks are represented by paraunitary matrices in the polyphase domain. We define special paraunitary matrices as paraunitary matrices with unit determinant. We show that every paraunitary matrix can be characterized by a special paraunitary matrix and a phase factor. Therefore, the design of paraunitary matrices (and thus of orthogonal filter banks) becomes the design of special paraunitary matrices, which requires a smaller set of nonlinear equations. Moreover, we provide a complete characterization of special paraunitary matrices in the Cayley domain, which converts nonlinear constraints into linear constraints. Our method greatly simplifies the design of MD orthogonal filter banks and leads to complete characterizations of such filter banks.

The limitations of commonly used separable extensions of one-dimensional transforms, such as the Fourier and wavelet transforms, in capturing the geometry of image edges are well known. In this paper, we pursue a "true" two-dimensional transform that can capture the intrinsic geometrical structure that is key in visual information. The main challenge in exploring geometry in images comes from the discrete nature of the data. Thus, unlike other approaches, such as curvelets, that first develop a transform in the continuous domain and then discretize for sampled data, our approach starts with a discrete-domain construction and then studies its convergence to an expansion in the continuous domain. Specifically, we construct a discrete-domain multiresolution and multidirection expansion using nonseparable filter banks, in much the same way that wavelets were derived from filter banks. This construction results in a flexible multiresolution, local, and directional image expansion using contour segments, and, thus, it is named the contourlet transform. The discrete contourlet transform has a fast iterated filter bank algorithm that requires an order N operations for N-pixel images. Furthermore, we establish a precise link between the developed filter bank and the associated continuous-domain contourlet expansion via a directional multiresolution analysis framework. We show that with parabolic scaling and sufficient directional vanishing moments, contourlets achieve the optimal approximation rate for piecewise smooth functions with discontinuities along twice continuously differentiable curves. Finally, we show some numerical experiments demonstrating the potential of contourlets in several image processing applications. Index Terms-Contourlets, contours, filter banks, geometric image processing, multidirection, multiresolution, sparse representation, wavelets.

We present a complete characterization and design of orthogonal infinite impulse response (IIR) and finite impulse response (FIR) filter banks in any dimension using the Cayley transform (CT). Traditional design methods for one-dimensional orthogonal filter banks cannot be extended to higher dimensions directly due to the lack of a multidimensional (MD) spectral factorization theorem. In the polyphase domain, orthogonal filter banks are equivalent to paraunitary matrices and lead to solving a set of nonlinear equations. The CT establishes a one-to-one mapping between paraunitary matrices and para-skew-Hermitian matrices. In contrast to the paraunitary condition, the para-skew-Hermitian condition amounts to linear constraints on the matrix entries which are much easier to solve. Based on this characterization, we propose efficient methods to design MD orthogonal filter banks and present new design results for both IIR and FIR cases.

This paper presents novel coding algorithms based on tree-structured segmentation, which achieve the correct asymptotic rate-distortion (R-D) behavior for a simple class of signals, known as piecewise polynomials, by using an R-D based prune and join scheme. For the one-dimensional case, our scheme is based on binary-tree segmentation of the signal. This scheme approximates the signal segments using polynomial models and utilizes an R-D optimal bit allocation strategy among the different signal segments. The scheme further encodes similar neighbors jointly to achieve the correct exponentially decaying R-D behavior (D(R) - c(o)2(-c1R)), thus improving over classic wavelet schemes. We also prove that the computational complexity of the scheme is of O(N log N). We then show the extension of this scheme to the two-dimensional case using a quadtree. This quadtree-coding scheme also achieves an exponentially decaying R-D behavior, for the polygonal image model composed of a white polygon-shaped object against a uniform black background, with low computational cost of O(N log N). Again, the key is an R-D optimized prune and join strategy. Finally, we conclude with numerical results, which show that the proposed quadtree-coding scheme outperforms JPEG2000 by about 1 dB for real images, like cameraman, at low rates of around 0.15 bpp.

The ridgelet transform was introduced as a sparse expansion for functions on continuous spaces that are smooth away from discontinuities along lines. We propose an orthonormal version of the ridgelet transform for discrete and finite-size images. Our construction uses the finite Radon transform (FRAT) as a building block. To overcome the periodization effect of a finite transform, we introduce a novel ordering of the FRAT coefficients. We also analyze the FRAT as a frame operator and derive the exact frame bounds. The resulting finite ridgelet transform (FRIT) is invertible, nonredundant and computed via fast algorithms. Furthermore, this construction leads to a family of directional and orthonormal bases for images. Numerical results show that the FRIT is more effective than the wavelet transform in approximating and denoising images with straight edges.

We present a statistical view of the texture retrieval problem by combining the two related tasks, namely feature extraction (FE) and similarity measurement (SM), into a joint modeling and classification scheme. We show that using a consistent estimator of texture model parameters for the FE step followed by computing the Kullback-Leibler distance (KLD) between estimated models for the SM step is asymptotically optimal in term of retrieval error probability. The statistical scheme leads to a new wavelet-based texture retrieval method that is based on the accurate modeling of the marginal distribution of wavelet coefficients using generalized Gaussian density (GGD) and on the existence a closed form for the KLD between GGDs. The proposed method provides greater accuracy and flexibility in capturing texture information, while its simplified form has a close resemblance with the existing methods which uses energy distribution in the frequency domain to identify textures. Experimental results on a database of 640 texture images indicate that the new method significantly improves retrieval rates, e.g., from 65% to 77%, compared with traditional approaches, while it retains comparable levels of computational complexity.