Search

The Blind Image Deconvolution/Deblurring (BID) problem was realised in the early 1960s but it still remains a challenging task for the image processing research community to find an efficient, reliable and most importantly a diversely applicable deblurring scheme. The main challenge arises from little or no prior information about the image or the blurring process as well as the lack of optimal restoration filters to reduce or completely eliminate the blurring effect. Moreover, restoration can be marred by the two common side effects of deblurring; namely the noise amplification and ringing artefacts that arise in the deblurred image due to an unrealizable or imperfect restoration filter. Also, developing a scheme that can process different types of blur, especially for real images, is yet to be realized to a satisfactory level. This research is focused on the development of blind restoration schemes for real life blurred images. The primary objective is to design a BID scheme that is robust in term of Point Spread Function (PSF) estimation, efficient in terms of restoration speed, and effective in terms of restoration quality. A desired scheme will require a deblurring measure to act as a feedback of quality regarding the deblurred image and lead the estimation of the blurring PSF. The blurred image and the estimated PSF can then be passed on to any classical restoration filter for deblurring. The deblurring measures presented in this research include blind non-Gaussianity measures as well as blind Image Quality Measures (IQMs). These measures are blind in the sense that they are able to gauge the quality of an image directly from it without the need to reference a high quality image. The non-Gaussianity measures include spatial and spectral kurtosis measures; while the image quality analysers include the Blind/Reference-less Image Spatial QUality Evaluator (BRISQUE), Natural Image Quality Evaluator (NIQE) index and Reblurring based Peak Signal to Noise Ratio (RPSNR) measure. BRISQUE, NIQE and spectral kurtosis, are introduced for the first time as deblurring measures for BID. RPSNR is a novel full reference yet blind IQM designed and used in this research work. Experiments were conducted on different image datasets and real life blurred images. Optimization of the BID schemes has been achieved using a gradient descent based scheme and a Genetic Algorithm (GA). Quantitative results based on full-reference and non-reference IQMs, present BRISQUE as a robust and computationally efficient blind feedback quality measure. Also, parametric and arbitrarily shaped (non-parametric or generic) PSFs were treated for the blind deconvolution of images. The parametric forms of PSF include uniform Gaussian, motion and out-of-focus blur. The arbitrarily shaped PSFs comprise blurs that have a much more complex blur shape which cannot be easily modelled in the parametric form. A novel scheme for arbitrarily shaped PSF estimation and blind deblurring has been designed, implemented and tested on artificial and real life blurred images. The scheme provides a unified base for the estimation of both parametric and arbitrarily shaped PSFs with the BRISQUE quality measure in conjunction with a GA. Full-reference and non-reference IQMs have been utilised to gauge the quality of deblurred images for the BID schemes. In the real BID case, only non-reference IQMs can be employed due to the unavailability of the reference high quality image. Quantitative results of these images depict the restoration ability of the BID scheme. The significance of the research work lies in the BID scheme‘s ability to handle parametric and arbitrarily shaped PSFs using a single algorithm, for single-shot blurred images, with enhanced optimization through the gradient descent scheme and GA in conjunction with multiple feedback IQMs.

Image Deconvolution is a well-studied problem that seeks to restore the original sharp image from a blurry image formed in the imaging system. The Point Spread function(PSF) of a particular system can be used to infer the original sharp image given the blurred image. However, such a problem is usually simplified by making the shift-invariant assumption over the Field of View (FOV). Realistic systems are shift-variant; the optical system’s point spread function depends on the position of the object point from the principal axis. For example, asymmetrical lenses can cause space variant aberration. In this paper, we first simulate our shift-variant aberrations by generating Point Spread Functions using the Seidel Aberration polynomial and use a shift-variant forward blur model to generate our shift-variant blurred image pairs. We then introduce, ShiVaNet. It is a two-stage architecture that builds upon the Learnable Wiener Deconvolution block as described in Yanny, Monakhova, Shuai, and Waller (Yanny et al.) by introducing Simplified Channel Attention and Transpose Attention to improve the performance of the module. We also devise a novel UNet refinement block by fusing a ConvNext-V2 block with Channel Attention and coupling with Transposed Attention Zamir, Arora, Khan, Hayat, Khan, and Yang (Zamir et al.). Our model performs better than state-of-the-art restoration models by a factor of 0.2 dB Peak Signal to Noise Ratio.

Image deblurring tries to eliminate degradation elements of an image causing blurriness and improve the quality of an image for better texture and object visualization. Blind image deblurring is one type of image deblurring that occurs when there is no information about blur effect. Such a deblurring process handles an ill-posed problem, so it requires imposing restrictions on a latent image or a blur kernel that represents blurriness. Various algorithms have been proposed to restore a latent image out of a blurred one, including prior-based optimization methods and, most recently, deep neural networks. Different from recent studies in prior-based optimization methods that impose some priors on the latent image, we regulate the structure of the blur kernel and demonstrate the benefits of imposing a structure on kernels. For modeling a kernel, we propose a kernel mixture structure with the Gaussian kernel as a base kernel. Albeit simple, by combining multiple Gaussian kernels with enhanced structures in terms of scales and centers, the kernel mixture becomes capable of modeling nearly non-parametric shape of blurriness. Recently, deep neural networks brought a major breakthrough in the image deblurring domain. Hence, we comprehensively review the recent progress of deep neural architectures in both blind and non-blind image deblurring. We outline the most popular deep neural network structures used in deblurring applications, describe their strengths and novelties, summarize performance metrics, and introduce broadly used datasets. In addition, we discuss the current challenges and research gaps in this domain and suggest potential research directions for future works. Comprehending the current models and research gaps in deep neural image deblurring networks, we propose a novel blur extraction module (BEM) to extract blur and semantic information and improve the feature extraction procedure. Our proposed module has two sub-modules: cascaded dense dilated convolution (CDDC) block, which is proposed to capture the multi-scale semantic information, and residual integrated poolings (RIPS) block introduced to extract the dynamic blur context. We trained and evaluated the network with our proposed module based on well-known image deblurring datasets. Our results quantitatively and qualitatively show the superiority of our blur extraction module in comparison with state-of-the-art neural image deblurring networks.

This dissertation presents algorithms for restoring some of the major corruptions observed in archived film or video material. The two principal problems of impulsive distortion (Dirt and Sparkle or Blotches) and noise degradation are considered. There is also an algorithm for suppressing the inter-line jitter common in images decoded from noisy video signals. In the case of noise reduction and Blotch removal the thesis considers image sequences to be three dimensional signals involving evolution of features in time and space. This is necessary if any process presented is to show an improvement over standard two-dimensional techniques. It is important to recognize that consideration of image sequences must involve an appreciation of the problems incurred by the motion of objects in the scene. The most obvious implication is that due to motion, useful three dimensional processing does not necessarily proceed in a direction 'orthogonal' to the image frames. Therefore, attention is given to discussing motion estimation as it is used for image sequence processing. Some discussion is given to image sequence models and the 3D Autoregressive model is investigated. A multiresolution BM scheme is used for motion estimation throughout the major part of the thesis. Impulsive noise removal in image processing has been traditionally achieved by the use of median filter structures. A new three dimensional multilevel median structure is presented in this work with the additional use of a detector which limits the distortion caused by the filters . This technique is found to be extremely effective in practice and is an alternative to the traditional global median operation. The new median filter is shown to be superior to those previously presented with respect to the ability to reject the kind of distortion found in practice. A model based technique using the 3D AR model is also developed for detecting and removing Blotches. This technique achieves better fidelity at the expense of heavier computational load. Motion compensated 3D IIR and FIR Wiener filters are investigated with respect to their ability to reject noise in an image sequence. They are compared to several algorithms previously presented which are purely temporal in nature. The filters presented are found to be effective and compare favourably to the other algorithms. The 3D filtering process is superior to the purely temporal process as expected. The algorithm that is presented for suppressing inter-line jitter uses a 2D AR model to estimate and correct the relative displacements between the lines. The output image is much more satisfactory to the observer although in a severe case some drift of image features is to be expected. A suggestion for removing this drift is presented in the conclusions. There are several remaining problems in moving video. In particular, line scratches and picture shake/roll. Line scratches cannot be detected successfully by the detectors presented and so cannot be removed efficiently. Suppressing shake and roll involves compensating the entire frame for motion and there is a need to separate global from local motion. These difficulties provide ample opportunity for further research.

We present two novel multiframe image super-resolution (SR) algorithms that employconvolutional neural networks (CNNs) to generate high-resolution (HR) images from mul-tiple low-resolution (LR) input frames. The first algorithm, named Fusion of InterpolatedFrames Network (FIFNET), utilizes motion-based multiframe SR by fusing multiple inputframes in a single CNN based on Random and Fixed shifts. The second algorithm, called theExponential weighted Fusion of Interpolated Frames Network (EWF-FIFNET), presents twovariations, Externally Exponential Weighted Fusion-FIFNET (EEWF-FIFNET) and Inter-nally Exponential Weighted Fusion-FIFNET (IEWF-FIFNET) based on affine motion. Acustom layer called the Exponential Weighted Fusion (EWF) layer is developed to combineinput frames using a technique inspired by the fusion interpolation frame SR frameworkwithin the IEWF-FIFNET model. The EWF-FIFNET network utilizes a modified ResidualChannel Attention Network architecture with residual in residual (RIR) structures.The proposed algorithms are trained and tested using a realistic observation cameramodel that incorporates optical and sensor degradation. Affine motion is also incorporatedto address a challenging degradation problem. The experimental results show that theproposed algorithms outperform the existing state-of-the-art methods using both simulatedand real camera data. It is noteworthy that the real data is not artificially downsampled ordegraded, making the proposed algorithms a promising solution for practical applications.This research contributes significantly to the field of multiframe image SR, particularly inmotion-based and exponentially weighted fusion approaches using CNNs.

The challenge of restoring real world low-quality images is due to a lack of appropriate training data and difficulty in determining how the image was degraded. Recently, generative models have demonstrated great potential for creating high- quality images by utilizing the rich and diverse information contained within the model’s trained weights and learned latent representations. One popular type of generative model is the generative adversarial network (GAN). Many new methods have been developed to harness the information found in GANs for image manipulation. Our proposed approach is to utilize generative models for both understanding the degradation of an image and restoring it. We propose using a combination of cycle consistency losses and self-attention to enhance face images by first learning the degradation and then using this information to train a style-based neural network. We also aim to use the latent representation to achieve a high level of magnification for face images (x64). By incorporating the weights of a pre-trained StyleGAN into a restoration network with a vision transformer layer, we hope to improve the current state-of-the-art in face image restoration. Finally, we present a projection-based image-denoising algorithm named Noise2Code in the latent space of the VQGAN model with a fixed-point regularization strategy. The fixed-point condition follows the observation that the pre-trained VQGAN affects the clean and noisy images in a drastically different way. Unlike previous projection-based image restoration in the latent space, both the denoising network and VQGAN model parameters are jointly trained, although the latter is not needed during the testing. We report experimental results to demonstrate that the proposed Noise2Code approach is conceptually simple, computationally efficient, and generalizable to real-world degradation scenarios.

Abstract: The objective of signal decomposition is to extract and separate distinct signal components from a composite signal. Signal decomposition has been studied in many applications, such as image, video, audio, and speech signals. This thesis focuses on the category of signal decomposition on Interferometric Synthetic Aperture Radar (InSAR), a remote sensing technology that can monitor the earth from space. It provides measurements for thousands of square kilometres of ground, with a spatial resolution of around 10 m per pixel and a 1 mm precision on ground deformation estimation over time. For wide-area monitoring, algorithms must handle tens of thousands of radar satellite images annually to measure ground stability over time. This thesis' primary focus is to combine traditional signal/image processing techniques with recent deep learning approaches to improve the InSAR processing pipeline to deliver faster and better results. The task is very challenging because ground surface displacement or deformation signals are encoded in observed InSAR phase measurements with other contaminant signals (e.g., atmospheric distortion, orbital error, and digital elevation model error and noise). Each type of signal could be spatially correlated, temporally correlated, or both. It is also possible for the signals to be neither spatially nor temporally correlated. The phase values are wrapped by 2pi, which causes a non-continuous processing domain. Moreover, there is no real-world ground truth to reference in the training or validation stages. This thesis explores and addresses the deformation signal extraction problem using different strategies. We start by focusing on the image filtering problem of removing spatially independent noise components. We demonstrate a novel deep learning model for Gaussian denoising in natural images and then adapt it to data from the InSAR modality. We designed a teacher-student paradigm for supervised training in the absence of real-world ground truth data. The framework uses a standard stack-based filtering method as the "teacher" (requiring more than 30 observations) and a deep differentiable model to learn the behaviour of the teacher method. After training, the student model can produce results comparable to, or even better than, those produced by its teacher method. Moreover, the student model relies on just a single pair of observations. Additionally, the proposed model is designed to provide a coherence map, which indicates the signal quality at the pixel level. Furthermore, we present an extension in the form of a novel self-supervised framework. This framework can be used to remove noise signals and estimate pixel-level quality using only noisy observations for training and inference. In addition to the previous outcome, we investigate how to separate deformation and DEM error signals using a 2D optimization problem for each spatial location in a time series. In general, current approaches suffer from a non-continuous solution space. They are limited to small-scale displacement use cases, making them unsuitable for high-velocity scenarios such as mining, construction, and earthquakes. We propose a two-stage optimization strategy that effectively locates global optima by combining an iterative global coarse search with a stochastic derivative-free local fine search. Almost all of the research on InSAR deforming signal estimation is based solely on temporal analysis and requires pre-removal of the atmospheric phase. We further investigate the spatial-temporal cross-domain optimization by developing an adaptive kernel that performs convolutional optimization on the entire 3D InSAR stack, resulting in accurate and robust deformation and DEM error signal extraction. The approach should be capable of processing wrapped phases directly and even working on phases that have not had their atmospheric component removed. Despite these signal decomposition processes, accurately validating and optimizing the developed algorithms remains a challenge due to the lack of relevant ground truth data in a real-world environment. We developed a stochastic InSAR simulator to address this problem. The simulator provides a highly flexible modeling framework for generating various phase fringes and coherence distributions. This simulator is suitable for conducting thorough quantitative evaluations of various filtering and coherence estimation algorithms. The simulator features 2D and 3D modes that support stack and non-stack analysis. The 3D version is expected to simulate time-series deformation signals to evaluate signal separation methods. Additionally, to mimic realistic signals, we also study the intelligent generative InSAR simulator with adversarial training to learn the real-world deformation signal's distribution and its correlations to the DEM error.

Computational efficiency is an essential factor that influences the applicability of computer vision algorithms. Although deep neural networks have reached state-of-the-art performances in a variety of computer vision tasks, there are a couple of efficiency related problems of the deep learning based solutions. First, the overparameterization of deep neural networks results in models with millions of parameters, which lowers the parameter efficiency of the designed networks. To store the parameters and intermediate feature maps during the computation, a large device memory footprint is required. Secondly, the massive computation in deep neural networks slows down their training and inference. This limits the application of deep neural networks to latency-demanding scenarios and low-end devices. Thirdly, the massive computation consumes significant amount of energy, which leaves a large carbon footprint of deep learning models. The aim of this thesis is to improve the computational efficiency of current deep neural networks. This problem is tackled from three perspective including neural network compression, neural architecture optimization, and computational procedure optimization. In the first part of the thesis, we reduce the model complexity of neural networks by network compression techniques including filter decomposition and filter pruning. The basic assumption for filter decomposition is that the ensemble of filters in deep neural networks constitutes an overcomplete set. Instead of using the original filters directly during the computation, they can be approximated by a linear combination of a set of basis filters. The contribution of this thesis is to provide a unified analysis of previous filter decomposition methods. On the other hand, a differentiable filter pruning method is proposed. To achieve differentiability, the layers of neural networks is reparameterized by a meta network. Sparsity regularization is applied to the input of the meta network, i.e. latent vectors. Optimizing with the introduced regularization leads to an automatic network pruning method. Additionally, a joint analysis of filter decomposition and filter pruning is presented from the perspective of compact tensor approximation. The hinge of the two techniques is the introduced sparsity inducing matrix. By simply changing the way the group sparsity regularization is enforced to the matrix, the two techniques can be derived accordingly. Secondly, we try to improve the performance of a baseline network by a fine-grained neural architecture optimization method. Different from network compression methods, the aim of this method is to improve the prediction accuracy of neural networks while reducing their model complexity at the same time. Achieving the two targets simultaneously makes the problem more challenging. In addition, a nearly cost-free constraint is enforced during the architecture optimization, which differs from current neural architecture search methods with bulky computation. This can be regarded as another efficiency-improving technique. Thirdly, we optimize the computational procedure of graph neural networks. By mathematically analyzing the operations in graph neural network, two methods are proposed to improve the computational efficiency. The first method is related to the simplification of neighbor querying in graph neural network while the second involves shuffling the order of graph feature gathering and an feature extraction operations. To summarize, this thesis contributes to multiple aspects of improving the computational efficiency of neural networks during the optimization, training, and test phase.

Deep Learning has become one of the cornerstones of today’s AI advancement and research. Deep Learning models are used for achieving state-of-the-art results on a wide variety of tasks, including image restoration problems, specifically image denoising. Despite recent advances in applications of deep neural networks and the presence of a substantial amount of existing research work in the domain of image denoising, this task is still an open challenge. In this thesis work, we aim to summarize the study of image denoising research and its trend over the years, the fallacies, and the brilliance. We first visit the fundamental concepts of image restoration problems, their definition, and some common misconceptions. After that, we attempt to trace back where the study of image denoising began, attempt to categorize the work done till now into three main families with the main focus on the neural network family of methods, and discuss some popular ideas. Consequently, we also trace related concepts of over-parameterization, regularisation, low-rank minimization and discuss recent untrained networks approach for single image denoising, which is fundamental towards understanding why the current state-of-art methods are still not able to provide a generalized approach for stabilized image recovery from multiple perturbations.

Removing rainy artefacts (e.g., rainstreaks and raindrops) can benefit advanced computer vision systems, such as intelligent surveillance systems and self-driving cars. Data-driven models produce more promising results than model-based approaches. However, by evaluating representative deep-learning based deraining models on benchmark datasets, two unfavorable results are obtained: (1) they cannot handle various rainy conditions (i.e., rains of various directions and densities) equally well, and (2) the models trained on synthetic images cannot generalize well to process real rainy images. This Ph.D. thesis has made progress towards solving both issues. A deraining network dissection tool (DNDT) was firstly designed to analyze the behavior of filters in deraining models. Task-specific issues are then identified and solved by formulating a series of novel deraining models: (1) it was found that the imbalanced distribution (between rains and backgrounds) results in biased representation, and then a context-enhanced representation learning and deraining network (CERLD-Net) was proposed to learn an explainable and controlled representation; (2) it was found that the representations learned in existing models suffer from the loss of features depicting the background, and then an entangled representation learning network (ERL-Net) was proposed to learn more complete features; (3) it was found that some negative factors are learned in the latent space of ERL-Net, and then a disentangled representation learning and enhancement network (DRLE-Net) was proposed to embed a rainy image into two separated spaces such that only the positive factors are selected for deraining; and (4) by dissecting DRLE-Net, the significance of using rain distribution map (RDM) as prior is verified, motivating the formulation of a novel cascaded attention guidance network (CAG-Net) for better exploiting RDM in a hierarchical manner. The issues of the coarse-to-fine deraining models are also identified and solved by proposing a novel attentive feature refinement network (AFR-Net) to remove heavy rainy artefacts. In summary, this thesis investigates and solves the problems of existing deraining models by proposing novel networks, which are designed towards learning unbiased and explainable representations for reconstructing clean images without over-/under- deraining artefacts. Extensive experiments demonstrate that the proposed methods outperform the state-of-the-arts by large margins.

Abstract: Detecting and interpreting image abnormalities and restoring images are essential to many processing pipelines in diverse fields. Challenges involved include randomness and unstructured nature of image artefacts (from signal processing perspective) and performance constraints imposed by resource-constrained systems (computational perspective). This thesis studies three such scenarios involving different sensor modalities, and proposes novel approaches to address such challenges using computer vision and machine learning techniques. The three scenarios are: GPU-friendly debanding for mobile HDR, MRI abnormality detection, and InSAR signal recovery for WAM. Below, we summarize each scenario, and outline the unique challenges involved in solving them. In the first scenario, in High-Dynamic-Range (HDR) imaging domain, we propose how to perceptually eliminate quantization artefacts resulting from dynamic range conversion, without distorting image contents, by adding noise patterns in the mobile Graphics Processing Unit (GPU) computing environment with limited computational resources. Traditional filtering methods are computationally non-ideal due to limited access to neighborhood information. We introduce a pixel based solution which does not rely on any neighborhood information. We impose a real-world transmission scenario where the receiving end cannot access the un-quantized (original) image. Most existing methods assume access to the original image, and this makes the image restoration problem easier to address. Our challenge is designing noise patterns that are perceptually pleasant, i.e. blended into the image content naturally based on the intensity profile and dynamic range conversion characteristics of the image. In the second scenario, we apply our methods to medical imaging: Magnetic Resonance (MR) images of preterm infant brains. We propose abnormality detection without using tissue priors (atlas) from a single acquisition sequence (T1-weighted). The rate of preterm births is increasing worldwide at an alarming rate. Preterm infant brains are at extremely high risk of developing abnormalities, which deter neuro-development. Traditional segmentation-based lesion detection methods rely on brain atlases to guide segmentation. For rapidly evolving preterm infant brains, such reliable atlases are unavailable, and low signal-to-noise ratio of small preterm infant brains complicates the restoration of tissue intensities. This motivates our solution design using more generic structural assumptions and heuristics for atlas-free WMI detection. In the third scenario, for Wide Area Monitoring (WAM) via ground movement prediction from noisy interferometric images, e.g. nation-wide monitoring for earthquake prediction or landslide prediction at mining sites, we propose image detail recovery through unsupervised learning-based filtering and filter output confidence prediction due to unavailability of clean training data. Filtering and confidence estimation are crucial steps in interferometric image processing pipelines. Challenges arise due to atmospheric factors, e.g. moisture, corrupting the images during acquisition. Also, corresponding pixels in time series images get decorrelated due to other undesired movements on the ground, e.g. moving vehicles on roads, vegetation and flowing water. Intermediate results of the filtering and confidence estimation steps can also influence the quality of the pipeline’s final result. Traditional algorithms in this area were not designed by prioritizing scalability via parallelism, without sacrificing accuracy, which are crucial for WAM, and addressed in this thesis. Thus, our main contribution is proposing novel solutions in scenarios where traditional image abnormality detection / restoration approaches are inadequate to address the signal processing and computational challenges involved.

In this dissertation we propose two novel image restoration schemes. The first pertains to automatic detection of damaged regions in old photographs and digital images of cracked paintings. In cases when inpainting mask generation cannot be completely automatic, our detection algorithm facilitates precise mask creation, particularly useful for images containing damage that is tedious to annotate or difficult to geometrically define. The main contribution of this dissertation is the development and utilization of a new inpainting technique, region hiding, to repair a single image by training a convolutional neural network on various transformations of that image. Region hiding is also effective in object removal tasks. Lastly, we present a segmentation system for distinguishing glands, stroma, and cells in slide images, in addition to current results, as one component of an ongoing project to aid in colon cancer prognostication.

Abstract: In the oil sands extraction process, bitumen (crude oil) is separated from the sands in the Primary Separation Vessel (PSV) through a water-based gravity separation process. The interface between froth (crude oil) and middlings (water and sand) is the most important control variable in the PSV operation. Bitumen recovery and downstream operations are critically dependent on interface level measurement and control. Most of the traditional PSV level instruments have de_cient service factors, limiting the implementation of automatic control. Therefore, we proposed novel and robust computer vision based methods to estimate the froth-middlings interface level on video frames captured from a PSV's sight glasses camera.The first chapter of the thesis discusses the computer vision as a knowledge basis for the proposed work. Typical image processing and analysis methods are described, and they provide the foundation for the subsequent chapters.The subsequent chapters propose several approaches for the interface level detection. As the first approach, we present the froth-middlings interface level detection on single frames (static image processing). The level is detected based on edge detection performed on the frames, in which traditional filters are proposed to smooth the images through model-based image restoration.Next, we develop and implement a robust computer vision based method to estimate the froth-middlings interface level in PSV, in which we additionally consider the dynamics of a set of consecutive frames to improve the level estimation. The algorithm processes the online video frames of a camera mounted on PSV sight glasses, and the level is detected based on a combined operation of edges and motion detection in a set of consecutive frames (static and dynamic image processing). In addition, the algorithm uses reliability analysis to detect the environmental conditions that may limit the level estimation, and a time-based sliding window analysis of the level measurements is proposed. Industrial application results show that the proposed computer vision algorithm is more accurate and reliable when compared to other instruments, as well as more robust against the process and environmental abnormalities.Finally, advanced filters with finite impulse response (FIR) structure are proposed and developed to improve the image restoration and object tracking process. We address the problem of smoother design for state estimation based on a finite number of measurements collected over a finite horizon. Three different FIR smoothing algorithms are proposed using the maximum likelihood FIR estimation, which is robust against uncertain noise statistics and modeling parameters. The FIR algorithms are applied to the image restoration and level tracking problem in PSV, and they show better robustness against modeling uncertainties than traditional IRR filtering approaches.

Electron microscopes have the capability to examine specimens at much finer detail than a traditional light microscope. Higher electron beam voltages correspond to higher resolution, but some specimens are sensitive to beam damage and charging at high voltages. In the scanning electron microscope (SEM), low voltage imaging is beneficial for viewing biological, electronic, and other beam-sensitive specimens. However, image quality suffers at low voltage from reduced resolution, lower signal-to-noise, and increased visibility of beam-induced contamination. Most solutions for improving low voltage SEM imaging require specialty hardware, which can be costly or system-specific. Point spread function (PSF) deconvolution for image restoration could provide a software solution that is cost-effective and microscope-independent with the ability to produce image quality improvements comparable to specialty hardware systems. Measuring the PSF (i.e., electron probe) of the SEM has been a notoriously difficult task until now. The goals of this work are to characterize the capabilities and limitations of a novel SEM PSF determination method that uses nanoparticle dispersions to obtain a two-dimensional measurement of the PSF, and to evaluate the utility of the measured PSF for restoration of low voltage SEM images. The presented results are meant to inform prospective and existing users of this technique about its fundamental theory, best operating practices, the expected behavior of output PSFs and image restorations, and factors to be aware of during interpretation of results.

Field robotics refers to the deployment of robots and autonomous systems in unstructured or dynamic environments across air, land, sea, and space. Robust sensing and perception can enable these systems to perform tasks such as long-term environmental monitoring, mapping of unexplored terrain, and safe operation in remote or hazardous environments. In recent years, deep learning has led to impressive advances in robotic perception. However, state-of-the-art methods still rely on gathering large datasets with hand-annotated labels for network training. For many applications across field robotics, dynamic environmental conditions or operational challenges hinder efforts to collect and manually label large training sets that are representative of all possible environmental conditions a robot might encounter. This limits the performance and generalizability of existing learning-based approaches for robot vision in field applications. This thesis focuses on developing approaches for unsupervised learning to advance perceptual capabilities of robots in underwater environments. The underwater domain presents unique environmental conditions to robotic systems that exacerbate the challenges to perception for field robotics. To address these challenges, the proposed approaches leverage physics-based models and cross-disciplinary knowledge about the physical environment and the data collection process to provide constraints that relax the need for ground truth labels in learning-based frameworks. This leads to a hybrid model-based, data-driven solution. Experiments are carried out using data from real field experiments to validate the proposed approaches. Although this work specifically focuses on the underwater domain, its aim is to present novel frameworks for using cross-disciplinary knowledge and physics-based models of environmental conditions in computer vision and unsupervised learning contexts to work towards robust perception systems for deploying robotic platforms in natural and unstructured environments.

On-line Wear Debris Analysis (WDA) with its rapid fault detection capability in a non-destructive manner, has been increasingly employed in machine condition monitoring. However, the available debris features obtained from current on-line WDA processes are mostly restricted to statistical indicators due to the limited data quality. Morphological features extraction from individual debris, hence, has not been fully successful in practice. Moreover, wear debris detections are currently relying on single-side or two-dimensional (2-D) debris features, while the three-dimensional (3-D) features that contain valuable morphological information are still not accessible in on-line WDA and its further application is thus impeded. To extract more individual debris features in current on-line WDA, this research aims at formulating an efficient scheme with improved debris observation, robust features extraction and 3-D debris shape measurement. First, an automatic framework is established to collect debris information in multi-views by observing the moving debris in a flow cell. Since the debris profiles are captured when it is moving, motion blur inevitably occurs and degrades the image quality. A fusion based restoration method is then developed to remove the motion blur by integrating multiple deblurring processes over several localised Point Spread Function (PSF). Furthermore, out-of-focus blur often occurs as another predominant source of degradation that leads to low feature extraction accuracy. This problem is tackled here by utilising a Convolutional Neural Networks (CNN) to model the defocus process and then remove the degradation. Finally, given the improved debris profiles, a debris shape measurement procedure is constructed to estimate 3-D debris features by minimising the discrepancies between multiple potential reconstructions. Through validation experiments and comparisons, it indicates that the proposed 3-D measurement framework could enable the observed debris information to be extended into the third dimension that is rarely achieved by available on-line detectors for WDA. Compared with other debris imaging approaches, the developed method allows the estimation of material loss based on the measured debris volume. Furthermore, the classification of debris shape into chunk, laminar and sphere is now accomplishable with no specialised equipment. The industrial practicability of WDA can be improved considerably.

This thesis proposes new techniques as well as considers state-of-the-art and recently developed methods to analyse images acquired through different turbulent media, such as atmospheric and underwater turbulence. The techniques include some challenging image processing tasks, such as image restoration, object detection and identification. In particular, it deals with both space-variant and space-invariant blur removal and supress the distortions. In addition, two effective techniques are proposed for object detection in images acquired through long-range atmospheric turbulent path, and object identification in images acquired through underwater turbulent media. Possible applications of these techniques can include different forms, such as military surveillance, detecting and identifying objects, imaging problems in remote sensing, and various ocean applications. A faster image pre-processing technique by employing the k-means clustering technique is proposed in this thesis for selecting the important frames rather than considering random frames. This technique is used in imaging for both turbulent media. A recent restoration method is reviewed and improved by employing a spatio-temporal kernel regression based fusion method for removing space-invariant blur from the images and supress deformations. To further improve the image quality a blind deconvolution procedure is employed. In practice, the images acquired through turbulent media are distorted due to space-variant blur problem. In-order to address this problem a patch-wise deconvolution process is carried out on the distorted image, and the turbulence corrected image is given as input to the dictionary learning algorithm for further denoising. Motivated by the saliency technique, object detection technique is proposed considering the small cluttered objects present in the images acquired. These objects makes other detection problems difficult such as moving object detection. Thus, it is important to detect the small cluttered objects. This thesis also focuses on the imaging issues considering underwater turbulence media. Images are distorted mainly due to non-uniform illumination that occurs as a result of scattering and absorption of various submerged organic substances, and space-variant blur. The Retinex model is employed to correct the non-uniform illumination problem prior to space-variant blur removal. A patch-wise operation is then carried out for further restoration purpose. Image segmentation algorithm by considering the efficiency of suppression factor is then implemented on the restored image for identifying the objects. The potential and the performance of the proposed approaches are investigated considering both real-world and synthetic datasets, and shows better restoration, detection and identification results. The results show practical interest for considering the proposed approaches in different computer vision and image processing systems, and machine learning and pattern analysis applications.

In the decades that followed World War II, tuition at American colleges was well within the financial reach of most families. Since 1980, however, it has grown more expensive to attend both public and private colleges, as tuition has surged at a rate that has far outpaced inflation. At the same time, the economic and lifestyle disparities between those who earn four-year degrees and those who do not have reached record levels. As a result, students have to go to college in order to have a realistic shot at prosperity, but must borrow significantly in order to afford the cost of attendance. Colleges are aware that whenever the subject of increased tuition comes up, be it a proposed increase or an official one, it is a threat to their image and is likely to be viewed as offensive by students, who are already straining from the high cost of college. Thus, colleges employ a range of image restoration theory strategies at all phases of the conflict management life cycle, in order to restore, repair, and protect their images. While the rhetorical strategies taken by colleges may be given a great deal of thought by college spokespersons, they are not always strategically appropriate. This thesis uses mental accounting to extend image restoration theory, and offers rhetorical strategies that colleges may consider in order to minimize the threat to their images posed by increased tuition.

Imaging through a turbulent medium, such as the atmosphere or the wavy surface of water, is highly desired in many scientific and military applications. This is a very challenging task due to the time-varying shifts and blurs captured in the images. This thesis deals with the geometrical restoration of such images captured as video sequences. These ordinarily undesirable geometrical distortions also act as information compressors and can be exploited to extract further bandwidth from the images to produce high-quality images from their lower resolution counterparts. The research investigations cover both the atmospheric as well as underwater imaging. First, a simple and robust method is reviewed and improved upon to restore warped frames using motion vector fields (shiftmaps) obtained through a motion estimation technique. The centroid of the pixel shiftmaps is then calculated to generate individual restoration shiftmaps for each warped frame. The centroid shiftmap is updated iteratively to take the restored frames closer to their likely ground-truth. Furthermore, the image restoration method is made predictive by the use of a generalized regression neural network (GRNN), where the pixel shiftmaps amongst successive frames are used for training the network to determine the underlying warping functions, which in turn, are used to predict the upcoming warped frame. Moreover, the accurate motion analysis along with video stabilization method is utilized for reliable segmentation of video frames into stable and moving components and subsequently stabilizing frames, keeping real moving objects unaltered. Motivated by the successful application of GRNN in warp prediction, finally, a new and more efficient target tracking algorithm is proposed that works based on determining the centre and the area of moving objects, using those features for GRNN training, and employing the trained network to estimate the objects’ locations in the next frame. Both the accuracy and the potential of the proposed algorithms have been investigated. The results presented are of both theoretical and practical interest and offer new efficient tools for substantial improvement of infrastructure of machine vision-based systems in general and of intelligent surveillance systems in particular.

Images degraded by light scattering and absorption such as hazy, sandstorm, and underwater images often suffer from color distortion and low contrast because of light traveling through turbid media. This can prevent systems that operate outdoors in different lighting conditions from functioning properly, for example, video surveillance systems, autopilot systems and intelligent transportation systems, which include automatic license plate recognition, automatic traffic counting, etc. Therefore, it is desirable to develop an effective method to restore color and enhance contrast for these images. This thesis presents novel work to advance research on image restoration and enhancement for such images.To enhance or restore such a degraded image, the image formation model is often used to describe it as a ``clear" image blended with an ambient light based on the scene transmission computed using the scene depth from the camera. The transmission describes the portion of the scene radiance which is not scattered or absorbed and which reaches the camera. By reversing the image formation process, one can attain the scene radiance from a degraded image, which is a ``clear" image. However, it involves solving an ill-posed and under-constrained problem because we need to estimate both the ambient light and scene transmission from a single degraded image.To attack this problem, we proposed to use image blurriness to estimate ambient light and scene depth for underwater images. Furthermore, we extended it by combining light absorption and blurriness to estimate scene depth for underwater scenes in different lighting conditions and color tones. For any images degraded by light scattering and absorption, not limited to underwater ones, we proposed a generalization of the common dark channel prior approach for ambient light and transmission estimation. Additionally, adaptive color correction is incorporated into the image formation model for removing color casts while restoring contrast. Based on the experimental results, our proposed algorithms outperform, both subjectively and objectively, other state-of-the-art algorithms based on the image formation model.

With the explosion in the number of digital images taken every day, people are demanding more accurate and visually pleasing images. However, the captured images by modern cameras are inevitably degraded by noise. Besides deteriorating image visual quality, noise also degrades the performance of high-level vision tasks such as object recognition and tracking. Therefore, image denoising is a critical preprocessing step. This thesis presents novel contributions to the field of image denoising.Image denoising is a highly ill-posed inverse problem. To alleviate the ill-posedness, an effective prior plays an important role and is a key factor for successful image denoising. With abundance of images available online, we propose to obtain priors from external image databases. In this thesis, we perform statistical analyses and rigorous derivations on how to obtain effective priors by utilizing external databases. For three denoising applications under different external settings, we show how we can explore effective priors and accordingly we present adaptive patch-based image denoising algorithms. In specific, we propose three adaptive algorithms: (1) adaptive non-local means for multiview image denoising; (2) adaptive image denoising by targeted databases; (3) adaptive image denoising by mixture adaption.In (1), we present how to improve the non-local prior by finding more relevant patches in the multiview image denoising setting. We propose a method that uses a robust joint-view distance metric to measure the similarity of patches and derive an adaptive procedure to determine the optimal number of patches for final non-local means denoising. In (2), we propose to switch from generic database to targeted database, i.e., for specific objects to be denoised, only targeted databases with relevant images should be used. We explore both the group sparsity prior and the localized Bayesian prior, and show how a near optimal and adaptive denoising filter can be designed so that the targeted database can be maximally utilized. In (3), we propose an adaptive learning procedure called Expectation-Maximization (EM) adaptation. The adaptive process takes a generic prior learned from a generic database and transfers it to the image of interest to create a specific prior. This adapted prior better captures the distribution of the image of interest and is consistently better than the un-adapted one. For all the three denoising applications, we conduct various denoising experiments. Our proposed adaptive algorithms have some superior denoising performance than some state-of-the-art algorithms.

Digital photography has experienced great progress during the past decade. A lot of people are recording their moments via digital hand-held cameras. Pictures taken with digital cameras usually undergo some sort of degradation in the form of noise/blur depending on the camera hardware and environmental conditions in which the photos are taken. This leads to an ever-increasing demand for effective and effcient image enhancement algorithms to achieve high quality output images in digital photography systems. In this dissertation, a new graph-based framework is introduced for different image restoration applications. This framework is based on exploiting the existing self-similarity in images. We introduce a new definition of normalized graph Laplacian matrix for image processing. We use this new definition to develop effective enhancement algorithms for image deblurring, image denoising, and image sharpening.First, we develop a regularization framework for image deblurring by constructing a new graph-based cost function. Minimizing the corresponding cost function yields effective outputs for different blur types including out-of-focus and motion blurs. Our proposed deblurring algorithm based on the new definition of normalized graph Laplacian provides performance and analysis advantages over previous methods. We have shown its effectiveness for several synthetic and real deblurring examples.Second, we develop a new graph-based framework for image denoising. The proposed denoising method exploits the similarity information in images by constructing the similarity matrix which in turn is used to derive the corresponding graph Laplacian. A graph-based objective function with new data fidelity and smoothness terms is constructed and minimized. We also establish the relationship between our proposed regularized framework and two well-known iterative methods for improving the performance of kernel-based denoising methods; namely, diusion and boosting iterations.We compare the performance of the proposed denoising method with that of NLM algorithm [11] and demonstrate that our proposed algorithm is able to enhance over NLM. Furthermore, we present a graph-based analysis framework for multi-layer image decomposition using diffusion and boosting iterations.Third, we propose a new data-adaptive sharpening algorithm based on the notion of difference of smoothing operators. We provide an interpretation of our proposed sharpening method as the image-derived version of difference of Gaussians (DoG) operator broadly used in numerous image processing and computer graphics tasks [66, 121, 122].Finally, we provide a theoretical study on the reported range of the eigenvalues of various definitions of normalized graph Laplacian for different graph structures. This sheds light on the existing ambiguity on the spectral range of such matrices in different applications.

Ill-posed inverse problems have wide applications in many fields such as oceanography, signal processing, machine learning, biomedical imaging, remote sensing, geophysics, and others. In this dissertation, we address the problem of solving unstable operator equations with iteratively regularized Newton-type algorithms. Important practical questions such as selection of regularization parameters, construction of generating (filtering) functions based on a priori information available for different models, algorithms for stopping rules and error estimates are investigated with equal attention given to theoretical study and numerical experiments.

Multi-frame super-resolution (SR) image reconstruction methods seek to overcome sampling limitations in staring cameras, leading to the reduction or elimination of aliasing artifacts in imaging systems. Prior research has predominantly been focused on single band or panchromatic imaging applications. Sampling limitations are further exacerbated for cameras that add functionality by implementing spatial filters across the cameras focal plane array. The types of cameras, generally called division-of-focal-plane (DoFP) cameras, include the Bayer color filter array (CFA) design common to commercial three-color cameras. Spatial filtering is accomplished by repeating a spatial filter pattern across the focal plane array. In CFA cameras, each detector element only collects a single color channel, resulting in a mosaic pattern. The other color channels must be estimated at that sample location. This is accomplished by a process called, demosaicing. Extensive research has been accomplished in demosaicing. However, these processes are typically insufficient in eliminating aliasing artifacts.The research described in this work is developed to overcome sampling limitations in DoFP cameras and reduce or eliminate aliasing in images. Two new methods are developed and are presented. Both methods create SR image estimates from multiple image frames in the presence of random frame-to-frame motion. The first method is an interpolation-restoration SR approach that places all frame samples onto a common non-uniform sampling space and applies an optimal filter based on sample pattern and image statistics. This fast method combines multiple SR steps as a spatially-adaptive weighted sum and is called the color adaptive Wiener filter (AWF) SR approach. By leveraging a newly-developed approach to capture channel cross-correlation, this fast approach rivals and exceeds performance of much more computationally-intensive variational methods.The second method in this work is a new and computationally simpler method to address practical challenges of multi-frame SR, including frame-to-frame affine motion and the presence of local motion objects within the image set, common to many airborne imaging applications. The method is a non-iterative, sequential non-uniform interpolation process that fuses multi-frame data using a new and unique weighted sum. This method is called Fusion of Interpolated Frames (FIF) SR. With its combination of simplicity and robustness, FIF SR moves beyond color AWF SR and other DoFP-based SR approaches in terms of performance and computational efficiency. No other method that we know of has been developed to handle DoFP data in these complex imaging scenarios while operating relatively fast and achieving this level of performance.This work provides the background of the problem and describes detailed mathematical formulations for the two proposed SR methods. Multiple experiments using synthetically generated and real image datasets are provided. The proposed methods are compared against prior research SR approaches with both qualitative and quantitative results detailed. Finally, conclusions and recommendations for future work are discussed.

In this dissertation, we have presented a novel patch-based algorithm using an adaptive Wiener filter (AWF) for image restoration as well as for multi-frame super-resolution (SR). The new filter structure is referred to as a collaborative adaptive Wiener filter (CAWF), which can be thought of an extension of the AWF using multiple patches. Starting with CAWF for image restoration, in each reference window, the most similar patches are identified. The output is formed as a single-pass weighted sum of all of the pixels from the multiple selected patches. Wiener weights which are derived from a novel spatial-domain multi-patch correlation model, are used to provide a minimum mean square error (MSE) estimate for this filter structure. While the first step of the CAWF for multi-frame super-resolution (CAWF SR) method is to register all low-resolution (LR) frames one a common high-resolution (HR) grid. Then, for each LR pixel position on the HR grid, we also identify the most similar patches within a given search window about the reference patch. The output HR pixels in the estimate window at the reference patch are obtained using a weighted sum of all of LR pixels of the selected similar patches. The weights are based on a new nonuniformly sampled multi-patch correlation model. A key aspect of the CAWF method for both restoration and multi-frame SR is the new spatial-domain multi-patch correlation model. This model attempts to capture the spatial correlation among the samples within a given patch, and also the correlations among the patches. The CAWF is able to jointly perform nonuniform interpolation, denoising and deblurring. We believe this type of joint restoration and interpolation is advantageous, compared with decoupling these operations. The CAWF algorithm is also capable of adapting to local signal and noise variance. Bad or missing pixels can easily be accommodated by leaving them out of the multi-patch observation vector and corresponding correlation statistics. We provide several experiments on the CAWF for image restoration and multi-frame SR using simulated and real data. The experimental results presented show that the proposed method delivers high performance in image restoration as well as in multi-frame SR.

This study looked at the effectiveness of William Benoit's image restoration strategies by using a post-test only experiment which featured four treatments. It compared consumers' feelings following responses made by a company after an image threat. Each response consisted of a combination of image restoration strategies, and while not all of Benoit's strategies were tested, the ones which seemed to be most commonly used were chosen. The present study showed two strategies, mortification and corrective action, were more effective than others tested. In addition, there were outside factors, such as perceived sincerity, which also had a role in the effectiveness of the restoration attempts.

The purpose of this study was to apply Benoit's theory of image restoration to apologetic discourse offered by those involved in the crisis surrounding Phil Robertson following his January 2014, GQ Magazine interview. This content analysis applied the theoretical framework to a collection of news segments, articles, and letters that contained apologetic discourse and analyzed them to identify the strategies of image restoration used. The results indicated that the most prevalent strategy used was transcendence- placing the act that caused offense in a different context. Robertson and those involved in his image restoration directed the audience's attention to higher moral values when they insisted that he was merely answering questions based on his personal beliefs that are strongly rooted in his faith. This was the theme of apologia that was issued. Robertson did not intend to offend the audience when he quoted the Bible. He was doing what he does regularly: sharing his faith.

The defense industry has numerous detectors that provide critical imaging capability on tactical and reconnaissance platforms and have been shown to be susceptible to permanent damage from high energy pulsed lasers in both laboratory and field testing. Much of the materials research into this involves two different methods of providing pulsed laser damage protection: extrinsic limiter implementation and intrinsic detector hardening. This thesis focused on what gains could be made using another method: system defocus and detector redundancy. The work of this thesis revolved around hardening a camera system through defocusing the focal plane array (FPA) and then using image restoration algorithms to regain the image quality of the degraded images. This system, a three channel image splitting prism with lens mount, provided a unique opportunity to test multiple images of an identical scene with slight spatial misalignments, varying sensor defocus and precisely measured optical degradation as measured by the Point Spread Function. These defocused images were then restored using filters that utilized information from only a single channel (the Wiener Filter, Regularized Least Squares Filter, and Constrained Least Squares Filter) and across multiple channels (Multichannel Constrained Least Squares Filter). Results from the single channel filters were excellent and allowed significant sensor hardening without image degradation when compared to the unfiltered image. Results from the multichannel RLS filter as tested were disappointing when compared to those from the single channel however and could be expanded upon in future work.

We propose a foundation for assessing visual quality with "corrupted reference"(CR-QA) - a new quality assessment(QA) paradigm for reasoning about human vision and image restoration problems jointly. The visual quality of a processed image signal is assessed relative to an ideal reference image (not provided) with the help of observed image. This is in contrast to today's QAs, which are optimized for a "post-hoc" usage (process first, assess quality second) and are unequipped to handle the assessment of the processed data relative to the ideal reference that exist only in theory and not in practice.

Progress in multimedia technologies has resulted in a plethora of services and devices that capture, compress, transmit and display audiovisual stimuli. Humans -- the ultimate receivers of such stimuli -- now have access to visual entertainment at their homes, their workplaces as well as on mobile devices. With increasing visual signals being received by human observers, in the face of degradations that occur to due the capture, compression and transmission processes, an important aspect of the quality of experience of such stimuli is the \emph{perceived visual quality}. This dissertation focuses on algorithm development for assessing such visual quality of natural images, without need for the `pristine' reference image, i.e., we develop computational models for no-reference image quality assessment (NR IQA). Our NR IQA model stems from the theory that natural images have certain statistical properties that are violated in the presence of degradations, and quantifying such deviations from \emph{naturalness} leads to a blind estimate of quality. The proposed modular and easily extensible framework is distortion-agnostic, in that it does not need to have knowledge of the distortion afflicting the image (contrary to most present-day NR IQA algorithms) and is not only capable of quality assessment with high correlation with human perception, but also is capable of identifying the distortion afflicting the image. This additional distortion-identification, coupled with blind quality assessment leads to a framework that allows for blind general-purpose image repair, which is the second major contribution of this dissertation. The blind general-purpose image repair framework, and its exemplar algorithm described here stem from a revolutionary perspective on image repair, where the framework does not simply attempt to ameliorate the distortion in the image, but to ameliorate the distortion, so that visual quality at the output is maximized. Lastly, this dissertation describes a large-scale human subjective study that was conducted at UT to assess human behavior and opinion on visual quality of videos when viewed on mobile devices. The study lead to a database of 200 distorted videos, which incorporates previously studied distortions such as compression and wireless packet-loss, and also dynamically varying distortions that change as a function of time, such as frame-freezes and temporally varying compression rates. This study -- the first of its kind -- involved over 50 human subjects and resulted in 5,300 summary subjective scores and time-sampled subjective traces of quality for multiple displays. The last part of this dissertation analyzes human behavior and opinion on time-varying video quality, opening up an extremely interesting and relevant field for future research in the area of quality assessment and human behavior.

.