Search

“Seeing through” occluders is one of the most important effects that can be achieved with synthetic aperture imaging. As well, the occlusion problem, a challenging task for many computer vision applications, can be easily handled. Synthetic aperture imaging takes advantage of the property that only objects on the focal plane are sharp. The resulting image that is obtained by averaging images from different views consists of blurry objects away from the focal plane and sharp objects on the focal plane. Removing the blurriness caused by defocusing in synthetic aperture images to achieve an all-in-focus “seeing through” image is a challenging research problem. In this paper, we propose a novel method to improve the image quality of synthetic aperture imaging using image matting via energy minimization by estimating the foreground and the background. In particular, we first estimate the out-of-focus region by focusing on the background objects in each camera view using energy minimization. Next, we utilize a labeling method to create a sharp “see through” synthetic aperture image of the hidden objects. Then, image matting is used to extract the alpha matte of the hidden objects. Finally, by compositing the hidden objects with the estimated background regions, a sharp “see through” synthetic aperture image is created. The experimental results show that the proposed method outperforms the traditional synthetic aperture imaging method [1] as well as its improved versions [2]–[4], which simply dim and blur the area in the image that is out of focus, and a recent all-in-focus method [5]. We show that both the occluded objects and the background can be combined using our method to create a sharp synthetic aperture image. [ABSTRACT FROM PUBLISHER]

• We propose a new online learnable keyframe extraction module (OKFEM) to extract keyframes in a video sequence. • The proposed OKFEM module is the first online module that can process frames sequentially and select the keyframes based on processed frames only. • Our OKFEM module can be used for transfer learning as well. Our method achieves the best state-of-the-art results on summarization datasets in the transfer learning environment. • We also propose a plugin module and a novel train/test strategy called ITTS for the first time to incorporate the semantic word vector along with keyframe features as input to the classification models. Our proposed approach of action recognition further improves the best state-of-the-art results on the action recognition datasets. Video processing has become a popular research direction in computer vision due to its various applications such as video summarization, action recognition, etc. Recently, deep learning-based methods have achieved impressive results in action recognition. However, these methods need to process a full video sequence to recognize the action, even though many of the frames in the video sequence are similar and non-essential to recognizing a particular action. Additionally, these non-essential frames increase the computational cost and can confuse a method in action recognition. Instead, the important frames called keyframes not only are helpful in recognizing an action but also can reduce the processing time of each video sequence in classification or in other applications, e.g. summarization. As well, current methods in video processing have not yet been demonstrated in an online fashion. Motivated by the above, we propose an online learnable module for keyframe extraction. This module can be used to select key shots in video and thus, can be applied to video summarization. The extracted keyframes can be used as input to any deep learning-based classification model to recognize action. We also propose a plugin module to use the semantic word vector as input along with keyframes and a novel train/test strategy for the classification models. To our best knowledge, this is the first time such an online module and train/test strategy have been proposed. The experimental results on many commonly used datasets in video summarization and in action recognition have demonstrated the effectiveness of the proposed module. [ABSTRACT FROM AUTHOR]

Many computer vision problems can be formulated as optimization problems. Presented in this paper is a new framework based on the quadtree-based genetic algorithm that can be applied to solve many of these problems. The proposed algorithm incorporates the quadtree structure into the conventional genetic algorithm. The solutions of image-related problems are encoded through encoding the corresponding quadtrees, and therefore, the 2D locality within a solution can be preserved. Examples addressed using the proposed framework include image segmentation, stereo vision, and motion estimation. In all cases, encouraging results are obtained. [Copyright &y& Elsevier]

• The proposed network contains an attentive recurrent network and a U-recurrent encoder-decoder network. • The attentive recurrent network generates a haze attention map for highlighting the haze regions. • The U-recurrent encoder-decoder network predicts the clear image with the guidance of the haze attention map. • A new residual pyramid pooling module enlarges the receptive field of the whole network. Haze removal is an important pre-processing step in many computer vision tasks. Convolutional neural networks, especially the U-shaped networks, have shown to be effective in image dehazing. Nevertheless, these networks have three main limitations. First, the relevant haze information, e.g. concentration of haze, is totally ignored. Second, spatial inconsistency and information dilution usually occur when the networks refine the dehazed results with a coarse-to-fine strategy. Third, the receptive field of the network is not large enough to capture structural information. Motivated by these problems, a new attentive U-recurrent encoder-decoder dehazing network is presented, which consists of an attentive recurrent network and a U-recurrent encoder-decoder network. By assuming that haze layers with different depths can be detected by multiple stages, we use an attentive recurrent network to generate the haze attention map for guiding the U-recurrent encoder-decoder network with the concentration of haze to better estimate the clear image. Meanwhile, the features for dehazing are further enhanced and the dehazing results are refined in the U-recurrent encoder-decoder network. This design not only enables spatial consistency but also reduces information dilution with short recurrent pathways. Furthermore, a novel residual pyramid pooling module is also proposed and used in the U-recurrent encoder-decoder network, which provides the network with structural information and with an enlarged receptive field. The experimental results demonstrate that our method outperforms state-of-the-art dehazing algorithms on both synthetic and real hazy images. [ABSTRACT FROM AUTHOR]

Please refer to full text. [ABSTRACT FROM AUTHOR]

Underwater images suffer from different types of degradation, where color degradation occurs in the spatial domain and edge degradation in the frequency domain. The high-quality underwater image enhancement represents a crucial milestone in advancing computer vision systems tailored for marine environments. This foundational endeavor encompasses a wide array of applications in computer vision tasks, including underwater inspection, underwater archaeology, and environmental monitoring. However, current convolutional neural network (CNN)-based pyramid frameworks primarily focus on capturing local features, often overlooking the significance of global semantic features that play a crucial role in understanding underwater scenes. Moreover, these frameworks handle spatial and frequency features independently, failing to enhance images by exploring correlation among domain-specific attributes for enabling information consistency. Besides, optimizing the model using a loss function with the same domain attributes from the ground truth may not lead to a better generalization ability. To solve these problems, we propose a new Convolution-Transformer Blend Pyramid Network (CTPN), which consists of a spatial branch and several frequency branches. The CTPN has four key components: a Swin transformer encoder, a CNN-Transformer aggregated encoder–decoder (CTED) and a blend pyramid framework. The Swin transformer encoder is employed to capture global semantic features, benefiting from its ability to extract long-range and global dependencies among features. The CTED fuses local features captured by CNN layers and global semantic features captured by the Swin transformer encoder in the spatial branch, with the help of the Cross-Model Fusion Module (CFM) and Skip-Aggregation Module (SAM). Subsequently, a blend pyramid framework is designed which not only progressively expands the transformed information of the previous domain branch to the current domain branch via the CTED-based refining operation, but also utilizes the proposed Domain Affinity Block (DAB) to explore the connection between domain attributes, ensuring information consistency. The experimental results demonstrate that the proposed method outperforms existing underwater image enhancement methods quantitatively and qualitatively. • A Convolution-Transformer blend pyramid network with spatial and frequency branches is designed for image enhancement. • A Convolution-Transformer aggregated encoder–decoder fuses local and global features. • A blend pyramid framework with the Domain Affinity Block is designed for enabling information consistent. [ABSTRACT FROM AUTHOR]

• An end-to-end network with a parallel spatial/channel-wise attention block is proposed for image dehazing. • The spatial attention module and the channelwise attention module are combined in parallel for obtaining powerful features. • The accuracy of the dehazing results is improved by using one parallel spatial/channel-wise attention block in the network. Image dehazing is a very important pre-processing step to many computer vision tasks such as object recognition and tracking. However, it is a challenging problem because the physical parameters of imaging, e.g. the depth information of scene pixels and the attenuation model, are usually unknown. Based on a physical model, different methods have been proposed to recover these parameters. Existing convolutional neural networks (CNNs) based methods try to solve the image dehazing problem using an end-to-end network to learn a direct mapping between a hazy image and its corresponding clear image. But the representational ability, spatial variant ability and dehazing capability of these network models are hindered by treating all the spatial and channel-wise features indiscriminately. Hence, we propose an end-to-end dehazing network with a parallel spatial/channel-wise attention block for capturing more informative spatial and channel-wise features respectively. Specifically, based on the encoder-decoder framework with a pyramid pooling operation, a novel parallel spatial/channel-wise attention block is proposed and applied to the end of the encoder for guiding the decoder to reconstruct better clear images. In the spatial/channel-wise attention block, the spatial attention module and the channel-wise attention module are connected in parallel, where the spatial attention module highlights important spatial positions of features. Meanwhile, the channel-wise module exploits inter-dependencies among the channel-wise features. Extensive experiments demonstrate that our network with a parallel spatial /channel-wise attention block can achieve better accuracy and visual results over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Amyotrophic Lateral Sclerosis (ALS) is a complex neurodegenerative disorder characterized by motor neuron degeneration. Significant research has begun to establish brain magnetic resonance imaging (MRI) as a potential biomarker to diagnose and monitor the state of the disease. Deep learning has emerged as a prominent class of machine learning algorithms in computer vision and has shown successful applications in various medical image analysis tasks. However, deep learning methods applied to neuroimaging have not achieved superior performance in classifying ALS patients from healthy controls due to insignificant structural changes correlated with pathological features. Thus, a critical challenge in deep models is to identify discriminative features from limited training data. To address this challenge, this study introduces a framework called S F 2 F o r m e r , which leverages the power of the vision transformer architecture to distinguish ALS subjects from the control group by exploiting the long-range relationships among image features. Additionally, spatial and frequency domain information is combined to enhance the network's performance, as MRI scans are initially captured in the frequency domain and then converted to the spatial domain. The proposed framework is trained using a series of consecutive coronal slices and utilizes pre-trained weights from ImageNet through transfer learning. Finally, a majority voting scheme is employed on the coronal slices of each subject to generate the final classification decision. The proposed architecture is extensively evaluated with multi-modal neuroimaging data (i.e., T1-weighted, R2*, FLAIR) using two well-organized versions of the Canadian ALS Neuroimaging Consortium (CALSNIC) multi-center datasets. The experimental results demonstrate the superiority of the proposed strategy in terms of classification accuracy compared to several popular deep learning-based techniques. • We propose a novel vision transformer model to classify ALS from healthy controls. • We analyze two independent and extensive datasets of 120 and 232 MRI scans. • We leverage multi-center and multi-modal neuroimaging data (T1W, R2*, and FLAIR). • The proposed method demonstrates state-of-the-art classification accuracy. [ABSTRACT FROM AUTHOR]

Occlusion poses as a critical challenge in computer vision for a long time. Recently, the technique of synthetic aperture photography using a camera array has been regarded as a promising way to address the problem of occluded object imaging. Unfortunately, the expensive cost of a standard camera array system with the required calibration procedure still limits the widespread popularity of this technique. To make synthetic aperture photography practical, in this paper, we propose a novel approach based on a mobile camera-IMU system moving on a planar surface. The contributions of our work include: (1) with geometric property assumptions, we mathematically prove that the transformation matrices for synthetic aperture image generation can be factorized into an infinite homography and a shift; (2) the IMU data are utilized to estimate the homography by selecting the plane at infinity as the reference plane, which is scene independent and does not require any calibration; (3) compared to the conventional camera array based approaches, the proposed method effectively improves the solution for synthetic aperture images capturing and generation by using a more flexible, convenient and inexpensive system. Extensive experimental results with qualitative and quantitative evaluations have demonstrated the see-through-occlusion performance with satisfactory accuracy of the proposed approach in different scenarios. [ABSTRACT FROM AUTHOR]

Abstract: Synthetic aperture imaging using an array of cameras, which has become popular recently, can easily handle the occlusion problem by “seeing through” occluders. Unfortunately, the resulting image is still blurry because it combines information not only from the region of interest but also from the occluding regions. Removing the blurriness of synthetic aperture images has become a challenging task for many computer vision applications. In this paper, we propose a novel method to improve the image quality of synthetic aperture imaging using energy minimization. Unlike the conventional synthetic aperture imaging method, which averages images from all the camera views, we reformulate the problem as a labeling problem. In particular, we use the energy minimization method to label each pixel in each camera view to decide whether or not it belongs to an occluder. After that, the focusing at the desired depth is by averaging pixels that are not labeled as occluder. The experimental results show that the proposed method outperforms the traditional synthetic aperture imaging method as well as its improved versions, which are simply dim and blur occluders in the resulting image. To the best of our knowledge, our proposed method is the first one for improving the results of synthetic aperture image without using a training set from the input sequence. As well, it is the first method that makes no assumptions on whether or not the objects in the scenes are static. [Copyright &y& Elsevier]

• The proposed network contains a feature extraction network, a recurrent refinement network and an encoder-decoder network. • The recurrent refinement network generates and refines the haze attention map using low-level and high-level features as input alternatively. • The encoder-decoder network predicts the dehazing result with the guidance of a haze attention map and the fused multi-level features. Image dehazing is very important for many computer vision tasks. However, typical CNN-based methods learn a direct mapping from a hazy image to a clear image, ignoring relevant haze priors and multi-level features. In this paper, a new Visual Attention Dehazing Network (VADN) with multi-level refinement and fusion is proposed, which leverages a haze attention map as a haze relevant prior and learns complementary haze information among multi-level features. The VADN contains a feature extraction network, a recurrent refinement network and an encoder-decoder network. The feature extraction network captures the multi-level features. The recurrent refinement network generates and refines the haze attention map by taking low-level features and high-level features as inputs alternatively. Then, the haze attention map is injected into the encoder-decoder network to obtain the clear image with the help of complementary information learned from informative multi-level features. The experimental results demonstrate that the average PSNR of VADN is 32.50 dB which outperforms most state-of-the-art methods by up to 5.14 dB. Besides, the run time of VADN is 0.067 s, only 55 % of the run time spent by the recent enhanced pix2pix dehazing network. [ABSTRACT FROM AUTHOR]

• Data from multiple centers for ALS research has some challenges such as intensity variations in MRI scans. • A gradient-based texture method called M-CoHOG is proposed to discriminate ALS patients from controls. • A feature normalization method along with an ensemble classifier is also proposed to accommodate the multicenter issues. • Our method achieves state-of-the-art results by accommodating the variation in multicenter data. • Thus, texture analysis using M-CoHOG shows promise as a potential biomarker for ALS. Gradient-based texture analysis methods have become popular in computer vision and image processing and has many applications including medical image analysis. This motivates us to develop a texture feature extraction method to discriminate Amyotrophic Lateral Sclerosis (ALS) patients from controls. But, the lack of data in ALS research is a major constraint and can be mitigated by using data from multiple centers. However, multi-center data gives some other challenges such as differing scanner parameters and variation in intensity of the medical images, which motivate the development of the proposed method. To investigate these challenges, we propose a gradient-based texture feature extraction method called Modified Co-occurrence Histograms of Oriented Gradients (M-CoHOG) to extract texture features from 2D Magnetic Resonance Images (MRI). We also propose a new feature-normalization technique before feeding the normalized M-CoHOG features into an ensemble of classifiers, which can accommodate for variation of data from different centers. ALS datasets from four different centers are used in the experiments. We analyze the classification accuracy of single center data as well as that arising from multiple centers. It is observed that the extracted texture features from downsampled images are more significant in distinguishing between patients and controls. Moreover, using an ensemble of classifiers shows improvement in classification accuracy over a single classifier in multi-center data. The proposed method outperforms the state-of-the-art methods by a significant margin. [ABSTRACT FROM AUTHOR]