Search

Deep Metric Learning (DML) is a crucial machine learning method in computer vision, and constructing an effective loss function. Average Precision (AP) is a well-known evaluation metric for image retrieval tasks. However, AP's non-differentiable and non-decomposable nature constrains its potential for adoption as an optimization goal. We propose a deep metric learning method with learned parametric metric learning loss and a contextual memory block (LM-Metric) for large-scale image retrieval tasks, which overcome AP's drawbacks and integrate AP within DML loss. We first introduce a parametric pairwise weighting scheme via policy gradient optimization and model the batch-wise inter-sample relationship via a Gated Recurrent Unit (GRU). Furthermore, a conditional Normalizing Flow-based contextual memory feature block to learn a compact single embedding for each image containing the contextual information during retrieval. We perform experiments on retrieval benchmark datasets and improve performance over the state-of-the-arts. • We propose a policy gradient-based meta-learned parametric metric learning scheme (LM-Metric). • We consider the inter-sample relationship during training to solve the non-decomposability problem of AP optimization. • We validate the effectiveness of our method, by applying it to variouis (DML) loss, e.g., Triplet, Margin, and Proxy-anchor losses. [ABSTRACT FROM AUTHOR]

Graph matching is an important problem that has received widespread attention, especially in the field of computer vision. Recently, state-of-the-art methods seek to incorporate graph matching with deep learning. However, there is no research to explain what role the graph matching algorithm plays in the model. Therefore, we propose an approach integrating a MILP formulation of the graph matching problem. This formulation is solved to optimal and it provides inherent baseline. Meanwhile, similar approaches are derived by releasing the optimal guarantee of the graph matching solver and by introducing a quality level. This quality level controls the quality of the solutions provided by the graph matching solver. In addition, several relaxations of the graph matching problem are put to the test. Our experimental evaluation gives several theoretical insights and guides the direction of deep graph matching methods. • We propose a MILP formulation integrated into a deep graph matching architecture. • We propose a quality-aware heuristic to study the impact of quality solutions. • We compare an exact method to heuristics methods. • We compare the graph matching problem to two continuous and topological relaxations. • Our experiments provide a clear picture to successfully relax deep graph matching. [ABSTRACT FROM AUTHOR]

Convolutional Neural Networks (CNNs) are frequently employed for image pattern recognition and other computer vision tasks. When over-parameterized deep learning models are used for inference, resource-constrained edge devices may struggle. As a result, model compression, particularly filter pruning, has become critical. A reduction in model size might result in less calculation, resulting in faster hardware execution and lower energy consumption. One of the drawbacks of current pruning strategies is that once the filters are pruned, their weights are permanently lost. To address this constraint, we propose a unique two-phase pruning technique in which the filters to be pruned are selected using two criteria: l 2 -norm and redundancy. Second, rather than omitting the selected filters for all future epochs, we restore them to their original value with some stochasticity. Retaining the most optimal filter weights in earlier epochs enables the survival of the fittest filters, resulting in higher model convergence. Experiments on three benchmark datasets, CIFAR-10, CIFAR-100, and ILSVRC-2012, reveal that our strategy outperforms other state-of-the-art pruning methods by a minimum reduction of 57% FLOPs with an accuracy loss as minimal as 0.08 %. • Two-criteria (l 2 -norm and redundancy) imposed for pruning non-critical filters. • Stochastic optimization technique implemented for restoring most promising filters. • Proposed pruning method accelerates CNNs with more than 50% reduction in FLOPS. • Pruned models achieve nearly the same accuracy as baseline models. [ABSTRACT FROM AUTHOR]

Human action recognition is a well-established task in the field of computer vision. However, accurately representing spatio-temporal information remains a challenge due to the complex interplay between human actions, video timing, and scene changes. To address this challenge and improve the efficiency of temporal modeling in videos, we propose MLENet, a novel approach that eliminates contextual data and eliminates the need for laborious optical flow extraction.MLENet incorporates a Temporal Feature Refinement Extraction Module (TFREM) that utilizes Optical Flow Guided Features to enhance attention to local deep detail information. This refinement process significantly enhances the network's capacity for feature learning and expression. Moreover, MLENet is designed to be trained end-to-end, facilitating seamless integration into existing frameworks. Additionally, our model adopts a temporal segmentation structure for sampling, effectively reducing redundant information and improving computational efficiency. Compared to existing video-based action recognition models that require optical flow or other modalities, MLENet achieves substantial performance enhancements while requiring fewer inputs. We validate the effectiveness of our proposed approach on benchmark datasets, including Something-Something V1&V2, UCF-101, and HMDB-51, where MLENet consistently outperforms state-of-the-art models. [Display omitted] • MLENet can increase the ability of modeling temporal structure over long distances. • TFREM increases attention to local deep detail information. • The differential extraction algorithm is used to obtain optical flow guided feature. • MLENet can extract fine-grained high-level temporal features. • MLENet can be trained end-to-end and it only requires RGB input. [ABSTRACT FROM AUTHOR]

Scene understanding is an important area in robotics and autonomous driving. To accomplish these tasks, the 3D structures in the scene have to be inferred to know what the objects and their locations are. To this end, semantic segmentation and disparity estimation networks are typically used, but running them individually is inefficient since they require high-performance resources. A possible solution is to learn both tasks together using a multi-task approach. Some current methods address this problem by learning semantic segmentation and monocular depth together. However, monocular depth estimation from single images is an ill-posed problem. A better solution is to estimate the disparity between two stereo images and take advantage of this additional information to improve the segmentation. This work proposes an efficient multi-task method that jointly learns disparity and semantic segmentation. Employing a Siamese backbone architecture for multi-scale feature extraction, the method integrates specialized branches for disparity estimation and coarse and refined segmentations, leveraging progressive task-specific feature sharing and attention mechanisms to enhance accuracy for solving both tasks concurrently. The proposal achieves state-of-the-art results for joint segmentation and disparity estimation on three distinct datasets: Cityscapes, TrimBot2020 Garden, and S-ROSeS, using only 1 / 3 of the parameters of previous approaches. [Display omitted] • End-to-end network that learns semantic segmentation and disparity from stereo data. • Progressive learning extracts task-specific features and multi-scale knowledge. • State-of-the-art results for multitasking co-learning of disparity and segmentation. • Competitive results against state-of-the-art single-task methods. • The network solves both tasks with less than 1/3 of the parameters of previous works. [ABSTRACT FROM AUTHOR]

Salient object detection (SOD) is of high significance for various computer vision applications but is a challenging task due to the complicated scenes in real-world images. Most state-of-the-art SOD methods aim to build long-range dependency for improving global contrast modeling in complicated scenes. However, most of them suffer from the prior assumption of treating image patches as visual tokens for building long-range dependency. This is because this assumption leads to localizing salient regions with uncertain boundaries due to the lost object structure information. In this paper, to address this issue, we re-construct the prior assumption of treating both patches and superpixels as visual tokens for building long-range dependency, which takes into account the properties of superpixels and patches in preserving detailed structural-aware information and local context information, respectively. Based on the re-constructed prior assumption, we propose a Collaborative Compensative Transformer Network (CCTNet) for the SOD task. CCTNet firstly alternates the computation within the same kind of vision tokens and among different vision tokens to build their dependencies. By this means, the relationship between multi-level global context and detailed structure representation can be explicitly modeled for consistent semantic and object structure understanding. Then, CCTNet performs feature joint decoding for SOD by fusing the complementary global context and detailed structure for locating objects with certain boundaries. Extensive experiments were conducted to validate the effectiveness of the proposed modules. Furthermore, the experiments on ten benchmark datasets demonstrated the state-of-the-art performance of CCTNet on both RGB and RGB-D SOD. • Re-constructing the vision prior to address complicated scenes in the SOD task. • Modeling relationships between global semantics and object structures explicitly. • Feature joint decoding for predicting salient objects. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) are ubiquitous in computer vision and natural language processing, but suffer from high inference cost. This problem can be addressed by quantization, which consists in converting floating point operations into a lower bit-width format. With the growing concerns on privacy rights, we focus our efforts on data-free methods. However, such techniques suffer from their lack of adaptability to the target devices, as a hardware typically only support specific bit widths. Thus, to adapt to a variety of devices, a quantization method shall be flexible enough to find good accuracy v.s. speed trade-offs for every bit width and target device. To achieve this, we propose PIPE , a quantization method that leverages residual error expansion, along with group sparsity and an ensemble approximation for better parallelization. PIPE is backed off by strong theoretical guarantees and achieves superior performance on every benchmarked application (from vision to NLP tasks), architecture (ConvNets, transformers) and bit-width (from int8 to ternary quantization). • Deep neural networks benefits from quantization for inference. • Quantization techniques offer few trade-offs between speed and accuracy given a target hardware. • Residual expansions of quantized network enable multiple speed and accuracy trade-offs. • PIPE derives multiple networks from a single expanded network. • PIPE leverages ensembles of quantized neural networks. [ABSTRACT FROM AUTHOR]

Shape contour is a key low-level characteristic, making shape description an important aspect in many computer vision problems, with several challenges such as variations in scale, rotation, and noise. In this paper, we introduce an approach for shape analysis and classification from binary images based on representations learned by applying Randomized Neural Networks (RNNs) on feature maps derived from a Complex Network (CN) framework. Our approach models the contour in a complex network and computes their topological measures using a dynamic evolution strategy. This evolution of the CN provides significant information into the physical aspects of the shape's contour. Therefore, we propose embedding the topological measures computed from the dynamics of the CN evolution into a matrix representation, which we have named the Topological Feature Map (TFM). Then, we employ the RNN to learn representations from the TFM through a sliding window strategy. The proposed representation is formed by the learned weights between the hidden and output layers of the RNN. Our experimental results show performance improvements in shape classification using the proposed method across two generic shape datasets. We also applied our approach to the recognition of plant leaves, achieving high performance in this challenging task. Furthermore, the proposed approach has demonstrated robustness to noise and invariance to transformations in scale and orientation of the shapes. • A complex representation for shape classification is proposed. • It is based on combining a complex network model and randomized neural networks. • Shape contours are modeled as complex networks. • Randomized neural networks learn the features from the modeled complex networks. • The Complex Representations are the output weights of the neural network. [ABSTRACT FROM AUTHOR]

Image keypoint detection and feature matching are fundamental steps in computer vision tasks. However, variations in environment, time, and viewpoint pose a challenge to the stability of image keypoint detection and matching. Most traditional and deep learning-based methods cannot accurately and efficiently extract highly repeatable keypoints and robust match pairs in low-luminance environments. Therefore, we propose a two-step 'detection + matching' framework, which consists of deep neural networks in each step. Firstly, we design a self-supervised robust keypoint detection network, which utilizes multi-scale, multi-angle, and multi-luminance transformation techniques to create pseudo-labeled datasets to improve the model's keypoint detection repeatability and luminance invariance. Secondly, we propose a descriptor-free cross-fusion matching network, which uses the cross-fusion attention mechanism to establish connections between keypoint-centered image patches and converts the feature-matching task into an image patch assignment task to improve the accuracy and efficiency of matching. Thirdly, the proposed framework is used to replace traditional SIFT in SfM. Experimental results on testing datasets show that the self-supervised robust keypoint detection network achieves higher keypoint repeatability in low luminance environments compared to SIFT, ORB, LIFT, and Superpoint. The descriptor-free cross-fusion matching network's mean matching accuracy and efficiency are higher than the mainstream Superglue algorithm. Also, SfM achieves better performance regarding the number of sparse point clouds and accuracy. • Propose a keypoint detection and feature-matching algorithm for low-luminance images. • Employ self-supervised methods to improve the robustness of keypoint detection. • Employ the attention mechanism to improve feature matching. • A novel framework for image matching applies to SfM in low-luminance environments. [ABSTRACT FROM AUTHOR]

Pedestrian detection is a crucial field of computer vision research which can be adopted in various real-world applications (e.g., self-driving systems). However, despite noticeable evolution of pedestrian detection, pedestrian representations learned within a detection framework are usually limited to particular scene data in which they were trained. Therefore, in this paper, we propose a novel approach to construct versatile pedestrian knowledge bank containing representative pedestrian knowledge which can be applicable to various detection frameworks and adopted in diverse scenes. We extract generalized pedestrian knowledge from a large-scale pretrained model, and we curate them by quantizing most representative features and guiding them to be distinguishable from background scenes. Finally, we construct versatile pedestrian knowledge bank which is composed of such representations, and then we leverage it to complement and enhance pedestrian features within a pedestrian detection framework. Through comprehensive experiments, we validate the effectiveness of our method, demonstrating its versatility and outperforming state-of-the-art detection performances. • We propose to obtain versatile pedestrian representations for pedestrian detection. • We exploit generalized pedestrian knowledge of a large-scale pretrained model. • We build versatile pedestrian knowledge bank and leverage it in detection framework. • It achieves state-of-the-art performances on four pedestrian detection datasets. [ABSTRACT FROM AUTHOR]

Visibility enhancement of outdoor images under complex imaging conditions has been a crucial task for computer vision and received growing attention. However, existing image enhancement methods could result in typical block-like artifacts or color distortion. The undesirable impurities might also be significantly magnified after the enhancement task, further reducing the image quality. For enhancing and super-resolving complex real-world degradation, we propose a simultaneous visual enhancement and resolution improvement (VERI) variational scene recovery model for jointly enhancing image visibility and improving the resolution of the degraded image. Particularly, we estimate the scattering light map for degradation images to achieve clean scene radiance and simultaneously seek a high-quality image through a deep super-resolution network. The semi-proximal alternating direction method of multipliers (sPADMM) algorithm is employed for efficiently solving the minimization problems in the proposed model. Extensive experiments illustrate the effectiveness and robustness of the proposed method in dealing with various scenes, such as haze, sandstorm, underwater or low illumination. • We develop a model that enhances visuals and resolution for real-world scenarios. • Scattering light map estimation and deep network yield high-quality scene radiance. • Minimization problems in our model are solved with semi-proximal ADMM algorithm. • Method's robustness and effectiveness were validated in diverse scenarios. [ABSTRACT FROM AUTHOR]

Source-Free Unsupervised Domain Adaptation (SFUDA) transfers knowledge from the source domain to the target domain with using the source domain model instead of the source data, as the source data cannot be accessed in data privacy scenarios. Our method is based on a clustering assumption: although there is a domain shift, target data with similar semantic still form a cluster in the source feature space. We identify two levels of clustering. One is class-level neighbor clustering: data with the same label tend to form a large cluster. The other is instance-level neighbor clustering: data and its neighbors tend to share the same label. Previous methods only consider one level, and we consider that both are complementary. We propose a new SFUDA method called unified multi-level neighbor clustering to address class and instance consistency in a complementary way. Our proposed method achieves competitive results on three domain adaptation benchmark datasets. [Display omitted] • An unified and complementary multi-level neighbor clustering method to solve SFUDA problem. • A dot-product loss is used to filter unreliable samples with ambiguous features. • The consistency of the labels of neighbors is used to reduce unconfident samples. [ABSTRACT FROM AUTHOR]

Image clustering is a difficult task with important application value in computer vision. The key to this task is the quality of images features. Most of current clustering methods encounter the challenge. That is, the process of feature learning and clustering operates independently. To address this problem, several researchers have been dedicated to performing feature learning and deep clustering together. However, the obtained features lack discriminability to address high-dimensional data successfully. To deal with this issue, we propose a novel model named as robust deep fuzzy K -means clustering (RD-FKC), which efficiently projects image samples into a representative embedding space and precisely learns membership degrees into a combined framework. Specifically, RD-FKC introduces Laplacian regularization technique to preserve locality properties of data. Moreover, by using an adaptive loss function, the model becomes more robust to diverse types of outliers. Furthermore, to avoid the latent space being distorted and make the extracted features retain the original information as much as possible, the model introduces reconstruction error and adds regularization to network parameters. Finally, an effective algorithm is derived to solve the optimization model. Numerous experiments have been conducted, illustrating the advantages and superiority of RD-FKC over existing clustering approaches. • Embedding clustering into deep convolutional autoencoder to learn compact and representative features. • Designing the Laplacian regularization to constrain the membership matrix for locality preserving. • Introducing the adaptive loss function to enhance the clustering robustness. • Presenting an effective algorithm to optimize the proposed model. [ABSTRACT FROM AUTHOR]

As an important carrier of data, images contain a huge amount of information. The purpose of image fusion is to integrate the information from source images into a single image. Since the source images are from the same scene, there is much redundant information between them. Common fusion methods do not filter this information and the fusion process is often disturbed. This leads to degradation in the quality of the reconstructed fused image. To solve this problem, this paper explores the strategies of information filtering and fusion control, and proposes a universal image fusion method based on the mask attention mechanism. It can be divided into pre-training stage and formal fusion stage. In the pre-training stage, coarse-grained mask maps are generated which are employed to improve the mask autoencoder and the mask attention mechanism. In the image formal fusion stage, with the help of coarse-grained mask maps, the mask autoencoder changes the process of random masking and discards redundant features between source images. Meanwhile the mask attention mechanism focuses on the distinctions between various source images and retains effective complementary information. Qualitative and quantitative extension experiments on different modal datasets validate the applicability of the model in multi-focus image fusion, infrared and visible image fusion, and medical image fusion. Our method achieves excellent performance in all these tasks and performs better than existing fusion methods. Our code is publicly available at https://github.com/xiangxiang-wang/MMAE. • A universal fusion method for generating high-quality fused images is proposed. • Information filtering and fusion control strategy is effective. • Improved mask autoencoder filters redundant information. • The patch attention mechanism corrects the masked regions. • The mask attention mechanism amplifies modal differences and reconstructs images. [ABSTRACT FROM AUTHOR]

Object matching can be viewed as an image patch matching problem, which is widely employed in image fusion, image retrieval and other computer vision fields. In this paper, we regard image matching as a regression and classification task and propose a visible and infrared image matching network named AMFFNet. AMFFNet adopts a Siamese network structure and utilizes a residual network with an attention mechanism to extract features from the input images to obtain feature maps, and fuses these feature maps. AMFFNet then performs classification and regression on the fused feature maps to achieve matching. The classification operation identifies whether the predicted area is the object, while the regression operation determines the coordinates and size of the predicted box. To improve the matching performance of the network, we utilize the center-ness branch in the network and use Generalized Intersection over Union (GIoU) loss during training. We rearrange the existing dataset to provide a sufficient visible–infrared image matching dataset. Experimental results demonstrate that the proposed method achieves superior matching mean Average Precision (mAP) compared to other methods. • A network for matching visible–infrared image is proposed. • Image matching is viewed as a regression and classification task. • The attention module is added to the feature extraction network. • Features of different levels are fused through the feature fusion mechanism. [ABSTRACT FROM AUTHOR]

Super-resolution (SR), aiming to super-resolve degraded low-resolution image to recover the corresponding high-resolution counterpart, is an important and challenging task in computer vision, and with various applications. The emergence of deep learning (DL) has significantly advanced SR methods, surpassing the performance of traditional techniques. This paper presents a comprehensive survey of DL-based SR methods encompassing single image super resolution (SISR) and multiple image super resolution (MISR) methods, along with their applications and limitations. In SISR methods, addressing individual images independently, we review blind and non-blind SR methods. Additionally, within MISR, we delve into multi-frame, multi-view, and reference-based SR methods. DL-based SR methods are categorized from the application perspective and a taxonomy is proposed. Finally, we present research prospects and future directions. • This paper provides a comprehensive survey on deep-learning-based super-resolution methods along with their applications and limitations. • This paper categorizes the methods from an application perspective and a taxonomy is proposed. • The key points of super-resolution are discussed, based on which we make prospects for further research. [ABSTRACT FROM AUTHOR]

Video anomaly detection presents a significant challenge in computer vision, with the aim of distinguishing various anomaly events from numerous normal ones. Weakly supervised video anomaly detection has recently emerged as a promising solution, enabling the detection of anomaly snippets with only video-level annotations. However, knowledge about anomaly annotation remains underutilized, resulting in a gap between visual space and semantic understanding of anomalies, thus failing to capture the clear boundary between anomalies and normalities. Therefore, we propose a weakly supervised paradigm of cross-modal detection and consistency learning, leveraging dual consistency to provide discriminative representations for anomalies at both the semantic-to-target and target-to-snippet levels. Specifically, we introduce a cross-modal detection network, which detects the targets in each frame according to given semantic rules, to derive semantic-consistent visual embeddings. To depict the clear boundary between anomalies and normalities, a cross-domain alignment module is proposed to enhance the discriminative representation of abnormal targets by learning the contextual consistency between the target and snippet embeddings. Our architecture integrates the detection of semantic-consistent targets based on variable semantic rules, ensuring transferable deployment across scenarios and enabling comprehensive identification, localization, and recognition of abnormal events through a "when-where-which" pipeline. The evaluation of our approach is conducted on four widely used public benchmarks: ShanghaiTech, UCSD Ped2, CUHK Avenue, and UBnormal through extensive qualitative and quantitative analyzes. The results demonstrate the remarkable performance of our approach in dealing with the VAD task. • In this paper, we propose a weakly supervised paradigm of cross-modal detection and consistency learning, leveraging dual consistency to provide discriminative representations for anomalies at both the semantic-to-target and target-to-snippet levels. • Specifically, we introduce a cross-modal detection network, which detects the targets in each frame according to given semantic rules, to derive semantic-consistent visual embeddings. • To depict the clear boundary between anomalies and normalities, a cross-domain alignment module is proposed to enhance the discriminative representation of abnormal targets by learning the contextual consistency between the target and snippet embeddings. • Our architecture integrates the detection of semantic-consistent targets based on variable semantic rules, ensuring transferable deployment across scenarios and enabling comprehensive identification, localization, and recognition of abnormal events through a "when-where-which" pipeline. [ABSTRACT FROM AUTHOR]

Surveillance systems are essential in computer vision, with video anomaly detection (VAD) critical for real-world security. Conventional approaches anticipate abnormal frames using normal patterns learned with normal training data and determine anomalies by comparing the output and input. However, they often incur high computational costs due to pre-trained networks and complex modules. Also, they suffer from the tendency to overlook anomalies due to strong generalizing ability. To address these, we propose two innovations: Anomaly Distance Learning (ADL) and Context-Aware Skip Connection (CONASkip), maximizing the distinction between normal and abnormal samples. ADL increases the feature gap by learning to separate normal and abnormal features, while CONASkip selectively connects layers to preserve normal information and differentiate output quality. These methods operate on self-supervised signals, replacing complex and costly modules. Additionally, ADL is excluded during testing, enabling fast inference. Our model achieves 130 frames per second and superior performance on three benchmarks, addressing real-world efficiency challenges. [Display omitted] • Patch anomaly generation enforces normality learning, in terms of appearance and motion. • Anomaly distance learning enlarges the feature distance of normal and abnormal frames. • Context-aware shortcut widens the quality gap between outputs for normal and abnormal samples. • Our method is fast, accurate, and free from pre-trained networks. [ABSTRACT FROM AUTHOR]

• We propose a new fractional-order moments named as FRSBFMS. • We integrate FRSBFMS and quaternion theory to develop QFrSBFMs. • The rotation invariance and parameter α are discussed in detail. Moment and moment invariants as effective feature descriptors have been widely applied in image analysis, pattern recognition and computer vision applications. Scholars have demonstrated that fractional-order orthogonal moments outperform their corresponding moments in representing the fine details of a given image. To this end, we propose a new fractional-order orthogonal moments based on fractional-order weighted spherical Bessel polynomial in this paper, which is named as Fractional-order weighted Spherical Bessel-Fourier Moments (FrSBFMs). Moreover, to address the issue of color image analysis, FrSBFMs are integrated with quaternion theory to develop Quaternion FrSBFMs (QFrSBFMs). In addition, the rotation invariance and parameter α analysis of FrSBFMs and QFrSBFMs are discussed in detail. The experimental results show the effectiveness of the proposed FrSBFMs and QFrSBFMs in terms of image reconstruction capability, pattern recognition accuracy and zero-watermark verification. [ABSTRACT FROM AUTHOR]

Local and sparse attention effectively reduce the high computational cost of global self-attention. However, they suffer from non-global dependency and coarse feature capturing, respectively. While some subsequent models employ effective interaction techniques for better classification performance, we observe that the computation and memory of these models overgrow as the resolution increases. Consequently, applying them to downstream tasks with large resolutions takes time and effort. In response to this concern, we propose an effective backbone network based on a novel attention mechanism called Concatenating glObal tokens in Local Attention (COLA) with a linear computational complexity. The implementation of COLA is straightforward, as it incorporates global information into the local attention in a concatenating manner. We introduce a learnable condensing feature (LCF) module to capture high-quality global information. LCF possesses the following properties: (1) performing a function similar to clustering, aggregating image patches into a smaller number of tokens based on similarity. (2) a constant number of aggregated tokens regardless of the image size, ensuring that it is a linear complexity operator. Based on COLA, we build COLAFormer, which achieves global dependency and fine-grained feature capturing with linear computational complexity and demonstrates impressive performance across various vision tasks. Specifically, our COLAFormer-S achieves 84.5% classification accuracy, surpassing other advanced models by 0.4% with similar or less resource consumption. Furthermore, our COLAFormer-S can achieve a better object detection performance while consuming only 1/4 of the resources compared to other state-of-the-art models. The code and models will be made publicly available. • We introduce a novel backbone called COLAFormer, which is based on Concatenating global tokens in Local Attention (COLA). COLA communicates between local attention and sparse attention following a concatenating manner. Additionally, it achieves linear complexity and can significantly contribute to downstream tasks. • In order to provide high-quality global information for COLA, we introduce an effective downsampling module called learnable condensing feature (LCF) module. LCF can perform a function similar to clustering and downsample tokens. • Extensive experiments across various vision tasks demonstrate that our COLAFormer achieves impressive classification accuracy on ImageNet1K and strikes a balance between performance and resource consumption on downstream tasks at large input resolutions such as object detection. [ABSTRACT FROM AUTHOR]

• Introduction: This paper proposes an enhanced AOD-Net for real-time dehazing, aimed at improving intelligent ship navigation in complex haze environments. • Hybrid Dilated Convolution (HDC): The dehazing network uses HDC to expand the receptive field and improve feature extraction without adding computational overhead. • Attention Mechanism: The ECPA module is designed to enhance dehazing by focusing on varying haze densities, improving detail retention and network performance. • Mixing Loss Function: A mixing loss function is designed to address dimness and color distortion post-dehazing, enhancing image restoration quality. • Experimental Results and Conclusion: Experiments show superior visual quality with highest PSNR and SSIM on synthetic datasets, and excellent performance in real-world conditions. As an essential preprocessing step in computer vision tasks, image dehazing is preparatory work in many practical applications. To ensure the navigation safety of intelligent ships in complex haze weather, this paper proposes a lightweight real-time dehazing method for the natural environment based on an all-in-one dehazing network. Specifically, a hybrid dilated convolution is used to construct a dehazing network, which effectively expands the receptive field and improves the ability to extract characteristic information without increasing the calculation amount of the model. Secondly, a hybrid attention module is developed to enhance the model's ability to remove haze while retaining more detailed features by giving weights to light haze and heavy haze areas in the image. Finally, considering the subjective perception of human eyes, a mixed loss function is designed to address image dimness and color distortion after dehazing. Through experiments on both synthetic and real-world haze datasets, the proposed method attains ideal dehazing results and has high real-time performance. It can dehaze under different haze conditions, address the critical challenge of degraded visual perception in severe weather conditions of intelligent ship navigation on inland rivers in complex environments, and promote intelligent ships' development. [ABSTRACT FROM AUTHOR]

Edge detection plays a fundamental role in computer vision tasks and gains wide applications. In particular, semantic edge detection recently draws more attention due to the high demand for a fine-grained understanding of visual scenes. However, detecting high-level semantic edges hidden in visual scenes is quite challenging. Existing semantic edge detection methods focus on category-aware semantic edges and require elaborate category annotations. Instead, we first propose the category-agnostic semantic edge detection task without additional semantic category annotations. To achieve this goal, we propose to utilize only edge position annotations and leverage the information randomness of semantic edges. Specifically, we align semantic edge positions to the ground truth by maximizing randomness on edge regions and minimizing randomness on non-edge regions in the training process. In the inference process, we first obtain neural representations by the trained network, and then generate semantic edges by measuring neural randomness. We evaluate our method by comparisons with alternative methods on two well-known datasets: Cityscapes (Cordts et al., 2016) and SBD (Hariharan et al., 2014). The results demonstrate our superiority over the alternatives, which is more significant under weak annotations. We also provide comprehensive mechanism studies to verify the generalizability, rationality, and validity of our working mechanism. [Display omitted] • We first propose the category-agnostic semantic edge detection task. • We reveal the intrinsic information properties of semantic edges. • We propose a two-stage method under the guidance of information randomness metrics. • We provide thorough comparisons and mechanism studies to prove our superiority. [ABSTRACT FROM AUTHOR]

Noise modeling is an important research field in computer vision; the design of an accurate model for imaging sensor noise depends on not only a comprehensive benchmark dataset of the real world, but also a precise design of the noise modeling algorithm. However, due to the inaccurate estimation method of noise-free images and limited shooting scenes, the current realistic datasets could not describe the diverse noise properties sufficiently. Moreover, popular parametric noise models are not sophisticated enough to characterize the real-world noise exactly. In this work, we first construct a more comprehensive dataset of the real world by capturing more indoor and outdoor scenes under different lighting conditions using diverse smartphones, then we propose a non-parametric noise estimation method capable of modeling the spatial heterogeneity of real-world noise patterns. Specifically, in order to characterize the spatial heterogeneity of real-world noise, we assume a non-i.i.d Gaussian distribution and propose a deep convolutional neural network (DCNN)-based approach for learning pixel-wise noise variance maps. To learn the pixel-wise variance map, we have constructed a variance estimation network mapping from the conditional signals (clean image, ISO, and camera model) to surrogate labels obtained from the nonlocal search of similar patches from the clean-noisy image pair. Finally, we conducted denoising and classification experiments using different kinds of simulated noisy images, compared to the Poisson-Gaussian and Noise Flow noise models, the proposed method achieves denoising performance improvements (PSNR) of 1.13 dB and 2.51 dB respectively on the proposed real-world test dataset, denoising and classification results on the real noisy data captured by mobile phones have verified that our approach is more accurate than current noise modeling methods. • Current realistic datasets could not describe diverse noise properties sufficiently. • Parametric noise models cannot characterize the real-world noise exactly. • Constructing a more comprehensive real-world benchmark dataset. • Developing a pixel-wise variance estimation network. • Improved real-world denoising and classification performance. [ABSTRACT FROM AUTHOR]

In this paper, we introduce the single image glare removal (SIGR) task. SIGR aims to eliminate glare or light caused by external environment in lighting scene. To efficiently improve natural image quality and usability, many research tasks, such as image deraining and shadow removal, have been investigated a lot. However, SIGR is still underexplored. Therefore, we propose to construct a dataset and explore deep learning-based models for SIGR. Our contributions can be summarized as follows: (1) We establish a new benchmark dataset for SIGR, termed De-Glare, aiming to propel research of SIGR. This dataset comprises pairs of {glare, glare-free} images sourced from both real-world and synthetic data, utilized for training and evaluating models. (2) We conduct a comprehensive benchmarking of extensive state-of-the-art (SoTA) methods on the constructed De-Glare dataset and provide insightful analyses based on the results. (3) An innovative approach for single image glare removal employing a multi-scale generative adversarial network (GR-GAN) model is proposed. Glare images typically exhibit irregular glare shapes and cluttered backgrounds. To address these irregular glare patterns, we introduce a deformable convolution-based glare attention detector (GAD) designed to generate an attention map which specifics glare spots or rays in the input image. In pursuit of enhancing the perceptual quality of output image, GR-GAN adaptively filters out irrelevant noises and enhances salient features through a generator with cascaded pyramid neck (CPN) network. This work can provide useful insights for developing better SIGR models. Without specific tuning, our method achieves the SoTA results on multiple computer vision tasks, including the image deraining and image shadow removal. • A new image denoising task is introduced, focusing on single image glare removal, and a corresponding dataset is constructed. • A novel adversarial generative network is proposed for single image glare removal. • It aggregates multiscale spatial and semantic visual features, enhancing the quality of the generated images. • Extensive qualitative and quantitative experiments on four types image denoising benchmark datasets demonstrate the superiority of the proposed method over state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The propagation of light through turbulent media results in refractive index fluctuations, causing distortion and blurring of images captured by imaging equipment. Consequently, it is of paramount importance to mitigate turbulence effects. Deep convolutional neural networks incorporating attention mechanisms have demonstrated remarkable success in dynamic video restoration. However, in most networks, the attention mechanism is limited to capturing simple contextual features, failing to adequately exploit multi-level image features. This paper introduces IDSSI, an effective model that utilizes semantic and spatio–temporal information for turbulence mitigation. A novel two-branch feature extraction structure is proposed, capable of extracting multi-scale enhanced features with semantic information under limited supervision. These perceived semantic features are subsequently fused into global features, enhancing multi-scale perception for turbulence repair. Furthermore, a new Spatial–Temporal feature learning strategy is proposed. This approach facilitates the extraction and modulation of temporal information by obtaining edge cues, effectively concatenating and merging with spatial features. This strategy serves as an efficient alternative to 3D convolution. Experimental results on relevant datasets demonstrate the superiority of the proposed model over current state-of-the-art turbulence repair methods. • We propose a Two-stage feature extraction structure to extract semantic features and global features. This work is trying to introduce semantic information to low computer vision tasks. • We propose a temporal feature modulation module, which effectively replaces 3D convolution, overcomes information loss, and reduces computation. • Extensive experiment results on both synthetic and real world video deturbulence benchmarks demonstrate the effectiveness of IDSSI. [ABSTRACT FROM AUTHOR]

• We employ U-Net as the framework and refine the generated point cloud features using a cross-attention mechanism at multiple-resolutions. It first generates the object's contour, and then the complete point cloud with rich texture features using the improved Foldingnet (FoldingNet++). • We propose the voting-based feature downsampling module (VFDS) for disordered point cloud feature vectors. • We propose the Generation-based feature upsampling module (GFUS) for unordered point cloud feature vector. In the field of 3D vision, 3D point cloud completion is a crucial task in many practical applications. Current methods use Transformer's Encoder-Decoder framework to predict the missing part of the point cloud features at low resolution, which does not fully utilize the feature information at multiple resolutions and can result in the loss of the object's geometric details. In this paper, we present a novel point cloud completion method, CarvingNet, which, to the best of our knowledge, is the first to apply the U-Net architecture to the point cloud completion task by operating directly on unordered point cloud features at multiple resolutions. Firstly, we gradually expand the receptive field and use cross-attention to purify the features of the missing part of the point cloud at each resolution and to generate the contour features of the complete point cloud at the last obtained resolution. Then, we gradually reduce the receptive field and use cross-attention to refine the features of the complete point cloud at each resolution and to generate the features of the complete point cloud with rich details at the last obtained resolution. To obtain point cloud features at different resolutions, we specifically design the up-sampling module and down-sampling module for disordered point cloud features. Furthermore, we improve the FoldingNet network to make it more suitable for generating high-quality dense point clouds. The experimental results demonstrate that our proposed CarvingNet achieves the performance of existing state-of-the-art methods on the ShapeNet-55, ShapeNet-34, and KITTI benchmarks. [ABSTRACT FROM AUTHOR]

Shadow detection and removal are important yet challenging computer vision tasks. Existing methods simultaneously contend with the brightness and color information of input image while treating different regions of input images equally. We argue that these operations fail to effectively extract the relationship between shadow and non-shadow regions. To relieve these problems, this paper proposes a shadow-aware decomposed transformer network for shadow detection and removal. The network decomposes the input image into brightness and color maps using its bright channel, which are concatenated with the original image as the input data, enabling the network to pay balanced attention to the brightness and color information when calculating the relationship between regions. Additionally, given that the correlation matrix of the transformer measures the relative dependency between regions, the proposed shadow-aware transformer block can extract the relationship between shadow and non-shadow regions more effectively by retaining the specific elements of the correlation matrix. We conduct extensive experiments on three shadow detection benchmark datasets and two shadow removal benchmark datasets. Experimental results show that the proposed method performs favorably against state-of-the-art methods. Codes have been made available at https://github.com/XIAOWANG914/SADT. • The brightness and color are essential for shadow detection and removal. • Performance of network declines when different regions are treated equally. • The input is decomposed into brightness and color map. • The network focuses on the relationship between shadow and non-shadow regions. [ABSTRACT FROM AUTHOR]

Sketch-based 3D shape retrieval (SBSR) is an active research area in the computer vision community, but it is still very challenging. One main reason is that existing deep learning-based methods usually treat sketches as 2D images, neglecting the sparsity and diversity. In this paper, we propose a novel Dual-level Dual-scale Graph Learning (D2GL) method to effectively enhance structural information and produce robust representations for sparse and diverse hand-drawn sketches. Specifically, in addition to the traditional branches for SBSR, we introduce a Dual-level Dual-scale Graph Self-attention (DLDS-GSA) as an auxiliary branch. DLDS-GSA further consists of two levels of encoders, i.e., a local structural encoder and a dual-scale global structural encoder, to capture both local discriminative and multi-scale global structures while minimizing the impact of various sketch drawing details. Comprehensive experiments on SHREC'13 and SHREC'14 datasets demonstrate the superiority of D2GL for SBSR, with extended experiments on PART-SHREC'14 confirming its generalization for unseen classes in SBSR. • Designing a network architecture incorporates the DLDS-GSA as an auxiliary network. • Designing a dual-level module DLDS-GSA for sketch representation. • The results on different datasets demonstrate the superiority of DLDS-GSA. [ABSTRACT FROM AUTHOR]

Domain Generalization (DG) studies the capability of a deep learning model to generalize to out-of-training distributions. In the last decade, literature has been massively filled with training methodologies that claim to obtain more abstract and robust data representations to tackle domain shifts. Recent research has provided a reproducible benchmark for DG, pointing out the effectiveness of naive empirical risk minimization (ERM) over existing algorithms. Nevertheless, researchers persist in using the same outdated feature extractors, and little to no attention has been given to the effects of different backbones yet. In this paper, we go "back to the backbones", proposing a comprehensive analysis of their intrinsic generalization capabilities, which so far have been overlooked by the research community. We evaluate a wide variety of feature extractors, from standard residual solutions to transformer-based architectures, finding an evident linear correlation between large-scale single-domain classification accuracy and DG capability. Our extensive experimentation shows that by adopting competitive backbones in conjunction with effective data augmentation, plain ERM outperforms recent DG solutions and achieves state-of-the-art accuracy. Moreover, our additional qualitative studies reveal that novel backbones give more similar representations to same-class samples, separating different domains in the feature space. This boost in generalization capabilities leaves marginal room for DG algorithms. It suggests a new paradigm for investigating the problem, placing backbones in the spotlight and encouraging the development of consistent algorithms on top of them. The code is available at https://github.com/PIC4SeR/Back-to-Bones. • We propose a novel definition of backbone for DG as a function of three key elements. • We present a new benchmark to compare the latest image classification models on DG. • We find a correlation between large-scale classification accuracy and DG performance. • We prove that DG algorithms do not beat ERM when using state-of-the-art backbones. • We inspect backbones and highlight the effect of DG training on extracted features. [ABSTRACT FROM AUTHOR]

• Based on the idea of collaborative learning, we propose a robust end-to-end framework CL-TransFER for facial expression recognition. • Our CL-TransFER co-trains a CNN feature extractor and a transformer feature extractor jointly, which can extract both rich local semantic features as well as global structural information from facial images. In addition, the consensus between the two networks can also alleviate the impact of noisy annotations. • A simple yet efficient SSMR pre-training task is constructed to enhance the representation learning ability of the transformer feature extractor, so as to extract fine-grained discriminative expression features in subsequent FER task. • Experimental results on three public benchmarks demonstrate the advancement of our method. Facial expression recognition (FER) has attracted intensive attention due to its critical role in various computer vision tasks. However, existing FER approaches suffer from either noisy annotations or expression ambiguity (high inter-class and low intra-class similarity), limiting the FER performance. To this end, we propose a robust end-to-end collaborative learning based transformer for FER (CL-TransFER) in this paper. Specifically, CL-TransFER co-trains a CNN feature extractor and a transformer feature extractor jointly to extract both rich local semantic features as well as global structural information from facial images. By enforcing the consensus between the predictions of two extractors, the CL-TransFER could suppress the influence of noisy annotations. To further tackle the expression ambiguity problem, we design a simple yet efficient self-supervised masked reconstruction (SSMR) task to pre-train the transformer feature extractor of CL-TransFER. This enhances the model's capability of learning fine-grained discriminative representations. Extensive experiments on three popular benchmarks have demonstrated the effectiveness and superiority of our method. [ABSTRACT FROM AUTHOR]

As an unnoticed specialized task in image restoration, rain-by-snow weather removal aims to eliminate the complicated coexisting rain streaks and snow particles. In this work, we propose a simple attention-based sampling cross-hierarchy Transformer (SASCFormer). Initially, we explore the proximity of convolution network and Transformer in hierarchical architectures and experimentally find they perform approximately for intra-stage feature representation. On this basis, we utilize a Transformer-like convolution block (TCB) to replace the computation-heavy self-attention while preserving the attention characteristics for adapting to the input content. Meanwhile, we demonstrate that cross-stage sampling progression is critical for the performance improvement in rain-by-snow weather removal, and propose a global–local self-attention sampling mechanism (GLSASM) that samples the features while preserving both the global and local dependencies. Finally, we synthesize two novel rain-by-snow weather-degraded benchmarks, RSCityscapes and RS100K datasets. Extensive experiments verify that our proposed SASCFormer achieves the best trade-off between the performance and inference time. In particular, our approach advances existing methods by 1. 14 dB ∼ 4. 89 dB in peak signal-to-noise ratio. Related resources are available at https://github.com/chdwyb/Rain-by-snow. • We first focus on rain-by-snow weather-degraded image restoration. • We propose a simple attention-based sampling cross-hierarchy Transformer. • We demonstrate that cross-stage progression is crucial for performance improvement. • Two novel rain-by-snow datasets are synthesized for rain-by-snow weather removal. [ABSTRACT FROM AUTHOR]

Lightweight network design is crucial for optimizing speed and accuracy in computer vision tasks on mobile platforms with limited resources. Widely adopted models, such as EfficientNet and RegNet have achieved significant success by integrating key elements like Pointwise Convolutions (PWConvs) and Squeeze-and-Excitation (SE) blocks. However, a notable observation is that the output feature of the PWConv closely resembles its input, particularly in the absence of an activation function. This similarity and redundancy lead to wasted computational complexity and adversely affect the inference speed. To address these issues, we propose an efficient lightweight network called Efficient Feature-Multiplex Group Network (FMGNet). FMGNet is composed of two key components: the Cross-layer Feature-multiplex Group (CFG) block and the CFG-aligned Cross-layer Attention (CCA) block. The CFG block enables more compact feature learning with fewer parameters by multiplexing the input features of the PWConv. Meanwhile, the CCA block leverages the pre-modified features derived from the CFG block's PWConv, allowing for simultaneous and parallel channel attention modeling. Our extensive experiments across various tasks, including image classification (ImageNet), object detection (PASCAL VOC), human pose estimation (MPII), person re-identification (Market-1501, DukeMTMC-ReID, CUHK03-NP), and semantic segmentation (Cityscapes), indicate that FMGNet achieves comparable performance to state-of-the-art lightweight convolutional neural networks, offering faster inference times. Remarkably, FMGNet even surpasses recent transformer-based models, such as SwiftFormer and EfficientFormerV2, achieving superior results with lower inference latency. • We propose a novel CFG block that effectively utilizes the feature redundancy before and after the PWConv without any accompanying activation function. And we introduce a more efficient attention mechanism, the CCA block, which is compatible with the CFG block. • We build a more efficient lightweight network FMGNet based on CFG block and CCA block. FMGNet achieves comparable performance with state-of-the-art models in image classification and even outperforms transformer-based models with lower latency. • Extensive experiments on wide tasks including object detection, human pose estimation, person re-identification, and semantic segmentation also indicate that our network is an excellent backbone. [ABSTRACT FROM AUTHOR]

Few-shot image classification is a challenging topic in pattern recognition and computer vision. Few-shot fine-grained image classification is even more challenging, due to not only the few shots of labelled samples but also the subtle differences to distinguish subcategories in fine-grained images. A recent method called task discrepancy maximisation (TDM) can be embedded into the feature map reconstruction network (FRN) to generate discriminative features, by preserving the appearance details through reconstructing the query image and then assigning higher weights to more discriminative channels, producing the state-of-the-art performance for few-shot fine-grained image classification. However, due to the small inter-class discrepancy in fine-grained images and the small training set in few-shot learning, the training of FRN+TDM can result in excessively flexible boundaries between subcategories and hence overfitting. To resolve this problem, we propose a simple scheme to amplify inter-class discrepancy and thus improve FRN+TDM. To achieve this aim, instead of developing new modules, our scheme only involves two simple amendments to FRN+TDM: relaxing the inter-class score in TDM, and adding a centre loss to FRN. Extensive experiments on five benchmark datasets showcase that, although embarrassingly simple, our scheme is quite effective to improve the performance of few-shot fine-grained image classification. The code is available at https://github.com/Airgods/AFRN.git. • We propose AFRN, a simple scheme to amplify inter-class discrepancy. • We relax the inter-class score in TDM to mitigate the negative impact of overfitting. • We incorporate the guidance of the centre loss to FRN to enhance the discriminative power of learnt features. • The experimental results demonstrate that our scheme is simple yet effective to improve the few-shot classification performance. [ABSTRACT FROM AUTHOR]

Capturing feature information effectively is of great importance in the field of computer vision. With the development of convolutional neural networks, concepts like residual connection and multiple scales promote continual performance gains in diverse deep learning vision tasks. In this paper, novel residual feature-reutilization inception and split-residual feature-reutilization inception are proposed to improve performance on various vision tasks. It consists of four parallel branches, each with convolutional kernels of different sizes. These branches are interconnected by hierarchically organized channels, similar to residual connections, facilitating information exchange and rich dimensional variations at different levels. This structure enables the acquisition of features with varying granularity and effectively broadens the span of the receptive field in each network layer. Moreover, according to the network structure designed above, split-residual feature-reutilization inceptions can adjust the split ratio of the input information, thereby reducing the number of parameters and guaranteeing the model performance. Specifically, in image classification experiments based on popular vision datasets, such as CIFAR10 (97.94%), CIFAR100 (85.91%), Tiny Imagenet (70.54%) and ImageNet (80.83%), we obtain state-of-the-art results compared with other modern models under the premise that the models' sizes are approximate and no additional data is used. • A novel multi-scale convolutional neural network architecture residual feature-reutilization inception and split-residual feature-reutilization inception are proposed to fully utilize features from different scales and enlarge the receptive fields. • Split-residual feature-reutilization inception reduces parameter amount and Flops with acceptable accuracy loss by dividing features into multiple groups with ratios referring to the setting of channel number of residual feature-reutilization inception. [ABSTRACT FROM AUTHOR]

Recently, appearance-based gaze estimation has been attracting attention in computer vision, and remarkable improvements have been achieved using various deep learning techniques. Despite such progress, most methods aim to infer gaze vectors from images directly, which causes overfitting to person-specific appearance factors. In this paper, we address these challenges and propose a novel framework: Stochastic subject-wise Adversarial gaZE learning (SAZE), which trains a network to generalize the appearance of subjects. We design a Face generalization Network (Fgen-Net) using a face-to-gaze encoder and face identity classifier and a proposed adversarial loss. The proposed loss generalizes face appearance factors so that the identity classifier inferences a uniform probability distribution. In addition, the Fgen-Net is trained by a learning mechanism that optimizes the network by reselecting a subset of subjects at every training step to avoid overfitting. Our experimental results verify the robustness of the method in that it yields state-of-the-art performance, achieving 3.89°and 4.42°on the MPIIFaceGaze and EyeDiap datasets, respectively. Furthermore, we demonstrate the positive generalization effect by conducting further experiments using face images involving different styles generated from the generative model. • We suggest adversarial training for gaze estimation to generalize facial appearances. • We propose a stochastic strategy for subject-wise training to improve generalization. • Our framework achieves state-of-the-art performance on the MPIIFaceGaze and Eyediap. [ABSTRACT FROM AUTHOR]

Fast and efficient fitting of accurate ellipses from data points has many applications in pattern recognition, machine vision, and robotics. However, the fitting accuracy may significantly degrade in the existence of outliers, such as the least-squares-based approaches. Despite robust methods attaining more accurate results than the least-squares manner under the contamination of outliers, they typically require the careful tuning of the hyper-parameters for good results. To mitigate the outlier disturbance, in this paper, we propose a conceptually simple yet quite useful preprocessing framework for high-precision ellipse fitting. Firstly, we leverage multi-scale operators to shrink the input image, by which a large number of outliers can be removed, followed by the smoothing of the sub-image to further improve the data quality. Then, we propose a key-point searching method to enhance the fitting precision via the analysis of the discrete pixel data in images. We prove that key-point-based ellipse fitting gives the upper bound of the approximation error generated by other sampled points with the same ellipse. Based on the key-point pairs inside and outside the ellipse, we further calculate their barycentric points and then perform fitting on these points to attain high-precision ellipses. We conduct extensive experiments on synthetic and real-world images to validate the proposed method and compare it with representative state-of-the-art approaches. Quantitative and qualitative results demonstrate that our method has more accurate and robust performance than competitors. Additionally, we employ the proposed method to compared approaches as a preprocessing step. Experiments demonstrate that our method is effective to significantly improve their fitting accuracy. Our source code is freely available at https://github.com/ChengQian09/MSKPF. • A simple yet quite useful method to improve the ellipse fitting performance. • A multi-scale smoothing module to remove the outlier disturbance. • A key-point searching technique to enhance the fitting accuracy. • Extensive experiments are implemented to demonstrate our method's advantages. [ABSTRACT FROM AUTHOR]

Template matching is a fundamental problem in computer vision with applications in fields including object detection, image registration, and object tracking. Current methods rely on nearest-neighbour (NN) matching, where the query feature space is converted to NN space by representing each query pixel with its NN in the template. NN-based methods have been shown to perform better in occlusions, appearance changes, and non-rigid transformations; however, they scale poorly with high-resolution data and high feature dimensions. We present an NN-based method that efficiently reduces the NN computations and introduces filtering in the NN fields (NNFs). A vector quantization step is introduced before the NN calculation to represent the template with k features, and the filter response over the NNFs is used to compare the template and query distributions over the features. We show that state-of-the-art performance is achieved in low-resolution data, and our method outperforms previous methods at higher resolution. • Existing methods scale poorly with high-resolution images and high-dimensional features. • Vector quantization in the template features is used to reduce nearest neighbour computations. • Filtering is introduced in the nearest neighbour space to encode spatial information. • State-of-the-art results are generated in runtime and performance for high-resolution datasets. [ABSTRACT FROM AUTHOR]

Stereo matching is a fundamental research area in the field of computer vision. In recent years, iterative methods based on Gated Recurrent Units (GRUs) have showcased remarkable achievements in this domain. Despite their high accuracy, these methods suffer from notable limitations such as a reliance on a large number of iterations and a tendency to lose high-frequency details. To address these issues, we propose a novel uncertainty-aware framework that combines 3D convolution and GRU-based iterations, aiming to improve efficiency and accuracy. Specifically, we first introduce a probabilistic method to jointly train the disparity map and its corresponding uncertainty map using 3D convolutions. Next, leveraging the uncertainty map as a guide, we develop a novel uncertainty reweighting iterative module to assist in identifying errors in the coarse disparity and cost volume, thereby refining the disparity estimation process and significantly improving the iteration efficiency. Moreover, we introduce a high-resolution refinement module that utilizes Pixel Difference Convolution (PDC) to incorporate additional gradient information. This module can fine-tune the disparity estimation to enhance accuracy. Finally, our network is evaluated on multiple widely-used benchmark datasets. The results demonstrate its proficiency in predicting precise boundaries and effectively reduce iterations. Our model achieves comparable performance to other state-of-the-art methods, ranking 1st on KITTI 2015, and 2nd on KITTI 2012. These results validate its strong performance and generalizability. • We propose an innovative uncertainty-based framework with coarse-to-fine manner that produces more accurate disparity with less iterations. • We introduce a high-resolution Pixel Difference Convolution (PDC)-based refinement module to recover more high-frequency detail information. [ABSTRACT FROM AUTHOR]

• A novel local feature descriptor called FApSH is proposed. It fully encodes the relevancy of the five attributes at each neighbor point based on an LRA, presenting strong robustness and high descriptiveness. • A priori distribution-based partition method is proposed, which can significantly reduce the length and also validly improve the descriptiveness of FApSH. • A radial distance-based histogram assignment method is proposed, which can enhance the robustness of FApSH. • The priori distribution-based partition and radial distance-based histogram assignment methods can be applied to other descriptors for improving their performance. Three-dimensional (3D) local feature descriptor plays an important role in 3D computer vision because it is widely used to build point-to-point correspondences in many 3D vision applications. However, existing descriptors are difficult to have both high descriptiveness and strong robustness to various nuisances (e.g., noise and occlusion). To address this problem, a descriptor named Fully Attribute-pairs Statistical Histogram (FApSH) is proposed. FApSH is constructed on a local reference axis (LRA), and fully encodes the relevancy information of five attributes at each neighbor point by ten attribute-pair statistics. In this process, a priori distribution-based partition strategy is proposed for evenly distributing the attribute values of all neighbor points, and a radial distance-based histogram assignment method is proposed to improve the robustness to noise and outliers. The proposed methods are rigorously evaluated on six benchmark datasets with different application scenarios and nuisances. The results show that FApSH has high descriptiveness and strong robustness. It obviously outperforms the existing handcrafted descriptors, and is comparable to some superior learning-based descriptors. The results also show that the proposed priori distribution-based partition strategy significantly reduces the length and also improves the descriptiveness of FApSH, and the proposed radial distance-based histogram assignment method improves the robustness of FApSH on various datasets. [ABSTRACT FROM AUTHOR]

Current methods aggregate multi-level features or introduce auxiliary information to get more refined saliency maps. However, little attention is paid to how to obtain complete salient objects in cluttered background. To address this problem, we propose an integrity segmentation network (ISNet) with a novel detection paradigm that first separates the targets completely and then segment them finely. Specifically, the ISNet architecture consists of a target separation (TS) branch and an object segmentation (OS) branch, trained using a hierarchical difference-aware (HDA) loss. The TS branch equipped with a fractal structure is utilized to produce saliency features with expanded boundary (SF w/ EB), which can enlarge the difference of border details to separate the target from background completely. Compared with the edge and skeleton information, the SF w/ EB contains a more complete structure, which can supplement the defect of salient objects. The OS branch is leveraged to generate complementary features, which gradually integrates the SF w/ EB and aggregated features to segment complete saliency maps. Moreover, we propose the HDA loss to further improve the structural integrity of prediction, which hierarchically assigns weight to pixels according to their differences. Hard pixels will be given more attention to discriminate the similar parts between foreground and background. Comprehensive experimental results on five datasets show that the proposed ISNet outperforms the state-of-the-art methods both quantitatively and qualitatively. Concretely, compared with three typical models, the average gain percentage reaches 2.6% in terms of F β , S m and MAE on two large complex datasets. The improvements demonstrate that the proposed ISNet are beneficial for improving the integrity of prediction. Besides, the ISNet is efficient and runs at a real-time speed of 39.5 FPS when processing an image with size of 320 × 320. Furthermore, the proposed model has better generalization, which can also be applied to other vision tasks to handle complex scenes. Codes are available at https://github.com/lesonly/ISNet. • We propose an integrity segmentation network for salient object detection, which can integrate the expanded saliency features and aggregated ones to generate more complete but less noisy saliency maps. • It is a novel detection paradigm that involves a process from separation to segmentation, thereby enhancing the integrity of prediction. • We design a loss function to further improve the integrity and details of salient objects by focusing on the hard pixels and suppressing the easy pixels. • Experimental results on five datasets demonstrate that our method outperforms state-of-the-art ones both in quantitative and qualitative evaluations. • The proposed model is efficient and has advantages in generalization, which can be utilized to other vision tasks for dealing with complex scenes. [ABSTRACT FROM AUTHOR]

Noise introduced due to weather can reduce the efficiency of computer vision applications as the visibility of the objects in images is greatly affected. Haze and rain are the most common weather conditions seen in nature. However, most of the algorithms found in the literature apply rain and haze removal approaches separately. To this end, in this paper, we propose a novel Wavelet-based deep Auto-encoder, called WAE, for simultaneously removing the haze and rain effects from images. The proposed network uses wavelet transformation and inverse wavelet transformation as an alternative to down-sampling and up-sampling operations, respectively, in order to add sparsity to the network. By training the model on both spatial and frequency domains, it learns non-stationary features that are found to be useful to remove haze and rain effects from images. The proposed model is tested on several rain and haze-affected image datasets, and it performs well in terms of standard evaluation metrics like structural similarity index measure and peak signal-to-noise ratio. The code can be found at : https://github.com/asfakali/WAE.git. • WAE, a new wavelet based Auto-Encoder for both haze and rain removal. • The network used wavelet transformation as an alternative to the upsampling and downsampling. • The model has the ability to localize features in both time and frequency domains. • The proposed model achieves state-of-the-art performance, on rain and haze datasets. [ABSTRACT FROM AUTHOR]

This paper introduces TrpViT, a novel triple attention vision transformer that efficiently captures both local and global features. The proposed architecture tackles global information acquisition by employing three complementary attention mechanisms in a unique attention block: Window, Dilated, and Channel attention. This attention block extracts spatially local features while expanding the receptive field to capture richer global context. By integrating this attention block with convolution, a new C-C-T-T architecture is formed. We rigorously evaluate TrpViT, demonstrating state-of-the-art performance on various computer vision tasks, including image classification, 2D and 3D object detection, instance segmentation, and low-level image colorization. Notably, TrpViT achieves strong accuracy across all parameter scales, highlighting its computational efficiency and effectiveness. • A Triple Attention Vision Transformer captures both global and local features. • TrpViT integrates convolution into transformer to provide induction bias. • TrpViT compensates non-local information using multiple complementary attentions. • TrpViT achieves state-of-the-art results across both high-level and low-level tasks. [ABSTRACT FROM AUTHOR]

Deep learning techniques have been widely used for two-dimensional (2D) and three-dimensional (3D) computer vision problems, such as object detection, super-resolution (SR) and classification to name a few. Radar images suffer from poor resolution as compared to optical images, hence developing a high-accuracy model to solve computer vision problems, such as a classifier, is a challenge. This is because of the lack of high-frequency details in the input images which makes it difficult for the classifier model to generate accurate predictions. Ways of addressing this challenge include training the learning model with a large dataset or using a more complicated model, such as deeper layer architecture. However, employing such approaches might result in the overfitting of the model, where the model might not generalize well on previously unseen data. Also, generating a large dataset for training the model is a challenging task, especially in the case of radar images. An alternate solution for achieving high accuracy in radar classification problems is provided in this paper wherein a CNN-enabled super-resolution (SR) model is integrated with the classifier model. The SR model is trained to generate high-resolution (HR) millimeter-wave (mmW) images from any input low-resolution (LR) mmW images. These resolved images from the SR model will be used by the classifier model to classify the input images into appropriate classes, consisting of threat and non-threat objects. The training data for the dual CNN layers are generated using a numerical model of a near-field coded-aperture computational imaging (CI) system. This trained dual CNN model is tested with simulated data generated from the CI numerical model wherein a high classification accuracy of 95% and a fast inference time of 0.193 s are obtained, making it suitable for real-time automated threat classification applications. For fair comparison, the developed CNN model is also validated with experimentally generated reconstruction data, in which case, a classification accuracy of 94% is obtained. • An integrated convolutional neural network is developed which jointly performs super-resolution and classification tasks. The super-resolution part enhances the resolution of millimeter-wave input images and the classifier part predicts a class for the enhanced image. • The super-resolution part of the model enhances both the cross-range and range resolutions of the input images. In addition, the model uses both the real and imaginary parts of the reconstruction data to perform the super-resolution task. • The super-resolution and the classifier models are trained on synthesized data instead of experimentally generated data. To offer a realistic approach, additive Gaussian noise is incorporated into the synthesized data. • Validation of the integrated model with both synthesized and experimental data shows a real-time accurate super-resolution and multi-label classification performance. [ABSTRACT FROM AUTHOR]

Modern convolutional neural networks (CNNs) are used in various applications, including computer vision, speech recognition, and robotics. However, practical usage in various applications requires large-scale datasets, and real-world data contains various corruptions that degrade the model's performance owing to the inconsistencies in the training and testing distributions. In this study, we propose Frequency re-Scaling Data Augmentation (FSDA) to improve the classification performance, robustness against corruption, and localizability of classifiers trained on various image classification datasets. Our method consists of two processes: mask generation process (MGP) and pattern re-scaling process (PSP). MGP clusters the frequency domain spectra to produce similar frequency patterns, and then PSP scales frequency by learning rescaling parameters from frequency patterns. Because the CNN classifies images by focusing on their structural features highlighted with FSDA, CNN trained with the proposed method has more robustness against corruption than that with the other data augmentations (DAs). Our technique outperforms the existing DAs on four public image classification datasets, including the CIFAR-10/100, STL-10, and ImageNet. Particularly, our strategy increases the robustness of the classifier against the different corruption errors by an average of 5.04% over the baseline. • MGP clusters similar frequencies in an image and re-scales multiple ranges simultaneously. • PSP is a learning-based method for frequency re-scaling to enhance corruption robustness. • Our method enhances robustness on CIFAR-10/100, STL-10, and ImageNet. [ABSTRACT FROM AUTHOR]

The detection of tiny objects is a challenging task in computer vision. Conventional object detection methods have difficulties in finding the balance between high detection rate and low false alarm rate. In the literature, some methods have addressed this issue by enhancing the feature map responses for small objects, but without guaranteeing robustness with respect to the number of false alarms induced by background elements. To tackle this problem, we introduce an a contrario decision criterion into the learning process to take into account the unexpectedness of tiny objects. This statistic criterion enhances the feature map responses while controlling the number of false alarms (NFA) and can be integrated as an add-on into any semantic segmentation neural network. Our add-on NFA module not only allows us to obtain competitive results for small target, road crack and ship detection tasks respectively, but also leads to more robust and interpretable results. [Display omitted] • Detecting tiny objects is challenging; they are often hidden in background clutter. • We design an a contrario -based module to describe tiny object unexpectedness. • Our add-on module can guide the training loop of any segmentation neural network. • It leads to competitive results on infrared small target and road crack detection. • It allows for an intuitive control of false alarms and the results are interpretable. [ABSTRACT FROM AUTHOR]

The field of eXplainable Artificial Intelligence (XAI) has witnessed significant advancements in recent years. However, the majority of progress has been concentrated in the domains of computer vision and natural language processing. For time series data, where the input itself is often not interpretable, dedicated XAI research is scarce. In this work, we put forward a virtual inspection layer for transforming the time series to an interpretable representation and allows to propagate relevance attributions to this representation via local XAI methods. In this way, we extend the applicability of XAI methods to domains (e.g. speech) where the input is only interpretable after a transformation. In this work, we focus on the Fourier Transform which, is prominently applied in the preprocessing of time series, with Layer-wise Relevance Propagation (LRP) and refer to our method as DFT-LRP. We demonstrate the usefulness of DFT-LRP in various time series classification settings like audio and medical data. We showcase how DFT-LRP reveals differences in the classification strategies of models trained in different domains (e.g., time vs. frequency domain) or helps to discover how models act on spurious correlations in the data. [Display omitted] • We propose a new form of explanation for models trained on time series data. • We derive a closed-form formula for relevance propagation through the DFT and STDFT. • We extend evaluations to allow for comparison of explanations in different formats. • We show how our method gives insights into strategies of audio and ECG classifiers. [ABSTRACT FROM AUTHOR]

• A local topology-based lightweight method for computing mismatch probability is proposed. • A sampling-refined mismatch removal method by combining local consensus with sampling tightly is proposed. • The experimental result demonstrated its high performance compared with the state-of-art methods. Feature point matching between two images is a fundamental and important process in machine vision. In many cases, mismatches are inevitable, and removing mismatches is an indispensable task. The existing methods attempt to find comprehensive constraints or sampling model to achieve better performance, which results in the increasingly complexity and may cause the weakness of the generality and scalability. To address this issue, a method called Local Topology similarity guided probabilistic Sampling consensus (LTS) is proposed. It constructs a topological network, then quantifies the mismatch probability in a concise approach based on comparing the topological relationship with neighbourhoods. Then, it detects and removes the mismatches by sampling guided by the mismatch probability. Compared with the state-of-the-art methods, LTS has an excellent performance in accuracy and robustness. [ABSTRACT FROM AUTHOR]

Head pose estimation (HPE) is a problem of interest in computer vision to improve the performance of face processing tasks in semi-frontal or profile settings. Recent applications require the analysis of faces in the full 360° rotation range. Traditional approaches to solve the semi-frontal and profile cases are not directly amenable for the full rotation case. In this paper we analyze the methodology for short- and wide-range HPE and discuss which representations and metrics are adequate for each case. We show that the popular Euler angles representation is a good choice for short-range HPE, but not at extreme rotations. However, the Euler angles' gimbal lock problem prevents them from being used as a valid metric in any setting. We also revisit the current cross-data set evaluation methodology and note that the lack of alignment between the reference systems of the training and test data sets negatively biases the results of all articles in the literature. We introduce a procedure to quantify this misalignment and a new methodology for cross-data set HPE that establishes new, more accurate, SOTA for the 300W-LP/Biwi benchmark. We also propose a generalization of the geodesic angular distance metric that enables the construction of a loss that controls the contribution of each training sample to the optimization of the model. Finally, we introduce a wide range HPE benchmark based on the CMU Panoptic data set. code: https://github.com/pcr-upm/opal23_headpose [Display omitted] • Head pose estimation (HPE) problem revisited for short and wide range configurations. • Short range HPE best representation: Euler angles (minimal and Gimbal Lock learnable). • Wide range HPE good representation: 6D (continuous in the full 360° rotation range). • Proposed a method for reference system alignment in cross-data set experiments. • New loss function, Opal, developed upon the Geodesic error loss for 6D representation. • New wide range benchmark based on the CMU Panoptic data set. [ABSTRACT FROM AUTHOR]

Multiple Object Tracking is an important vitally important fundamental task in computer vision. Visual tracking becomes challenging when objects move in groups and are obscured from each other. There are two mainstream solution strategies for these group models. One is to transform the data association problem into a graph matching problem for solving, while the other is to apply the social power model as an advanced constraint for group tracking. In the former case, the solving difficulty geometric growth as the number of tracked objects increases, and the computing efficiency for real-time tracking demand cannot be met. The latter strategy tends to set up fixed-size groups or offline training rules, resulting in a lack of flexibility that limits their scenario generalization. According to the shortcomings of existing methods, this paper proposes a novel multiple object tracking method with spatio-temporal correlation and graph neural networks. Firstly, the relational features of the historical trajectories are extracted through the spatio-temporal relationship learning module, which models the spatio-temporal correlations of the objects and dynamically constructs the group structure online. Then, the graph neural network is combined with appearance and motion information, and the similarity between each detection and tracklet is used as a weight in node feature aggregation to make powerful distinctions between node features. Meanwhile, the spatio-temporal correlation method is also used to solve target loss issues caused by occlusion. Even collocated with linearly assigned data association method, good tracking results are still achieved, with a low computational complexity. Experiments on three challenging public datasets, namely MOT16, MOT17, and MOT20, validated the accuracy and efficiency of the proposed tracking method. • Spatio-temporal correlation is modeled based on historical trajectories. • Undetected objects can be tracked with spatio-temporal correlation constraints. • Appearance and motion information are used to select graph nodes for aggregation. • Secondary matching enhances target association between detections and tracklets. [ABSTRACT FROM AUTHOR]

Handwritten mathematical expression recognition (HMER) is a challenging task in the field of computer vision due to the complex two-dimensional spatial structure and diverse handwriting styles of mathematical expressions (MEs). Recent mainstream approach treats MEs as objects with tree structures, modeled by sequence decoders or tree decoders. These decoders recognize the symbols and relationships between symbols in MEs in depth-first order, resulting in long decoding steps that can harm their performance, particularly for MEs with complex structures. In this paper, we propose a novel tree-based model with branch parallel decoding for HMER, which parses the structures of ME trees by explicitly predicting the relationships between symbols. In addition, a query constructing module is proposed to assist the decoder in decoding the branches of ME trees in parallel, thus reducing the number of decoding time steps and alleviating the problem of long sequence attention decoding. As a result, our model outperforms existing models on three widely-used benchmarks and demonstrates significant improvements in HMER performance. • Proposing a tree-based model for handwritten mathematical expression recognition. • Performing explicit symbol recognition and relationship prediction between symbols. • Proposing a query constructing module to decode the branches of trees in parallel. • Reducing the number of decoding steps and improving the recognition performance. [ABSTRACT FROM AUTHOR]