Search

Stereo matching acts a crucial role in computer vision and robotics applications. An accurate cost volume and robust disparity regression method are essential for stereo matching of high accuracy. Following GCNet and PSMNet, constructing 4D cost volume and then using the soft argmin method to regress has been dominated. However, it will encounter many difficulties due to the multi-modal distribution of cost volume. One of the reasons for this multi-modal distribution is the occlusion area which not be possible to find a matching region on the reference image and rarely discussed. In this paper, we propose to use global context information could improve the performance of model in occluded regions. Recently, novel recurrent neural network regression methods are proposed, but most of them regress disparity maps from 3D cost volume. In this paper, we propose the new combinatorial paradigm that combine stacked hourglass modules and recurrent neural networks to further aggregate 4D cost volume and regress disparity respectively. The proposed method can be seamlessly integrated into most stereo matching networks, we improved the accuracy by 45% for PSMNet and 38% for GwcNet in our experiment. Experimental results on Scene Flow, KITTI2012, KITTI2015, and ETH3D datasets show our method is competitive. The code is available at: https://github.com/truman1211/HCRnet. • Replacing the soft argmin method with GRU modules. • Different context information is embedded in different stages of the GRU. • The GRU model is regressed on the cost volume at different scales. • Attention mechanism is used to provide global information for the occlusion region. [ABSTRACT FROM AUTHOR]

Object detection, as one of the most fundamental and essential tasks in the field of computer vision, has been the focus of unremitting efforts by researchers, who are committed to modifying the neural network structure in order to improve the accuracy of object detection and expedite task execution. As the application scope continues to expand, small object detection has gradually emerged as a crucial branch in the field of object detection. In this paper, the development history of object detection algorithms is introduced, the concept of small objects is introduced, and the current problems and challenges faced by small object detection are outlined. In this paper, the network structure is disassembled from a macroscopic point of view, and improved algorithms such as enhanced data augmentation, improved feature extraction, superior feature fusion, and refined loss functions are described in detail. Furthermore, the paper explores a series of emerging and improved algorithms for small object detection. It encompasses the introduction of advanced strategies such as unsupervised learning, end-to-end training, density cropping, transfer learning, and anchor-free approaches. The paper provides a comprehensive list of commonly used general-purpose datasets and domain-specific datasets for small object detection tasks, offering performance comparisons of the mentioned improved algorithms. In conclusion, the paper summarizes and provides an outlook on current small object detection algorithms, furnishing the reader with a thorough understanding of the field and insights into future directions. • GAN can improve the resolution of images containing small objects. • The receptive field can be expanded by dilated convolution. • Multi-scale feature extraction is often used in small object detection. • Small object datasets are mostly concentrated in remote sensing. [ABSTRACT FROM AUTHOR]

Shape descriptors provide valuable prior information in many tomographic image reconstruction methods. Such descriptors include, among others, centroid, circularity, orientation, and elongation. Shape descriptor measures are often analytically expressed as a composition of certain geometric moments. Building upon this fact, this paper suggests preserving the values of a specific geometric moment in the reconstruction process, instead of preserving entire descriptors, as it has been suggested so far. Reconstructions from two natural projection directions (vertical and horizontal) are considered with special attention. The provided theoretical analysis demonstrates that preserving the value of a specific geometric moment, provided as prior information for the reconstruction process, simultaneously ensures the preservation of the true measures of all four abovementioned descriptors. Based on this result, a novel regularized energy minimization reconstruction model is proposed. The minimization task of the new model is solved using gradient-based optimization algorithm. Performance evaluation of the proposed method is supported by experimental results obtained through comparisons with other well-known reconstruction methods. • The paper introduces a new geometrical moment preserving based tomographic reconstruction model. • The preserved moment implies preservation of following descriptors: orientation, elongation, centroid and circularity. • Analytical gradient of the proposed energy functions is derived, which makes the minimization process more efficient. [ABSTRACT FROM AUTHOR]

Edge-preserving image smoothing is vital in the field of image processing and computational photography. The state-of-the-art filters based on optimization models have achieved promising performance. However, most of them fail to consider the spatial support in the regularization term, thus limiting the edge-preserving capabilities. In this paper, inspired by the bilateral filter, which consists of a range kernel and a spatial kernel. we propose to leverage bilateral kernel as a penalty function, and embed it into an optimization model for edge-preserving image smoothing. Furthermore, we propose to incorporate an edge-aware weighted scheme in the data term design, which further improves the edge-preserving capability. The bilateral function is non-convex and can be non-trivial to solve. In this paper, we propose a novel iterative solution based on fixed point iteration, where the main burden in each iteration is a bilateral filtering process. We have conducted extensive experiments to evaluate the proposed filter. Experiment results indicate that our filter benefits a variety of image processing tasks. Moreover, we propose an efficient approximation of the proposed filter, which is able to significantly accelerate the filtering process with neglectable sacrifice of smoothing quality. J is the objective function in this paper. Our solver is essentially an iterative procedure, where each iteration selectively adding the normalized detail layer C. ∗ I − C. ∗ O to the bilateral filtered result P. / Q. k indicates the k -th iteration. [Display omitted] • We propose a bilateral regularized optimization model for edge-preserving smoothing. • We incorporate the spatial support in the design of the regularization term. • We incorporate an edge-aware weighted scheme in the design of the data term. • We propose a novel iterative solution for our model based on fixed point iteration. [ABSTRACT FROM AUTHOR]

Underwater images serve as crucial mediums for conveying marine information. Nevertheless, due to the inherent complexity of the underwater environment, underwater images often suffer from various quality degradation phenomena such as color deviation, low contrast, and non-uniform illumination. These degraded underwater images fail to meet the requirements of underwater computer vision applications. Consequently, effective quality optimization of underwater images is of paramount research and analytical value. Based on whether they rely on underwater physical imaging models, underwater image quality optimization techniques can be categorized into underwater image enhancement and underwater image restoration methods. This paper provides a comprehensive review of underwater image enhancement and restoration algorithms, accompanied by a brief introduction to underwater imaging model. Then, we systematically analyze publicly available underwater image datasets and commonly-used quality assessment methodologies. Furthermore, extensive experimental comparisons are carried out to assess the performance of underwater image optimization algorithms and their practical impact on high-level vision tasks. Finally, the challenges and future development trends in this field are discussed. We hope that the efforts made in this paper will provide valuable references for future research and contribute to the innovative advancement of underwater image optimization. • A comprehensive review of underwater image enhancement and restoration methods is provided. • The publicly available underwater image datasets and underwater image quality metrics are introduced. • The extensive experiments on underwater image enhancement and restoration methods are conducted. • The challenges in the field are discussed and the promising insights for future directions are provided. [ABSTRACT FROM AUTHOR]

The inherent complexity and extensive architecture of deep neural networks often lead to overfitting, compromising their ability to generalize to new, unseen data. One of the regularization techniques, data augmentation, is now considered vital to alleviate this, and mixup, which blends pairs of images and labels, has proven effective in enhancing model generalization. Recently, incorporating saliency in mixups has shown performance gains by retaining salient regions in mixed results. While these methods have become mainstream at the input level, their applications at the feature level remain under-explored. Our observations indicate that outcomes from naive applications of input saliency-based methods did not consistently lead to enhancements in performance. In this paper, we attribute these observations primarily to two challenges: 'Hard Boundary Issue' and 'Saliency Mismatch.' The Hard Boundary Issue describes a situation where masks with distinct, sharp edges work well at the input level, but lead to unintended distortions in the deeper layers. The Saliency Mismatch points to the disparity between saliency masks generated from input images and the saliency of feature maps. To tackle these challenges, we present a novel method called 'attention-based mixup mask adaptation' (MMA). This approach employs an attention mechanism to effectively adapt mixup masks, which are designed to maximize saliency at the input level, for feature augmentation purposes. We reduce the Saliency Mismatch problem by incorporating the spatial significance of the feature map into the mixup mask. Additionally, we address the Hard Boundary Issue by applying softmax to smoothen the adjusted mixup mask. Through comprehensive experiments, we validate our observations and confirm the effectiveness of applying MMA to saliency-aware mixup approaches at the feature level, as evidenced by the performance improvements on multiple benchmarks and the robustness improvements against corruption and deformation. • Saliency-aware mixups have not been applied and less explored at the feature level. • Naive feature-level application often fails from 'Hard Boundary' and 'Saliency Mismatch' issues. • The paper introduces 'MMA' to adapt mixup masks for feature-level augmentation utilizing attention. • The proposed MMA addresses these issues and shows the performance improvement. [ABSTRACT FROM AUTHOR]

Recently, it was found that deep neural networks (DNNs) are susceptible to adversarial input perturbations. Most defense strategies adopt the denoising method based on preprocessing, which mitigates the impacts of adversarial perturbations on DNNs by learning the distributions of nonadversarial datasets and projecting adversarial inputs into the learned nonadversarial manifolds. However, existing defense strategies commonly focus on reconstructing clean images while ignoring the role of adversarial perturbations, which results in the reconstructed images failing to achieve the visual quality and classification accuracy of the original clean images, and the induced adversarial robustness improvement is limited. This paper proposes a feature decoupling-interaction network (FDIN), which introduces the concepts of clean features and adversarial features to separate the two kinds of features from the input adversarial examples (AEs) in a feature decoupling-interaction manner. The clean features are used to reconstruct the input image so that it is infinitely close to the original clean image, and the adversarial features are used to reconstruct the adversarial perturbations. Adversarial perturbations are removed from the adversarial examples across multiple cross cycles to improve further the reconstructed image's visual quality and classification accuracy. The features of the original clean image are used as prior knowledge to guide the network to learn the clean features of the adversarial examples and improve the classification accuracy of the model on the clean examples. In addition, a classification loss function based on the Carlini & Wagner (CW) attack algorithm is used instead of the conventional cross-entropy loss function to improve the adversarial robustness of the FDIN. The experimental results show that the proposed method achieves better defense performance than the current state-of-the-art methods on both standard tests and various attack tests and even exceeds the test accuracy of the target classifier on the original test set. • This paper proposes a feature decoupling-interaction network (FDIN). • We introduces the concepts of clean features and adversarial features. • The FDIN separates the clean features and adversarial features. • We use the CW loss instead of the CE loss to improve the adversarial robustness. • The experimental results show that FDIN achieves better defense performance. [ABSTRACT FROM AUTHOR]

Though a large body of research has focused on tracking and identifying objects from the domain of colour or grey-scale images, there is a relative dearth in the literature on complex articulated/non-rigid motion reconstruction from a collection of low density feature points. In this paper, we propose a segment-based articulated matching algorithm to establish a crucial self-initialising identification in model-based point-feature tracking of articulated motion with near-rigid segments. We avoid common assumptions such as pose similarity or small motion with respect to the model, and assume no prior knowledge of a specific movement from which to restrict pose identification. Experimental results based on synthetic pose and real-world human motion capture data demonstrate the ability of the algorithm to perform the identification task. [Copyright &y& Elsevier]

In the realm of Tiny AI, we introduce "You Only Look at Interested Cells" (YOLIC), an efficient method for object localization and classification on edge devices. Through seamlessly blending the strengths of semantic segmentation and object detection, YOLIC provides improved computational efficiency and precision compared to traditional methods. By adopting Cells of Interest for classification instead of individual pixels, YOLIC encapsulates relevant information, reduces computational load, and enables rough object shape inference. Importantly, the need for bounding box regression is obviated, as YOLIC capitalizes on the predetermined cell configuration that provides information about potential object location, size, and shape. To tackle the issue of single-label classification limitations, a multi-label classification approach is applied to each cell for effectively recognizing overlapping or closely situated objects. This paper presents extensive experiments on multiple datasets to demonstrate that YOLIC achieves detection performance comparable to the state-of-the-art YOLO algorithms while surpassing in speed, exceeding 30fps on a Raspberry Pi 4B CPU. • YOLIC: efficient object localization and classification method for edge devices. • Seamlessly blends strengths of semantic segmentation and object detection. • Utilizes predefined Cells of Interest for object classification and localization. • Achieves real-time performance over 30 FPS on Raspberry Pi 4B. [ABSTRACT FROM AUTHOR]

Neural Radiance Fields (NeRF) have attracted increasing interest in 3D talking portrait synthesis, which is a crucial problem in the field of digital humans and the metaverse. The synthesis of high-fidelity talking portraits remains a challenging task due to the intricacies of capturing and reproducing subtle facial expressions. In this paper, we propose an innovative approach termed Expression-Aware Neural Radiance Fields (EA-NeRF) for the talking portraits synthesis with remarkable realism and expressiveness. Our method leverages the power of NeRF to model complex scene appearance and illumination, while incorporating expression-awareness to accurately capture and reproduce nuanced facial dynamics. Specifically, we introduce a novel Expression-Aware Module (EAM) that enables our model to seamlessly blend between different facial expressions, yielding convincing and natural transitions during synthesis. Moreover, we present a Local–Global Attention Module (LGAM) that dynamically focuses on salient regions of the face, allowing the model to allocate more resources to areas exhibiting significant expression changes. This attention-guided synthesis process enables our model to generate talking portraits with unparalleled realism and expressiveness, accurately preserving fine-grained details and subtle nuances of facial dynamics. Both qualitative and quantitative experimental results demonstrate the effectiveness of our proposed method in generating talking portraits with superior fidelity and expressiveness compared to existing methods. • We propose Expression-Aware Neural Radiance Fields for talking portrait synthesis. • Expression-Aware Module seamlessly blends between different facial expressions. • Local–Global Attention Module dynamically focuses on salient regions of the face. • The proposed method performs better in terms of expressiveness and realism. [ABSTRACT FROM AUTHOR]

Reversible Adversarial Example (RAE) has been widely researched for its ability to ensure authorized access while preventing unauthorized recognition. Existing RAE schemes focus on Reversible Data Hiding techniques and white-box attacks. However, white-box attacks might be impractical due to the unknown parameters of the target model. Besides, these methods suffer massive loss during the embedding of perturbations, impacting the RAE's quality. In this paper, we propose I-RAE scheme to generate black-box RAE with minimal loss based on Invertible Neural Network (INN). Specifically, Black-box Attack Flow (BAFlow) is introduced to generate perturbations on a Gaussian distribution that are more easily embeddable. Furthermore, to enhance the embedding capability of RAE, we innovatively treat the embedding of perturbation as an image hiding and propose Perturbation Hiding Network (PHN) to reversibly hide the entire perturbation into the adversarial example. We also implement wavelet high-frequency hiding to reduce the degradation in the visual quality of RAE. Experimental results on the ImageNet and CIFAR-10 datasets demonstrate that I-RAE achieves state-of-the-art black-box attack ability and visual quality. • The proposed I-RAE scheme focuses on black-box Reversible Adversarial Example (RAE) generation. • The proposed I-RAE applies invertible neural network for generation. • The proposed I-RAE reduces the embedding loss during RAE generation. • The proposed I-RAE outperforms other RAE schemes. [ABSTRACT FROM AUTHOR]

In this paper, we introduce a novel approach to improve the performance of Neural Radiance Fields (NeRF) from limited input views. NeRF has exhibited impressive capabilities in producing photo-realistic renderings when trained on dense input views, but its performance degrades as the number of training views decreases. Our key insight is that the original NeRF lacks geometric regularization and appearance information due to limited inputs, resulting in an over-fitting issue. To address this challenge, we present a novel method: first, a global sampling method with geometric regularization is employed by utilizing warped images as additional pseudo-views, which optimizes the multi-view consistency during the training. Second, we introduce a local patch sampling technique with perceptual regularization to ensure pixel correspondence in appearance. Furthermore, we incorporate depth information for explicit geometry regularization. We evaluate our method on the DTU dataset and LLFF dataset from a different number of inputs. Extensive evaluations demonstrate that our approach outperforms existing benchmarks across various metrics, achieving state-of-the-art results. • A novel method aims at improving the few-shot neural radiance field from sparse inputs. • A global patched-based sampling scheme together with geometry regularization is introduced. • We propose a local patch sampling technique with perceptual regularization to ensure the pixel correspondence in appearance. • The proposed approach outperforms sparse NeRF baselines and achieves state-of-the-art rendering quality. [ABSTRACT FROM AUTHOR]

Although learning-based stereo matching methods have achieved remarkable performance, accurately recovering disparity maps for boundary areas (e.g., thin structures) remains an intractable issue. Existing stereo pipelines usually employ the standard L1 loss function, averaging pixel-wise losses equally despite variable difficulty. This mechanism inevitably overwhelms thin structures due to their limited proportion. In this paper, we propose reweighting the L1 loss to focus on pixels of varying hardness. First, we explicitly model the uncertainty of each pixel to gauge the confidence in its prediction. By aggregating the volume, uncertainty is obtained effortlessly. Second, uncertainty is mapped to weights, which reweight the L1 loss accordingly. The core of our approach lies in leveraging uncertainty to personalize the loss map adjustment, progressively optimizing challenging regions during training. Notably, our method requires no extra parameters or inference computations. Finally, we introduce the Boundary Pixel Error (BPE), a novel metric targeting boundary quality. Extensive experiments with the SceneFlow, KITTI 2012, and KITTI 2015 datasets demonstrate the effectiveness and universality of our elegant framework, seamlessly integrating it into existing models as a plug-and-play component, leading to substantial performance improvements. [Display omitted] [ABSTRACT FROM AUTHOR]

Reliable multi-vehicle detection in unregulated traffic environments is a crucial computer vision task in the development of Intelligent Transportation Systems (ITS). Despite the promising potential of Deep Learning (DL) methods for vehicle detection, the presence of uncertainties such as varying vehicle shapes and sizes, intricate background clutter, and unpredictable vehicle flow contribute to the chaotic unregulated traffic. For real-time vehicle detection in an unregulated traffic environment this paper proposes the Hybrid Learning Multi-Vehicle Detection framework (HL-MVD) based on the C onvolutional M ulti-head A ttention T ransformer D etector (CMATDet). The primary objective of this study is to generate a dataset containing highly informative video frames using a Pool-based Active Learning Strategy (PALS). Additionally, transfer learning will be implemented to train CMATDet to achieve improved accuracy and reduced detection latency. The proposed approach restructures the baseline YOLOv5x and incorporates a Multi-head Attention transformer encoder to effectively extract global features and a Scale Specific Bidirectional Feature Pyramid Network (SS-BiFPN) to facilitate multi-scale feature representation. Simplified Optimal Transport Algorithm with top-q approximation technique (Sim-OTA) is utilized for label assignment approach. Heatmap analysis demonstrated the suitability of the newly generated dataset "AU-INV-P-PALS" for detecting specific Indian native vehicles. The performance of the proposed framework is evaluated on our custom-developed AU-INV-P-PALS vehicle dataset and the IITM-HeTra Dataset 1. In comparison with contemporary detection models, the proposed HL-MVD framework resulted with higher mAP scores (91.1% on mAP@0.5 and 78.3% on mAP@0.5:0.95) for AU-INV-P-PALS. The proposed model demonstrated lower inference latency (8.1 ms), higher precision score (82.7% for IoU = 0.5), and higher recall score (90.8% for IoU = 0.5) than recent deep learning-based detection models in the literature. The top-q approximation technique in the detection head results in a reduced false-positive rate compared to conventional models. Finally, the performance of the proposed framework is tested on CCTV traffic footage captured on city roads in Chennai, India. • Hybrid Learning framework for reliable vehicle detection in unregulated traffic. • Active Learning to generate compact and informative dataset (AU-INV-P-PALS). • CNN-Transformer based network (CMATDet) to improve detection capabilities. • Dynamic label assignment to reduce false positives during vehicle occlusion. • Validation on real-time CCTV traffic footages for reliability. [ABSTRACT FROM AUTHOR]

Neural Radiance Fields (NeRF) can achieve state-of-the-art new view results when given a sufficient number of training views. However, NeRF's rendering is based on minimizing photometric consistency loss, and during the optimization process, it can suffer from overfitting due to factors such as lighting and texture, resulting in poor geometric and color reconstruction. The less data there is in the overlapping areas of the images, the greater the impact on the results, such as insufficient data caused by occlusion relationships. In this paper, we observed through experiments that introducing low-frequency images into NeRF during training can quickly obtain approximate geometric structures, which can guide NeRF to achieve more stable view synthesis results. Therefore, we propose a progressive wavelet mask to assist in the training of neural radiance fields. By first constructing a gaussian pyramid for the training images and then applying wavelet decomposition to them, we can obtain a series of low-frequency region masks to guide the neural radiance field to focus on learning low-frequency pixel regions and gradually introduce high-frequency lighting and texture changes. Our experiments on the LLFF and Blender datasets show that using a progressive wavelet mask in NeRF training can achieve more realistic generation effects with almost no additional computational overhead. At the same time, we also tested our method in sparse scenes, where it still performs well in avoiding overfitting. • NeRF get more stable geometric structures and rendering quality with blur image. • Designed a new training strategy to enhance the rendering performance of NeRF. • NeRF model with neural network is influenced by the ray training strategy. • The method is more effective in NeRF with neural network. [ABSTRACT FROM AUTHOR]

Medical image segmentation is a crucial aspect of medical image processing, and has been widely used in the detection and clinical diagnosis for brain, lung, liver, heart and other diseases. In this paper, we propose a novel multimodal parallel attention network, called MPA-Net, for medical image segmentation. MPA-Net is divided into two parts. The first part extracts more high-dimensional features by improved network structure, which contains the skip connection, the output of the multimodal parallel attention and the output of the previous upsampling layer. The second part incorporates a multimodal parallel attention mechanism, encompassing feature parallel attention, spatial parallel attention and channel parallel attention. This mechanism facilitates the effective fusion of high-dimensional and low-dimensional features, leading to enhanced context information. Experimental results on Kagglelung dataset, Liver dataset, Cell dataset, Drive dataset and Kvasir-SEG dataset show that MPA-Net has achieved better segmentation performance than that of other baseline methods, on lung, liver, cell contour, retinal vessel and polyps. • Architecture integrates multimodal parallel attention for segmentation. • Network employs feature, spatial, and channel parallel attention. • Radiologists aided in precise lesion localization across organs. • Identification rate of lesions improved, aiding disease diagnosis. [ABSTRACT FROM AUTHOR]

Attention is an important component of modern Convolutional Neural Networks (CNNs) that has been shown to improve baseline model performance for a wide variety of tasks. Attention has shown specific promise in the classification and segmentation of mammograms, but we have a limited understanding of why attention improves performance in these domains. In this paper, we present a robust comparison of different combinations of baseline models and attention methods at two resolutions for whole mammogram classification of masses and calcifications. We find that attention generally helps to improve baseline model performance. However, the extent of improvement is governed by a combination of model architecture and the statistical characteristics of the data. Specifically, we show that high amounts of pooling and model complexity may result in decreased performance for data with high variability. To better understand the effect of attention on mammogram classification, we used LayerCAM, a hierarchical Class Activation Map (CAM) approach, to visualize where the network pays attention in the input image. This research provides statistical evidence that attention can improve the correlation between model performance and LayerCAM activation in the region of interest (ROI). However, these correlations are weak and variable, indicating that improvements in model performance due to attention are not necessarily caused by increased model activation near the ROI. Overall, our work provides novel insights to help guide future efforts in incorporating attention-based mechanisms for mammogram classification. • Attention generally improves mammogram classification. • Newer attention methods don't consistently perform better. • Improvements due to attention depend on model architecture/complexity and characteristics of the image data. • Attention doesn't necessarily improve focus on the region of interest. • CNNs outperform transformers for high resolution mammogram classification. [ABSTRACT FROM AUTHOR]

Spatial and temporal action detection aims to precisely locate actions while predicting their respective categories. The existing solution, TubeR (Zhao et al., 2022), is designed to directly detect action tubes in videos by recognizing and localizing actions using a unified representation. However, a potential challenge arises during the decoding stage, leading to a gradual decrease in the model's performance in action detection, specifically in terms of the confidence associated with detected actions. In this paper, we propose TQRFormer: Tubelet Query Recollection Transformer, enabling the subsequent decoder to obtain information from the previous stage. Specifically, we designed Query Recollection Attention to correct errors and output the synthesized results, effectively breaking the limitations of sequential decoding. During the training stage, TubeR (Zhao et al., 2022) generates a limited number of positive sample queries through a one-to-one matching strategy, potentially impacting the effectiveness of training with positive samples. To enhance the quantity of positive samples, we propose a stage matching approach that combines both one-to-many matching and one-to-one matching without additional queries. This approach serves to boost the overall number of positive samples for improved training outcomes. We also propose a more elegant classification head that contains the start and end frames of the small tubes information, eliminating the necessity for a separate action switch. The performance of TQRFormer is superior to previous state-of-the-art technologies on public action detection datasets, including AVA, UCF101–24, JHMDB-21 and MultiSports. The code will available at https://github.com/ykyk000/TQRFormer. • Query Recollection Attention integrates decoding information from previous stages. • Stage matching strategy improves the overall quality of the query. • Classification Head integrates the start and end times of action tubes. • Long-Term Contextual module to compress temporal information. [ABSTRACT FROM AUTHOR]

Medical image segmentation is an important task in medical imaging and diagnosis. Data augmentation can substantially improve the accuracy of medical image segmentation when the dataset has a small amount of medical images. However, the data augmentation methods for medical image are usually based on big models that require extensive search space. Furthermore, excessively complex models often have a heavy burden for the general healthcare organization or researcher. To address this problem, we propose a method of data augmentation that is simple to implement even for the general researcher and simple to transplant across various models. Here we introduce our new methods called KeepMask and KeepMix, which can be simply ported to a variety of models and provide high performance. These methods allow data augmentation without any effect on the target organ or lesion and can also be adapted to multi-class segmentation. KeepMask and KeepMix can not only perturb the background of an existing medical image but also add target organs that are not present to it and generate new images based on the image. In this paper, we performed our methods on both binary class datasets and multi-class datasets and obtained better performance. We conducted numerous experiments showing the predicted segmentation images using our proposed methods obtained more accurate boundaries. • Higher accuracy and predict more accurate boundaries. • Simple to transplant across various models. • Can be used for binary and multiclass tasks. • The images generated using the proposed data augmentation method are more consistent with the original dataset distributions. [ABSTRACT FROM AUTHOR]

Although deep learning-based RGB-D salient object detection methods have achieved impressive results in the recent years, there are still some issues need to be addressed including multi-modal fusion and multi-level aggregation. In this paper, we propose a bifurcated multi-modal fusion network (BMFNet) to address these two issues cooperatively. First, we design a multi-modal feature interaction (MFI) module to fully capture the complementary information between the RGB and depth features by leveraging the channel attention and spatial attention. Second, unlike the widely used layer-by-layer progressive fusion, we adopt a bifurcated fusion strategy for all the multi-level unimodal and cross-modal features to effectively reduce the gaps between features at different levels. For the intra-group feature aggregation, a multi-modal feature fusion (MFF) module is designed to integrate the intra-group multi-modal features to produce a low-level/high-level saliency feature. For the inter-group aggregation, a multi-scale feature learning (MFL) module is introduced to exploit the contextual interactions between different scales to boost fusion performance. Experimental results on five public RGB-D datasets demonstrate the effectiveness and superiority of our proposed network. The code and prediction maps will be available at https://github.com/ZhangQing0329/BMFNet • We propose a bifurcated multi-modal fusion network for RGB-D saliency detection. • We design an MFI module to capture the complementary information of RGB and depth. • We use a bifurcated fusion strategy to reduce the gaps between multi-level features. • The proposed method performs competitively against the recent methods. [ABSTRACT FROM AUTHOR]

In Semi-supervised Video Object Segmentation (SVOS), there is a critical emphasis on enhancing the memory and readout mechanisms for frame matching, especially in relation to temporal dynamics. Current methods predominantly use 2D CNNs for encoding video frames, which unfortunately neglects the crucial aspect of addressing temporal variations in individual frames and their associated masks during the encoding process. One potential solution would be to implement temporal models such as 3D CNNs instead of 2D CNNs, but this significantly increases computational requirements, making it impractical for real-world SVOS applications. In this paper, we introduce the Grouped Temporal Recalibration with Attention for Convolutional Encoders (G-TRACE), a novel plug-and-play module that is compatible with various existing SVOS frameworks. G-TRACE uses hierarchical memory-centric attention and integrates effortlessly with 2D CNNs, offering a novel approach to temporal modeling that operates orthogonally to traditional frame matching methods. Extensive evaluations on four widely-used benchmarks demonstrate that our method consistently delivers significant performance improvements over various baseline models. • Captures Temporal Variations: Effectively handles previously overlooked time-based changes in frame encoding. • Innovates Frame Encoding: Improves spatial and temporal context capture, potentially replacing self-attention mechanisms. • Hierarchical Memory Grouping: Introduces a two-tier strategy for efficient feature capture and memory utilization. • Enhances Feature Quality: Provides a richer understanding of video sequences through advanced memory-based attention. • Empirical Superiority: Demonstrates consistent performance improvements across key SVOS benchmarks. [ABSTRACT FROM AUTHOR]

Glaucoma is a progressive eye condition that can lead to permanent vision loss. Therefore, on-time detection of glaucoma is critical for making an effective treatment plan. In recent years, enormous attempts have been made to develop automated glaucoma classification systems using CNNs through images. In contrast, limited methods have been proposed for diagnosing different glaucoma stages. It is mainly owing to the lack of large publicly available labeled datasets. Also, fundus images exhibit a high inter-stage resemblance, redundant features and minute size variations of lesions, making the conventional CNNs difficult to classify multiple stages of glaucoma accurately. To address these challenges, this paper proposes a novel adapter and enhanced self-attention based CNN framework named AES-Net for effective classification of glaucoma stages. In particular, we propose a spatial adapter module on top of the backbone network for learning better feature representations and an enhanced self-attention module (ESAM) to capture global feature correlations among the relevant channels and spatial positions. The ESAM assists in capturing stage-specific and detailed-lesion features from the fundus images. Extensive experiments on two multi-stage glaucoma datasets indicate that our AES-Net surpasses CNN-based existing approaches. The Grad-CAM ++ visualization maps further confirm the effectiveness of our AES-Net. • A spatial adapter and enhanced self-attention based CNN, AES-Net, is proposed. • An enhanced self-attention module is proposed to learn class-specific features. • A comparative analysis with state-of-the-art attention mechanisms is performed. • Results on two datasets indicate the effectiveness of AES-Net over current methods. • The visualization results demonstrate the explainability and interpretability of the model. [ABSTRACT FROM AUTHOR]

The growing demand for mobile devices has generated interest in lightweight human pose estimation. Currently, lightweight estimation generally uses heatmap-based methods, which has demonstrated exceptional performance. However, their use of non-differentiable post-processing imposes considerable inference latencies. Conversely, integral-based approaches expedite the inference process by employing a soft-argmax operation but compromise in accuracy. Integrating explicit heatmap knowledge learned using the heatmap-based method into the implicit heatmap generated by the integral-based method, thereby combining the best of both worlds, offers a promising avenue. However, owing to the disparities in supervision and inference processes, the explicit and implicit heatmaps are heterogeneous. Consequently, direct transfer of knowledge presents difficulties in ensuring consistencies in heat value and location. In this paper, we propose a novel Heterogeneous Heatmap Distillation (HHD) framework that effectively tackles these challenges. The framework seamlessly integrates explicit heatmap knowledge that contains high-precision localization information into implicit heatmaps. The framework revolves around an unbiased heatmap alignment scheme encompassing two steps: heterogeneous heatmap normalization and unbiased cropping. Heterogeneous heatmap normalization separately normalizes the output feature maps of both the teacher and student models, alleviating potential heat value bias during the knowledge transfer. Unbiased cropping applies closed-form computation on the normalized teacher and student heatmap to eliminate location bias. Additionally, mirror expansion is implemented to handle potential cases wherein the cropped region extends beyond the image boundary. Extensive experiments demonstrate the efficiency and effectiveness of our methods on the MSCOCO and MPII datasets compared to other integral-based lightweight networks. Our source codes and pre-trained models are available at https://github.com/ducongju/HHD. • We propose a heterogeneousheatmap distillation(HHD) forhuman pose estimation. • The HHDtransfers knowledge from explicit heatmaps to implicit heatmaps. • Heterogeneous heatmap normalization mitigatesthe heat value bias between heatmaps. • Unbiasedcropping and mirror expansion mitigatethe location bias between heatmaps. [ABSTRACT FROM AUTHOR]

In the task of three-dimensional human motion posture recognition, there are problems such as target loss, inaccurate target positioning, and high computational complexity. This article designs a recognition evaluation algorithm to address these issues. Design a LiteHRNet model for extracting skeleton sequences from action videos, and propose a graph convolutional structure that combines residual networks and attention mechanisms. This network can effectively enhance the expression ability of node key features. Introducing second-order velocity information and spatial position information of joint points to improve positioning accuracy. Improve the TCN and Transformer network models to simultaneously extract local and long-term features throughout the entire model, and more accurately model the temporal correlation between nodes in the entire action sequence. The fusion of Transformer networks can reduce the computational complexity of the model while ensuring its accuracy. The experiment shows that the model has good evaluation performance on multiple datasets. • This paper proposes a technique for 3D human motion pose recognition algorithm. • This method improves the efficiency of human pose estimation models based on GCN. • This method solves the problems of low accuracy in action recognition models. • This method solves the problems of high resource consumption. [ABSTRACT FROM AUTHOR]

The main challenge in deceiving face recognition (FR) models lies in the target model under the black-box setting. Existing works seek to generate adversarial examples to improve the adversarial transferability for black-box attacks. However, the attack performance and quality of the crafted image still have room for improvement. In this work, we propose a novel method called TransMix to improve the transferability of adversarial face examples based on data augmentation. Our approach leverages the mixture of the original image with a mixed sample image that is randomly mixed using images from different identities or the same identities, incorporating information from diverse categories. Then, we perform random transformations N times to create diverse input patterns, exploiting the gradient from various images and other identities in the same iteration. Extensive experiments conducted on the CelebA dataset demonstrate that TransMix achieves a significantly higher attack success rate against different FR models and Vision Transformers (ViTs), outperforming the best competitor by a large margin of 5.6% and 8.8% when attacking the ViTs using adversarial images generated on the ArcFace model. Our results also confirm that adversarial examples crafted by TransMix exhibit good adversarial transferability against defense models, achieving an attack success rate of 52.3% on the Bit-Red model. • This paper presents TransMix as a solution to enhance adversarial transferability. • TransMix leverages image augmentation techniques for adversarial transferability. • TransMix generates mixed images and integrates them into the original images. • We calculate the gradient of both the original and mixed image. • TransMix exhibits superior cross-model transferability in face recognition models. [ABSTRACT FROM AUTHOR]

Deep learning techniques have gained significant interest due to their success in large model scenarios. However, large models often require massive computational resources, which can challenge end devices with limited storage capabilities. Transferring knowledge from big to small models and achieving similar results with limited resources requires further research. Knowledge distillation techniques, which involve using teacher-student models to migrate large model capabilities to small models, have been widely used in model compression and knowledge transfer. In this paper, a novel knowledge distillation approach is proposed, which utilizes the sparse attention mechanism (SAKD). SAKD computes attention using student features as queries and teacher features as key values and performs sparse attention values by random deactivation. Then, this sparse attention value is used to reweight the feature distance of each teacher-student feature pair to avoid negative transfer. Comprehensive experiments demonstrate the effectiveness and generality of our approach. Moreover, our SAKD method outperforms previous state-of-the-art methods on image classification tasks. • Propose a novel knowledge distillation approach utilizing the sparse attention mechanism (SAKD). • Use student features as queries and teacher features as key values and performs sparse attention values by random deactivation. • Use sparse attention value to reweight the feature distance of each teacher-student feature pair to avoid negative transfer. • SAKD method outperforms previous state-of-the-art methods on image classification tasks (CIFAR-100 and ImageNet). [ABSTRACT FROM AUTHOR]

Image recognition is an important task in computer vision with broad applications. In recent years, with the advent of deep learning, lightweight convolutional neural network (CNN) has brought new opportunities for image recognition, which allows high-performance recognition algorithms to run on resource-constrained devices with strong representation and generalization capabilities. This paper first presents an overview of several classical lightweight CNN models. Then, a comprehensive review is provided on recent image recognition techniques using lightweight CNN. According to the strategies applied to optimize image recognition performance, existing methods are classified into three categories: (1) model compression, (2) optimization of lightweight network, and (3) combining Transformer with lightweight network. In addition, some representative methods are tested on three commonly used datasets for performance comparison. Finally, technical challenges and future research trends in this field are discussed. • A comprehensive review of recent advances in lightweight CNN for image recognition. • Classification according to different optimization directions. • Present the evaluation results of different representative lightweight CNN methods. • Discuss future research trends of image recognition. [ABSTRACT FROM AUTHOR]

Adverse weather conditions would decrease the image quality, leading to a sharp decline in detection accuracy. Most of the researches focus on object detection in fine weather conditions, rather than in adverse weather conditions. Recently, some methods attempt to reduce the gap between degraded images and clean images to improve the detection accuracy in adverse weather conditions. Specifically, these methods usually conduct image restoration and object detection in a sequential way or by joint learning. While these methods can improve detection accuracy to a certain extent, image restoration models may introduce noise or artifacts and increase computational burden, limiting the accuracy and efficiency of object detection in adverse weather conditions. In this paper, we propose a knowledge distillation-based method, Localization-aware Logit Mimicking (LaLM), to improve detection accuracy in adverse weather conditions by reducing the gap between degraded images and clean images at the prediction level, rather than the image level. Moreover, the localization quality is designed as the mimicking target to make the knowledge distillation more effective. Experiments conducted on three popular benchmarks (i.e., RTTS, ExDark, and RID) demonstrate that our LaLM can achieve the state-of-the-art detection accuracy and inference speed in foggy, rainy, and low-light conditions. Code is available at: https://github.com/VIPLab-CQU/LaLM. • We are the first to utilize knowledge distillation for detection in adverse weather. • Localization quality is deigned as the mimicking target to ensure the effectiveness. • Our proposed method achieves the SOTA accuracy and speed in adverse weather. [ABSTRACT FROM AUTHOR]

This paper proposes a novel framework for generating synthesized PET images from MRIs to fill in missing PETs and help with Alzheimer's disease (AD) diagnosis. This framework employs a 3D multi-scale image-to-image CycleGAN architecture for the end-to-end translation of MRI and PET domains together. A hybrid loss function is also proposed to enforce structural similarity while preserving voxel-wise similarity and avoiding blurry images. As shown by the quantitative and visual assessment of the synthesized PETs, this framework is superior to the state-of-the-art. Moreover, using these synthesized PETs helps improve the ternary classification of AD subjects (AD vs. MCI vs. NC). Specifically, assuming an extreme case where none of the subjects has a PET, feeding the classifier with MRIs and their corresponding synthetic PETs results in a more accurate diagnosis than feeding it with just available MRIs. Accordingly, the proposed framework can help improve AD diagnosis, which is the final goal of the current study. Ablation investigation of the proposed multi-scale framework as well as the proposed loss function, is also conducted to study their contribution to the quality of synthesized PETs. Furthermore, other factors, such as stopping criteria, the type of normalization layer, the activation function, and dropouts, are examined, concluding that the appropriate use of these factors can significantly improve the quality of synthesized PETs. • A novel 3D multi-scale CycleGAN is proposed for MRI to PET translation. • A hybrid loss is introduced to enforce structural and voxel-wise similarity. • The proposed framework outperforms state-of-the-art based on different criteria. • Using generated synthesized PETs can improve Alzheimer's disease (AD) diagnosis. • Ablation studies confirm the impact of the proposed framework and loss function. [ABSTRACT FROM AUTHOR]

Although deep convolutional neural networks have achieved remarkable success in removing synthetic fog, it is essential to be able to process images taken in complex foggy conditions, such as dense or non-homogeneous fog, in the real world. However, the haze distribution in the real world is complex, and downsampling can lead to color distortion or loss of detail in the output results as the resolution of a feature map or image resolution decreases. Moreover, the over-stacking of convolutional blocks might increase the model complexity. In addition to the challenges of obtaining sufficient training data, overfitting can also arise in deep learning techniques for foggy image processing, which can limit the generalization abilities of the model, posing challenges for its practical applications in real-world scenarios. Considering these issues, this paper proposes a Transformer-based wavelet network (WaveletFormerNet) for real-world foggy image recovery. We embed the discrete wavelet transform into the Vision Transformer by proposing the WaveletFormer and IWaveletFormer blocks, aiming to alleviate texture detail loss and color distortion in the image due to downsampling. We introduce parallel convolution in the Transformer block, which allows for the capture of multi-frequency information in a lightweight mechanism. Such a structure reduces computational expenses and improves the effectiveness of the network. Additionally, we have implemented a feature aggregation module (FAM) to maintain image resolution and enhance the feature extraction capacity of our model, further contributing to its impressive performance in real-world foggy image recovery tasks. Through extensive experiments on real-world fog datasets, we have demonstrated that our WaveletFormerNet achieves superior performance compared to state-of-the-art methods, as shown through quantitative and qualitative evaluations of minor model complexity. Additionally, our satisfactory results on real-world dust removal and application tests showcase the superior generalization ability and improved performance of WaveletFormerNet in computer vision-related applications compared to existing state-of-the-art methods, further confirming our proposed approach's effectiveness and robustness. Our code is available at https://github.com/shengli666666/WaveletFormerNet. [Display omitted] • Combining Vision Transformer and wavelet transform alleviates image detail loss. • Parallel convolution captures frequency information in a lightweight mechanism. • The feature aggregation module enhances the model feature extraction capability. • Experimental results demonstrate WaveletFormerNet's superior dehazing performance. [ABSTRACT FROM AUTHOR]

In this paper, we propose a comprehensive framework for detecting anomalies in videos based on autoencoder (AE). Traditional AE models solely rely on input and final reconstruction, potentially limiting their capacity to fully utilize the intermediate neural network layers. To mitigate this limitation, we introduce a novel approach that concurrently trains the model using corresponding intermediate layers from both the encoder and decoder. This allows the model to capture more intricate features, thus enhancing its anomaly detection capabilities. Furthermore, we introduce a motion loss function that exclusively relies on original video frames rather than optical flow, rendering it more efficient and capable of extracting motion features. Additionally, we have devised a variance attention strategy that is parameter-free and can automatically directs our model's focus towards moving objects, further boosting the performance of our approach. Our experiments on three public datasets demonstrate the effectiveness and efficiency of our method in identifying abnormal events in complex scenarios. The code is publicly available at https://github.com/lsf2008/multRecLossAEPub. • We propose a multi-layer reconstruction AE framework for anomaly detection in videos. • We present a motion reconstruction loss function designed to capture motion behavior by relying on original video frames. • We propose a parameter-free variance attention strategy that directs the model to focus on the moving target. [ABSTRACT FROM AUTHOR]

Moving image recognition has become a well-explored problem in computer vision. However, it is difficult for the traditional convolutional neural network (CNN) model to effectively capture timing information in motion. For better use of video sequence features and to improve the accuracy of action recognition, Therefore, this paper proposes a Three-dimensional CNN (3DCNN) model based on Gated Recurrent Unit (GRU) with an attention mechanism. The model leverages 3DCNN for deep feature extraction from video frames, employs GRU to capture the temporal dynamics of feature sequences and incorporates an attention mechanism to emphasize key frames, which improves moving image recognition. Demonstrating superior accuracy in'Cross-Subject' and'Cross-View' evaluations, our model surpasses standard benchmarks with accuracies of 83.2% and 87.3% respectively. • We combined 3 DCNN and GRU to classify the sequences of the motion images. • The framework adds attention mechanisms to improve recognition capabilities. • Experimental results show that the action recognition accuracy is increased by 6%. [ABSTRACT FROM AUTHOR]

Weakly supervised object localization (WSOL) aims to localize objects with only image-level labels. As a common WSOL method, adversarial erasing always masks the most discriminative region in the feature space to compel the network to localize more regions of the object. However, with the discriminative region vanishing, the localizer is confused when distinguishing the regions of object from the background. In this paper, we propose a new feature disparity learning (FDL), which encourages the network to learn more distinctive features from the object region with similarity measurement after feature enhancement. Specifically, we first introduce a Spatial Vector Cross Attention (SVCA) module. This module enhances responses in less discriminative region of erased feature maps by reintegrating the spatial distribution of features through the capture of interdependencies among spatial vectors on each channel. Furthermore, we propose a feature complementarity loss to measure the similarity between unerased features and erased features, guiding the network to learn feature disparities caused by adversarial erasing for improved localization and classification. Several experimental studies demonstrate a significant increase in localization performance over the existing state-of-the-art erasing methods on the CUB 200–2011 and ILSVRC 2016 datasets. • A novel adversarial erasing feature disparity learning method. • Introduce a Spatial Vector Cross Attention module to enlargement feature disparity. • The network implicitly learns feature disparities by similarity regularization. [ABSTRACT FROM AUTHOR]

With the widespread adoption of object detection models across various industries, the interpretability of these detectors has become an important research topic. The interpretability of a detector helps humans understand which areas significantly contribute to the model's decision. Furthermore, the interpretability enhances the credibility of detectors and helps identify their strengths and weaknesses. Due to the ability to provide intuitive explanations of models, the saliency map has been widely employed in the field of interpreting deep models. Model-agnostic interpretability methods are more general approaches as they treat the model as a black box without considering its internal complexity structure. However, existing model-agnostic interpretability methods often introduce "noise" into saliency maps by applying random masking and fixed masking granularity. This noise reduces the quality and interpretability of the generated saliency maps. To address this challenge and obtain more interpretable saliency maps for object detection models, this paper proposes a model-agnostic progressive saliency map generation method based on a hierarchical framework called MAPSM. In MAPSM, an adaptive masking partition mechanism is introduced to adapt the masking granularity to different object sizes. Additionally, MAPSM employs a saliency-driven mask generation strategy to effectively reduce the "noise". Utilizing a hierarchical framework, MAPSM progressively discovers and refines the saliency areas of objects, resulting in more interpretable saliency maps. To evaluate the quality of the saliency maps generated by MAPSM, we compare it with other methods in multiple metrics. Experimental results demonstrate that our method produces saliency maps with better quality and interpretability. • A progressive saliency map generation method for explaining object detectors. • The saliency map visualizes the contribution of individual pixels to a prediction. • The quality of the saliency map is validated by multiple metrics. • The method is usable to detectors without accessing their internal information. [ABSTRACT FROM AUTHOR]

UAV tracking is a burgeoning task with vast application prospects in various fields such as agriculture, navigation, and public safety. However, the computational constraints and limited processing speed of drones hinder the deployment of Siamese tracking algorithms. In order to better apply tracking algorithms to drone devices, this paper proposes a novel Ranking-Based SiamCAR (RB-SiamCAR) tracker. The RB-SiamCAR tracker achieves model compression by utilizing a ranking-based filter pruning method, which sorts the filters in the backbone network based on their importance and prunes the filters with low ranks for efficient feature extraction. Additionally, considering existing Siamese trackers overlook the correlation between positive and negative samples and the coherence between classification and localization, we introduce two ranking-based losses. The classification ranking loss ensures that the ranking of positive samples is higher than that of hard negative samples, allowing the tracker to successfully select foreground samples without being fooled by distractors. The IoU-guided ranking loss aims to align the classification with the intersection over union (IoU) of the corresponding localizations of positive samples, enabling well-localized predictions to be represented by high classification confidence. To further enhance the tracking performance, we employ an effective channel attention module (ECA) that allows the network to automatically learn and focus on the most important channels for capturing more discriminative features. Experimental evaluations on UAV tracking benchmarks demonstrate that the proposed RB-SiamCAR outperforms existing state-of-the-art trackers. It is noteworthy that our RB-SiamCAR achieves an impressive tracking speed of nearly 100 fps. The experimental results validate its effectiveness and efficiency in UAV tracking applications. • This paper develops a ranking-based filter pruning technique for effective model compression. • We further incorporate an effective channel attention module to capture more discriminative features. • We introduce a classification ranking loss and IoU-guided ranking loss to facilitate the consistent optimization of classification and localization. • The proposed RB-SiamCAR model outperforms other trackers, showing its superiority in UAV tracking scenarios. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNN) are mainly used for image recognition tasks. However, some huge models are infeasible for mobile devices because of limited computing and memory resources. In this paper, feature maps of DenseNet and CondenseNet are visualized. It could be observed that there are some feature channels in locked state and some have similar distribution property, which could be compressed further. Thus, in this work, a novel architecture — RSNet is introduced to improve the computing efficiency of CNNs. This paper proposes Relative-Squeezing (RS) bottleneck design, where the output is the weighted percentage of input channels. Besides, RSNet also contains multiple compression layers and learned group convolutions (LGCs). By eliminating superfluous feature maps, relative squeezing and compression layers only transmit the most significant features to the next layer. Less parameters are employed and much computation is saved. The proposed model is evaluated on three benchmark datasets: CIFAR-10, CIFAR-100 and ImageNet. Experiment results show that RSNet performs better with less parameters and FLOPs, compared to the state-of-the-art baseline, including CondenseNet, MobileNet and ShuffleNet. [ABSTRACT FROM AUTHOR]

This paper presents a topology-based global descriptor that allows for efficient 3D point cloud processing tasks associated with the analysis of shapes. The descriptor is called the Signature of Topologically Persistent Points (STPP). By using persistent homology, STPP is formed by the computation of topological invariants involving the zeroth and first homology groups. Persistent homology is a methodology that finds the features of a topological space at different spatial resolutions. STPP requires no preprocessing and uses a single tuning parameter. It is an effective 3D point cloud descriptor with robustness to noisy sensor data. The paper highlights this aspect with experimental comparisons to the state of the art. Our research has been validated on a publicly available RGB-D dataset. The results show that STPP can be used as a distinctive signature by employing a small number of features with object detection and classification benefiting from its usage. • STPP is a global shape descriptor for 3D point cloud data. • It encodes a topological signature based on the pairing of homology generators. • STPP can be computed quickly and efficiently, and uses a single tuning parameter. [ABSTRACT FROM AUTHOR]

The purpose of camouflaged object detection (COD) focuses on detecting objects that seamlessly blend into their surroundings. Camouflaged objects pose a substantial challenge in the realm of computer vision due to various factors, including occlusion, limited illumination, and diminutive dimensions. In this paper, we propose a cross-level refinement and interaction network (CRI-Net) to capture camouflaged objects. Specifically, we advance the concept of a semantic amplification module (SAM), which simulates human visual processes through multi-scale parallel convolution in a way of progressive aggregation with the aim of obtaining rich semantic information. Subsequently, we propose a cross-level refinement unit (CRU), which focuses on multivariate information at different levels in an attention-induced manner to facilitate the fusion refinement of features between levels and the exploration of feature similarity. Finally, we design a semantic-texture interaction module (SIM) to facilitate the interaction between high-level semantics and low-level textures while mining rich fine-grained spatial information to improve the integrity of camouflaged objects. By conducting comprehensive experiments on four benchmark camouflaged datasets, our CRI-Net demonstrates significantly superior performance compared to 20 cutting-edge competing methods. • A cross-level refinement and interaction network (CRI-Net) focuses on camouflaged object. • We earnestly design three key components for better study of COD missions to accurately predict camouflaged objects. The Semantic Amplification Module (SAM) is used to extract multi-scale semantic information, the Cross-level Refinement Unit (CRU) is proposed to focus on multivariate information at different levels, and the Semantic-texture Interaction Module (SIM) is designed to facilitate the interaction between high-level semantics and low-level textures. • Experimental results on four challenging COD datasets suggest that our (CRI-Net) is extremely superior in four widely used evaluation indicators of 20 state-of-the-art models. In addition, our approach is more generalized in many other visual tasks with similar COD. [ABSTRACT FROM AUTHOR]

Semantic information and attention mechanism play important roles in the task of image captioning. Semantic information can strengthen the relationship between images and languages, while attention operation can steer the relevant regions spatially in the image. However, in most current works, semantic attributes are always confined to be learned from pairs of images and sentences, which ignore to fully utilize more semantic attributes and the structure information of sentences, thus limit the variety of sentences to be generated. Meanwhile, current attention models usually lack the ability to learn the positional information in an explicit way during attention generation, and have the problem of vanishing gradient in the training process. This paper proposes a Semantic Selection Unit (SSU) and a Stacked Residual Attention (SRA) to remedy these drawbacks. Specifically, the SSU is designed to capture selectively semantic information from expanding attributes or guidance sentences. With the help of expanding vocabulary and the structure information in sentences, the SSU can improve the quality of the generated sentences. The SRA is constructed to solve the problem of positional information missing and vanishing gradient problem during attention generation. Architecturally, the SSU and SRA work together in a jointed framework with end-to-end learning for image captioning. Extensive experiments have been conducted on the public dataset of the MS COCO, achieving 139.7 CIDEr score on the test set. • Semantic information plays an important role in processing image captioning tasks. • Improving the performance of attention models by proposing the stacked residual attention mechanism. • The Transformer models use self-attention mechanisms to ensure the model trained in parallel. • Performance of the attention model declines due to the effect of gradient vanishing problem. [ABSTRACT FROM AUTHOR]

Learning fine-grained discriminative information is essential to address the challenges of small inter-class differences and large intra-class differences in vehicle re-identification (Re-ID). Attentional mechanism is often used to capture important global information in images rather than fine-grained discriminative information. Studies have shown that the multi-axis interaction of information can enhance the feature representation ability of networks. This paper explores how to use the multi-axis interaction of information to facilitate more effective learning of attention and how to capture important detailed information in local regions. We propose a multi-axis interactive multidimensional attention network (MIMA-Net) for vehicle Re-ID. The network allows information to interact on multiple axes and calibrates the weight distribution of features from multiple dimensions to learn subtle discriminative information in vehicle parts/regions. The window-channel attention module (W-CAM) in MIMA-Net facilitates the learning of channel attention by interacting first across locations and then across channels, while the channel group-spatial attention module (CG-SAM) facilitates the learning of spatial attention by interacting first across channels and then across locations. These two modules perform window partitioning in a priori manner and channel semantic aggregation in an adaptive manner to learn discriminative semantic features in parts, respectively. These two approaches complement each other to strengthen the feature representation ability of MIMA-Net. Extensive experiments on three large public datasets, VeRi-776, VehicleID, and VERI-Wild, verify the effectiveness of our MIMA-Net and show that our method achieves state-of-the-art performance. • Utilizing multi-axis interaction of information to promote attention learning. • Learning subtle semantics via multidimensional calibration of weights. • A priori and adaptive ways are combined to learn part-level features. [ABSTRACT FROM AUTHOR]

Multi-object tracking (MOT) is a challenging task in computer vision that aims to estimate the trajectories of multiple objects in a video sequence. Observation-Centric SORT (OCSORT) is a pure motion-based MOT algorithm that uses the Kalman filter as the motion model and three observation-centric techniques: Re-Update, Momentum and Recovery, to enhance the data association. However, OCSORT is limited by camera motion error, constant measurement noise and lack of appearance information. In this paper, we propose three methods to address these limitations and improve the performance of OCSORT. First, we use Enhanced Correlation Coefficient Maximization (ECC) to compensate for the camera motion between adjacent frames. Second, we adjust the measurement noise scale for the Kalman filter according to the detection confidence. Third, we introduce a deep visual feature model to extract appearance information and propose a method to effectively use both motion and appearance information. The proposed method first filters out the inappropriate appearance information based on motion information and then combines the filtered appearance information with the motion information by minimization. We evaluate our algorithm on three MOT benchmarks: MOT17, MOT20 and DanceTrack. The results show that our algorithm achieves state-of-the-art performance on all datasets, especially on DanceTrack, where the objects have highly nonlinear motion and frequent occlusion. Compared to OCSORT, our algorithm improves Higher Order Tracking Accuracy (HOTA) by 1.1%, 0.8%, and 3.5%, and ID F1 Score (IDF1) by 1.7%, 1.9%, and 4.3% on MOT17, MOT20 and DanceTrack, respectively. [Display omitted] • Identify three limitations of OCSORT: camera motion error, constant measurement noise and lack of appearance information. • Incorporate a camera motion compensation module into OCSORT to mitigate camera motion errors. • Propose a method to dynamically adjust the measurement noise scale of the motion model based on the detection confidence. • Introduce a deep visual feature model to OCSORT and propose a method to utilize both motion and appearance information. [ABSTRACT FROM AUTHOR]

Thanks to the wide view field, the fisheye camera can get much more visual information. Thus, it is widely used in the field of computer vision. However, projection is often required for fisheye images to be used for object detection. Meanwhile, the projection will lead to distortion in fisheye images, and the discontinuous image edges will make the objects incomplete. Fisheye images are characterized by objects that are large near and small far. These problems are still challenges for the existing advanced object detector YOLOv7. Therefore, in this paper, we propose an improved YOLOv7 model. First, Modulated Deformable Convolution is introduced into the YOLOv7 model to automatically adapt to distortion changes of distorted objects in fisheye images. It not only adjusts the sampling position of the convolutional kernel but also further extends the deformation range. The improved model can efficiently extract features of distorted and edge-discontinuous objects. In addition, fisheye images are characterized by objects close to the fisheye lens being large, while objects farther away from the fisheye lens will be smaller. To further optimize the detection performance of small objects in fisheye images, Swin Transformer is also introduced into the YOLOv7 model, and Swin Transformer Block with Window Multi-head Self-Attention (W-MSA) Effectively enhances Network Local Perception. Finally, our proposed model achieves up to 2.4% improvement in mAP compared to the original YOLOv7 model on the ERP-360 dataset. Also, the proposed model achieves the best results compared to other state-of-the-art object detection methods for equirectangular projection images. On the VOC-360 dataset, our proposed model improves the mAP by up to 5.9% compared to the original YOLOv7 model. The experimental results show that the proposed models achieve good results for object detection in both fisheye images and equirectangular projection images. The ERP-360 dataset, source code and pre-trained models for related tasks can be found at https://github.com/xiaoxiaomichong/ERP-360dataset. • Modulated Deformable Convolution is introduced into YOLOv7 to effectively detect distorted objects in fisheye images. • Swin Transformer Block is introduced into YOLOv7 to effectively recognize small objects in fisheye images. • A SMDC-YOLOv7 model is proposed which can improve the overall detection performance of objects in fisheye images. [ABSTRACT FROM AUTHOR]

Emotion recognition from facial expressions is a fundamental human ability that can be harnessed and transferred to machines. The ability to differentiate between spontaneous and posed emotions holds significant importance in various domains, including behavioral biometrics, forensics, and security. This paper introduces a novel method, called POsed vs Spontaneous Emotion Recognition (POSER), which leverages a modified version of the Partitioned Iterated Functions System (PIFS) to obtain a Fractal Encoding. This encoding is used for the first time as facial features to train a machine learning approach for the classification of emotions as either spontaneous or posed. Furthermore, by adapting the original architecture, we demonstrate the effectiveness of these features in distinguishing seven different emotions in controlled as well as wild environments, within a framework referred to as POSER-EMO. Experimental results are presented on the SPOS and DISFA + datasets for the first classification problem, where POSER outperforms the state of the art, and on the CK + and SFEW datasets for the second classification problem. [Display omitted] • PIFS modified to extract features in emotion recognition tasks. • Poser vs Spontaneous recognition of emotions by Machine Learning techniques on PIFS. • Emotion classification by a cascade of ML techniques using PIFS as features. • Extensive experiments on CK +, SPOS, DISFA, DISFA + and SFEW. [ABSTRACT FROM AUTHOR]

With the rapid progress of computer vision and artificial intelligence, the accuracy of estimating and analyzing human body movements and postures has always been a highly focused research field. However, current methods still have some shortcomings in accurately estimating the pose of 3D human movements. This study aims to propose an effective method to accurately estimate the motion posture of 3D human activities using new technologies of deep learning and neural networks. Firstly, based on previous research, this paper analyzes the problems and challenges of insufficient accuracy in current 3D human motion pose estimation methods, limited capture of 3D spatial information in deep video data, and inability to capture subtle motion details. Then, in response to the problem of disorder in point clouds, an innovative SMPL human Point-2 s reinforcement learning framework was constructed using the VIBE network to estimate the pose of RGB in the NTU dataset. 24 Point-2 s biomimetic algorithm joints were sampled from a distance using the FPS of PointNet ++in SMPL, and an index was established based on relative positions to ensure that other joint points are in the same position. Finally, these joint points were input into the 2 s-AGCN network to construct a complete Point-2 s model. The research results indicate that the proposed Point-2 s model has achieved good results in accurately estimating the motion posture of 3D human activities, and effectively solves the problem of disorder in point clouds converted from deep videos. Compared to traditional methods, this research model has significantly improved accuracy and stability. The practical application of this method will provide useful references and guidance for research and method development in related fields. • Point-2 s: An RL model for human pose estimation. • Point-2 s: Mitigating disorder in deep video-to-point cloud conversion. • Point-2 s: Directly processing raw point clouds to eliminate data transformation. [ABSTRACT FROM AUTHOR]

Wild Facial Expression Recognition (FER) task has been a long-standing challenge due to the various forms of uncertainty exist in expression data. When expression data is fed into a convolutional neural network (CNN), the model's estimated parameters also become uncertain. This uncertainty gives rise to concerns regarding the reliability of the recognition results. To quantify these uncertainties and achieve robust performance in the presence of noisy data, this paper introduces a novel model for Wild Facial Expression Recognition: the Bayesian Convolutional Neural Network with Perturbed Multi-Branch Structure (BPMB). This model aims to address uncertainty issues, enabling the network's decisions to become more deterministic with increasing training accuracy. Specifically, BPMB incorporates variational inference (VI) to introduce a probability distribution for the weights. A variational approximation to the true posterior is derived using Bayes by Backprop, involving two convolution operations: one for classification and another for quantifying uncertainty. Furthermore, an exploration is conducted into a lightweight multi-branch structure that leverages Dropout as a random generator to introduce perturbations during the training process, enhancing the model's robustness while extracting deep features. Extensive experiments validate the superiority of the proposed BPMB algorithm over the majority of existing mainstream algorithms on three widely utilized wild datasets. • An effective method for measuring uncertainty in facial expression recognition. • Variational inference derives a variational approximation to the true posterior. • Disturbance of multi branch structure enhances the robustness of the model while extracting deep features. • A study provides a probabilistic framework for convolutional neural networks. [ABSTRACT FROM AUTHOR]

The performance of few-shot object detection has seen marked improvement through fine-tuning paradigms. However, existing methods often depend on shared parameters to implicitly transfer knowledge without explicit induction. This results in novel-class representations that are easily confused with similar base classes and poorly suited to diverse patterns of variation in the truth distribution. In view of this, the present paper focuses on mining transferable base-class knowledge, which is further subdivided into inter-class correlation and intra-class diversity. First, we design a graph to dynamically capture the relationship between base and novel class representations, and then introduce distillation techniques to tackle the shortage of correlation knowledge in few-shot labels. Furthermore, an efficient diversity knowledge transfer module based on the data hallucination is proposed, which can adaptively disentangle class-independent variation patterns from base-class features and generate additional trainable hallucinated instances for novel classes. Experiments on VOC and COCO datasets confirmed that our proposed method effectively reduces the reliance on novel-class samples and demonstrates superior performance compared to other state-of-the-art baseline methods. [Display omitted] • Optimizing the knowledge transfer process by mining the transferable knowledge. • Explicitly calibrate and distill inter-class correlation to alleviate class confusion. • Transferring sharable patterns to fill the restricted intra-class variance. • This method is used only in the fine-tuning phase, with no extra inference cost. [ABSTRACT FROM AUTHOR]

The loss of pixel-level information in the multi-class segmentation task based on the U-net model results in unclear boundaries and low semantic segmentation accuracy. Aiming at this, a deep multi-branch residual Unet (IWG-MRUN) with fused inverse weight gated-control is proposed to improve the quality of image semantic segmentation. Specifically, we first introduce a deep multi-branch residual module, which used parallel convolution mode to capture the contextual feature to extract the detailed features of the input image at a deeper level. Then, we adopt an inverse weight gated-control module to enhance the diversity of up-sampling information by counterclockwise transmitting attention horizontally to improve the restoration accuracy of up-sampled image pixels. Finally, to obtain finer granularity features from low spatial resolution images, we adopt the different receptive field pyramid attention mechanisms at the highest level of the U-shaped encoder to capture high-level context information at different scales, thereby improving the accuracy of semantic segmentation. The experimental results show that the segmentation accuracy of the proposed algorithm reaches 91.80% and the CCE loss is reduced to 0.21. When compared to the Unet, BiSeNet, DeeplabV3 + and U-net + BLR model, the pixel accuracy of semantic segmentation is improved by 15.0%, 1.98%, 0.9% and 6.5%, respectively. The semantic segmentation model proposed in this paper provides an end-to-end semantic segmentation capability with the enriched finer granularity features of the target boundary and realizes the accurate segmentation of the objects in different categories. • The loss of pixel-level information results in low semantic segmentation accuracy. • Parallel convolution extracts the detailed features of the image at a deeper level. • Inverse weight gated-control module enhances the expression of important feature. • High-level contextual feature at different scales improves segmentation accuracy. [ABSTRACT FROM AUTHOR]

Human re-identification (re-ID) is nowadays among the most popular topics in computer vision, due to the increasing importance given to safety/security in modern societies. Being expected to sun in totally uncontrolled data acquisition settings (e.g., visual surveillance) automated re-ID not only depends on various factors that may occur in non-controlled data acquisition settings, but - most importantly - performance varies with respect to different subject features (e.g., gender, height, ethnicity, clothing, and action being performed), which may result in highly biased and undesirable automata. While many efforts have been putted in increase the robustness of identification to uncontrolled settings, a systematic assessment of the actual variations in performance with respect to each subject feature remains to be done. Accordingly, the contributions of this paper are threefold: 1) we report the correlation between the performance of three state-of-the-art re-ID models and different subject features; 2) we discuss the most concerning features and report valuable insights about the roles of the various features in re-ID performance, which can be used to develop more effective and unbiased re-ID systems; and 3) we leverage the concept of biometric menagerie , in order to identify the groups of individuals that typically fall into the most common menagerie families (e.g., goats, lambs, and wolves). Our findings not only contribute to a better understanding of the factors affecting re-ID performance, but also may offer practical guidance for researchers and practitioners concerned on human re-identification development. • We announce the results of an analysis about the subject features that provide the largest variations in re-id performance. • We compare the variations in performance of three well known re-identification methods with respect to each of these features (covariates). • We analyze the relation between each feature and the most important biometric families, regarding the well-known menagerie taxonomy. [ABSTRACT FROM AUTHOR]

Person search has become one of the most critical and challenging applications in today's video surveillance systems. It helps in locating a person in surveillance videos, which is plausible only with advanced deep learning models, large scale datasets and high compute power GPUs. This survey features exhaustive analysis of deep learning based person search through image, textual and attributes based description. The image based person search is reviewed based on aspects such as region proposal consideration, feature representation, context information and compute complexity. The text based person search is reviewed based on the aspects of feature representation and alignment. The attribute based person search is reviewed based on the aspect of high level and low level feature representation. The paper summarizes more than 100 research works and provides future perspectives for enhancements with the objective of guiding and facilitating the development of better solutions in future. We believe that this exclusive review on deep learning based person search for video surveillance security systems will facilitate better systematization. • Examines deep learning methods applied to person search tasks. • Explores feature representation, loss functions, datasets, and metrics. • Evaluates image-based person search and person re-identification techniques. • Summarizes the state-of-the-art in text-based and attribute-based person search. • Provides several challenging future directions in the field of person search. [ABSTRACT FROM AUTHOR]

Automatic detection of abnormal behavior in video sequences is a fundamental and challenging problem for intelligent video surveillance systems. However, the existing state-of-the-art Video Anomaly Detection (VAD) methods are computationally expensive and lack the desired robustness in real-world scenarios. The contemporary VAD methods cannot detect the fundamental features absent during training, which usually results in a high false positive rate while testing. To this end, we propose a Constrained Generative Adversarial Network (CVAD-GAN) for real-time VAD. Adding white Gaussian noise to the input video frame with constrained latent space of CVAD-GAN improves its fine-grained features learning from the normal video frames. Also, the dilated convolution layers and skip-connection preserve the information across layers to understand the broader context of complex video scenes in real-time. Our proposed approach achieves a higher Area Under Curve (AUC) score and a lower Equal Error Rate (EER) with enhanced computational efficiency than the existing state-of-the-art VAD methods. CVAD-GAN achieves an AUC and EER score of 98.0% and 6.0% on UCSD Peds1, 97.8% and 7.0% on UCSD Peds2, 94.0% and 8.1% on CUHK Avenue, and 76.2% and 21.7% on ShanghaiTech dataset, respectively. Also, it detects 63 and 19 abnormal events, with false alarms of 3 and 1, respectively, on the Subway-Entry and Subway-Exit datasets. The source code to replicate the results of the proposed CVAD-GAN is available at https://github.com/Rituraj-ksi/CVAD-GAN. • The paper introduced a novel GAN-based framework for real-time video anomaly detection. • The proposed model (CVAD-GAN) is robust and attained competitive AUC score on datasets. • The model used dilated convolution layers and skip connections to enhance feature representation. • It calculates the anomaly score based on discriminator score and pixel differences between real and generated images. [ABSTRACT FROM AUTHOR]