Search

The success of deep learning-based techniques in solving various computer vision problems motivated the researchers to apply deep learning to predict the optical flow of a video in the next frame. However, the problem of predicting the motion of an object in the next few frames remains an unsolved and less explored problem. Given a sequence of frames, predicting the motion in the next few frames of the video becomes difficult in cases where the displacement of optical flow vector across frames is large. Traditional CNNs often fail to learn the dynamics of the objects across frames in case of large displacements of objects in consecutive frames. In this paper, we present an efficient CNN based on the concept of feature pyramid for extracting the spatial features from a few consecutive frames. The spatial features extracted from consecutive frames by a modified PWC-Net architecture are fed into a bidirectional LSTM for obtaining the temporal features. The proposed spatiotemporal feature pyramid is able to capture the abrupt motion of the moving objects in video, especially when displacement of the object is large across the consecutive frames. Further, the proposed spatiotemporal pyramidal feature can effectively predict the optical flow in next few frames, instead of predicting only the next frame. The proposed method of predicting optical flow outperforms the state of the art when applied on challenging datasets such as “MPI Sintel Final Pass,” “Monkaa” and “Flying Chairs” where abrupt and large displacement of the moving objects in consecutive frames is the main challenge. [ABSTRACT FROM AUTHOR]

Accurate segmentation of pulmonary nodule is essential for subsequent pathological analysis and diagnosis. However, current U-Net architectures often rely on a simple skip connection scheme, leading to the fusion of feature maps with different semantic information, which can have a negative impact on the segmentation model. In response to this challenge, this study introduces a novel U-shaped model specifically designed for pulmonary nodule segmentation. The proposed model incorporates features such as the U-Net backbone, semantic aggregation feature pyramid module, and reverse attention module. The semantic aggregation module combines semantic information with multi-scale features, addressing the semantic gap between the encoder and decoder. The reverse attention module explores missing object parts and captures intricate details by erasing the currently predicted salient regions from side-output features. The proposed model is evaluated using the LIDC-IDRI dataset. Experimental results reveal that the proposed method achieves a dice similarity coefficient of 89.11%and a sensitivity of 90.73 %, outperforming state-of-the-art approaches comprehensively., Competing Interests: Declaration of competing interest No conflict of interest. We declare that we have no financial and personal relationships with other people or organizations that can inappropriately influence our work, there is no professional or other personal interest of any nature or kind in any product, service and company that could be construed as influencing the position presented in the manuscript., (Copyright © 2024. Published by Elsevier Ltd.)

Recent advances in object detection play a key role in various industrial applications. However, a fully convolutional one-stage detector (FCOS), a conventional object detection method, has low detection accuracy given the calculation cost. Thus, in this study, we propose a half-inverted stage FCOS (HISFCOS) with improved detection accuracy at a computational cost comparable to FCOS based on the proposed half inverted stage (HIS) block. First, FCOS has low detection accuracy owing to low-level information loss. Therefore, an HIS block that minimizes feature loss by extracting spatial and channel information in parallel is proposed. Second, detection accuracy was improved by reconstructing the feature pyramid on the basis of the proposed block and improving the low-level information. Lastly, the improved detection head structure reduced the computational cost and amount compared to the conventional method. Through experiments, the proposed method defined the optimal HISFCOS parameters and evaluated several datasets for fair comparison. The HISFCOS was trained and evaluated using the PASCAL VOC and MSCOCO2017 datasets. Additionally, the average precision (AP) was used as an evaluation index to quantitatively evaluate detection performance. As a result of the experiment, the parameters were increased by 0.5 M compared to the conventional method, but the detection accuracy was improved by 3.0 AP and 1.5 AP in the PASCAL VOC and MSCOCO datasets, respectively. in addition, an ablation study was conducted, and the results for the proposed block and detection head were analyzed.

Small object detection becomes a challenging problem in computer vision due to low resolution and less feature information. Making full use of high-resolution features is an important factor in improving small object detection. In this paper, to improve the utilization of high-resolution features, this work proposes the Bidirectional Multi-scale Feature Enhancement Network (BFE-Net) based on RetinaNet. First, this work introduces a bidirectional feature pyramid structure to shorten the propagation path of high-resolution features. Then, this work utilizes residually connected dilated convolutional blocks to fully extract high-resolution features of low-feature layers. Finally, this work supplements the high-resolution features lost in the high-level feature propagation process by leveraging the high-level guided lower-level features. Experiments show that our proposed BFE-Net achieves stable performance gains in the object detection task. Specifically, the improved method improves RetinaNet from 34.4 AP to 36.3 AP on the challenging MS COCO dataset and especially achieves excellent results in small object detection with an improvement of 2.8%.

Abstract Histopathology image analysis is a gold standard for cancer recognition and diagnosis. But typical problems with histopathology images that hamper automatic analysis include complex clinical features, insufficient training data, and large size of a single image (always up to gigapixels). In this paper, an image semantic segmentation algorithm based on feature Pyramid (ResNet50-GICN-GPP) is proposed. Firstly, the patch sampling method is adopted to resample the image, reduce the size of a single sample, and expand the number of training samples. Secondly, design the whole convolution network based on ResNet50 learning feature location information, then use GICN structure and deconvolution network to integrate multi-level features. Finally, in order to solve the problem that the GICN structure may lose the small object, the GPP structure should be joined to explore the multi-scale semantic information. The proposed method achieves 63% of the average segmentation accuracy (Dice coefficient) on Camelyon16 and Gastric WSIs Data, compared with U-Net, FCN and SegNet which has 10~20% improvement, and fully demonstrates the effectiveness of this method in different types of cancer. By experimentally comparing the segmentation accuracy of various scales of cancerous tissues, the performance of ResNet50-GICN-GPP is balanced and the multi-scale information localization is more accurate.

In view of the current deficiencies that the previous saliency detection algorithms are difficult to segment the complete salient region and sharp edge details in complex scenes, a novel feature fusion of saliency detection model is proposed in this paper. The proposed algorithm utilizes full convolution neural network to obtain the initial features of multi-level roughness and combines the feature pyramid structure to analyze its depth. In order to realize the gradual fusion and transmission of features, the progressive structural receptive field module is designed to transform features to different scales of space for optimization. The global attention mechanism is used to eliminate the background noise and establish the long-distance dependence between the saliency pixels, so as to improve the effectiveness of the saliency region, highlight the saliency region, and then obtains the saliency map by learning and fusing the hierarchical features. The comprehensive experiment show that the F-measure index is far beyond the other seven mainstream methods when the absolute error is reduced. The proposed saliency model combines the advantages of full convolution neural network and feature pyramid structure, and combines the gradual structure receptive field and global attention mechanism designed in this study to make the saliency map closer to the truth map. [ABSTRACT FROM AUTHOR]

Measurement of anatomical structures from ultrasound images requires the expertise of experienced clinicians. Moreover, there are artificial factors that make an automatic measurement complicated. In this paper, we aim to present a novel end-to-end deep learning network to automatically measure the fetal head circumference (HC), biparietal diameter (BPD), and occipitofrontal diameter (OFD) length from 2D ultrasound images. Fully convolutional neural networks (FCNNs) have shown significant improvement in natural image segmentation. Therefore, to overcome the potential difficulties in automated segmentation, we present a novelty FCNN and add a regression branch for predicting OFD and BPD in parallel. In the segmentation branch, a feature pyramid inside our network is built from low-level feature layers for a variety of fetal head in ultrasound images, which is different from traditional feature pyramid building methods. In order to select the most useful scale and reduce scale noise, attention mechanism is taken for the feature's filter. In the regression branch, for the accurate estimation of OFD and BPD length, a new region of interest (ROI) pooling layer is proposed to extract the elliptic feature map. We also evaluate the performance of our method on large dataset: HC18. Our experimental results show that our method can achieve better performance than the existing fetal head measurement methods. [ABSTRACT FROM AUTHOR]

Deep learning methods for optical flow estimation usually increase the receptive field of convolution through reducing image resolution, which results in loss of spatial detail information during feature extraction. In this paper, we introduce dilated convolution into feature pyramid network, which can extract multi-scale features containing more motion details and can further improve the accuracy of optical flow estimation. The unsupervised loss function is based on forward–backward consistency check and robust census transform that has a good constraint performance in the case of illumination changes, which can train an unsupervised learning optical flow model with higher accuracy. Our network is trained on FlyingChairs and KITTI raw datasets with an unsupervised manner and tested on MPI-Sintel, KITTI 2012 and KITTI 2015 benchmarks. The experimental results show the advantages of our method in unsupervised learning approaches. [ABSTRACT FROM AUTHOR]

Recent deep learning-based methods for saliency detection have proved the effectiveness of integrating features with different scales. They usually design various complex architectures of network, e.g., multiple networks, to explore the multi-scale information of images, which is expensive in computation and memory. Feature maps produced with different subsampling convolutional layers have different spatial resolutions; therefore, they can be used as the multi-scale features to reduce the costs. In this paper, by exploiting the in-network feature hierarchy of convolutional networks, we propose a novel multi-scale network for saliency detection (MSNSD) consisting of three modules, i.e., bottom-up feature extraction, top-down feature connection and multi-scale saliency prediction. Moreover, to further boost the performance of MSNSD, an input image-aware saliency aggregation method is proposed based on the ridge regression, which combines MSNSD with some well-performed handcrafted shallow models. Extensive experiments on several benchmarks show that the proposed MSNSD outperforms the state-of-the-art saliency methods with less computational and memory complexity. Meanwhile, our aggregation method for saliency detection is effective and efficient to combine deep and shallow models and make them complementary to each other. [ABSTRACT FROM AUTHOR]

In recent years, the field of object detection has made significant progress. The success of most state-of-the-art object detectors is derived from the use of feature pyramid and the carefully designed anchor boxes. However, the existing methods for constructing feature pyramid blindly integrate multi-scale representations on each feature hierarchy. Furthermore, these detectors also suffer from some drawbacks brought by the hand-designed anchors. To mitigate the adverse effects caused thereby, we propose a semi-anchor-free network with enhanced feature pyramid for object detection, named SAFNet. Specifically, to better construct feature pyramid, we propose a novel enhanced feature pyramid generation paradigm, which consists of two modules, i.e., adaptive feature fusion module (AFFM) and self-enhanced module (SEM). The paradigm adaptively integrates multi-scale representations in a non-linear way meanwhile suppresses the redundant semantic information for each pyramid level. Thus, a clean and enhanced feature pyramid could be obtained. In addition, an adaptive anchor generator (AAG) is designed to yield fewer but more suitable anchor boxes for each input image. Benefiting from the enhanced feature pyramid, AAG is capable of generating more accurate anchor boxes by introducing few priors. With this semi-anchor-free method, our detector has the ability to alleviate the drawbacks of hand-designed anchors meanwhile retain the merits of anchor-based methods. Extensive experiments demonstrate the effectiveness of our approach. Profited from the proposed modules, SAFNet significantly boosts the detection performance, i.e., achieving 2 points and 2.1 points higher Average Precision (AP) than RetinaNet on PASCAL VOC and MS COCO respectively. [ABSTRACT FROM AUTHOR]

With the development of driverless technology, we are in dire need of a method to understand traffic scenes. However, it is still a difficult task to detect traffic signs because of the tiny scale of signs in real-world images. In complex scenarios, some traffic signs could be very elusive due to the awful weather and lighting conditions. To implement a more comprehensive detection and recognition system, we develop a two-stage network. At the region proposal stage, we adopt a deep feature pyramid architecture with lateral connections, which makes the semantic feature of small object more sensitive. At the classification stage, densely connected convolutional network is used to strengthen the feature transmission and multiplexed, which leads to more accurate classification with less number of parameters. We test on GTSDB detection benchmark, as well as the challenging Tsinghua-Tencent 100K benchmark which is pretty difficult for most traditional networks. Experiments show that our proposed method achieves a very great performance and surpasses the other state-of-the-art methods. Implementation source code is available at https://github.com/derderking/Traffic-Sign. [ABSTRACT FROM AUTHOR]

Image denoising is 1 of the fundamental problems in the image processing field since it is the preliminary step for many computer vision applications. Various approaches have been used for image denoising throughout the years from spatial filtering to model-based approaches. Having outperformed all traditional methods, neural-network-based discriminative methods have gained popularity in recent years. However, most of these methods still struggle to achieve flexibility against various noise levels and types. In this paper, a deep convolutional autoencoder combined with a variant of feature pyramid network is proposed for image denoising. Simulated data generated by Blender software along with corrupted natural images are used during training to improve robustness against various noise levels. Experimental results show that the proposed method can achieve competitive performance in blind Gaussian denoising with significantly less training time required compared to state of the art methods. Extensive experiments showed the proposed method gives promising performance in a wide range of noise levels with a single network. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Image & Graphics is the property of Editorial Office of Journal of Image & Graphics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Pavement crack detection is a critical task for insuring road safety. Manual crack detection is extremely time-consuming. Therefore, an automatic road crack detection method is required to boost this progress. However, it remains a challenging task due to the intensity inhomogeneity of cracks and complexity of the background, e.g., the low contrast with surrounding pavements and possible shadows with a similar intensity. Inspired by recent advances of deep learning in computer vision, we propose a novel network architecture, named feature pyramid and hierarchical boosting network (FPHBN), for pavement crack detection. The proposed network integrates context information to low-level features for crack detection in a feature pyramid way, and it balances the contributions of both easy and hard samples to loss by nested sample reweighting in a hierarchical way during training. In addition, we propose a novel measurement for crack detection named average intersection over union (AIU). To demonstrate the superiority and generalizability of the proposed method, we evaluate it on five crack datasets and compare it with the state-of-the-art crack detection, edge detection, and semantic segmentation methods. The extensive experiments show that the proposed method outperforms these methods in terms of accuracy and generalizability. Code and data can be found in https://github.com/fyangneil/pavement-crack-detection. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Zhengzhou University: Engineering Science is the property of Editorial Office of Journal of Zhengzhou University: Engineering Science and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Pedestrian detection is one of the challenging tasks in the urban traffic environments. A natural urban traffic environments include different objects like buildings, vehicles, pedestrians and so on. The conventional approach only used for a particular traffic scenario and it does not suitable for different scenarios. A novel approach is required to model these traffic scenarios. Multiresolution Morlet Decomposition Based Iterative Learning Deformable Part Model (MMD-ILDP) is proposed for improving the performance of multiresolution pedestrian detection to control the traffic in the urban area with higher accuracy. The MMD-ILDP Model uses Morlet wavelet transformation for decomposing the image into subbands with multiple resolutions. After wavelet decomposition, histogram of oriented gradients (HOG) feature pyramid is generated. Then, feature matching is performed between the pedestrian objects in the image and the feature pyramid generated with HOG and the root and part scores are computed. Finally, the root and part scores are combined to compute the final score of objects in the image. The performance measures used in evaluating the proposed algorithm are detection accuracy, time and space complexity. The simulation results show that the MMD-ILDP Model gives improved pedestrian detection in urban traffic environment where healthcare systems find more difficult to reach to the people and also reduces the time complexity on detecting the pedestrians in road traffic scenes when compared to the existing DPM and RealBoost methods. [ABSTRACT FROM AUTHOR]

Automatic sleep scoring is essential for the diagnosis and treatment of sleep disorders and enables longitudinal sleep tracking in home environments. Conventionally, learning-based automatic sleep scoring on single-channel electroencephalogram (EEG) is actively studied because obtaining multi-channel signals during sleep is difficult. However, learning representation from raw EEG signals is challenging owing to the following issues: (1) sleep-related EEG patterns occur on different temporal and frequency scales and (2) sleep stages share similar EEG patterns. To address these issues, we propose an automatic Slee p scoring framework that incorporates (1) a feature Py ramid and (2) supervised Co ntrastive learning, named SleePyCo. For the feature pyramid, we propose a backbone network named SleePyCo-backbone to consider multiple feature sequences on different temporal and frequency scales. Supervised contrastive learning allows the network to extract class discriminative features by minimizing the distance between intra-class features and simultaneously maximizing that between inter-class features. Comparative analyses on four public datasets demonstrate that SleePyCo consistently outperforms existing frameworks based on single-channel EEG. Extensive ablation experiments show that SleePyCo exhibited an enhanced overall performance, with significant improvements in discrimination between sleep stages, especially for N1 and rapid eye movement (REM). Source code is available at https://github.com/gist-ailab/SleePyCo. • SleePyCo uses a feature pyramid and supervised contrastive learning for sleep scoring. • We incorporate feature pyramid and propose SleePyCo-backbone for diverse EEG scales. • Our training framework is based on supervised contrastive learning for sleep scoring. • SleePyCo outperforms state-of-the-art frameworks on four public datasets. • Feature pyramid and supervised contrastive learning synergize in SleePyCo. [ABSTRACT FROM AUTHOR]

In recent years, unsupervised person re-identification (re-ID) has attracted a lot of attention because it can save manual annotation costs and adapt to new scenes more conveniently in real-world applications. We focus on fully unsupervised learning-based re-ID, which aims to train a discriminative model based on unlabeled person images. In unsupervised learning, we need to generate pseudo labels by clustering convolutional features and then train convolutional neural network (CNN) models with these pseudo labels. The features used in the clustering process play an important role to ensure the quality of pseudo labels. Hence, we propose an enhancing feature extraction method to increase the reliability of generated pseudo labels and thus facilitate CNN model training. In order to enrich the obtained features, we carry out the feature extraction from both global and local aspects. The global features are extracted with ResNet50 backbone as many existing methods do. The local features are extracted by an additional part-based feature pyramid structure, in which the person image is divided into three parts and the features are extracted from each part with a pyramid structure. Then, we fuse the multi-layer pyramid features for each part as the local features. According to the joint global features and local features, the pseudo labels are predicted using clustering algorithms and further refined based on the similarity between global and local features. In addition, we design an inter-camera association learning component to effectively learn the ID discrimination ability across cameras. Extensive experiments on three large and representative person re-ID datasets demonstrate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

For fine-scale prediction tasks such as semantic segmentation, existing segmentation models cannot support detailed segmentation due to the difficulty of assigning deep feature semantics generated by the encoder to shallow features, thus making the segmentation of details ambiguous in semantic segmentation scenarios. In addition, high-precision models often require large quantities of computational resources. To solve the above problems, we design an efficient cross-information fusion decoder (ECFD). In the ECFD, we design a cross-information fusion block (CFB), and contextual information is used to assign semantic information to the shallow features in spatial domain, thus facilitating the classification of the details of segmented objects. To reduce the computational effort of the model, we choose the same decoder structure as used by the efficient SenFormer: the feature pyramid structure. Compared with SenFormer, ECFD-Swin-Large reduces the numbers of parameters and floating-point operations by 1/3, and achieves 83.61% and 64.98% of mIoU values for the benchmark datasets Cityscapes and Pascal Context, respectively, outperforming SenFormer, especially for in detailed segmentation. In addition, 69.19% is obtained on BDD100K. The code is publicly available at https://github.com/songyang-xiaobai/ECFD-main. • Cross-information Fusion Module addresses the problems of blurring boundaries or small object segmentation in scenes. • Cross-information Fusion Module ensures model accuracy while reducing the required computational resources. • Advanced test results have been achieved in three datasets: Cityscapes, Pascal Context, and BDD100K. [ABSTRACT FROM AUTHOR]

Repeated observation mechanism can effectively solve the problem of low efficiency of feature extraction. By extracting features for many times to strengthen target features, this paper proposed a multi-scale switchable atrous convolution based on feature pyramid, SPC. The head of the detector adopted pyramid convolution mode, constructs 3-D convolution in the feature pyramid, and detected the same target in different pyramid levels by using the shared convolution with different stride changes, which realized the repeated observation of target features on multi-scale. The module optimized the convolution layer, extracted the features of the same image by convolution check of different sizes, and then selected and integrated the extracted results by using switch function, which effectively expanded the field of view of convolution kernel. In this paper, we choosed retinanet as the baseline network, and improved the loss function of focal loss proposed by retinanet to further solved the problem of unbalanced number of samples and sample distribution in the network model. The proposed method performed well on MS coco data set, improved the average accuracy of 9.8% on the basis of retinanet to 48.9%, and achieved FPS of 5.1 in 1333 * 800 images.

Unlike single geospatial objects extraction from high-resolution remote sensing images, the task of road extraction faces more challenges, including its narrowness, sparsity, diversity, multiscale characteristics, and class imbalance. Focusing on these challenges, this paper proposes an end-to-end framework called the multiple feature pyramid network (MFPN). In MFPN, we design an effective feature pyramid and a tailored pyramid pooling module, taking advantage of multilevel semantic features of high-resolution remote sensing images. In the optimization stage, a weighted balance loss function is presented to solve the class imbalance problem caused by the sparseness of roads. The proposed novel loss function is more sensitive to the misclassified and the sparse real labeled pixels and helps to focus on the spare set of hard pixels in the training stage. Compared with the cross-entropy loss function, the weighted balance loss can reduce training time dramatically for the same precision. Experiments on two challenging datasets of high-resolution remote sensing images which illustrate the performance of the proposed algorithm have achieved significant improvements, especially for narrow rural roads.

Feature Pyramid Network (FPN) is used as the neck of current popular object detection networks. Research has shown that the structure of FPN has some defects. In addition to the loss of information caused by the reduction of the channel number, the features scale of different levels are also different, and the corresponding information at different abstract levels are also different, resulting in a semantic gap between each level. We call the semantic gap level imbalance. Correlation convolution is a way to alleviate the imbalance between adjacent layers; however, how to alleviate imbalance between all levels is another problem. In this article, we propose a new simple but effective network structure called Scale-Equalizing Feature Pyramid Network (SEFPN), which generates multiple features of different scales by iteratively fusing the features of each level. SEFPN improves the overall performance of the network by balancing the semantic representation of each layer of features. The experimental results on the MS-COCO2017 dataset show that the integration of SEFPN as a standalone module into the one-stage network can further improve the performance of the detector, by ∼1AP, and improve the detection performance of Faster R-CNN, a typical two-stage network, especially for large object detection APL∼2AP.

Convolutional-neural-network (CNN)-based methods are continuously used in various industries with the rapid development of deep learning technologies. However, an inference efficiency problem was reported in applications that require real-time performance, such as a mobile device. It is important to design a lightweight network that can be used in general-purpose environments such as mobile environments and GPU environments. In this study, we propose a lightweight network efficient shot detector (ESDet) based on deep training with small parameters. The feature extraction process was performed using depthwise and pointwise convolution to minimize the computational complexity of the proposed network. The subsequent layer was formed in a feature pyramid structure to ensure that the extracted features were robust to multiscale objects. The network was trained by defining a prior box optimized for the data set of each feature scale. We defined an ESDet-baseline with optimal parameters through experiments and expanded it by gradually increasing the input resolution for detection accuracy. ESDet training and evaluation was performed using the PASCAL VOC and MS COCO2017 Dataset. Moreover, the average precision (AP) evaluation index was used for quantitative evaluation of detection performance. Finally, superior detection efficiency was demonstrated through the experiment compared to the conventional detection method.

Deep learning approaches have been widely used in building automatic extraction tasks and have made great progress in recent years. However, the missing detection and wrong detection causing by spectrum confusion is still a great challenge. The existing fully convolutional networks (FCNs) cannot effectively distinguish whether the feature differences are from one building or the building and its adjacent non-building objects. In order to overcome the limitations, a building multi-feature fusion refined network (BMFR-Net) was presented in this paper to extract buildings accurately and completely. BMFR-Net is based on an encoding and decoding structure, mainly consisting of two parts: the continuous atrous convolution pyramid (CACP) module and the multiscale output fusion constraint (MOFC) structure. The CACP module is positioned at the end of the contracting path and it effectively minimizes the loss of effective information in multiscale feature extraction and fusion by using parallel continuous small-scale atrous convolution. To improve the ability to aggregate semantic information from the context, the MOFC structure performs predictive output at each stage of the expanding path and integrates the results into the network. Furthermore, the multilevel joint weighted loss function effectively updates parameters well away from the output layer, enhancing the learning capacity of the network for low-level abstract features. The experimental results demonstrate that the proposed BMFR-Net outperforms the other five state-of-the-art approaches in both visual interpretation and quantitative evaluation.

This paper proposes a novel method for hand-raising detection in real classroom environments. Different from traditional motion detection, the hand-raising detection is quite challenging due to complex backgrounds, various gestures and low resolutions. To solve these challenges, we build up a large-scale dataset from videos of real classrooms, and propose a novel neural network architecture based on region-based, fully convolutional networks (R-FCN). Specifically, we first design an adaptive templates selection algorithm for various gestures of hand-raising detection. Secondly, for better detection of small-size hands, we design a feature pyramid to simultaneously capture the detail and highly semantic features. Compared with state-of-the-art object detection algorithms, our method achieves impressive results on our hand-raising dataset. After extensive testing, the accuracy of the hand-raising detection achieves 90% in mean Average Precision (mAP), which can satisfy real applications. [ABSTRACT FROM AUTHOR]

• We propose a new spatial pyramid module to aggregate inherent multi-scale features of a CNN for action recognition. • We propose a new temporal pyramid module to reason about different timescale relations for action recognition. • We propose S-TPNet to learn an effective spatiotemporal pyramid feature for action recognition. Convolutional Neural Networks (CNNs) usually use top-level appearance features of video frames for action recognition. However, these methods discard the implicit complementary advantages across different-scale appearance representations which are effective for object detection, instance segmentation and person re-identification. In this paper, a new spatial pyramid module is proposed to take full use of inherent multi-scale information of CNNs with nearly cost-free by which a bottom-up architecture with lateral connections is constructed for combining high-, mid-, low-level representations of CNNs into a hierarchical frame-level feature elaborately. Additionally, temporal relations at appropriate timescale are contributed to the identification of an action. To this end, we also propose a new temporal pyramid module in which frame-level features belonged to one video are reused by various timescale pooling approaches to get different time-grained features of snippets efficiently. Followed by snippet-relation reasoning, different timescale temporal relations are derived and accumulated for the comprehensive prediction. Unifying the proposed spatial and temporal pyramid modules, a novel network, Spatial-Temporal Pyramid Network (S-TPNet), is proposed to extract spatial-temporal pyramid features for action recognition in videos. Unlike previous models which boost performance at the cost of computation, S-TPNet can be trained in an end-to-end fashion with great efficiency. Extensive experiments on Kinetics, UCF101, and HMDB51 demonstrate that S-TPNet displays significant performance improvements compared with existing frameworks and obtains competitive performance with the state-of-the-arts. 1 1 The code is available at https://www.github.com/ZhenxingZheng/S-TPNet. [ABSTRACT FROM AUTHOR]

Due to the high proportion of aircraft faults caused by cracks in aircraft structures, crack inspection in aircraft structures has long played an important role in the aviation industry. The existing approaches, however, are time-consuming or have poor accuracy, given the complex background of aircraft structure images. In order to solve these problems, we propose the YOLOv3-Lite method, which combines depthwise separable convolution, feature pyramids, and YOLOv3. Depthwise separable convolution is employed to design the backbone network for reducing parameters and for extracting crack features effectively. Then, the feature pyramid joins together low-resolution, semantically strong features at a high-resolution for obtaining rich semantics. Finally, YOLOv3 is used for the bounding box regression. YOLOv3-Lite is a fast and accurate crack detection method, which can be used on aircraft structure such as fuselage or engine blades. The result shows that, with almost no loss of detection accuracy, the speed of YOLOv3-Lite is 50% more than that of YOLOv3. It can be concluded that YOLOv3-Lite can reach state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Copyright of Chinese Journal of Liquid Crystal & Displays is the property of Chinese Journal of Liquid Crystal & Displays and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Copyright of Journal of Image & Graphics is the property of Editorial Office of Journal of Image & Graphics and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Pedestrian detection is a challenging research task, and it is widely applied in automatic driving and intelligent surveillance fields. Although many approaches based on deep learning have shown effectiveness for detecting pedestrian, these approaches are difficult to achieve a good trade-off between real time and accuracy. In this paper, a new pedestrian detection algorithm is proposed to address the above problem, and then, a new pedestrian dataset is introduced to evaluate detection performance in our experiment. Our model contains region generation module and region prediction module, and our model allows for parallel processing of two modules for speed. The feature pyramid strategy is adopted in generation module to make full use of features, and deconvolution layers are used to obtain more high-level feature contextual. The deep supervision idea is introduced to prediction module to guide the detection results toward ground truth. Eventually, the proposed method is evaluated on three different datasets (INRIA, ETH and Caltech) and compared with other existing state-of-the-arts, and the experimental results present the competitive accuracy and real time of the proposed method. [ABSTRACT FROM AUTHOR]

The proposed work introduces a novel, lightweight feature fusion network model based on the attention mechanism to address the issues of high time consumption and poor effectiveness in wind turbine blade surface defect detection. A bidirectional feature fusion network enhances the YOLOX network. The model employs an attention mechanism module to learn channel and spatial feature information. A classification loss function with an attenuation factor is designed to tackle the unbalanced distribution of learning weights for multiple samples in the blade surface defect detection task. Additionally, to mitigate recognition loss caused by target occlusion or overlap, the Soft-NMS method is utilized to eliminate redundant detection boxes. Sample imbalance is addressed by creating new samples using a geometric transformation-based multi-sample fusion data enhancement method and unsupervised learning MFF-GAN image editing techniques. A feature fusion network comparison test, classification loss function comparison test, and attention mechanism module position ablation test were completed using the blade surface defect dataset. The results indicate that the blade surface defect detection model incorporating the attention mechanism can effectively identify five types of defects with an average detection accuracy of mAP-0.5, achieving approximately 95.03 % and a detection frame rate of about 54.56 frame·s−1. Compared with the YOLOX-s network, YOLOv7-tiny network, and YOLOv8-s network, this model can improve the recognition rate and shorten the detection time, achieving high precision and rapid identification and localization of blade surface defects. [ABSTRACT FROM AUTHOR]

Animal pose estimation serves as an upstream task for recognizing and understanding animal behavior. Over the last year, the accuracy of the deep learning-based method has steadily improved, but at the expense of the model's inference speed. This paper uses an efficient and powerful model to improve inference speed and accuracy. The classic encoder–decoder architecture is chosen. For estimating animal pose, our model based on a feature pyramid and a multi-scale asymmetric convolution attention mechanism is developed and named MAPoseNet (Animal Pose Estimation Network Via Multi-scale Convolutional Attention). MAPoseNet consists of an encoder and a decoder. Rather than typical self-attention, the encoder's attention mechanism comprises multi-scale, asymmetric convolutions that are lightweight and instrumental in improving inference speed. A feature pyramid and a feature balance module make up the decoder. The public dataset AP-10K is used to train and test MAPoseNet. A series of experimental results demonstrate that the MAPoseNet model provides cutting-edge performance. MAPoseNet outperforms HRFormer by 1.3 AP and 0.8 AR, with 33.7% fewer FLOPs and 66% faster inference speed. And our model surpasses HRNet and HRFormer on the Animal Pose dataset as well. Our model has achieved a win-win situation regarding inference speed and accuracy. [Display omitted] • A faster and more accurate model is built for animal pose estimation. • Multi-scale convolutional attention helps our model capture long-range information. • Fuse the feature maps from each stage and find how to keep the features balanced. [ABSTRACT FROM AUTHOR]

• Using low-level features for pedestrian Re-ID has significant advantages. • The bidirectional fusion branch networks can preserve low-level features effectively. • The model's clustering performance is improved through using penalty terms. • The optimized sample space has better generalization ability. Person re-identification (Re-ID) is the recognition of the same person in different camera views. Because of the existence of highly similar persons and great differences of the same person in different scenes, and the fact that the features extracted by current mainstream models lose some fine-grained information, it is likely for the models to misidentify the query person. To tackle these challenges, we introduce a bidirectional fusion branch network with penalty term-based trihard loss (BFB-PTT). The BFB-PTT constructs a bidirectional fusion branch (BFB) network based on feature pyramid, where low-level features are transferred to a high-level feature space through fewer convolutional layers than most of the traditional CNN-based models have, thus retaining more local features to discriminate different pedestrians more accurately and effectively. Meanwhile, we propose using the penalty term-based trihard loss (PTT) to optimize the spatial structure of pedestrian's samples, so that the similar samples are drawn closer together in order to reduce the variabilities of the same person in different scenes. We have conducted comprehensive experiments and analyses on the proposed method's effectiveness on three challenging benchmarks, and the results show that our approach achieves competitive performance with the state-of-art models. [ABSTRACT FROM AUTHOR]

Most traditional loop closure detection (LCD) methods rely on manual feature design, which is sensitive to environmental conditions. Convolutional neural networks (CNNs) cope better with illumination changes by extracting hierarchical features and ignoring the local spatial characteristics of images. We propose an LCD algorithm that combines VGG16, NetVLAD, and image pyramids to enhance its accuracy and robustness. In particular, a three-level image pyramid was constructed via downsampling, and then a feature pyramid (FP) layer was obtained by extracting features through VGG16 network on different image resolutions. The obtained FPs were then passed into the VLAD model, and this model outputted VLAD vectors by performing residual summation with L2 normalization. Finally, a triplet loss function was employed for training. Experimental results on two benchmark datasets and a real scenario dataset demonstrated that this algorithm outperforms the NetVLAD baseline and the VGG16 network, exhibiting superior feature-learning capabilities and achieving a higher LCD accuracy. Further, it maintained real-time performance with only a 2% increase in processing time. The results indicate that the proposed method detects loop closures even in complex environments with varying conditions and perspectives. Hence, the approach can be used for large-scale visual simultaneous localization and mapping applications, such as autonomous driving, where LCD plays a crucial role in mapping. [ABSTRACT FROM AUTHOR]

Convolution-based networks show impressive performance in Single Image Super-Resolution (SISR) task by making full use of local patterns. However, it requires deeper structure to capture the long-range dependencies which is inefficient. Hence, the Non-Local Attention (NLA) is introduced to search similar patterns globally and effectively exchange information globally. Nevertheless, the naïve NLA only exchanges information in original spatial scale and has quadratic computational complexity with spatial size of input image. Besides, the global fusion of all similar patterns may lead to blurring, which deviates from the goal of SISR to accurately recover image details. Hence, the naïve NLA is not suit for SISR. To solve this problem, we propose a novel Cross-Scale Collaborative Attention (CSCA) which is designed for SISR. Due to similar patterns tend to occur at different locations and scales, we explore self-similarity and cross-scale similarity simultaneously on multi-scale feature pyramid to utilize more rich details for improving the performance of recovery. To guarantee the recovered details are accurate, we aggregate features in each layer of pyramid into groups and only exchange information between the most relevant groups across multi-scale. Hence, CSCA is both efficient and can accurately recover details using the rich information contained in multi-scale. Furthermore, we propose a Cross-Scale Collaborative Network (CSCN) by inserting a few CSCA modules into a simple backbone network. Extensive experiments show the effectivity and efficiency of CSCA, and state-of-the-art performance of CSCN in SISR task. • Constructing a cross-scale collaborative network for image super-resolution. • Designing an efficient region-wise collaborative attention module. • Building multi-layer feature pyramid to catch self and cross-scale similarities. • Reducing computational complexity with spherical locality sensitive hashing. [ABSTRACT FROM AUTHOR]

Hyperspectral images reconstruction focuses on recovering the spectral information from a single RGBimage. In this paper, we propose two advanced Generative Adversarial Networks (GAN) for the heavily underconstrained inverse problem. We first propose scale attention pyramid UNet (SAPUNet), which uses U-Net with dilated convolution to extract features. We establish the feature pyramid inside the network and use the attention mechanism for feature selection. The superior performance of this model is due to the modern architecture and capturing of spatial semantics. To provide a more accurate solution, we propose another distinct architecture, named W-Net, that builds one more branch compared to U-Net to conduct boundary supervision. SAPUNet and scale attention pyramid WNet (SAPWNet) provide improvements on the Interdisciplinary Computational Vision Lab at Ben Gurion University (ICVL) datasetby 42% and 46.6%, and 45% and 50% in terms of root mean square error (RMSE) and relative RMSE, respectively. The experimental results demonstrate that our proposed models are more accurate than the state-of-the-art hyperspectral recovery methods

To solve the low detection accuracy of SSD for the small size object, this paper proposed an improved algorithm of SSD object detection based on the feature pyramid (FP-SSD). In the deep convolutional neural network, the high-level features contain well semantic information but are not sensitive to the translations. The low-level features have high resolutions but could not represent the features well. The feature pyramid structure contains multi-scale features. To combine the high and low-level features of the pyramid, the algorithm of this paper applied the deconvolution network to the high-level features of the feature pyramid to get the semantic information, dilated convolution network to learn the position information of the low-level features and used convolution for the middle level features to reduce the feature channels, then used convolution to fuse the features. After using the algorithm, a multi-scale detection structure is constructed. FP-SSD achieves a mean accuracy of 79% on PASCAL VOC2007, and 47% on MSCOCO, which has a great improve compared with SSD. We compared the detection accuracy and results with all kinds of scales by experiments, compared with SSD, the accuracy of FP-SSD is higher, which has more accurate location and higher recognition confidence. [ABSTRACT FROM AUTHOR]

Copyright of Computer Measurement & Control is the property of Magazine Agency of Computer Measurement & Control and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

The PASCAL VOC Challenge performance has been significantly boosted by the prevalently CNN-based pipelines like Faster R-CNN. However, directly applying the Faster R-CNN to the small remote sensing objects usually renders poor performance. To address this issue, this paper investigates on how to modify Faster R-CNN for the task of small object detection in optical remote sensing images. First of all, we not only modify the RPN stage of Faster R-CNN by setting appropriate anchors but also leverage a single high-level feature map of a fine resolution by designing a similar architecture adopting top-down and skip connections. In addition, we incorporate context information to further boost small remote sensing object detection performance while we apply a simple sampling strategy to solve the issue about the imbalanced numbers of images between different classes. At last, we introduce a simple yet effective data augmentation method named ‘random rotation’ during training. Experimental results show that our modified Faster R-CNN algorithm improves the mean average precision by a large margin on detecting small remote sensing objects. [ABSTRACT FROM AUTHOR]

Automatic building segmentation from aerial imagery is an important and challenging task because of the variety of backgrounds, building textures and imaging conditions. Currently, research using variant types of fully convolutional networks (FCNs) has largely improved the performance of this task. However, pursuing more accurate segmentation results is still critical for further applications such as automatic mapping. In this study, a multi-constraint fully convolutional network (MC-FCN) model is proposed to perform end-to-end building segmentation. Our MC-FCN model consists of a bottom-up/top-down fully convolutional architecture and multi-constraints that are computed between the binary cross entropy of prediction and the corresponding ground truth. Since more constraints are applied to optimize the parameters of the intermediate layers, the multi-scale feature representation of the model is further enhanced, and hence higher performance can be achieved. The experiments on a very-high-resolution aerial image dataset covering 18 km2 and more than 17,000 buildings indicate that our method performs well in the building segmentation task. The proposed MC-FCN method significantly outperforms the classic FCN method and the adaptive boosting method using features extracted by the histogram of oriented gradients. Compared with the state-of-the-art U-Net model, MC-FCN gains 3.2% (0.833 vs. 0.807) and 2.2% (0.893 vs. 0.874) relative improvements of Jaccard index and kappa coefficient with the cost of only 1.8% increment of the model-training time. In addition, the sensitivity analysis demonstrates that constraints at different positions have inconsistent impact on the performance of the MC-FCN. [ABSTRACT FROM AUTHOR]

Voice anti-spoofing is an important step for secure speaker verification in voice-enabled Internet of Things (IoT) systems. Most voice spoofing detection methods require significant computational costs and lack good generalization capability, making them unsuitable for deployment in IoT systems. To alleviate these problems, this work proposes a voice spoofing detection framework, which contains four parts, namely the compact model, the feature pyramid module (FPM), the feature aggregation module (FAM) and online knowledge distillation. FPM leverages multi-scale features from different layers of the compact model and then produces four embeddings with different resolutions. Instead of using frequently-used aggregation operations, FAM adaptively assigns weights for the four embeddings above and thus gains the high-quality soft embedding that can optimize the compact model in reverse. Moreover, the Kullback–Leibler (KL) divergence is employed to minimize the difference between the embedding predicted by the compact model and the soft embedding, which enables the compact model to learn dark knowledge in an online manner. We carry out a series of experiments on LA and PA corpora of ASVspoof 2019 dataset. These comprehensive experiments show that the proposed framework gives better performance than other methods without sacrificing inference complexity, demonstrating the superiority of our framework. [ABSTRACT FROM AUTHOR]

Bridge detection from Synthetic Aperture Radar (SAR) images has very important strategic significance and practical value, but there are still many challenges in end-to-end bridge detection. In this paper, a new deep learning-based network is proposed to identify bridges from SAR images, namely, multi-resolution attention and balance network (MABN). It mainly includes three parts, the attention and balanced feature pyramid (ABFP) network, the region proposal network (RPN), and the classification and regression. First, the ABFP network extracts various features from SAR images, which integrates the ResNeXt backbone network, balanced feature pyramid, and the attention mechanism. Second, extracted features are used by RPN to generate candidate boxes of different resolutions and fused. Furthermore, the candidate boxes are combined with the features extracted by the ABFP network through the region of interest (ROI) pooling strategy. Finally, the detection results of the bridges are produced by the classification and regression module. In addition, intersection over union (IOU) balanced sampling and balanced L1 loss functions are introduced for optimal training of the classification and regression network. In the experiment, TerraSAR data with 3-m resolution and Gaofen-3 data with 1-m resolution are used, and the results are compared with faster R-CNN and SSD. The proposed network has achieved the highest detection precision (P) and average precision (AP) among the three networks, as 0.877 and 0.896, respectively, with the recall rate (RR) as 0.917. Compared with the other two networks, the false alarm targets and missed targets of the proposed network in this paper are greatly reduced, so the precision is greatly improved.

Jointly using spatial and spectral information has been widely applied to hyperspectral image (HSI) classification. Especially, convolutional neural networks (CNN) have gained attention in recent years due to their detailed representation of features. However, most of CNN-based HSI classification methods mainly use patches as input classifier. This limits the range of use for spatial neighbor information and reduces processing efficiency in training and testing. To overcome this problem, we propose an image-based classification framework that is efficient and straightforward. Based on this framework, we propose a multiscale spatial-spectral CNN for HSIs (HyMSCN) to integrate both multiple receptive fields fused features and multiscale spatial features at different levels. The fused features are exploited using a lightweight block called the multiple receptive field feature block (MRFF), which contains various types of dilation convolution. By fusing multiple receptive field features and multiscale spatial features, the HyMSCN has comprehensive feature representation for classification. Experimental results from three real hyperspectral images prove the efficiency of the proposed framework. The proposed method also achieves superior performance for HSI classification.

Due to the high proportion of aircraft faults caused by cracks in aircraft structures, crack inspection in aircraft structures has long played an important role in the aviation industry. The existing approaches, however, are time-consuming or have poor accuracy, given the complex background of aircraft structure images. In order to solve these problems, we propose the YOLOv3-Lite method, which combines depthwise separable convolution, feature pyramids, and YOLOv3. Depthwise separable convolution is employed to design the backbone network for reducing parameters and for extracting crack features effectively. Then, the feature pyramid joins together low-resolution, semantically strong features at a high-resolution for obtaining rich semantics. Finally, YOLOv3 is used for the bounding box regression. YOLOv3-Lite is a fast and accurate crack detection method, which can be used on aircraft structure such as fuselage or engine blades. The result shows that, with almost no loss of detection accuracy, the speed of YOLOv3-Lite is 50% more than that of YOLOv3. It can be concluded that YOLOv3-Lite can reach state-of-the-art performance.

Over the past few years, deep learning-based image registration methods have achieved remarkable performance in medical image analysis. However, many existing methods struggle to ensure accurate registration while preserving the desired diffeomorphic properties and inverse consistency of the final deformation field. To address the problem, this paper presents a novel symmetric pyramid network for medical image inverse consistent diffeomorphic registration. Specifically, we first encode the multi-scale images to the feature pyramids via a shared-weights encoder network and then progressively conduct the feature-level diffeomorphic registration. The feature-level registration is implemented symmetrically to ensure inverse consistency. We independently carry out the forward and backward feature-level registration and average the estimated bidirectional velocity fields for more robust estimation. Finally, we employ symmetric multi-scale similarity loss to train the network. Experimental results on three public datasets, including Mindboggle101, CANDI, and OAI, show that our method significantly outperforms others, demonstrating that the proposed network can achieve accurate alignment and generate the deformation fields with expected properties. Our code will be available at https://github.com/zhangliutong/SPnet. • Propose symmetric pyramid network for inverse consistent diffeomorphic registration. • Encode images to feature pyramids and implement feature registration progressively. • Formulate symmetric feature registration by bidirectional velocity fields average. • Experimental results show our network consistently outperforms other methods. [ABSTRACT FROM AUTHOR]

• A deep learning model with PAN as the base structure was designed for the low definition of ultrasound breast images, and the effectiveness of this model was verified on BUSI and OASBUD. • A multi-scale feature extraction module is designed, which can acquire deep image local information while preserving shallow image global information. • The attention module called SCA is added to the decoder based on PAN to prompt the model to pay attention to the edges and details in the image and strengthen the segmentation ability of the model. Breast ultrasound medical images are characterized by poor imaging quality and irregular target edges. During the diagnosis process, it is difficult for physicians to segment tumors manually, and the segmentation accuracy required for diagnosis is high, so there is an urgent need for an automated method to improve the segmentation accuracy as a technical tool to assist diagnosis. This study designed an improved Pyramid Attention Network combining Attention mechanism and Multi-Scale features (AMS-PAN) for breast ultrasound image segmentation. On the encoding side, the model adopts the depthwise separable convolution strategy to achieve a multi-scale receptive field with cumulative small-size convolution, which performs multi-dimensional feature extraction and forms a feature pyramid. The model uses Global Attention Upsample (GAU) feature fusion on the decoding side. In order to further process the fused feature information, the proposed method uses a Spatial and Channel Attention (SCA) module to shift the model's segmentation focus to the edge texture information. The good segmentation performance of our method is verified through experiments on BUSI and OASBUD. All the designed parts have contributed to the segmentation performance in practical applications. Compared with the traditional non-deep learning methods and the current mainstream deep learning methods, the improvement of the model in Dice and IoU metrics is pronounced. AMS-PAN has high computational efficiency, and its good performance has been proven to play a role in ultrasound detection tasks of breast tumors for physicians to specific auxiliary diagnostic roles to guide the subsequent diagnosis and treatment services for patients. [ABSTRACT FROM AUTHOR]