Search

Model filter pruning has shown efficiency in compressing deep convolutional neural networks by removing unimportant filters without sacrificing the performance. However, most existing criteria are empirical, and overlook the relationship between channel saliencies and the non-linear activation functions within the networks. To address these problems, we propose a novel channel pruning method coined gradient flow based saliency (GFBS). Instead of relying on the magnitudes of the entire feature maps, GFBS evaluates the channel saliencies from the gradient flow perspective and only requires the information in normalization and activation layers. Concretely, we first integrate the effects of normalization and ReLU activation layers into convolutional layers based on Taylor expansion. Then, through backpropagation, the derived channel saliency of each layer is indicated by of the first-order Taylor polynomial of the scaling parameter and the signed shifting parameter in the normalization layers. To validate the efficiency and generalization ability of GFBS, we conduct extensive experiments on various tasks, including image classification (CIFAR, ImageNet), image denoising, object detection, and 3D object classification. GFBS could feasibly cooperate with the baseline networks and compress them with only negligible performance drop. Moreover, we extended our method to pruning scratch networks and GFBS is capable to identify subnetworks with comparable performance with the baseline model at an early training stage. Our code has been released at https://github.com/CUHK-AIM-Group/GFBS. [ABSTRACT FROM AUTHOR]

There are frequently a large number of strip objects in segmentation scenarios, and the use of conventional square convolution may yield redundant information. Based on our previously proposed SA-FFNet (Zhou et al. in Neurocomputing 453:50–59, 2021), we study the effect of strip sub-region information extraction on semantic segmentation and propose a network. Our method is conducive to extracting multi-scale strip objects that often appear in segmentation scenes, and using strip dilated convolution to further extract contextual dependencies in other directions. First, we propose a multi-scale strip pooling module that enables the backbone network to effectively obtain multi-scale contexts; Then, we introduce a strip dilated convolution module, which supplements the vertical contexts of the strip pooling by using strip dilated convolution; Finally, we construct a novel network integrating the proposed two modules. The method explicitly takes horizontal and vertical contexts of multi-scale strip objects into consideration, so that scene understanding could benefit from long-range dependencies. The experimental results on the widely used PASCAL VOC 2012 and Cityscapes scene analysis benchmark datasets, which are better than the existing OCRNet, DeeplabV3+, SPNet, etc, both qualitatively and quantitatively. [ABSTRACT FROM AUTHOR]

Image semantic segmentation is a meaningful task that requires both accuracy and efficiency in computer vision. At present, most current deep learning based semantic segmentation methods needs extensive computational resources, and knowledge distillation may reduce such a computational burden due to its model compression ability. In this paper, different from previous knowledge distillation methods that directly transfer the knowledge of the teacher network to the student network, we propose a novel intermediate feature refinement method for semantic segmentation based on adversarial learning, which reduces the error and redundant information contained in the teacher network in the process of knowledge distillation, enhances the correct information contained in the teacher network and transfers it to the student network. Then we improve the conventional discriminator in adversarial learning to help the student network align more correct intermediate features in the teacher network. Our method can make the feature distribution of the student network closer to that of the teacher network, and finally improve the segmentation performance of the student network. Finally, we conducted experiments on three popular benchmarks to verify the effectiveness of our proposed method, including Pascal VOC, Cityscapes and CamVid. Compared with the competitive baseline, our proposed method can improve the performance of the student network by up to 1.43% (the mIOU increases from 67.14% to 68.57% on the Cityscapes val set). [ABSTRACT FROM AUTHOR]

Unifying object detection and re-identification (ReID) into a single network enables faster multi-object tracking (MOT), while this multi-task setting poses challenges for training. In this work, we dissect the joint training of detection and ReID from two dimensions: label assignment and loss function. We find previous works generally overlook them and directly borrow the practices from object detection, inevitably causing inferior performance. Specifically, we identify a qualified label assignment for MOT should: 1) have the assignment cost aware of ReID cost, not just detection cost; 2) provide sufficient positive samples for robust feature learning while avoiding ambiguous positives (i.e., the positives shared by different ground-truth objects). To achieve the above goals, we first propose Identity-aware Label Assignment, which jointly considers the assignment cost of detection and ReID to select positive samples for each instance without ambiguities. Moreover, we advance a novel Discriminative Focal Loss that integrates ReID predictions with Focal Loss to focus the training on the discriminative samples. Finally, we upgrade the strong baseline FairMOT with our techniques and achieve up to 7.0 MOTA / 54.1% IDs improvements on MOT16/17/20 benchmarks under favorable inference speed, which verifies our tailored label assignment and loss function for MOT are superior to those inherited from object detection. [ABSTRACT FROM AUTHOR]

Semantic segmentation is a popular research topic in computer vision, and many efforts have been made on it with impressive results. In this paper, we intend to search an optimal network structure that can run in real-time for this problem. Towards this goal, we jointly search the depth, channel, dilation rate and feature spatial resolution, which results in a search space consisting of about 2.78 × 10 324 possible choices. To handle such a large search space, we leverage differential architecture search methods. However, the architecture parameters searched using existing differential methods need to be discretized, which causes the discretization gap between the architecture parameters found by the differential methods and their discretized version as the final solution for the architecture search. Hence, we relieve the problem of discretization gap from the innovative perspective of solution space regularization. Specifically, a novel Solution Space Regularization (SSR) loss is first proposed to effectively encourage the supernet to converge to its discrete one. Then, a new Hierarchical and Progressive Solution Space Shrinking method is presented to further achieve high efficiency of searching. In addition, we theoretically show that the optimization of SSR loss is equivalent to the L 0 -norm regularization, which accounts for the improved search-evaluation gap. Comprehensive experiments show that the proposed search scheme can efficiently find an optimal network structure that yields an extremely fast speed (175 FPS) of segmentation with a small model size (1 M) while maintaining comparable accuracy. [ABSTRACT FROM AUTHOR]

In domain adaptation (DA), label-induced losses generally occupy a dominant position and most previous models regard hard or soft labels as their inputs. However, these two types of labels may mislead the modeling process of label-induced losses since hard label is sensitive to a wrongly-predicted sample while soft label may introduce label noise, thus they may cause negative transfer. To relieve this problem, we propose a novel label learning approach namely confidence regularized label propagation (CRLP) that regularizes the confidence of predicted soft labels with constraints of F-norm or L21-norm. It is validated that maximizing either one of these two constraints equals to minimizing entropy loss. Specially, we illustrate that L21-norm is more suitable for DA than F-norm when the dataset contain a large number of categories. Then, we leverage the regularized soft labels produced by CRLP to reformulate some popular label-induced losses that consider feature transferability and discriminability such as class-wise maximum mean discrepancy, intra-class compactness and inter-class dispersion in a probability manner to present a novel DA method (i.e., CRLP-DA). Comprehensive analysis and experiments on four cross-domain object recognition datasets verify that the proposed CRLP-DA outperforms some state-of-the-art methods, especially 59.5% for Office10+Caltech10 dataset with SURF features. For others to better reproduce, our preliminary Matlab code will be available at https://github.com/WWLoveTransfer/CRLP-DA/. [ABSTRACT FROM AUTHOR]

In recent years, deep convolutional neural networks (DCNN) have been widely used in the field of video action recognition. Attention mechanisms are also increasingly utilized in action recognition tasks. In this paper, we want to combine temporal and spatial attention for better video action recognition. Specifically, we learn a set of sparse attention by computing class response maps for finding the most informative region in a video frame. Each video frame is resampled with this information to form two new frames, one focusing on the most discriminative regions of the image and the other on the complementary regions of the image. After computing sparse attention all the newly generated video frames are rearranged in the order of the original video to form two new videos. These two videos are then fed into a CNN as new inputs to reinforce the learning of discriminative regions in the images (spatial attention). And the CNN we used is a network with a frame selection strategy that allows the network to focus on only some of the frames to complete the classification task (temporal attention). Finally, we combine the three video (original, discriminative, and complementary) classification results to get the final result together. Our experiments on the datasets UCF101 and HMDB51 show that our approach outperforms the best available methods. [ABSTRACT FROM AUTHOR]

Automatic sketch colorization is a challenging task in both computer graphics and computer vision since all the color, texture, shading generation have to be created based on the abstract sketch. Besides, it is a subjective task in painting process, which needs illustrators to comprehend drawing priori (DP), such as hue variation, saturation contrast and gray contrast and utilize them in the HSV color space which is closer to human visual cognition system. As such, incorporating supplementary supervision in the HSV color space may be beneficial to sketch colorization. However, previous methods improve the colorization quality only in the RGB color space without considering the HSV color space, often causing results with dull color, inappropriate saturation contrast, and artifacts. To address this issue, we propose a novel sketch colorization method, dual color space guided generative adversarial network (DCSGAN), that considers the complementary information contained in both the RGB and HSV color space. Specifically, we incorporate the HSV color space to construct dual color spaces for supervising our method with a color space transformation (CST) network that learns transformation from the RGB to HSV color space. Then, we propose a DP loss that enables the DCSGAN to generate vivid color images with pixel level supervision. Additionally, a novel dual color space adversarial (DCSA) loss is designed to guide the generator at global level to reduce the artifacts to meet audiences’ aesthetic expectations. Extensive experiments and ablation studies demonstrate the superiority of the proposed method over previous state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

Depth map super-resolution is an ill-posed inverse problem with many challenges. First, depth boundaries are generally hard to reconstruct particularly at large magnification factors. Second, depth regions on fine structures and tiny objects in the scene are destroyed seriously by downsampling degradation. To tackle these difficulties, we propose a progressive multi-branch aggregation network (PMBANet), which consists of stacked MBA blocks to fully address the above problems and progressively recover the degraded depth map. Specifically, each MBA block has multiple parallel branches: 1) The reconstruction branch is proposed based on the designed attention-based error feed-forward/-back modules, which iteratively exploits and compensates the downsampling errors to refine the depth map by imposing the attention mechanism on the module to gradually highlight the informative features at depth boundaries. 2) We formulate a separate guidance branch as prior knowledge to help to recover the depth details, in which the multi-scale branch is to learn a multi-scale representation that pays close attention at objects of different scales, while the color branch regularizes the depth map by using auxiliary color information. Then, a fusion block is introduced to adaptively fuse and select the discriminative features from all the branches. The design methodology of our whole network is well-founded, and extensive experiments on benchmark datasets demonstrate that our method achieves superior performance in comparison with the state-of-the-art methods. Our code and models are available at https://github.com/Sunbaoli/PMBANet_DSR/. [ABSTRACT FROM AUTHOR]

Wireless capsule endoscopy (WCE) has become a widely used diagnostic technique for the digestive tract, at the price of a large volume of data that needs to be analyzed. To tackle this problem, a new computer-aided system using novel features is proposed in this paper to classify WCE images automatically. In the feature learning stage, to obtain the representative visual words, we first calculate the color scale invariant feature transform from the bleeding, polyp, ulcer, and normal WCE image samples separately and then apply $K$ -means clustering on these features to obtain visual words. These four types of visual words are combined together to composite the representative visual words for classifying the WCE images. In the feature coding stage, we propose a novel saliency and adaptive locality-constrained linear coding (SALLC) algorithm to encode the images. The SALLC encodes patch features based on adaptive coding bases, which are calculated by the distance differences among the features and the visual words. Moreover, it imposes the patch saliency constraint on the feature coding process to emphasize the important information in the images. The experimental results exhibit a promising overall recognition accuracy of 88.61%, validating the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Wireless capsule endoscopy (WCE) needs computerized method to reduce the review time for its large image data. In this paper, we propose an improved bag of feature (BoF) method to assist classification of polyps in WCE images. Instead of utilizing a single scale-invariant feature transform (SIFT) feature in the traditional BoF method, we extract different textural features from the neighborhoods of the key points and integrate them together as synthetic descriptors to carry out classification tasks. Specifically, we study influence of the number of visual words, the patch size and different classification methods in terms of classification performance. Comprehensive experimental results reveal that the best classification performance is obtained with the integrated feature strategy using the SIFT and the complete local binary pattern (CLBP) feature, the visual words with a length of 120, the patch size of 8*8, and the support vector machine (SVM). The achieved classification accuracy reaches 93.2%, confirming that the proposed scheme is promising for classification of polyps in WCE images. [ABSTRACT FROM AUTHOR]

Wireless capsule endoscopy (WCE) enables noninvasive and painless direct visual inspection of a patient's whole digestive tract, but at the price of long time reviewing large amount of images by clinicians. Thus, an automatic computer-aided technique to reduce the burden of physicians is highly demanded. In this paper, we propose a novel color feature extraction method to discriminate the bleeding frames from the normal ones, with further localization of the bleeding regions. Our proposal is based on a twofold system. First, we make full use of the color information of WCE images and utilize K-means clustering method on the pixel represented images to obtain the cluster centers, with which we characterize WCE images as words-based color histograms. Then, we judge the status of a WCE frame by applying the support vector machine (SVM) and K-nearest neighbor methods. Comprehensive experimental results reveal that the best classification performance is obtained with YCbCr color space, cluster number 80 and the SVM. The achieved classification performance reaches 95.75% in accuracy, 0.9771 for AUC, validating that the proposed scheme provides an exciting performance for bleeding classification. Second, we propose a two-stage saliency map extraction method to highlight bleeding regions, where the first-stage saliency map is created by means of different color channels mixer and the second-stage saliency map is obtained from the visual contrast. Followed by an appropriate fusion strategy and threshold, we localize the bleeding areas. Quantitative as well as qualitative results show that our methods could differentiate the bleeding areas from neighborhoods correctly. [ABSTRACT FROM AUTHOR]

Ulcer is one of the most common symptoms of many serious diseases in the human digestive tract. Especially for the ulcers in the small bowel where other procedures cannot adequately visualize, wireless capsule endoscopy (WCE) is increasingly being used in the diagnosis and clinical management. Because WCE generates large amount of images from the whole process of inspection, computer-aided detection of ulcer is considered an indispensable relief to clinicians. In this paper, a two-staged fully automated computer-aided detection system is proposed to detect ulcer from WCE images. In the first stage, we propose an effective saliency detection method based on multi-level superpixel representation to outline the ulcer candidates. To find the perceptually and semantically meaningful salient regions, we first segment the image into multi-level superpixel segmentations. Each level corresponds to different initial region sizes of the superpixels. Then we evaluate the corresponding saliency according to the color and texture features in superpixel region of each level. In the end, we fuse the saliency maps from all levels together to obtain the final saliency map. In the second stage, we apply the obtained saliency map to better encode the image features for the ulcer image recognition tasks. Because the ulcer mainly corresponds to the saliency region, we propose a saliency max-pooling method integrated with the Locality-constrained Linear Coding (LLC) method to characterize the images. Experiment results achieve promising 92.65% accuracy and 94.12% sensitivity, validating the effectiveness of the proposed method. Moreover, the comparison results show that our detection system outperforms the state-of-the-art methods on the ulcer classification task. [ABSTRACT FROM AUTHOR]

Long term tracking is a challenging task for many applications. In this paper, we propose a novel tracking approach that can adapt various appearance changes such as illumination, motion, and occlusions, and owns the ability of robust reacquisition after drifting. We utilize a condensation-based method with an online support vector machine as a reliable observation model to realize adaptive tracking. To redetect the target when drifting, a cascade detector based on random ferns is proposed. It can detect the target robustly in real time. After redetection, we also come up with a new refinement strategy to improve the tracker's performance by removing the support vectors corresponding to possible wrong updates by a matching template. Extensive comparison experiments on typical and challenging benchmark dataset illustrate a robust and encouraging performance of the proposed approach. [ABSTRACT FROM PUBLISHER]

Magnetic tracking technology is emerging to provide an occlusion-free tracking scheme for the estimation of full pose (position and orientation) of various instruments. This brings substantial benefits for intracorporeal applications, such as for tracking of flexible or wireless endoscopic devices, and thus is significant for further computer-assisted diagnosis, interventions, and surgeries. Toward efficient magnetic tracking, a 6-D magnetic localization and orientation method is proposed in this paper. An annular permanent magnet is mounted on the exterior surface of a capsule. With a magnetic sensor array, the magnetic field can be measured and the capsule’s 3-D location and 3-D orientation information can be estimated based on proposed closed-form analytical model of annular magnet and particle swarm optimization algorithm. Magnetic dipole model and Levenberg–Marquardt algorithm are used to improve the speed and accuracy of estimation. Extensive simulation experiments show that the localization and orientation method works well with good position and orientation accuracy. [ABSTRACT FROM AUTHOR]

We propose an efficient and effective magnetic tracking method in this paper. The tracking method is based on tri-axial transmitting coils and uniaxial sensing of the generated electromagnetic field. Three mutually orthogonal transmitting coils are excited simultaneously with alternating current (AC) signals of different frequency. At a specific position, the sum of amplitude square of the three different frequency sensing signals will reach maximum when the uniaxial coil points to the tri-axial transmitting coils. The maximum value is reciprocally proportional to the cube of the distance between the transmitter and receiver. By processing the output signals from the uniaxial sensing coil when it is rotating, the direction and distance between the sensing coil and transmitting coils can be decided with an efficient method with low calculation overheads. Experiments were conducted to validate the proposed method. [ABSTRACT FROM PUBLISHER]

The electromagnetic localization and orientation method is based on 2-axis generation and 3-axis sensing of a quasi-static magnetic field. Two mutually orthogonal coils fed with phase-quadrature signals comprise the excitation source, which is equal to a rotating magnetic dipole. Using the amplitude and phase information of the sensing signals from the 3-axis sensing coils, the position of the sensor can be calculated directly by simple and explicit analytical expressions, then the orientations of the sensor can be determined by using rotation matrix. In this paper, a novel method is proposed to realize the direct calculation, which is easy and fast to determine the position and orientation of the 3-axis sensor in a single period of excitation. Experimental results demonstrate a promising performance of this method. [ABSTRACT FROM AUTHOR]

Tumor in digestive tract is a common disease and wireless capsule endoscopy (WCE) is a relatively new technology to examine diseases for digestive tract especially for small intestine. This paper addresses the problem of automatic recognition of tumor for WCE images. Candidate color texture feature that integrates uniform local binary pattern and wavelet is proposed to characterize WCE images. The proposed features are invariant to illumination change and describe multiresolution characteristics of WCE images. Two feature selection approaches based on support vector machine, sequential forward floating selection and recursive feature elimination, are further employed to refine the proposed features for improving the detection accuracy. Extensive experiments validate that the proposed computer-aided diagnosis system achieves a promising tumor recognition accuracy of 92.4% in WCE images on our collected data. [ABSTRACT FROM PUBLISHER]

This article presents a computer-aided detection system for capsule endoscopy (CE) images using contourlet-based color textural features to recognize tumors in the digestive tract. As tumor exhibits rich information in color texture, a novel color texture feature based on contourlet transform is proposed to describe characteristics of tumor in CE images. The proposed features are a hybrid of contourlet transform and uniform local binary pattern, yielding detailed and robust color texture features in multi-directions for CE images. Sequential floating forward search approach is further applied to refine the proposed features. With support vector machine for classification, comprehensive experiments on our present data reveal an encouraging accuracy of 93.6% for tumor detection in CE images using the proposed features. [ABSTRACT FROM AUTHOR]

Due to a few great advantages such as viewing the entire small intestine with almost non-invasiveness and no sedation over traditional endoscopies and other imaging techniques for gastrointestinal tract diseases, the wireless capsule endoscopy invented by Given Imaging has found its gradually wide applications in hospitals. However, one major issue concerning this new technology is that too many images to be examined by naked eyes cause a huge burden to physicians, so it is very necessary to ease the physician if we can do diseases detection using computerized methods. In this paper, we develop a new method by making use of color feature, also a very important clue for diagnosis by physicians, to discriminate between normal region and abnormal region. Exploiting the color histogram of the image, we can get the distribution of the color in the image. Then we use the minimum distance classifier to judge the status of the regions. Experimental results on our present data prove promising performance of the proposed scheme in detecting bleeding and ulcers. [ABSTRACT FROM AUTHOR]

To assist the diagnosis, this paper presents a new algorithm to enhance the capsule endoscopy images, which are now widely used to detect the diseases in the digestive tract because of its great advantages over traditional endoscopy technologies. The capsule endoscopy images have the characteristics such as low resolution and low contrast, which cause some difficulties to analysis and diagnosis for the physicians, especially to computer aided diagnosis. To overcome these shortcomings, we propose a contrast-based forward and backward anisotropic diffusion method to enhance the capsule endoscopy images. Based on the local analysis of the capsule endoscopy images, we put forward a new concept of contrast. Then, we employ the forward and backward anisotropic diffusion to enhance the capsule endoscopy images. Experimental results show that this new method can efficiently enhance the capsule endoscopy images, which provides a better visualization and an improved discrimination of region of interest from the background. [ABSTRACT FROM AUTHOR]