Search

Sketches, as a kind of probabilistic data structures, have been considered as the most promising solution for network measurement in recent years. Most sketches do not work well for skewed network traffic. To address this problem, we propose a family of sketch frameworks, namely the Pyramid family. The first member of our Pyramid family is the S-Pyramid framework, which includes two techniques: counter-pair sharing for high accuracy, and word acceleration for fast speed. The second member of our Pyramid family is the Mini-Pyramid framework, which projects the S-Pyramid framework into one counter, bringing more flexibility in application while keeping the accuracy. To demonstrate the generality of our Pyramid family, we apply both frameworks to sketches of CM, CU, Count, and Augmented. To demonstrate the flexibility of the Mini-Pyramid framework, we further apply Mini-Pyramid to SBF and the On-Off sketch. The experimental results show that, the S-Pyramid framework can reduce the ARE by up to 7.12 times compared with the original sketches, while improving the throughput by up to 2.37 times; the Mini-Pyramid framework can reduce the ARE by up to 29.2 times, at the cost of 21.3% lower throughput on average.

Both siamese network and correlation filter (CF)-based trackers have exhibited superior performance by formulating tracking as a similarity measure problem, where a similarity map is learned by the correlation between a target template and a region of interest (ROI) with a cosine window. Nevertheless, this window function is usually fixed for various targets and not changed, undergoing significant noise variations during tracking, which easily makes model drift. In this article, we focus on the study of a noise-aware (NA) framework for robust visual tracking. To this end, the impact of various window functions is first investigated in visual tracking. We identify that the low signal-to-noise ratio (SNR) of windowed ROIs makes the above trackers degenerate. At the prediction phase, a novel NA window customized for visual tracking is introduced to improve the SNR of windowed ROIs by adaptively suppressing the variable noise according to the observation of similarity maps. In addition, to further optimize the SNR of windowed pyramid ROIs for scale estimation, we propose to use the particle filter to dynamically sample several windowed ROIs with more favorable signals in temporal domains instead of this pyramid ROIs extracted in spatial domains. Extensive experiments on the popular OTB-2013, OTB-50, OTB-2015, VOT2017, TC128, UAV123, UAV123@10fps, UAV20L, and LaSOT datasets show that our NA framework can be extended to many siamese and CF trackers and our variants obtain superior performance than baseline trackers with a modest impact on efficiency.

Data stories integrate compelling visual content to communicate data insights in the form of narratives. The narrative structure of a data story serves as the backbone that determines its expressiveness, and it can largely influence how audiences perceive the insights. Freytag's Pyramid is a classic narrative structure that has been widely used in film and literature. While there are continuous recommendations and discussions about applying Freytag's Pyramid to data stories, little systematic and practical guidance is available on how to use Freytag's Pyramid for creating structured data stories. To bridge this gap, we examined how existing practices apply Freytag's Pyramid by analyzing stories extracted from 103 data videos. Based on our findings, we proposed a design space of narrative patterns, data flows, and visual communications to provide practical guidance on achieving narrative intents, organizing data facts, and selecting visual design techniques through story creation. We evaluated the proposed design space through a workshop with 25 participants. Results show that our design space provides a clear framework for rapid storyboarding of data stories with Freytag's Pyramid.

Discriminative correlation filters (DCF)-based trackers have been increasingly applied to visual tracking due to their high precision while running at high frame rates. However, most recent DCF-based methods solely concentrate on learning the correlation filter with spatial information and thus do not have sufficient descriptive power to discriminate the target from the background in the complex circumstances, such as full occlusion (OCC) and rapid target variation. In this article, we introduce a novel tracking framework that exploits the relationship between the target and its spatiotemporal context to improve tracking accuracy and robustness. Especially, we present our spatiotemporal context model in a hierarchical way, where each layer of the context pyramid is a spatial correlation filter learned from different temporal instances. For gaining an accurate spatiotemporal model, we propose an optimization fusion approach that can adaptively and efficiently learn the effect of each hierarchical layer and exploit these multiple temporal levels of correlation filters for visual tracking. Moreover, an adaptive model update strategy for correlation filters is introduced into the framework to dynamically select proper hierarchical layers, which boosts the temporal diversity of the target appearance, while radically reduces the number of model parameters and guarantees the real-time performance of the tracking method. The experimental results show that, with conventional handcrafted features, our tracker achieves the best success rates among available state-of-the-art trackers with handcrafted features, and provides state-of-the-art performance comparable to those of deep-learning-based trackers on OTB-2013, OTB-2015, VOT-2016, and UAV-20L benchmarks but runs significantly faster than deep trackers.

Modern autonomous vehicles are required to perform various visual perception tasks for scene construction and motion decision. The multiobject tracking and instance segmentation (MOTS) are the main tasks since they directly influence the steering and braking of the car. Implementing both tasks using a multitask learning neural network presents significant challenges in performance and complexity. Current work on MOTS devotes to improve the precision of the network with a two-stage tracking by detection model, which is difficult to satisfy the real-time requirement of autonomous vehicles. In this article, a real-time multitask network named YolTrack based on one-stage instance segmentation model is proposed to perform the MOTS task, achieving an inference speed of 29.5 frames per second (fps) with slight accuracy and precision drop. The YolTrack uses ShuffleNet V2 with feature pyramid network (FPN) as a backbone, from which two decoders are extended to generate instance segments and embedding vectors. Segmentation masks are used to improve the tracking performance by performing logic AND operation with feature maps, proving that foreground segmentation plays an important role in object tracking. The different scales of multiple tasks are balanced by the optimized geometric mean loss during the training phase. Experimental results on the KITTI MOTS data set show that YolTrack outperforms other state-of-the-art MOTS architectures in real-time aspect and is appropriate for deployment in autonomous vehicles.

Depth of field is one of the critical reasons to limit the richness of image information. Usually, in a scene with multiple targets, when the distance between each target and the lens is different, the clear scene image can be get within a certain distance range. This situation restricts the further image processing, such as semantic segmentation, object recognition and 3D reconstruction. Multi‐focus image fusion uses two or more images focused on different targets to fuse scene information, which can solve this problem to a great extent. In general, two or more multi‐focus images can cover almost all near/far targets. The fusion of more than two multi‐focus images can be accomplished by cascading the fusion results of the previous two images and the next image to be processed many times. Therefore, the paper focus on the fusion of two multi‐focus images. Inspired by this, new Pseudo‐Siamese neural network with several residual atrous convolution pyramids with multi‐level perception ability to perceive the multi‐level features and consistency relations of multi‐focus image pairs is proposed, and multi‐layer residual blocks are used to fuse the extracted features. In this process, the residual of the groundtruth and the generated image will be learned. Finally, a fully focused image without blur will be generated. After several ablation experiments and comparison experiments with other methods, the results show that the performance of the method proposed in this paper is state‐of‐the‐art, and overall better than other methods, which are advanced.

The road extraction from high resolution remote sensing image is of great importance in a variety of applications. Recently, the abundant deep convolutional neural networks are proposed for road extraction task. However, the existing approaches lack suitable strategy to utilize multiple views road features for road extraction, which fails to extract road with smooth appearance and accurate boundary under complex scenes. To address this problem, the authors propose a novel deep residual and pyramid pooling network (DRPPNet) for extracting road regions from high resolution remote sensing image. The DRPPNet consists of three parts: deep residual network (DResNet), pyramid pooling module (PPM) and deep decoder (DD). Specially, the DResNet uses several residual blocks to extract deep road features from input images, which can enhance learning ability of DRPPNet and avoid gradient vanish. Then, PPM is proposed to fuse road features from multiple views and it aims to address disadvantage of single view feature. Finally, the DD is used to recover size of feature maps to input size. Extensive experiments on two challenging road datasets demonstrate that proposed method outperforms the state‐of‐the‐art methods greatly on performance of road extraction task.

Airlines and researchers have been working to reduce the health risk to passengers from the novel coronavirus SARS-CoV2. Recent literature indicates that the Reverse Pyramid boarding method provides a lower health risk than other boarding methods using a limited number of boarding groups, while keeping the middle seats empty. With a primary objective of minimizing health risks, we determine the number of passengers to include in each boarding group when using the Reverse Pyramid method. We examine how the optimal performance varies with the volume of carry-on luggage, the social distance maintained between passengers walking down the aisle, and the number of boarding groups. An increased quantity of luggage increases health risk, while changing the aisle social distance from 1m to 2m brings provides negligible health value to seated passengers. Increasing the number of boarding groups decreases the risk to passenger health while increasing the time to complete boarding. [ABSTRACT FROM AUTHOR] Copyright of Transportmetrica B: Transport Dynamics is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

In the field of automatic driving, identifying vehicles and pedestrians is the starting point of other automatic driving techniques. Using the information collected by the camera to detect traffic targets is particularly important. The main bottleneck of traffic object detection is due to the same category of targets, which may have different scales. For example, the pixel‐level of cars may range from 30 to 300 px, which will cause instability of positioning and classification. In this paper, a multi‐dimension feature pyramid is constructed in order to solve the multi‐scale problem. The feature pyramid is built by developing a U‐shaped module and using a cascade‐method. In order to verify the effectiveness of the U‐shaped module, we also designed a new one‐shot detector U‐FPNDet. The model first extracts the basic feature map by using the basic network and constructs the multi‐dimension feature pyramid. Next, a pyramid pooling module is used to get more context information from the scene. Finally, the detection network is run on each level of the pyramid to obtain the final result by NMS. By using this method, a state‐of‐the‐art performance is achieved on both detection and classification on commonly used benchmarks.

Crowd counting is one of the most challenging issues in computer vision community for safety and security through surveillance systems. It has extensive range of applications, such as disaster management, surveillance event detection, intelligence gathering and analysis, public safety control, traffic monitoring, design of public spaces, anomaly detection and military. Early approaches still encounter many issues, like non-uniform density distribution, partial occlusion and discrepancies in scale and point of view. To address the above problems, Feature Pyramid Networks are introduced in deep convolution networks for counting the individuals in the Crowd. The designed network has extracted the features at all resolutions and is constructed rapidly from only one input image. This method achieves out-performance results compared to the well-known networks on three demanding standard crowd counting datasets.

Semantic segmentation is a valuable yet challenging research in the Internet of Things (IoT), especially for some real-time and resource-constrained applications. Recently we witness a strong tendency of fusing multi-level features or multi-scale context information to achieve promising segmentation performance . However, we find that existing literature has at least one of the following issues: 1) relying on more resource-consuming feature extraction operations, e.g., standard convolution with large kernels, for multiple information fusion; 2) seldom considering that how to narrow the semantic gap between multi-layer features, leading to sub-optimal performance. To tackle these issues, we propose a novel IoT-perceptive Dual Feature Fusion Network (DFFNet) for semantic segmentation, which aims to leverage multi-level features and multi-scale context information in an efficient yet effective manner. Specifically, the multi-level feature fusion module (MFFM), which enhances the semantic consistency between multi-level features by two attention refinement blocks, is proposed to exploit multi-layer features for jointly learning spatial and semantic information with small overheads. Moreover, a multi-scale component termed as lightweight semantic pyramid module (LSPM) is presented to improve the efficiency of context encoding by depthwise factorized convolution operations . Extensive experimental results on benchmarks have demonstrated that DFFNet achieves better performance than existing advanced methods.

Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi‐scale feature pyramid network (MFP‐Net) for dense crowd counting is proposed in this paper. The proposed MFP‐Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention‐aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP‐Net is evaluated on five publicly available datasets, and experiments show that the MFP‐Net not only provides better crowd counting results than comparative models, but also requires fewer parameters.

The retinal vessel is the only microvascular network that can be directly and non‐invasively observed in humans. Cardiovascular and cerebrovascular diseases, such as diabetes, hypertension, can lead to structural changes of the retinal microvascular network. Therefore, it is of great significance to study effective retinal vessel segmentation methods and assist doctors in early diagnoses with quantitative results for vascular networks. In this study, we propose a novel convolutional neural network named feature pyramid U‐Net (FPU‐Net) that extracts multiscale representations by constructing two feature pyramids both on the encoder and the decoder of U‐Net. In this representation, objects features with different size like micro‐vessels and pathology will be fused for better vessel segmentation. The experimental results show that compared with state‐of‐the‐art methods, FPU‐Net is superior in terms of accuracy, sensitivity, F1‐score, and area under the curve and capable of stronger domain generalisation across different datasets.

Real-time semantic segmentation aims to generate high-quality prediction in limited time. Recently, with the development of many related potential applications, such as autonomous driving, robot se...

Pedestrian attributes recognition is an important issue in computer vision and has a special role in the field of video surveillance. The previous methods presented to solve this issue are mainly based on multi‐label end‐to‐end deep neural networks. These methods neglect to apply attributes for defining local feature areas and they suffer from the problems of the bounding box presence. A new framework for jointly human semantic parsing and pedestrian attribute recognition to achieve effective attribute recognition is proposed. By extracting human parts via semantic parsing, both semantic and spatial information can be explored with eliminating of background. The framework also uses multi‐scale features to employ rich details and contextual information through proposed attribute recognition‐bidirectional feature pyramid network. For baseline network that has a significant impact on the performance, EfficientNet‐B3 is selected as a baseline network from The EfficientNet family which provides an appropriate trade‐off between the three factors of CNNs scaling (depth/width/resolution). Finally, the proposed framework is tested on datasets PETA, RAP and PA‐100k. Experimental results show that our method has superior performance in both mean accuracy and instance‐based metrics compared to state‐of‐the‐art results.

RoIPool/RoIAlign is an indispensable process for the typical two-stage object detection algorithm, it is used to rescale the object proposal cropped from the feature pyramid to generate a fixed size feature map. However, these cropped feature maps of local receptive fields will heavily lose global context information. To tackle this problem, we propose a novel end-to-end trainable framework, called global context aware (GCA) RCNN, aiming at assisting the neural network in strengthening the spatial correlation between the background and the foreground by fusing global context information. The core component of our GCA framework is a context aware mechanism, in which both global feature pyramid and attention strategies are used for feature extraction and feature refinement, respectively. Specifically, we leverage the dense connection to improve the information flow of the global context at different stages in the top-down process of FPN, and further use the attention mechanism to refine the global context at each level in the feature pyramid. In the end, we also present a lightweight version of our method, which only slightly increases model complexity and computational burden. Experimental results on COCO benchmark dataset demonstrate the significant advantages of our approach.

Single image de‐raining based on convolutional neural network (CNN) has made considerable progress in recent years. However, usually the de‐rained result has dark artifacts and image textures tend to be over‐smoothed. In this paper, a pyramid non‐local enhanced residual dense network is proposed to reduce such distortion. Firstly, the down‐sampled images are input into the Laplacian pyramid, which can extract the overall and partial texture clues, and subsequently a set of images of different scales are produced. Secondly, these images are fed into a non‐local enhanced residual dense block, which can not only capture long‐distance dependencies of feature maps, but also fully utilizes the hierarchical features in every dense block, leading to high accuracy of rain streaks extraction and better preservation of image edge detail. Finally, the de‐rained image is gradually restored by Gaussian reconstruction pyramid. Experimental results on both synthetic data and real‐world data show that the artifacts distortion is obviously reduced by the proposed network. And the quality of de‐rained image is significantly improved compared with the state‐of‐the‐art methods.

Many deep learning detectors have been proposed to address the scale variation issue in pedestrian detection. However, it is still far from being fully solved because most of these detectors handle the issue by directly fusing multiple extracted features with the weighted average or concatenation. Therefore, this paper designs a novel and effective anchor-free network (namely MPAF-Net) for multi-scale pedestrian detection. Specifically, we first improve Res2Net as our backbone network by designing a novel bottleneck block to make the proposed detector has a strong multi-scale representation ability for pedestrians. Furthermore, based on Conditional Random Fields (CRFs), the multi-scale features obtained from three different feature pyramid levels of backbone network are refined with a designed message passing mechanism. By doing so, the complementary information from the features at other scales can be dynamically passed to enhance the scale-specific feature. Finally, pedestrian detection is simplified as a straightforward center and scale prediction task by performing convolution on the concatenation of refined multi-scale convolution features. This way, the proposed MPAF-Net can be trained in an end-to-end fashion. Experimental results show that MPAF-Net achieves state-of-the-art performance on CityPersons, Caltech, and CrowdHuman benchmarks.

In this paper, we propose a novel multi-scale fusion framework for low-illumination image enhancement, which effectively enhances images taken under various low-light conditions. Based on the high dynamic range imaging technique, we first employ a novel remapping function to generate a sequence of artificial multi-exposure images. The generated sequence of images ensures that the contrast of each intensity interval of the input image is enhanced at least once. Then three fusion-relevant features, namely, exposure, global contrast and local contrast, are selected as the weight maps. Combined with the weight maps, a pyramid fusion scheme is introduced to do a layer-by-layer integration of the different frequency bands of the image layer by layer. In addition, a strategy for extracting details from the original image is designed, which effectively maintains the detail information without causing colour distortions. The framework is very efficient and suitable for mobile devices because most of the calculations are at the pixel-level. Extensive experiments have shown that the proposed approach yields comparable and better performances in comparisons with the state-of-the-art competing techniques in both qualitative and quantitative evaluations.

Automatic facial expression recognition, which has many applications such as drivers, patients, and criminals' emotions recognition, is a challenging task. This is due to the variety of individuals and facial expression variability in different conditions, for instance, gender, race, colour and changing illumination. In addition, there are many regions in a face image such as forehead, mouth, eyes, eyebrows, nose, cheeks and chin, and extracting features of all these regions are expensive in terms of computational time. Each of the six basic emotions of anger, disgust, fear, happiness, sadness and surprise affect some regions more than the other regions. The goal of this study is to evaluate the performance of enhanced local binary pattern, pyramid histogram of oriented gradients feature‐extraction algorithms and their combination in terms of recognition accuracy, feature vector length and computational time on one, two and three combined regions of a face image. Our experimental results show that the combination of both feature‐extraction algorithms yields an average recognition accuracy of 95.33% using three regions, that is, the mouth, nose and eyes on Cohn–Kanade dataset. Besides, the mouth region is the most important part in terms of accuracy in comparison to eyes, nose and combination of both eyes and nose regions.

In this paper, we explore an issue that is to detect vehicle headlights from the nighttime traffic surveillance images with highly reflections. In the night, reflections on the road (water) surface, vehicles bodies, or some reflective objects (such as lane markings, traffic signs) will interfere the headlight detection seriously. Although the existing methods have achieved good results, however, most of them failed to detect the headlight when headlights are far from camera. In order to solve the issue, we propose a novel method for vehicle headlights detection. The proposed method makes full use of the brightness and gradient information of the headlights in the night. First, we propose an effective region-of-interest (ROI) segmentation method which is based on multi-scale local saliency detection. The method pre-serve faint or small-sized objects and retain the original shape of the object to the greatest extent. Then, we compute the pyramid histogram of oriented gradients (PHOG) features, which are used to train support vector machine (SVM) classifier. Finally, the extracted bright blocks are classified according to the pre-trained SVM classifier. Experimental results and quantitative evaluations in different scenes demonstrate that our proposed method can achieve a better result compared with previous methods.

Feature pyramids have delivered significant improvement in object detection. However, building effective feature pyramids heavily relies on expert knowledge, and also requires strenuous efforts to balance effectiveness and efficiency. Automatic search methods, such as NAS-FPN, automates the design of feature pyramids, but the low search efficiency makes it difficult to apply in a large search space. In this paper, we propose a novel search framework for a feature pyramid network, called AutoDet, which enables to automatic discovery of informative connections between multi-scale features and configure detection architectures with both high efficiency and state-of-the-art performance. In AutoDet, a new search space is specifically designed for feature pyramids in object detectors, which is more general than NAS-FPN. Furthermore, the architecture search process is formulated as a combinatorial optimization problem and solved by a Simulated Annealing-based Network Architecture Search method (SA-NAS). Compared with existing NAS methods, AutoDet ensures a dramatic reduction in search times. For example, our SA-NAS can be up to 30x faster than reinforcement learning-based approaches. Furthermore, AutoDet is compatible with both one-stage and two-stage structures with all kinds of backbone networks. We demonstrate the effectiveness of AutoDet with outperforming single-model results on the COCO dataset. Without pre-training on OpenImages, AutoDet with the ResNet-101 backbone achieves an AP of 39.7 and 47.3 for one-stage and two-stage architectures, respectively, which surpass current state-of-the-art methods.

The convolutional neural network (CNN)-based multi-focus image fusion methods which learn the focus map from the source images have greatly enhanced fusion performance compared with the traditional methods. However, these methods have not yet reached a satisfactory fusion result, since the convolution operation pays too much attention on the local region and generating the focus map as a local classification (classify each pixel into focus or de-focus classes) problem. In this article, a global-feature encoding U-Net (GEU-Net) is proposed for multi-focus image fusion. In the proposed GEU-Net, the U-Net network is employed for treating the generation of focus map as a global two-class segmentation task, which segments the focused and defocused regions from a global view. For improving the global feature encoding capabilities of U-Net, the global feature pyramid extraction module (GFPE) and global attention connection upsample module (GACU) are introduced to effectively extract and utilize the global semantic and edge information. The perceptual loss is added to the loss function, and a large-scale dataset is constructed for boosting the performance of GEU-Net. Experimental results show that the proposed GEU-Net can achieve superior fusion performance than some state-of-the-art methods in both human visual quality, objective assessment and network complexity.

The performance of state-of-the-art object skeleton detection (OSD) methods have been greatly boosted by Convolutional Neural Networks (CNNs). However, the most existing CNN-based OSD methods rely on a ‘skip-layer’ structure where low-level and high-level features are combined to gather multi-level contextual information. Unfortunately, as shallow features tend to be noisy and lack semantic knowledge, they will cause errors and inaccuracy. Therefore, in order to improve the accuracy of object skeleton detection, we propose a novel network architecture, the Multi-Scale Bidirectional Fully Convolutional Network (MSB-FCN), to better gather and enhance multi-scale high-level contextual information. The advantage is that only deep features are used to construct multi-scale feature representations along with a bidirectional structure for better capturing contextual knowledge. This enables the proposed MSB-FCN to learn semantic-level information from different sub-regions. Moreover, we introduce dense connections into the bidirectional structure to ensure that the learning process at each scale can directly encode information from all other scales. An attention pyramid is also integrated into our MSB-FCN to dynamically control information propagation and reduce unreliable features. Extensive experiments on various benchmarks demonstrate that the proposed MSB-FCN achieves significant improvements over the state-of-the-art algorithms.

Due to the advantages of real-time detection and improved performance, single-shot detectors have gained great attention recently. To solve the complex scale variations, single-shot detectors make scale-aware predictions based on multiple pyramid layers. Typically, small objects are detected on shallow layers while large objects are detected on deep layers. However, the features in the pyramid are not scale-aware enough, which limits the detection performance. Two common problems in single-shot detectors caused by object scale variations can be observed: (1) false negative problem, i.e. , small objects are easily missed due to the weak features; (2) part-false positive problem, i.e. , the salient part of a large object is sometimes detected as an object. With this observation, a new Neighbor Erasing and Transferring (NET) mechanism is proposed for feature scale-unmixing to explore scale-aware features in this paper. In NET, a Neighbor Erasing Module (NEM) is designed to erase the salient features of large objects and emphasize the features of small objects in shallow layers. A Neighbor Transferring Module (NTM) is introduced to transfer the erased features and highlight large objects in deep layers. With this mechanism, a single-shot network called NETNet is constructed for scale-aware object detection. In addition, we propose to aggregate nearest neighboring pyramid features to enhance our NET. Experiments on MS COCO dataset and UAVDT dataset demonstrate the effectiveness of our method. NETNet obtains 38.5% AP at a speed of 27 FPS and 32.0% AP at a speed of 55 FPS on MS COCO dataset. As a result, NETNet achieves a better trade-off for real-time and accurate object detection.

Video frame interpolation aims to improve users' watching experiences by generating high-frame-rate videos from low-frame-rate ones. Existing approaches typically focus on synthesizing intermediate frames using high-quality reference images. However, the captured reference frames may suffer from inevitable spatial degradations such as motion blur, sensor noise, etc. Few studies have approached the joint video enhancement problem, namely synthesizing high-frame-rate and high-quality results from low-frame-rate degraded inputs. In this paper, we propose a unified optimization framework for video frame interpolation with spatial degradations. Specifically, we develop a frame interpolation module with a pyramid structure to cyclically synthesize high-quality intermediate frames. The pyramid module features adjustable spatial receptive field and temporal scope, thus contributing to controllable computational complexity and restoration ability. Besides, we propose an inter-pyramid recurrent module to connect sequential models to exploit the temporal relationship. The pyramid module integrates the recurrent module, thus can iteratively synthesize temporally smooth results. And the pyramid modules share weights across iterations, thus it does not expand the model's parameter size. Our model can be generalized to several applications such as up-converting the frame rate of videos with motion blur, reducing compression artifacts, and jointly super-resolving low-resolution videos. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art methods on various video frame interpolation and enhancement tasks.

Object detection is one of the core tasks in computer vision. Object detection algorithms often have difficulty detecting objects with diverse scales, especially those with smaller scales. To cope with this issue, Lin et al. proposed feature pyramid networks (FPNs), which aim for a feature pyramid with higher semantic content at every scale level. The FPN consists of a bottom-up pyramid and a top-down pyramid. The bottom-up pyramid is induced by a convolutional neural network as its layers of feature maps. The top-down pyramid is formed by progressive up-sampling of a highly semantic yet low-resolution feature map at the top of the bottom-up pyramid. At each up-sampling step, feature maps of the bottom-up pyramid are fused with the top-down pyramid to produce highly semantic yet high-resolution feature maps in the top-down pyramid. Despite significant improvement, the FPN still misses small-scale objects. To further improve the detection of small-scale objects, this paper proposes scale adaptive feature pyramid networks (SAFPNs). The SAFPN employs weights chosen adaptively to each input image in fusing feature maps of the bottom-up pyramid and top-down pyramid. Scale adaptive weights are computed by using a scale attention module built into the feature map fusion computation. The scale attention module is trained end-to-end to adapt to the scale of objects contained in images of the training dataset. Experimental evaluation, using both the 2-stage detector faster R-CNN and 1-stage detector RetinaNet, demonstrated the proposed approach’s effectiveness.

While Mechanism of Objective Marking of Automated Marking is approaching the latest technique mainly; the approach has an important aspect; where assansoyn fellow complete the mechanical teaching curriculum by filling up the niche in the near term. While inexplicable isn't fully understood, validates discrimination decisions in mechanical teaching, thus instilling confide in ML goal calls. Alternatively, the approach via can act as a standalone element, especially in scenarios where a little amount of benefit, for example, "Today's the interactions of curricula via don't require train data, and thus prove different mechanical teaching curricula for accurate train data. So that an example-via approach, which screen shoot prominent goal shape data to identifying in a large-scale satellite imagery. The right mix of coarse three dimensions aims finds the abstract form and realism of the goals to provide strength against objective differences discrimination at the same time. The curriculum uses powerful new forms of image correlation to match the shape of expected objectives with the image. Look for shape projections about setting, and use engineering property objectives and shadow projections. Binding factors provide tolerance to lighting differences, temperate covers; where many true subjects. To provide distinguishing digital objective on realistic satellite imagery that illustrates performance.

Scale variation is one of the major challenges in object detection task. Modern region-based object detection architectures often adopt Feature Pyramid Network (FPN) as feature extraction neck to achieve multi-scale feature representation in solving scale variation problem. However, due to the rough feature selection strategy in Region of Interest (RoI) feature extraction step, these methods might not perform well on object detection under strong scale variation. In this work, we are motivated by the limitations of current FPN-based two-stage object detectors and then present a novel module, namely scale-aware feature selective (SAFS) module, that flexibly and adaptively selects feature levels in two-stage object detectors. Specifically, we firstly build the RoI Pyramid in standard FPN structure to extract RoI features from various scale levels. Next, in order to achieve scale-aware mechanism for solving scale variation issue, we develop a novel weighting gate function containing one set of trainable parameters to automatically learn the fusion weight for each RoI feature level, which relieves the limitation of hard feature selection strategy guided by online instance size. Outputs from the RoI features with the learned weights are fused for classification and bounding box regression. Furthermore, we design a multi-level SAFS architecture to obtain different types of RoI feature combinations that ensures our method is more robust to various instance scales. Experimental results show that our SAFS module is very compatible with most of two-stage object detectors and could achieve state-of-the-art results with Average Precision of 48.3 on COCO test-dev and other popular object detection benchmarks. Our code will be made publicly available.

Accurate and automatic segmentation of medical images is a crucial step for clinical diagnosis and analysis. The convolutional neural network (CNN) approaches based on the U-shape structure have achieved remarkable performances in many different medical image segmentation tasks. However, the context information extraction capability of single stage is insufficient in this structure, due to the problems such as imbalanced class and blurred boundary. In this paper, we propose a novel Context Pyramid Fusion Network (named CPFNet) by combining two pyramidal modules to fuse global/multi-scale context information. Based on the U-shape structure, we first design multiple global pyramid guidance (GPG) modules between the encoder and the decoder, aiming at providing different levels of global context information for the decoder by reconstructing skip-connection. We further design a scale-aware pyramid fusion (SAPF) module to dynamically fuse multi-scale context information in high-level features. These two pyramidal modules can exploit and fuse rich context information progressively. Experimental results show that our proposed method is very competitive with other state-of-the-art methods on four different challenging tasks, including skin lesion segmentation, retinal linear lesion segmentation, multi-class segmentation of thoracic organs at risk and multi-class segmentation of retinal edema lesions.

Semantic scene segmentation has become a key application in computer vision and is an essential part of intelligent transportation systems for complete scene understanding of the surrounding environment. While several methods based on deep fully Convolutional Neural Network (CNN) have been emerging, there are two main challenges: (i) They mainly focus on improvement of the accuracy than efficiency. (ii) They assume structured driving environment like in USA and Europe. While most of the current works focus on the well structured driving environment, we focus our research on India Driving Dataset (IDD) which contains data from unstructured traffic scenario. In this paper, we propose modifications in the DeepLabV3+ framework by using lower atrous rates in Atrous Spatial Pyramid Pooling (ASPP) module for dense traffic prediction. We propose to use dilated Xception network as the backbone for feature extraction. A lightweight segmentation framework is also presented by exploring the effectiveness of MobileNetV2 architecture, which achieves competitively high accuracy and is much smaller than other state-of-art architectures. The performance is evaluated in terms of mean Intersection over Union (mIoU) on 26 fine grained classes of IDD. Our proposed model with 24 M parameters achieves 68.41 mIoU on test set and efficient mobile model achieves mIoU of 61.6 by reducing the parameters to 2.2 M only.

Recognizing aerial image categories is of great significance in computer vision, which is widely utilized in geological analysis, agricultural production and urban planning. However, conventional approaches cannot explicitly exploit spatial information of aerial images, i.e., spatial relations among different components. To solve this problem, we propose a novel aerial image classification algorithm based on the well-known spatial pyramid model, where saliency maps are utilized for discriminative regions selection, and a visual quality model is leveraged to alleviate the impact of image distortion. More specifically, we first partition each aerial image into several fine subregions, each of which is represented using the computed spatial pyramid-based local features. Afterwards, each image is characterized by the most representative features engineered by the saliency map and quality assessment module. Subsequently, a regularized topic model based probabilistic learning is designed for recognizing different types of aerial images. Extensive experiments have demonstrated the effectiveness of our proposed method.

Road segmentation from remote sensing images is a critical task in many applications. In recent years, various approaches, particularly deep learning-based methods, have been proposed for accurate road segmentation. However, most existing road segmentation methods always obtain unsatisfactory results (e.g., heterogeneous pixels) due to the complex backgrounds and view occlusions of buildings and trees around a road; consequently, road segmentation remains a challenging problem. In this study, we propose a novel global context based dilated convolutional neural network (GC-DCNN) to address the aforementioned problem. The structure of GC-DCNN is similar to that of UNet. In particular, building the encoder of GC-DCNN with three residual dilated blocks is suggested to further enlarge the effective receptive field and learn additional discriminative features. Thereafter, a pyramid pooling module is used to capture the multiscale global context features and fuse them to achieve stronger feature representation. The decoder network upsamples the fused features to the same size as the input image, combining the high-resolution features with the contracting path of the encoder network. Moreover, the dice coefficient loss is adopted as the loss function. This function differs from those in most previous studies but is more suitable for road segmentation. Extensive experimental results on two benchmark datasets compared with several baseline models demonstrate the superiority of the proposed GC-DCNN algorithm.

High resolution medical images are expected for accurate analysis results in medical diagnosis. However, the resolution of these medical images is always restricted by the factors such as medical devices, time constraints. Despite these restrictions, the resolution of these medical images can be enhanced with a well-designed super-resolution(SR) algorithm. As a post-processing manner after medical imaging, the adoption of the SR algorithms has the advantages of low cost and high efficiency compared with upgrading medical devices. In this paper, we propose a network named LDSRN that combines the Laplacian pyramid structure and the dense network to reconstruct clear and convincing medical HR images. Our LDSRN can make full use of the information from different pyramid levels to recover faithful HR images by the dense connection. Specifically, the Laplacian structure decomposes the difficult SR task into several easy SR tasks to obtain the HR images step by step for better reconstruction. Experimental results demonstrate that our LDSRN can obtain better HR medical images than several state-of-the-art SR methods in terms of objective indices and subjective evaluations.

Low-to-high hierarchical convolutional features can significantly improve edge detection. This paper proposes a feature decoder-based algorithm that employs a Feature Decoder Network (FDN) to extract more information within limited Convolutional Neural Network (CNN) features. Previous studies applied convolutional elements by weight fusion, but we measure a feature decoder as a pyramid by qualifying convolutional layers. The feature decoder fuses CNN features of adjacent layers to judge the edge and non-edge pixels, which can learn the relationship and distinction between low-level edge hierarchies and high-level semantic hierarchies. Furthermore, we use Gaussian blur labels to train the network to optimize network convergence and training. From the experimental results, our proposed algorithm performs better on the BSDS500 (average accuracy (AP) of 0.865) and NYUD (OIS F-measure of 0.775) datasets compared to the state-of-the-art algorithms, including RCF.

It is a challenging task to distinguish between numerous species of flowers due to their visually similarities and variations of the pose and structure. Thanks to properly modeling of the local feature interactions, bilinear CNN has succeeded in classifying of many non-rigid fine-grained species including flowers. However, bilinear CNN only computes the feature in a straightforward way without exploring the interactions between features from multiple layers in the network. In this paper, we present a novel Bilinear Pyramid Network (BPN) for flower categorization. Instead of passing through the network and directly feeding the final classifier, features from a convolutional layer are resized and multiplied with that from the former layer, which alternates multiple times to generates prediction vectors using the features from distinct layers. These features encoded from the feature pyramid spontaneously carry multi-level semantic cues, which yields stronger discriminative powers than single-layer features. Experiments show that the proposed network obtains superior classification results on the challenging dataset of flowers.

The automatic binary classification of normal and abnormal subjects using magnetic resonance (MR) brain images has made remarkable progress in recent years. This automation method plays a central role in the early evaluation of degenerative brain diseases in patients. In recent years, various new robust classification methods have been proposed based on families of the wavelet transform (WT). These transforms are good at capturing edge points but lack smoothness along the contour of an image. Therefore, instead of using WT in our experiment, we used a pyramid directional filter bank contourlet transform (PDFB-CT). The key characteristic of this transform is that it is likely to manage 2D singularities efficiently, i.e., edges, unlike the wavelets, which deal with point singularities exclusively, and it also efficiently implements a wavelet-like structure using iterative filter banks. Moreover, we passed the outcome obtained from the PDFB-CT to the gray-level co-occurrence matrix (GLCM) to obtain 22 texture features from each MR brain image. Furthermore, we passed these extracted features to random tree embedding (RTE) to transform low-dimensional features to high-dimensional features before passing them to probabilistic principal component analysis (PPCA) for dimensionality reduction. Here, the multi-kernel support vector machine classifier with a grid search CV (to find optimal hyperparameters) method was used to perform binary classification of abnormal and normal images. As a result, our proposed system achieved an area under the receiver operating characteristic (AU-ROC) curve and classification accuracy of 100% for abnormal vs. normal group classification using a ten-fold stratified cross-validation technique. This experiment result exemplifies the significance of our proposed method compared with recently published state-of-the-art techniques, and hence our proposed method can be effectively used by a physician as a support tool for examining a patient’s brain.

How to add more context information and bring more accurate detection is an important problem to be considered in object detection. In this paper, we propose a new object detector with enriched global context information by a pyramid feature pool module and several global activation blocks, named EGCI-Net, which is a one-stage object detector from scratch as DSOD.The global activation blocks are added into the backbone sub network of the detector to weaken the local information of the detected object feature maps and increase the global context of them. And the pyramid feature pool module produces multi-scale global context features to supervise the pyramid features by multi-scale global average pooling. Then the features obtained by the main structure are fused with the pyramid pooling features to merge into the final multibox detector. We have evaluated our detector on the Pascal VOC and MS COCO datasets. The experimental results show that our proposed detector achieves better results than DSOD and exceeds most of the existing excellent detectors, especially detects partially occluded objects and small objects well.

Recently, scene text detection has become an active research field, which is an essential component of scene text reading. Especially, segmentation-based methods are commonly used, since the segmentation results can describe text of arbitrary shape. However, curve texts have a diversity of shapes, scales and orientations, which are difficult to locate, so the detector requires to adjust the local receptive fields size adaptively, which can aggregate multi-scale spatial information to accurately locate the curve text instance. Moreover, the low-level features are critical for localizing large text instances. When using Feature Pyramid Network (FPN) for multi-scale feature fusion, it will prevent the flow of accurate localization signals due to the long path from low-level to top-level. In order to solve these two problems, this paper proposes an Adaptive Convolution and Path Enhancement Pyramid Network (ACPEPNet), which can more accurately locate the text instances with arbitrary shapes. Firstly, an Adaptive Convolution Unit is introduced to improve the ability of backbone to aggregate multi-scale spatial information at the same stage. Specially, this unit is a lightweight component and without the cost of computations, based on this component we present a backbone network for text features extraction. Secondly, the original FPN structure is redesigned to build a short path from the low-level to top-level, in this way, we modify the path from one-way flow to two-way flow and add original features to the final stage of information fusion. Experiments on CTW1500, Total-Text, ICDAR 2015 and MSRA-TD500 validate the robustness of the proposed method. When there is no bells and whistles, this method achieves an F-measure of 80.8% without external training data on CTW1500.

Scene text recognition has become a research hotspot owing to its abundant semantic information and various applications. Recent methods of scene text recognition usually focus on handling shape distortion, attention drift, or background noise, ignoring that text recognition encounters character scale-variation problem. To address this issue, in this paper, we propose a new scale-aware hierarchical attention network (SaHAN) for scene text recognition. Inspired by feature pyramid network, we exploit the inherent pyramidal structure of a deep convolutional network to retain multi-scale features for flexible receptive fields. Then, we construct a hierarchical attention decoder that performs the attention mechanism twice on multi-scale features to collect the most fine-grained information for prediction. The SaHAN is trained in a weak supervision way, requiring only images and corresponding text labels. Extensive experiments on seven benchmarks reveal that SaHAN achieves state-of-the-art performance.

Despite enormous research efforts devoted by the research community to effectively and precisely perform video matching and retrieval among heterogeneous videos from large-scale video repositories still remains a complex and most challenging task. In order to address this complex challenge, a content based video retrieval technique is required, which can exploit the visual content of the videos for effective retrieval from the videos repositories. In our proposed method, we introduce a computer assisted video retrieval technique which can retrieve the visually similar videos stored in the repositories. To accomplish this task, video summarization based on motion vector is employed to select keyframes based on similar segments. To estimate the video content, salient foreground extraction is executed, and matching based on the spatial pyramid is employed for matching the keyframe features of query video with videos in the repositories. The contribution of the former process has two major sections for superior saliency map generation. Firstly, it heuristically integrates the regional property, contrast, and foreground descriptors together. Secondly, it introduces a new feature vector to characterize the foreground as an object descriptor, while the latter process is the extension of orderless bag-of-features representation, which has significant performance with respect to scene categorization. The video retrieval performance is compared with standard state-of-the-art techniques using real-time datasets. Experimental and usability studies provide satisfactory results for video retrieval based on evaluation metrics such as video sampling error, fidelity, precision, and recall.

Marine object detection is an appealing but challengeable task in computer vision. Even though recent popular object detection algorithms perform well on common classes, they cannot acquire satisfied detection performance on marine objects because underwater images are affected by color cast and blur, and scales of the target in underwater images are usually small. These phenomena aggravate the difficulty of detection. Thus, it is urgent to design a proper structure to settle marine object detection issues. To this end, this paper proposes a novel scale-aware feature pyramid architecture named SA-FPN to extract abundant robust features on underwater images and improve the performance on marine object detection. Specifically, we design a special backbone subnetwork to improve the ability of feature extraction, which could provide richer fine-grained features for small object detection. What is more, this paper proposes a multi-scale feature pyramid to enrich the semantic features for prediction. Each feature map is enhanced by the higher level layer with context information through a top-down upsampling pathway. Through obtaining ample feature maps on underwater images, our algorithm could generate multiple bounding boxes for each target. To mitigate the reduplicative boxes and avoid miss suppression, we replace the non-maximum suppression method with soft non-maximum suppression. In this paper, we evaluate our algorithm on underwater image datasets and achieve 76.27% mAP. Meanwhile, we conduct experiments on PASCAL VOC datasets and smart unmanned vending machines datasets and get 79.13% mAP and 91.81% mAP, respectively. The experimental results reveal that our approach achieves best performance not only on marine object detection, but also on common classes.

Single-shot multibox detector (SSD), one of the top-performing object detection algorithms, has achieved both high accuracy and fast speed. However, its performance is limited by two factors: (1) anchors are generated uniformly over the image by predefined manners, and (2) multiscale features from the feature pyramid are used to detect objects independently. In this paper, we propose a single-shot augmentation detector, called SSADet, that significantly improves the detection accuracy of the original SSD with a slight decrease in speed. SSADet mainly consists of two modules, namely the anchor prediction module and the feature fusion module. These two modules aim to generate anchors with any scale and aspect ratio and fuse multiscale features from different layers, respectively. Specifically, we define an anchor generator whose parameter weights are predicted dynamically by a small neural network and then use the anchor generator to generate optimal anchors over the image in the anchor prediction module. In the feature fusion module, multiscale features from the feature pyramid are concatenated to generate a new feature pyramid through a set of downsampling and upsampling operations. The new feature pyramid takes the generated anchors as the input from the anchor prediction module to predict the final detection results. Extensive experiments are conducted to demonstrate the effectiveness of SSADet on the PASCAL VOC 2007, PASCAL VOC 2012 and MS COCO detection datasets. The experimental results show that SSADet achieves state-of-the-art detection performance with high efficiency.

Because of the large variation in appearance, the existence of artifacts, the low contrast, skin lesion segmentation remains a challenging task. In this paper, we propose a novel Scale Attention based Atrous Spatial Pyramid Pooling (Scale-Att-ASPP) module for skin lesion segmentation with attentive boundary aware. Our network is based on the Generative Adversarial Network (GAN), which includes the segmentation network and the critic network. In the segmentation network, we design the Scale-Att-ASPP module to automatically select the optimal scale of the skin lesion feature of the intermediate convolution layer (Inter-CL) in the encoding path, meanwhile, the irrelevant artifacts features are automatically diminished without using complex pre-processing. After introducing the output of the Scale-Att-ASPP module to the same level layer in the decoding path through skip connection in pixel-wise addition way, the more meaningful semantic segmentation is gained. The Jaccard distance loss is employed to solve the problem of label imbalance in skin lesion segmentation. Our network is adversarially trained on ISBI 2017 dataset by the multi-scale L1 loss introduced by the critic network, which guides the Scale-Att-ASPP module learning to focus on the optimal scale of the skin lesion feature. Finally, our network significantly improves the segmentation performance compared with other state-of-the-art methods, especially for the JAC and SEN scores. Besides, our proposed network works efficiently and shows robustness for different datasets.

International audience; In recent times, the topic of human facial age estimation attracted much attention. This is due to its ability to improve biometrics systems. Recently, several applications that are based on the demographic attributes estimation have been developed. These include law enforcement, re-identification in videos, planed marketing, intelligent advertising, social media, and human-computer interaction. The main contributions of the paper are as follows. Firstly, it extends some handcrafted models that are based on the Pyramid Multi Level (PML) face representation. Secondly, it evaluates the performance of two different kinds of features that are handcrafted and deep features. It compares handcrafted and deep features in terms of accuracy and computational complexity. The proposed scheme of study includes the following three main steps: 1) face preprocessing; 2) feature extraction (two different kinds of features are studied: handcrafted and deep features); 3) feeding the obtained features to a linear regressor. In addition, we investigate the strengths and weaknesses of handcrafted and deep features when used in facial age estimation. Experiments are run on three public databases (FG-NET, PAL and FACES). These experiments show that both handcrafted and deep features are effective for facial age estimation.

Micro-expressions are brief and subtle facial expressions that go on and off the face in a fraction of a second. This kind of facial expressions usually occurs in high stake situations and is considered to reflect a human’s real intent. There has been some interest in micro-expression analysis, however, a great majority of the methods are based on classically established computer vision methods such as local binary patterns, histogram of gradients and optical flow. A novel methodology for micro-expression recognition using the Riesz pyramid, a multi-scale steerable Hilbert transform is presented. In fact, an image sequence is transformed with this tool, then the image phase variations are extracted and filtered as proxies for motion. Furthermore, the dominant orientation constancy from the Riesz transform is exploited to average the micro-expression sequence into an image pair. Based on that, the Mean Oriented Riesz Feature description is introduced. Finally the performance of our methods are tested in two spontaneous micro-expressions databases and compared to state-of-the-art methods.