Search

Recent technological developments have enabled computers to identify and categorize facial expressions to determine a person's emotional state in an image or a video. This process, called "Facial Expression Recognition (FER)", has become one of the most popular research areas in computer vision. In recent times, deep FER systems have primarily concentrated on addressing two significant challenges: the problem of overfitting due to limited training data availability, and the presence of expression-unrelated variations, including illumination, head pose, image resolution, and identity bias. In this paper, a comprehensive survey is provided on deep FER, encompassing algorithms and datasets that offer insights into these intrinsic problems. Initially, this paper presents a detailed timeline showcasing the evolution of methods and datasets in deep facial expression recognition (FER). This timeline illustrates the progression and development of the techniques and data resources used in FER. Then, a comprehensive review of FER methods is introduced, including the basic principles of FER (components such as preprocessing, feature extraction and classification, and methods, etc.) from the pro-deep learning era (traditional methods using handcrafted features, i.e., SVM and HOG, etc.) to the deep learning era. Moreover, a brief introduction is provided related to the benchmark datasets (there are two categories: controlled environments (lab) and uncontrolled environments (in the wild)) used to evaluate different FER methods and a comparison of different FER models. Existing deep neural networks and related training strategies designed for FER, based on static images and dynamic image sequences, are discussed. The remaining challenges and corresponding opportunities in FER and the future directions for designing robust deep FER systems are also pinpointed. [ABSTRACT FROM AUTHOR]

Crowd counting, as a basic computer vision task, plays an important role in many fields such as video surveillance, accident prediction, public security, and intelligent transportation. At present, crowd counting tasks face various challenges. Firstly, due to the diversity of crowd distribution and increasing population density, there is a phenomenon of large-scale crowd aggregation in public places, sports stadiums, and stations, resulting in very serious occlusion. Secondly, when annotating large-scale datasets, positioning errors can also easily affect training results. In addition, the size of human head targets in dense images is not consistent, making it difficult to identify both near and far targets using only one network simultaneously. The existing crowd counting methods mainly use density plot regression methods. However, this framework does not distinguish the features between distant and near targets and cannot adaptively respond to scale changes. Therefore, the detection performance in areas with sparse population distribution is not good. To solve such problems, we propose an adaptive multi-scale far and near distance network based on the convolutional neural network (CNN) framework for counting dense populations and achieving a good balance between accuracy, inference speed, and performance. However, on the feature level, in order to enable the model to distinguish the differences between near and far features, we use stacked convolution layers to deepen the depth of the network, allocate different receptive fields according to the distance between the target and the camera, and fuse the features between nearby targets to enhance the feature extraction ability of pedestrians under nearby targets. Secondly, depth information is used to distinguish distant and near targets of different scales and the original image is cut into four different patches to perform pixel-level adaptive modeling on the population. In addition, we add density normalized average precision (nAP) indicators to analyze the accuracy of our method in spatial positioning. This paper validates the effectiveness of NF-Net on three challenging benchmarks in Shanghai Tech Part A and B, UCF_ CC_50, and UCF-QNRF datasets. Compared with SOTA, it has more significant performance in various scenarios. In the UCF-QNRF dataset, it is further validated that our method effectively solves the interference of complex backgrounds. [ABSTRACT FROM AUTHOR]

Identification of human actions from video has gathered much attention in past few years. Most of the computer vision tasks such as Health Care Activity Detection, Suspicious Activity detection, Human Computer Interactions etc. are based on the principle of activity detection. Automatic labelling of activity from videos frames is known as activity detection. With the introduction of deep networks, the process of activity detection is clustered into two groups known as hand-crafted feature based approach and automatic feature extraction approach. This paper focuses on various approaches used in recent literature based on traditional and automatic approach. Moreover, hierarchy for different approaches under them such as space based, motion based, genetic based, fuzzy based, dictionary based are discussed. With introduction of Convolutional Neural Networks and Recurrent Neural Networks, automatic learning capability from input modality makes them first choice to be implemented for activity recognition. In this paper various approaches have been analyzed according to methodology, accuracy, classifier and datasets. [ABSTRACT FROM AUTHOR]

Image deblurring attracts research attention in the field of image processing and computer vision. Traditional deblurring methods based on statistical prior largely depend on the selected prior type, which limits their restoring ability. Moreover, the constructed deblurring model is difficult to solve, and the operation is comparatively complicated. Meanwhile, deep learning has become a hotspot in various fields in recent years. End-to-end convolutional neural networks (CNNs) can learn the pixel mapping relationships between degraded images and clear images. In addition, they can also obtain the result of effectively eliminating spatial variable blurring. However, conventional CNNs have some disadvantages in generalization ability and details of the restored image. Therefore, this paper presents an iterative dual CNN called IDC for image deblurring, where the task of image deblurring is divided into two sub-networks: deblurring and detail restoration. The deblurring sub-network adopts a U-Net structure to learn the semantical and structural features of the image, and the detail restoration sub-network utilizes a shallow and wide structure without downsampling, where only the image texture features are extracted. Finally, to obtain the deblurred image, this paper presents a multiscale iterative strategy that effectively improves the robustness and precision of the model. The experimental results showed that the proposed method has an excellent effect of deblurring on a real blurred image dataset and is suitable for various real application scenes. [ABSTRACT FROM AUTHOR]

Image recognition has long been one of the research hotspots in computer vision tasks. The development of deep learning is rapid in recent years, and convolutional neural networks usually need to be designed with fixed resources. If sufficient resources are available, the model can be scaled up to achieve higher accuracy, for example, VggNet, ResNet, GoogLeNet, etc. Although the accuracy of large-scale models has been improved, the following problems will occur with the expansion of model scale: (1) There may be over-fitting; (2) increasing model parameters; (3) slow model convergence. This paper proposes a design method for a modular convolutional neural network model which solves the problem of over-fitting and large model parameters by connecting multiple modules in parallel. Moreover, each module contains several submodules (three submodules in this paper) and fuses the features extracted from the submodules. The model convergence can be accelerated by using the fused features (the fused features contain more image information). In this study, we add a gate unit based on the attention mechanism to the model, which aims to optimize the structure of the model (select the optimal number of modules), allowing the model to select an optimum network structure by learning and dynamically reducing FLOPs (floating-point operations per second) of the model. Compared to VggNet, ResNet, and GoogLeNet, the structure of the model proposed in this paper is simple and the parameters are small. The proposed model achieves good results in the Kaggle datasets Cats-vs.-Dogs (99.3%), 10-Monkey Species (99.26%), and Birds-400 (99.13%). [ABSTRACT FROM AUTHOR]

Simple Summary: The use of computer vision technology has improved the effectiveness of individual sheep identification, but existing methods face challenges such as large parameter sizes, slow recognition speeds, and difficult deployment. To address this issue, we have made improvements and optimizations based on the Retinaface face recognition model to create a balanced speed-accuracy sheep face recognition model. The optimized model has fewer parameters, simpler computations, faster inference speeds, and higher recognition accuracy, making it well-suited for deployment on resource-constrained edge devices. This research is expected to promote the application of deep learning-based sheep face recognition methods in production. The recognition of sheep faces based on computer vision has improved the efficiency and effectiveness of individual sheep identification, providing technical support for the development of smart farming. However, current recognition models have problems such as large parameter sizes, slow recognition speed, and difficult deployment. Therefore, this paper proposes an efficient and fast basic module called Eblock and uses it to build a lightweight sheep face recognition model called SheepFaceNet, which achieves the best balance between speed and accuracy. SheepFaceNet includes two modules: SheepFaceNetDet for detection and SheepFaceNetRec for recognition. SheepFaceNetDet uses Eblock to construct the backbone network to enhance feature extraction capability and efficiency, designs a bidirectional FPN layer (BiFPN) to enhance geometric location ability, and optimizes the network structure, which affects inference speed, to achieve fast and accurate sheep face detection. SheepFaceNetRec uses Eblock to construct the feature extraction network, uses ECA channel attention to improve the effectiveness of feature extraction, and uses multi-scale feature fusion to achieve fast and accurate sheep face recognition. On our self-built sheep face dataset, SheepFaceNet recognized 387 sheep face images per second with an accuracy rate of 97.75%, achieving an advanced balance between speed and accuracy. This research is expected to further promote the application of deep-learning-based sheep face recognition methods in production. [ABSTRACT FROM AUTHOR]

Fruit image classification and recognition is a challenging application of computer vision. The computer vision system is used to recognize a fruit based on artificial neural networks. Deep neural network is widely used for various classification problems. In this paper convolutional neural network (CNN) is used to recognize the fruits. The dataset contains 1877 images of ten categories which are used for the experimental purpose. CNN is constructed with sixteen layers which are used to extract the features from images and support vector machine (SVM) classifier is used for classification. The proposed system has the classification accuracy of 99.2% and the recognition accuracy of 99.02%. [ABSTRACT FROM AUTHOR]

The recent boom of autonomous driving nowadays has made object detection in traffic scenes a hot topic of research. Designed to classify and locate instances in the image, this is a basic but challenging task in the computer vision field. With its powerful feature extraction abilities, which are vital for object detection, deep learning has expanded its application areas to this field during the past several years and thus achieved breakthroughs. However, even with such powerful approaches, traffic scenarios have their own specific challenges, such as real-time detection, changeable weather, and complex lighting conditions. This survey is dedicated to summarizing research and papers on applying deep learning to the transportation environment in recent years. More than 100 research papers are covered, and different aspects such as key generic object detection frameworks, categorized object detection applications in traffic scenario, evaluation metrics, and classified datasets are included. Some open research fields are also provided. We believe that it is the first survey focusing on deep learning-based object detection in traffic scenario. [ABSTRACT FROM AUTHOR]

Different from most current studies using convolutional neural network (CNN), a deep learning detection method for steel plate surface defects based on Transformer is researched. This paper presents a process, network structure and detection method for steel strip surface defect detection. A Swin Transformer was used to extract hierarchical features in the detection system. Then feature pyramid networks (FPN) was used to fuse the above features to form multi‐scale feature maps, and region proposal network (RPN) was adopted to generate the generate region of interest (ROI) of defects. Finally, the ROI head was used to generate defect category information and its precise location. Here, ablation experiments were conducted to explore the impact of different backbone networks and the number of stages of Swin Transformer and FPN on target detection performance to prove the rationality and efficiency of the network. On the NEU‐DET dataset, the proposed algorithm achieved 81.1% mAP, which is higher than that of classic CNN detection methods such as Faster R‐CNN, SSD, Yolo v3 and RepPoints. [ABSTRACT FROM AUTHOR]

It is considered that 1 in 10 adults worldwide have diabetes. Diabetic foot ulcers are some of the most common complications of diabetes, and they are associated with a high risk of lower-limb amputation and, as a result, reduced life expectancy. Timely detection and periodic ulcer monitoring can considerably decrease amputation rates. Recent research has demonstrated that computer vision can be used to identify foot ulcers and perform non-contact telemetry by using ulcer and tissue area segmentation. However, the applications are limited to controlled lighting conditions, and expert knowledge is required for dataset annotation. This paper reviews the latest publications on the use of artificial intelligence for ulcer area detection and segmentation. The PRISMA methodology was used to search for and select articles, and the selected articles were reviewed to collect quantitative and qualitative data. Qualitative data were used to describe the methodologies used in individual studies, while quantitative data were used for generalization in terms of dataset preparation and feature extraction. Publicly available datasets were accounted for, and methods for preprocessing, augmentation, and feature extraction were evaluated. It was concluded that public datasets can be used to form a bigger, more diverse datasets, and the prospects of wider image preprocessing and the adoption of augmentation require further research. [ABSTRACT FROM AUTHOR]

Few-shot hyperspectral classification is a challenging problem that involves obtaining effective spatial–spectral features in an unsupervised or semi-supervised manner. In recent years, as a result of the development of computer vision, deep learning techniques have demonstrated their superiority in tackling the problems of hyperspectral unmixing (HU) and classification. In this paper, we present a new semi-supervised pipeline for few-shot hyperspectral classification, where endmember abundance maps obtained by HU are treated as latent features for classification. A cube-based attention 3D convolutional autoencoder network (CACAE), is applied to extract spectral–spatial features. In addition, an attention approach is used to improve the accuracy of abundance estimation by extracting the diagnostic spectral features associated with the given endmember more effectively. The endmember abundance estimated by the proposed model outperforms other convolutional neural networks (CNNs) with respect to the root mean square error (RMSE) and abundance spectral angle distance (ASAD). Classification experiments are performed on real hyperspectral datasets and compared to several supervised and semi-supervised models. The experimental findings demonstrate that the proposed approach has promising potential for hyperspectral feature extraction and has better performance relative to CNN-based supervised classification under small-sample conditions. [ABSTRACT FROM AUTHOR]

With a large number of Local Binary Patterns (LBP) variants being currently used today, the significant and importance of visual descriptors in computer vision applications are prominent. This paper presents a novel visual descriptor, i.e., SIM-LBP. It employs a new matrix technique called the Symmetric Inline Matrix generator method, which acts as a new variant of LBP. The key feature that separates our variant from existing counterparts is that our variant is very efficient in extracting facial expression features like eyes, eye brows, nose and mouth in a wide range of lighting conditions. For testing our model, we applied SIM-LBP on the JAFFE dataset to convert all the images to its corresponding SIM-LBP transformed variant. These transformed images are then used to train a Convolution Neural Network (CNN) based deep learning model for facial expressions recognition (FER). Several performance evaluation metrics, i.e., recognition accuracy rate, precision, recall, and F1-score, were used to test mode efficiency in comparison with those using the traditional LBP descriptor and other LBP variants. Our model outperformed in all four matrices with the proposed SIM-LBP transformation on the input images against those of baseline methods. In comparison analysis with the other state-of-the-art methods, it shows the usefulness of the proposed SIM-LBP model. Our proposed SIM-LBP variant transformation can also be applied on facial images to identify a person's mental states and predict mood variations. [ABSTRACT FROM AUTHOR]

Semantic segmentation is a well-studied topic and one of the most challenging tasks in computer vision applications, such as autonomous driving. Deep learning approaches based on convolutional neural networks (CNN) have demonstrated exceptional success on this task in recent years. Despite this success, existing approaches are plagued by higher-order inconsistencies between the ground truth images and the ones predicted by the segmentation model. This paper proposes a novel post-processing scheme based on adversarial learning to counter these inconsistencies. Such a scheme can be combined with a variety of existing CNN-based semantic segmentation networks to improve their segmentation performances. The proposed scheme is a Two-Stream Conditional Generative Adversarial Network (TScGAN), with one stream having initial semantic segmentation masks predicted by an existing CNN, while the other stream utilizes scene images to retain high-level information under a supervised residual network structure. In addition, TScGAN incorporates a novel dynamic weighting mechanism, which leads to significant and consistent gains in segmentation performance. Several comparative tests on public benchmark driving databases, including Cityscapes, Mapillary, and Berkeley DeepDrive100K, demonstrate the effectiveness of the proposed method when used with state-of-the-art CNN-based semantic segmentation models. Furthermore, the ablation experiment proved the structural rationality of our two-stream structure. The code for TScGAN could be found at https://github.com/epan-utbm/TScGAN-for-Improving-Semantic-Predictions • Explores various post-processing techniques for enhancing semantic segmentation. • Novel Two-Stream Conditional GAN enhances urban drive scene segmentation. • Enhances feature extraction by implementing dynamic weighting within streams. • Evaluates the model against existing semantic segmentation methodologies. • Tests on Cityscapes, Mapillary, BDD100K reveal major IoU gains with proposed model. [ABSTRACT FROM AUTHOR]

Convolutional neural network (CNN) algorithm is a very important branch of deep learning research, which has been widely applied in many fields and achieved excellent results, especially in computer vision, where convolutional neural network has made breakthroughs in image classification and object detection. Convolutional neural network architecture can realize more efficient network training through the final combination of different modules, and the convolutional neural network training does not need to actively extract image features and can directly carry out end-to-end training and prediction. At first, this paper analyzed some problems of the current image recognition and expounds the progress of convolution neural network in image recognition and then studied the traditional algorithm of target recognition, including traditional recognition algorithm framework of target, the target orientation, feature extraction, classifier classification, etc., and the traditional target recognition algorithm is compared with those of the target recognition algorithm of deep learning. On the basis of the above research, an improved model of CNN is proposed, which focuses on the structural design and network optimization of convolutional neural network and designs a more efficient convolutional neural network. Test experiments verify the effectiveness of the proposed model, which not only achieves lower error rate, but also greatly reduces the number of network parameters and has stronger learning ability. [ABSTRACT FROM AUTHOR]

Video deblurring, which aims at restoring the sharp video from blurry video, is drawing increasing attention in the field of computer vision. In this paper, a method called Multi-Attention Convolutional Neural Network (MACNN) consisting of the temporal-spatial attention module, the frame channel attention module, and the feature extraction-reconstruction module is proposed. First, we use the temporal-spatial attention module and the frame channel attention module to capture features with temporal and spatial information existing across neighboring frames. Then, these captured features are fused and reconstructed to restore the sharp frame. Last but not least, we train MACNN together with a content loss and a perceptual loss in an end-to-end manner to recover realistic video details. Both quantitative and qualitative evaluation results on standard benchmarks demonstrate the proposed MACNN is superior to the state-of-the-art methods in terms of accuracy, efficiency, and visual effect. [ABSTRACT FROM AUTHOR]

In this paper, we first propose an accurate edge detector using a distance field-based convolutional neural network (DF-CNN). In recent years, CNNs have been proved to be effective in image processing and computer vision. As edge detection is a fundamental problem among them, we try to improve the accuracy of edge detection based on the deep learning framework. The proposed network combines a feature extraction backbone that can fully exploit the multiscale and multilevel information of the edge with the supervised training of the distance field branch to realize the accurate end-to-end object edge detection. The distance field branch is applied to predict the Euclidean distance from nonedge points to the nearest edge point in the feature maps. And the distance information embedded in the predicted distance field map can effectively improve the accuracy of edge detection. The network is trained to minimize the weighted sum of the distance field branch loss and the cross-entropy loss. Our experimental results show that the proposed edge detector achieves better performance than previous approaches and the effectiveness of the proposed distance field branch. [ABSTRACT FROM AUTHOR]

Video anomaly detection is the problem of detecting unusual events in videos. The challenges of this task lie mainly in the following aspects: first, unusual events tend to make up only a very small portion of a video, which means a large amount of useless information needs to be culled. It further aggravates the test of algorithm performance and the computing ability of devices. Second, anomaly detection techniques are always used in the surveillance system, which contains massive video data. The analysis of such large video data is difficult. Last, the feature extraction ability of the algorithm appears a high performance since unusual video streams may lie close to normal video. Benefiting from the development of deep learning-based in computer vision fields, the accuracy and the efficiency of video anomaly detection has been improved a lot during recent years. In this paper, we present a newly developed two-stream deep learning model, which uses a 3D convolutional neural network (C3D) structure as the feature extraction part, to handle this task. Both the sequence of frames and the optical flow are required as the input of the model. Then, features of these two streams will be extracted from C3D and traditional convolutional neural network (CNN). Finally, a fusion layer will be used to fuse both results of streams and generate a final detection. Our experimental results on UCF-Crime video dataset outperform other benchmark results such as traditional deep CNN and long short-term memory (LSTM) in terms of area under curve (AUC). As the result, our proposed method achieves the AUC of 85.18%, which is 3% higher than the second highest method. [ABSTRACT FROM AUTHOR]

Recently, food recognition has received more and more attention in image processing and computer vision for its great potential applications in human health. Most of the existing methods directly extracted deep visual features via convolutional neural networks (CNNs) for food recognition. Such methods ignore the characteristics of food images and are, thus, hard to achieve optimal recognition performance. In contrast to general object recognition, food images typically do not exhibit distinctive spatial arrangement and common semantic patterns. In this paper, we propose a multi-scale multi-view feature aggregation (MSMVFA) scheme for food recognition. MSMVFA can aggregate high-level semantic features, mid-level attribute features, and deep visual features into a unified representation. These three types of features describe the food image from different granularity. Therefore, the aggregated features can capture the semantics of food images with the greatest probability. For that solution, we utilize additional ingredient knowledge to obtain mid-level attribute representation via ingredient-supervised CNNs. High-level semantic features and deep visual features are extracted from class-supervised CNNs. Considering food images do not exhibit distinctive spatial layout in many cases, MSMVFA fuses multi-scale CNN activations for each type of features to make aggregated features more discriminative and invariable to geometrical deformation. Finally, the aggregated features are more robust, comprehensive, and discriminative via two-level fusion, namely multi-scale fusion for each type of features and multi-view aggregation for different types of features. In addition, MSMVFA is general and different deep networks can be easily applied into this scheme. Extensive experiments and evaluations demonstrate that our method achieves state-of-the-art recognition performance on three popular large-scale food benchmark datasets in Top-1 recognition accuracy. Furthermore, we expect this paper will further the agenda of food recognition in the community of image processing and computer vision. [ABSTRACT FROM AUTHOR]

This paper intends to present an automated mango grading system under four stages (1) pre‐processing, (2) feature extraction, (3) optimal feature selection and (4) classification. Initially, the input image is subjected to the pre‐processing phase, where the reading, sizing, noise removal and segmentation process happens. Subsequently, the features are extracted from the pre‐processed image. To make the system more effective, from the extracted features, the optimal features are selected using a new hybrid optimization algorithm termed the lion assisted firefly algorithm (LA‐FF), which is the combination of LA and FF, respectively. Then, the optimal features are given for the classification process, where the optimized deep convolutional neural network (CNN) is deployed. As a major contribution, the configuration of CNN is fine‐tuned via selecting the optimal count of convolutional layers. This obviously enhances the classification accuracy in grading system. For fine‐tuning the convolutional layers in the deep CNN, the LA‐FF algorithm is used so that the classifier is optimized. The grading is evaluated on the basis of healthydiseased, ripeunripe and bigmediumvery big cases with respect to type I and type II measures and the performance of the proposed grading model is compared over the other state‐of‐the‐art models. [ABSTRACT FROM AUTHOR]

Shadow removal can significantly improve the image visual quality and has many applications in computer vision. Deep learning methods based on CNNs have become the most effective approach for shadow removal by training on either paired data, where both the shadow and underlying shadow-free versions of an image are known, or unpaired data, where shadow and shadow-free training images are totally different with no correspondence. In practice, CNN training on unpaired data is more preferred given the easiness of training data collection. In this paper, we present a new Lightness-Guided Shadow Removal Network (LG-ShadowNet) for shadow removal by training on unpaired data. In this method, we first train a CNN module to compensate for the lightness and then train a second CNN module with the guidance of lightness information from the first CNN module for final shadow removal. We also introduce a loss function to further utilise the colour prior of existing data. Extensive experiments on widely used ISTD, adjusted ISTD and USR datasets demonstrate that the proposed method outperforms the state-of-the-art methods with training on unpaired data. [ABSTRACT FROM AUTHOR]

Image inpainting is a challenging computer vision task that aims to fill in missing regions of corrupted images with realistic contents. With the development of convolutional neural networks, many deep learning models have been proposed to solve image inpainting issues by learning information from a large amount of data. In particular, existing algorithms usually follow an encoding and decoding network architecture in which some operations with standard schemes are employed, such as static convolution, which only considers pixels with fixed grids, and the monotonous normalization style (e.g., batch normalization). However, these techniques are not well-suited for the image inpainting task because the random corrupted regions in the input images tend to mislead the inpainting process and generate unreasonable content. In this paper, we propose a novel dynamic selection network (DSNet) to solve this problem in image inpainting tasks. The principal idea of the proposed DSNet is to distinguish the corrupted region from the valid ones throughout the entire network architecture, which may help make full use of the information in the known area. Specifically, the proposed DSNet has two novel dynamic selection modules, namely, the validness migratable convolution (VMC) and regional composite normalization (RCN) modules, which share a dynamic selection mechanism that helps utilize valid pixels better. By replacing vanilla convolution with the VMC module, spatial sampling locations are dynamically selected in the convolution phase, resulting in a more flexible feature extraction process. Besides, the RCN module not only combines several normalization methods but also normalizes the feature regions selectively. Therefore, the proposed DSNet can illustrate realistic and fine-detailed images by adaptively selecting features and normalization styles. Experimental results on three public datasets show that our proposed method outperforms state-of-the-art methods both quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

For a long time, learning spatiotemporal features with deep neural networks has been a difficult task in the field of computer vision. In this paper, we present a novel deep architecture, termed as Bifurcated Convolutional Neural Network (BifurcatedNet) to learn the discriminative video representation in an end-to-end manner. In our work, the BifurcatedNet is built upon the stacking bifurcated blocks that aim at simultaneously capturing the static appearance information and the temporal dynamic from input data. Specifically, the bifurcated block is composed of two separated branch, i.e., an appearance branch and a dynamic branch. The appearance branch employs 2D convolutional operation to obtain the spatial responses of image pixels or filters of each input frame, while the design of the dynamic branch is based on the spatio-temporal convolutional operation to exploit the temporal dynamic between pixels and filter response across multiple frames. Multiple experiments are conducted on two popular action recognition benchmarks: UCF101 and HMDB51. With only RGB input, the BifurcatedNet obtains the superior performance over the existing state-of-the-art models under the same experimental setting. The proposed BifurcatedNet is also implemented in a two-stream fashion by using both RGB and optical flow input, and still achieves the state-of-the-art performance, demonstrating the effectiveness of the network design. Furthermore, in order to evaluate the generalization ability, we conduct experiments on the Chalearn LAP IsoGD dataset and find that our model works well in gesture recognition tasks. [ABSTRACT FROM AUTHOR]

Human detection is one of the long-standing computer vision tasks, and it has been a cornerstone for many real-world applications, such as photo album organization, video surveillance, and autonomous driving. Benefiting from deep learning technologies, such as convolutional neural networks and modern object detectors, have been achieving much improved accuracy in generic object detection tasks. In this paper, we aim to improve deep learning-based human detection. Our main idea is to exploit semantic context information for human detection by using deep-learnt semantic features provided by semantic segmentation masks. Segmentation masks play as an attention mechanism and enforce the detectors to focus on the image regions where potential object candidates are likely to appear. Meanwhile, the extra segmentation mask channel can also guide the convolutional kernels to automatically learn more discriminative features which make it easier to distinguish the background and foreground. We implement our methods with two popular detection frameworks, i.e., faster R-CNN and SSD and experimentally analyze the effectiveness of the proposed methods. Evaluation results on the widely used MS-COCO dataset and the very recent CrowdHuman dataset are provided. Our proposed methods outperform the baseline detectors and achieve better performance on highly occluded human detection. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have facilitated substantial progress on various problems in computer vision and image processing. However, applying them to image fusion has remained challenging due to the lack of the labelled data for supervised learning. This paper introduces a deep image fusion network (DIF-Net), an unsupervised deep learning framework for image fusion. The DIF-Net parameterizes the entire processes of image fusion, comprising of feature extraction, feature fusion, and image reconstruction, using a CNN. The purpose of DIF-Net is to generate an output image which has an identical contrast to high-dimensional input images. To realize this, we propose an unsupervised loss function using the structure tensor representation of the multi-channel image contrasts. Different from traditional fusion methods that involve time-consuming optimization or iterative procedures to obtain the results, our loss function is minimized by a stochastic deep learning solver with large-scale examples. Consequently, the proposed method can produce fused images that preserve source image details through a single forward network trained without reference ground-truth labels. The proposed method has broad applicability to various image fusion problems, including multi-spectral, multi-focus, and multi-exposure image fusions. Quantitative and qualitative evaluations show that the proposed technique outperforms existing state-of-the-art approaches for various applications. [ABSTRACT FROM AUTHOR]

Driven by recent computer vision and robotic applications, recovering 3D human poses has become increasingly important and attracted growing interests. In fact, completing this task is quite challenging due to the diverse appearances, viewpoints, occlusions and inherently geometric ambiguities inside monocular images. Most of the existing methods focus on designing some elaborate priors /constraints to directly regress 3D human poses based on the corresponding 2D human pose-aware features or 2D pose predictions. However, due to the insufficient 3D pose data for training and the domain gap between 2D space and 3D space, these methods have limited scalabilities for all practical scenarios (e.g., outdoor scene). Attempt to address this issue, this paper proposes a simple yet effective self-supervised correction mechanism to learn all intrinsic structures of human poses from abundant images. Specifically, the proposed mechanism involves two dual learning tasks, i.e., the 2D-to-3D pose transformation and 3D-to-2D pose projection, to serve as a bridge between 3D and 2D human poses in a type of “free” self-supervision for accurate 3D human pose estimation. The 2D-to-3D pose implies to sequentially regress intermediate 3D poses by transforming the pose representation from the 2D domain to the 3D domain under the sequence-dependent temporal context, while the 3D-to-2D pose projection contributes to refining the intermediate 3D poses by maintaining geometric consistency between the 2D projections of 3D poses and the estimated 2D poses. Therefore, these two dual learning tasks enable our model to adaptively learn from 3D human pose data and external large-scale 2D human pose data. We further apply our self-supervised correction mechanism to develop a 3D human pose machine, which jointly integrates the 2D spatial relationship, temporal smoothness of predictions and 3D geometric knowledge. Extensive evaluations on the Human3.6M and HumanEva-I benchmarks demonstrate the superior performance and efficiency of our framework over all the compared competing methods. [ABSTRACT FROM AUTHOR]

The catenary support components (CSCs) are the most important devices in high-speed railways to support contact lines for powering trains. To estimate the states of CSCs, it is very necessary to locate their positions in a monitoring system based on computer vision. Considering the application scenarios and characteristics of CSCs, an automatic and quick positioning system is designed in this paper to simultaneously position the multiscale CSCs with 12 categories. In the system, an effective framework called CSCs network (CSCNET) is presented, which cascades the coarse positioning network and the fine positioning network to reduce multiscale differences between different CSCs. In the coarse positioning network, a new unsupervised clustering algorithm based on the relative positioning information is proposed to classify the catenary images. Then, a convolutional neural network (CNN) classification network is trained to extract the structural features of catenary images and generate the proposal regions with labels. In the fine positioning network, a modified CNN positioning framework is applied to obtain the accurate positions of CSCs based on the coarse positioning results. Due to the special lightweight structure with a classification network, the relative position information is applied and makes the CSCNET sensitive to small-scale components. The experimental results from some high-speed railway lines in China show that the proposed system has obvious advantages in the CSCs positioning. The mean average precision and frames per second of CSCNET reach 0.837 and 2.17, respectively. Compared with some popular convolutional networks [faster region-based CNN (Faster R-CNN), etc.] and a typical positioning method, the proposed system significantly improves the AP without increasing the computational time. [ABSTRACT FROM AUTHOR]

Automatic target recognition (ATR) is an important part for many computer vision applications. Despite the extensive research which has been carried out in this area for many years, there is no ATR system which performs well on all applications. Recently, different object recognition frameworks have been proposed which yield a high performance in baseline databases. However, our experiments showed that they can fail in real-world scenarios, when dealing with a limited number of data samples. In this paper, we propose a new ATR system, based on deep convolutional neural network (DCNN), to detect the targets in forward looking infrared (FLIR) scenes and recognize their classes. In our proposed ATR framework, a fully convolutional network is trained to map the input FLIR imagery data to a fixed stride correspondingly-sized target score map. The potential targets are identified by applying a threshold on the target score map. Finally, the corresponding regions centered at these target points are fed to a DCNN to classify them into different target types while at the same time rejecting the false alarms. The proposed architecture achieves a significantly better performance in comparison with that of the state-of-the-art methods on two large FLIR image databases. [ABSTRACT FROM AUTHOR]

Moving object detection is an essential, well-studied but still open problem in computer vision and plays a fundamental role in many applications. Traditional approaches usually reconstruct background images with hand-crafted visual features, such as color, texture, and edge. Due to lack of prior knowledge or semantic information, it is difficult to deal with complicated and rapid changing scenes. To exploit the temporal structure of the pixel-level semantic information, in this paper, we propose an end-to-end deep sequence learning architecture for moving object detection. First, the video sequences are input into a deep convolutional encoder–decoder network for extracting pixel-wise semantic features. Then, to exploit the temporal context, we propose a novel attention long short-term memory (Attention ConvLSTM) to model pixelwise changes over time. A spatial transformer network and a conditional random field layer are finally appended to reduce the sensitivity to camera motion and smooth the foreground boundaries. A multi-task loss is proposed to jointly optimization for frame-based classification and temporal prediction in an end-to-end network. Experimental results on CDnet 2014 and LASIESTA show 12.15% and 16.71% improvement to the state of the art, respectively. [ABSTRACT FROM AUTHOR]

While action recognition has become an important line of research in computer vision, the recognition of particular events, such as aggressive behaviors, or fights, has been relatively less studied. These tasks may be extremely useful in several video surveillance scenarios, such as psychiatric wards, prisons, or even in personal camera smartphones. Their potential usability has led to a surge of interest in developing fight or violence detectors. One of the key aspects in this case is efficiency, that is, these methods should be computationally fast. “Handcrafted” spatio-temporal features that account for both motion and appearance information can achieve high accuracy rates, albeit the computational cost of extracting some of those features is still prohibitive for practical applications. The deep learning paradigm has been recently applied for the first time to this task too, in the form of a 3D convolutional neural network that processes the whole video sequence as input. However, results in human perception of other’s actions suggest that, in this specific task, motion features are crucial. This means that using the whole video as input may add both redundancy and noise in the learning process. In this paper, we propose a hybrid “handcrafted/learned” feature framework which provides better accuracy than the previous feature learning method, with similar computational efficiency. The proposed method is compared to three related benchmark data sets. The method outperforms the different state-of-the-art methods in two of the three considered benchmark data sets. [ABSTRACT FROM AUTHOR]

Usage of depth data in various computer vision applications gains popularity with cheap depth sensors available on the market. In this study, a method that utilizes Deep Learning is proposed for single human actions and dyadic actions. Depth data of the actions are used to construct a three-dimensional (3D) template. These templates are rotated in different directions and two-dimensional (2D) views from different angles are stored. Acquired 2D images are used in deep feature extraction. AlexNet (Krizhevsky, Sutskever, & Hinton, 2012) pre-trained convolutional neural network is used for deep feature extraction. For all viewpoints, deep features are extracted and concatenated. After that random forest classifier is used for action recognition. The contributions of this paper are as follows. First, a 3D isosurface models is constructed to figure actions sequence. Second, we project 3D shapes into a 2D feature space by taking its snapshots from different views and giving them to a pre-trained CNN as input for feature extraction. We explore more information about the actions by extracting features from different layers of the deep neural network. We compare the results of the deep features extracted from different layers of the pre-trained CNN. Various classifiers are trained with extracted deep features. Because of the complex structure of the 3D shapes, there are a limited number of feature extraction methods. Our method obtains close to state-of-the-art performance on various single and two-person action datasets. [ABSTRACT FROM AUTHOR]

Following the great success of deep convolutional neural networks (CNNs) in computer vision, this paper proposes a complex-valued CNN (CV-CNN) specifically for synthetic aperture radar (SAR) image interpretation. It utilizes both amplitude and phase information of complex SAR imagery. All elements of CNN including input-output layer, convolution layer, activation function, and pooling layer are extended to the complex domain. Moreover, a complex backpropagation algorithm based on stochastic gradient descent is derived for CV-CNN training. The proposed CV-CNN is then tested on the typical polarimetric SAR image classification task which classifies each pixel into known terrain types via supervised training. Experiments with the benchmark data sets of Flevoland and Oberpfaffenhofen show that the classification error can be further reduced if employing CV-CNN instead of conventional real-valued CNN with the same degrees of freedom. The performance of CV-CNN is comparable to that of existing state-of-the-art methods in terms of overall classification accuracy. [ABSTRACT FROM PUBLISHER]

• This paper proposes densely connected convolutional networks (DenseNets) for SHM. • DenseNets is applied to perform vibration based structural damage identification. • Dense block is used to alleviate the gradient vanishing and strengthen feature flow. • DenseNets is developed as an efficient feature and robust feature extractor. • Numerical and experimental studies are conducted to validate the proposed approach. Vibration-based damage identification has been a challenging task in structural health monitoring. The main difficulty lies on the reliable correlation between the measured vibration characteristics and damage states (e.g., stiffness reductions) of structures. Such states can ideally indicate the presence, locations, and severities of structural damages. The procedure is considered as a feature extraction process from the input measurement, mapping the selected features to damage states. Time domain vibration responses, e.g., acceleration responses, are used in this study for damage identification. To address this pattern recognition problem, many methods have been developed including various neural networks in recent years. This paper proposes a novel approach based on densely connected convolutional networks (DenseNets), which is one of the major breakthroughs in the computer vision community, for vibration based structural damage identification. It implements dense connectivity in the convolutional neural network architecture, which fits well for this study using acceleration responses. Both low-level and high-level features are learned and reused during training. It not only eases the information flow during training, but also preserves all levels of features and tends to be more effective for damage identification. Besides, the dense connectivity alleviates the gradient vanishing problem and strengthens feature propagation through the network. In the meantime, these designs substantially reduce the number of parameters, making the network easy to train. The performance of the proposed approach is evaluated through both numerical and experimental verifications. Both modelling uncertainties and measurement noises are considered in numerical studies. The results from numerical and experimental studies demonstrate that the damage localization and quantification are achieved with high accuracies (e.g., Regression value ≥ 96.0% on numerical datasets, and ≥ 94.9% on experimental datasets) and good robustness. [ABSTRACT FROM AUTHOR]

Using computer vision and deep learning (e.g., Convolutional Neural Networks) to automatically recognise unsafe behaviour from digital images can help managers identify and respond quickly to such actions and mitigate an adverse event. However, there has been a tendency for computer vision studies in construction to focus solely on detecting unsafe behaviour (i.e., object detection) or the regions of interest with pre-defined labels. Moreover, such approaches have been unable to consider rich semantic information among multiple unsafe actions in a digital image. The research we present in this paper uses a safety rule query to determine and locate several unsafe behaviours in a digital image by employing a visual grounding approach. Our approach consists of: (1) visual and text feature extraction, (2) recursive sub-query, and (3) generation of the bounding box. We validate our approach by conducting an experiment to demonstrate it is effectiveness. The results from an experimental study demonstrate an average precision, recall, and F1-score were 0.55, 0.85, and 0.65, respectively, suggesting our approach can accurately identify and locate different types of unsafe behaviours from digital images acquired from a construction site. [ABSTRACT FROM AUTHOR]

This paper presents alternative solutions for classifying concrete spall severity based on computer vision approaches. Extreme Gradient Boosting Machine (XGBoost) and Deep Convolutional Neural Network (DCNN) are employed for categorizing image samples into two classes: shallow spall and deep spall. To delineate the properties of a concrete surface subject to spall, texture descriptors including local binary pattern, center symmetric local binary pattern, local ternary pattern, and attractive repulsive center symmetric local binary pattern (ARCS-LBP) are employed as feature extraction methods. In addition, the prediction performance of XGBoost is enhanced by Aquila optimizer metaheuristic. Meanwhile, DCNN is capable of performing image classification directly without the need for texture descriptors. Experimental results with a dataset containing real-world concrete surface images and 20 independent model evaluations point out that the XGBoost optimized by the Aquila metaheuristic and used with ARCS-LBP has achieved an outstanding classification performance with a classification accuracy rate of roughly 99%. • Propose a computer vision method for classifying concrete spall severity. • Extreme Gradient Boosting Machine and deep neural networks are employed. • Use local binary pattern variants as image texture descriptors. • The boosting machine is finetuned by the Aquila optimizer. • The proposed method achieves a high classification accuracy rate = 0.99. [ABSTRACT FROM AUTHOR]

• The Hybrid-SSR framework is proposed to perform real-time hand gesture recognition. • The Enhanced Xception (E-Xception) architecture is utilized as a backbone network. • Mitigates the misclassification problem encountered by the conventional models. • The proposed model is evaluated on MITI-HD, NUSHP-II and Senz-3D datasets. In-situ detection and recognition of hand gestures in an unconstrained environment is a crucial task in the field of machine vision and video recognition. Hand gesture recognizing task is perhaps more complicated, with varying lighting conditions and background. Robust, accurate hand detection and classification becomes necessary to facilitate the Graphical User Interface, Tele-surgery, etc. In this paper, a Convolutional Neural Network (CNN) based Hybrid Single Stage Recognition (Hybrid-SSR) framework is proposed for real-time hand gesture recognition. The time-efficient Enhanced Xception (E-Xception) CNN model is proposed for feature extraction. The framework is evaluated using three separate datasets (NUSHP-II, Senz-3D, and MITI-HD). The performance of the framework is evaluated for the range of IoU from 0.5 to 0.95. In comparison with the conventional single-stage hand gesture recognition system, the Hybrid-SSR model resulted in higher precision values (99.60 % on AP 0.5 , 97.80 % on AP 0.75 , and 88.20 % on AP 0.5:0.95) for the MITI-HD dataset. The Hybrid-SSR model with Adam Optimizer surpasses the other optimization techniques. The proposed model achieved comparatively higher precision and recall values for focusing parameter (γ) = 2 and weighting parameter (α) = 0.5. The misclassification problem encountered by the conventional single-stage models is mitigated by implementing the focal loss function in the classification sub-network. The computation cost of Hybrid-SSR (12 ms) is significantly lower compared to other single-stage hand gesture recognition frameworks. [ABSTRACT FROM AUTHOR]

• A SEGmentation-free Writer Identification (SEG-WI) model based on CNN is proposed to identify the writer. • Region selection mechanism is also develop to improve the overall performance of the model. • The model utilizes the convolution layers with different kernel size and stride. • A new training strategy is suggested to train the model. • The model has been evaluated for various databases such as IAM, CVL, IFN/ENIT, Kannada and Devnagri (Hindi) script. Handwriting recognition is one of the desired aspects of document understanding and analysis. It deals with the writing style of the document and learns the features which differentiate the writers. In this paper, a SEGmentation-free Writer Identification (SEG-WI) model is proposed based on a convolution neural network and a weakly supervised region selection mechanism. The model, SEG-WI, takes an unsegmented text document and produces the writer-ID with the region probability map. The probability vectors at each cell location in the input document constitute a region probability map. To achieve the best performance, top 10% to 50% cell regions are selected for decision making and a voting mechanism among the selected regions is used to identify the writer. The region selection, voting mechanism for decision making, and loss calculation are the main contributions in this work, which enables the proposed system as segmentation free. The proposed model is evaluated on different datasets such as IAM handwriting database (IAM), and computer vision lab (CVL) for English, Institut of communications technology/Ecole Nationale d'Ing nieurs de Tunis (IFN/ENIT) for Arabic, Kannada, and Devanagari for Indic, and outperforms compare with the state-of-the-art results. Moreover, a comparative analysis of the proposed model with and without region selection is performed to validate the effect of the region selection mechanism and it improves the performance of the model. [ABSTRACT FROM AUTHOR]

Crowd counting, which is widely used in disaster management, traffic monitoring, and other fields of urban security, is a challenging task that is attracting increasing interest from researchers. For better accuracy, most methods have attempted to handle the scale variation explicitly. which results in huge scale changes of the object size. However, earlier methods based on convolutional neural networks (CNN) have focused primarily on improving accuracy while ignoring the complexity of the model. This paper proposes a novel method based on a lightweight CNN-based network for estimating crowd counting and generating density maps under resource constraints. The network is composed of three components: a basic feature extractor (BFE), a stacked à trous convolution module (SACM), and a context fusion module (CFM). The BFE encodes basic feature information with reduced spatial resolution for further refining. Various pieces of contextual information are generated through a short pipeline in SACM. To generate a context fusion density map, CFM distills feature maps from the above components. The whole network is trained in an end-to-end fashion and uses a compression factor to restrict its size. Experiments on three highly-challenging datasets demonstrate that the proposed method delivers attractive performance. [ABSTRACT FROM AUTHOR]