Search

This paper presents a novel deep learning-based method that integrates a Convolutional Auto-Encoder (CAE) with support vector machine (SVM) for finger-vein verification. The CAE is used to learn the features from finger-vein images, and the SVM is used to classify finger vein from these learned feature codes. The CAE consists of a finger-vein encoder, which extracts high-level feature representation from raw pixels of the images, and a decoder which outputs reconstruct finger-vein images from high-level feature code. As an effective classifier, SVM is introduced in this paper to classify the feature code which is obtained from CAE. Experiments prove that the proposed deep learning-based approach has superior performance in learning features than traditional method without any prior knowledge, presenting a good potential in the verification of finger vein. [ABSTRACT FROM AUTHOR]

Electrical capacitance tomography exhibits great potentials in the visualization measurement of industrial processes, and high-precision images are of great significance for the reliability and usefulness of measurement results. In this paper, we propose a deep learning-based inversion method to ameliorate the reconstruction accuracy. With the aid of the deep learning methodology, the prior from the images reconstructed by a certain imaging technique to the true images is abstracted and stored in the deep extreme learning machine. A new cost function is constructed to encapsulate the prior from the proposed deep learning model and the domain expertise about imaging targets, and the split Bregman algorithm and the fast iterative shrinkage thresholding technique are combined into a new numerical method to effectively solve it to get the final reconstruction. The numerical and experimental results validate that the inversion method proposed in this paper reduces the reconstruction artifacts and deformations and leads to the much improvement in the imaging quality. [ABSTRACT FROM AUTHOR]

This paper presents an effective diagnosis algorithm for permanent magnet synchronous motors running with an array of faults of varying severity over a wide speed range. The fault diagnosis is based on a current signature analysis. The complete fault motor diagnosis system requires the extraction of features based on the current method and a subsequent method for adding classifications. In this paper, we propose two feature extraction methods: the first involves a classification method that utilizes a wavelet packet transform and the second is a deep 1-D convolution neural network that includes a softmax layer. The experimental results obtained using real-time motor stator current data demonstrate the effectiveness of the proposed methods for real-time monitoring of motor conditions. The results also demonstrate that the proposed methods can effectively diagnose five different motor states, including two different demagnetization fault states and two bearing fault states. [ABSTRACT FROM AUTHOR]

Deep learning (DL) architecture, which exploits multiple hidden layers to learn hierarchical representations automatically from massive input data, presents a promising tool for characterizing fault conditions. This paper proposes a DL-based multi-signal fault diagnosis method that leverages the powerful feature learning ability of a convolutional neural network (CNN) in images. The proposed deep model is able to learn from multiple types of sensor signals simultaneously so that it can achieve robust performance and finally realize accurate induction motor fault recognition. First, the acquired sensor signals are converted to time–frequency distribution (TFD) by wavelet transform. Then, a deep CNN is applied to learning discriminative representations from the TFD images. Since then, a fully connected layer in deep architecture gives the prediction of induction motor condition based on learned features. In order to verify the effectiveness of the designed deep model, experiments are carried out on a machine fault simulator where both vibration and current signals are analyzed. Experimental results indicate that the proposed method outperforms traditional fault diagnosis methods, hence, demonstrating effectiveness in induction motor application. Compared with conventional methods that rely on delicate features extracted by experienced experts, the proposed deep model is able to automatically learn and select suitable features that contribute to accurate fault diagnosis. Compared with single-signal input, the multi-signal model has more accurate and stable performance and overcomes the overfitting problem to some degree. [ABSTRACT FROM AUTHOR]

For the data-driven remaining useful life (RUL) prediction for rolling bearings, the traditional machine learning-based methods generally provide insufficient feature representation and adaptive extraction. Although deep learning-based RUL prediction methods can solve these problems to some extent, they still do not yield satisfactory predictive results due to less degradation data and inconsistent data distribution among different bearings. To solve these problems, a new RUL prediction method based on deep feature representation and transfer learning is proposed in this paper. This method includes an off-line stage and an online stage. In the off-line stage, the Hilbert–Huang transform marginal spectra of the raw vibration signal of auxiliary bearings are first calculated as the input, and then contractive denoising autoencoder is introduced to extract deep features with good and stable fault representation. Second, by using the obtained deep features and Pearson’s correlation coefficient, a new health condition assessment method is proposed to divide the whole life of each bearing into a normal state and a fast-degradation state. Finally, using the extracted deep features and their RUL values, an RUL prediction model for the fast-degradation state is trained by means of a least-square support vector machine. In the online stage, a kind of transfer learning algorithm, i.e., transfer component analysis, is introduced to sequentially adjust the features of target bearing from auxiliary bearings, and then the corresponding RUL is predicted using the corrected features. Results using the PHM Challenging 2012 data set show a significant performance improvement when using the proposed method in terms of predictive accuracy and numerical stability. [ABSTRACT FROM AUTHOR]

A complete defect detection task aims to achieve the specific class and precise location of each defect in an image, which makes it still challenging for applying this task in practice. The defect detection is a composite task of classification and location, leading to related methods is often hard to take into account the accuracy of both. The implementation of defect detection depends on a special detection data set that contains expensive manual annotations. In this paper, we proposed a novel defect detection system based on deep learning and focused on a practical industrial application: steel plate defect inspection. In order to achieve strong classification ability, this system employs a baseline convolution neural network (CNN) to generate feature maps at each stage, and then the proposed multilevel feature fusion network (MFN) combines multiple hierarchical features into one feature, which can include more location details of defects. Based on these multilevel features, a region proposal network (RPN) is adopted to generate regions of interest (ROIs). For each ROI, a detector, consisting of a classifier and a bounding box regressor, produces the final detection results. Finally, we set up a defect detection data set NEU-DET for training and evaluating our method. On the NEU-DET, our method achieves 74.8/82.3 mAP with baseline networks ResNet34/50 by using 300 proposals. In addition, by using only 50 proposals, our method can detect at 20 ft/s on a single GPU and reach 92% of the above performance, hence the potential for real-time detection. [ABSTRACT FROM AUTHOR]

This paper presents a new online detection approach for rolling bearing’s incipient fault based on self-adaptive deep feature matching (SDFM). This approach includes offline and online stages. At the offline stage, a new health state assessment algorithm is first proposed based on singular value decomposition (SVD) and Kurtosis criterion. Based on the assessment results, a kind of deep learning algorithm, i.e., stacked denoising autoencoder (SDAE), is introduced to extract the common deep features of normal state and early fault state. Support vector data description (SVDD) is applied to establish the offline detection model using the obtained features. At the online stage, a self-adaptive matching strategy with 1-Dimensional anchor is proposed. By utilizing the SDAE model established at offline stage, this strategy can extract more representative deep features of the target bearing via generating various proposal fragments and then determining the fault occurrence time in a self-adaptive way by feeding the online features into the SVDD model. Experiments run on the bearing data set of IEEE prognostic and health management (PHM) Challenge 2012. The results show the proposed approach has good detection performance in real time and much lower false alarm rate, with no need to acquire fault characteristic frequency in advance. [ABSTRACT FROM AUTHOR]

Fault diagnosis is vital for modern industry, and an increasing number of intelligent methods have been proposed for the fault diagnosis. However, most of the studies focus on distinguishing different fault patterns while ignoring fault deterioration. In this paper, a new hierarchical convolutional neural network (HCNN) is proposed as the two-level hierarchical diagnosis network, and it has two characteristics: 1) the fault pattern and fault severity are modeled as one hierarchical structure and 2) the fault pattern and fault severity can be estimated at the same time. Based on these, a new structure of HCNN is designed, which has two classifiers. Then, a two-stage training method is developed for HCNN to train these two classifiers at once training. The proposed HCNN is conducted on three case studies and has achieved state-of-the-art results. The results show that HCNN outperforms traditional two-layer hierarchical fault diagnosis network, and other machine learning and deep learning methods. [ABSTRACT FROM AUTHOR]

This paper introduces a soft sensor (SS) for the estimation of the deflection of a polymeric mechanical actuator. The actuator is based on ionic polymer-metal composites (IPMCs). Applications of IPMCs have been proposed in fields such as robotics, surgery, and aerospace, to mention the most interesting ones. In such application fields, both the complexity and the size of the actuating system are of chief importance. An SS can be, therefore, preferred to hardware measuring the actuator output, for estimating the actuator motion. Also, low-order models are of interest to limit the computational load, which can be a constraint in real-time applications. To this aim, several data-driven nonlinear finite-impulse response (NFIR) models have been investigated. Data, used for the model identification, have been acquired, in controlled environmental conditions, by using swept signals as the input to the IPMC actuator. Linear and nonlinear models, based on principal component analysis, shallow, and deep neural networks (NNs), have been investigated, for different model orders. The best results have been obtained by an SS based on a fifth-order NFIR model, implemented by a deep belief NN. [ABSTRACT FROM AUTHOR]

Effective intelligent fault diagnosis has long been a research focus on the condition monitoring of rotary machinery systems. Traditionally, time-domain vibration-based fault diagnosis has some deficiencies, such as complex computation of feature vectors, excessive dependence on prior knowledge and diagnostic expertise, and limited capacity for learning complex relationships in fault signals. Furthermore, following the increase in condition data, how to promptly process the massive fault data and automatically provide accurate diagnosis has become an urgent need to solve. Inspired by the idea of compressed sensing and deep learning, a novel intelligent diagnosis method is proposed for fault identification of rotating machines. In this paper, a nonlinear projection is applied to achieve the compressed acquisition, which not only reduces the amount of measured data that contained all the information of faults but also realizes the automatic feature extraction in transform domain. For exploring the discrimination hidden in the acquired data, a stacked sparse autoencoders-based deep neural network is established and performed with an unsupervised learning procedure followed by a supervised fine-tuning process. We studied the significance of compressed acquisition and provided the effects of key factors and comparison with traditional methods. The effectiveness of the proposed method is validated using data sets from rolling element bearings and the analysis shows that it is able to obtain high diagnotic accuracies and is superior to the existing methods. The proposed method reduces the need of human labor and expertise and provides new strategy to handle the massive data more easily. [ABSTRACT FROM PUBLISHER]

The recent success of deep learning in 3-D data analysis relies upon the availability of large annotated data sets. However, creating 3-D data sets with point-level labels are extremely challenging and require a huge amount of human efforts. This paper presents a novel open-sourced method to extract light detection and ranging point clouds with ground truth annotations from a simulator automatically. The virtual sensor can be configured to simulate various real devices, from 2-D laser scanners to 3-D real-time sensors. Experiments are conducted to show that using additional synthetic data for training can: 1) achieve a visible performance boost in accuracy; 2) reduce the amount of manually labeled real-world data; and 3) help to improve the generalization performance across data sets. [ABSTRACT FROM AUTHOR]

The catenary support components (CSCs) are the most important devices in high-speed railways to support contact lines for powering trains. To estimate the states of CSCs, it is very necessary to locate their positions in a monitoring system based on computer vision. Considering the application scenarios and characteristics of CSCs, an automatic and quick positioning system is designed in this paper to simultaneously position the multiscale CSCs with 12 categories. In the system, an effective framework called CSCs network (CSCNET) is presented, which cascades the coarse positioning network and the fine positioning network to reduce multiscale differences between different CSCs. In the coarse positioning network, a new unsupervised clustering algorithm based on the relative positioning information is proposed to classify the catenary images. Then, a convolutional neural network (CNN) classification network is trained to extract the structural features of catenary images and generate the proposal regions with labels. In the fine positioning network, a modified CNN positioning framework is applied to obtain the accurate positions of CSCs based on the coarse positioning results. Due to the special lightweight structure with a classification network, the relative position information is applied and makes the CSCNET sensitive to small-scale components. The experimental results from some high-speed railway lines in China show that the proposed system has obvious advantages in the CSCs positioning. The mean average precision and frames per second of CSCNET reach 0.837 and 2.17, respectively. Compared with some popular convolutional networks [faster region-based CNN (Faster R-CNN), etc.] and a typical positioning method, the proposed system significantly improves the AP without increasing the computational time. [ABSTRACT FROM AUTHOR]

Fault diagnosis is of importance to guarantee the printing quality and to avoid unexpected downtime for 3-D printers. In this paper, a sparse echo autoencoder network (SEAEN) is proposed for the fault diagnosis of delta 3-D printers using attitude data. Considering the practicality and economy of the fault diagnosis, the attitude data, including three-axial angular velocity signals, three-axial vibratory acceleration ones, and three-axial magnetic field intensity ones, are collected by installing a low-cost attitude sensor on the moving platform of the delta 3-D printer. However, the low-cost sensor will increase the chaos of the attitude data. To make up this deficiency, the SEAEN approach featuring a sparse autoencoder (SAE) combined with an echo state network (ESN) is designed for the fault diagnosis. The SAE is employed to automatically learn features from the high-dimensional attitude data of the delta 3-D printer, while the ESN is used for fault recognition based on the extracted features. The diagnosis performance of the address approach was evaluated in the experiments and its superiority was demonstrated through comparing with other intelligent fault diagnosis techniques. [ABSTRACT FROM AUTHOR]

Fault detection and diagnosis (FDD) is crucial for stable, reliable, and safe operation of industrial equipment. In recent years, deep learning models have been widely used in data-driven FDD methods because of their automatic feature learning capability. In general, these models are trained on historical sensor data, and therefore, it is very difficult to meet the real-time requirement of online FDD applications. Since transfer learning can solve different but similar problems in the target domain efficiently and effectively with the knowledge learned from the source domain, this paper proposes an online fault diagnosis method based on a deep transfer convolutional neural network (TCNN) framework. The TCNN framework is made up of an online CNN based on LeNet-5 and several offline CNNs with a shallow structure. First, time-domain signal data are converted into images that contain abundant fault information and are suitable as the input of CNN. Then, the online CNN is constructed to automatically extract representative features from the converted images and classify faults. Finally, in order to improve the real-time performance of the online CNN, several offline CNNs are also constructed and pretrained on related data sets. By directly transferring the shallow layers of the trained offline CNNs to the online CNN, the online CNN can significantly improve the real-time performance and successfully address the issue of achieving the desired diagnostic accuracy within limited training time. The proposed method is validated on two bearing data sets and one pump data set, respectively. The prediction accuracy of the proposed method using three data sets are 99.88%, 99.13%, and 99.98%, respectively. The experimental results also indicate that the improvement of accuracy is 19.21% for the motor bearing case, 29.82% for the rolling mill bearing case, and 33.26% for the pump case during the early stage of learning. [ABSTRACT FROM AUTHOR]

In this paper, we propose a robust detection and tracking strategy for autonomous aerial refueling of unmanned aerial vehicles. The proposed framework includes two modules: a faster deep-learning-based detector (DLD) and a more accurate reinforcement-learning-based tracker (RLT). In the detection stage, the DLD achieves faster speed by combining the efficient MoblieNet with the you only look once framework. In the tracking stage, RLT is proposed to obtain target’s position accurately and fastly by performing hierarchically positioning and adjusting target bounding box according to the reinforcement learning. The precision of drogue object tracking is 98.7%, which is obviously higher than the other comparison methods. The speed of our network can achieve 15 frames/s on GPU Titan X. The experimental results validate the effectiveness and robustness of the proposed framework. [ABSTRACT FROM AUTHOR]

An enhanced intelligent diagnosis method for rotary equipment is proposed based on multi-sensor data-fusion and an improved deep convolutional neural network (CNN) models. An improved CNN based on LeNet-5 is constructed which can enhance the features of the samples by stacking bottleneck layers without changing the size of the samples. A new conversion approaches are also proposed for converting multi-sensor vibration signals into color images, and it can refine features and enlarge the differences between different types of fault signals by the fused images transformed in red–green–blue (RGB) color space. In the last stage of network learning, visual clustering is realized with t-distributed stochastic neighbor embedding (t-SNE) to evaluate the performance of the network. To verify the effectiveness of the proposed method, examples in practice such as the diagnosis for the wind power test rigs and industrial fan system are provided with the prediction accuracies of 99.89% and 99.77%, respectively. In addition, the efficiency of other comparative baseline approaches such as the deep belief network and support vector machine (SVM) is evaluated. In conclusion, the proposed intelligent diagnosis method based on multi-sensor data-fusion and CNN shows higher prediction accuracy and more obvious visualization clustering effects. [ABSTRACT FROM AUTHOR]

Pedestrian dead reckoning (PDR) is a popular indoor localization method due to its independence of additional infrastructures and the wide availability of smart devices. Step length estimation is a key component of PDR, which has an important influence on the performance of PDR. Existing step length estimation models suffer from various limitations such as requiring knowledge of user’s height, lack of consideration of varying phone carrying ways, and dependence on spatial constraints. To solve these problems, we propose a deep learning-based step length estimation model, which can adapt to different phone carrying ways and does not require individual stature information and spatial constraints. Experimental results show that the proposed method outperforms existing popular step length estimation methods. [ABSTRACT FROM AUTHOR]

Vision-based measurement (VBM) systems are powerful tool to extract quantitative information by acquiring video sequence or static images. When a VBM is applied to the evaluation of nominal properties, such as image characteristics, the term VBM is substituted with vision-based evaluation (VBE) by extending the framework of operation unit to a new concept of evaluation unit (EU) for the image analysis and machine learning phases. To this regard, deep learning (DL) approaches have gained an exponential interest in the research and industrial community, thanks to incredible flexibility toward visual words and the possibility to abandon subjective feature extraction procedures. From such an explosiveness of applications, it emerges the need to conduct studies on the capability of DL strategies to deal with uncertainty contributions, i.e., definitional uncertainty related to the measurand and reference uncertainty that may occur during the calibration process. In order to present a benchmark platform to analyze the effect of the major sources of uncertainties estimated, we use here an atomic force microscopy (AFM) imaging scenario for the evaluation of the effect of nanoparticles exposure on human cells in the laboratory. These studies are nowadays fundamental in toxicity analysis for monitoring the health conditions of workers and for protecting people from atherosclerosis disease. The performance of the proposed VBE-DL system to recognize cell alterations from the AFM images is related to three different sources of uncertainty and a critical analysis of the results achieved is provided. [ABSTRACT FROM AUTHOR]

Cabin pose measurement is one of the key procedures in the assembly and docking process of large cabins, which provides important feedback information for the subsequent docking control system. As the basis of cabin pose measurement, the accuracy and robustness of cabin assembly image segmentation are particularly important. However, traditional image segmentation method based on RGB sensor is extremely susceptible to interference from the external environment, which greatly weakens the recognition effect. In this article, an image segmentation method of cabin assembly scene based on improved red-green-blue-depth (RGB-D) Mask R-CNN is proposed, and its network structure is designed to be able to specifically process four-channel images. The method can accurately extract the corresponding area of the cabin under complex and severe environmental disturbances, with high robustness and generalization capability. Meanwhile, the excellence of deep learning segmentation algorithms with depth channel information input is highlighted. In experiments, improved classic segmentation network U-Net, SegNet, pyramid scene parsing network (PSPNet), and Deeplab-v3 based on RGB-D were constructed as control, and these models were tested and evaluated on the enhanced test sets to verify their segmentation accuracy and robustness performance. Comparing experiments fully demonstrate the superiority of the segmentation network model of RGB-D four-channel input over RGB input. At the same time, vision system using the proposed Mask R-CNN algorithm based on RGB-D has the best cabin segmentation accuracy, robustness, and generalization capability, which has practical significance for industrial applications. [ABSTRACT FROM AUTHOR]

In general, deep learning-based fault diagnosis methods need a large number of training samples, which are often not available in real applications. Aiming at this problem, this article develops a new data augmentation method, i.e., randomized wavelet expansion (RWE), to generate a set of synthesis samples that share similar characteristics with the original sample. The first key point is that the amplitudes of wavelet coefficients at a randomly selected frequency band are enlarged through random expansion. Another key point is that the synthesis samples are processed to have the same mean values and standard deviations as their corresponding original sample. Afterward, the synthesis samples are used as the training dataset to train a deep convolutional neural network (CNN) for implementing the few-shot fault diagnosis of aviation hydraulic pumps. Finally, the performance has been validated through a series of experiments. [ABSTRACT FROM AUTHOR]

Due to the unavailability of large-scale underwater depth image datasets and ill-posed problems, underwater single-image depth prediction is a challenging task. An unambiguous depth prediction for single underwater image is an essential part of applications like underwater robotics, marine engineering, and so on. This article presents an end-to-end underwater generative adversarial network (UW-GAN) for depth estimation from an underwater single image. Initially, a coarse-level depth map is estimated using the underwater coarse-level generative network (UWC-Net). Then, a fine-level depth map is computed using the underwater fine-level network (UWF-Net) which takes input as the concatenation of the estimated coarse-level depth map and the input image. The proposed UWF-Net composes of spatial and channel-wise squeeze and excitation block for fine-level depth estimation. Also, we propose a synthetic underwater image generation approach for large-scale database. The proposed network is tested on real-world and synthetic underwater datasets for its performance analysis. We also perform a complete evaluation of the proposed UW-GAN on underwater images having different color domination, contrast, and lighting conditions. Presented UW-GAN framework is also investigated for underwater single-image enhancement. Extensive result analysis proves the superiority of proposed UW-GAN over the state-of-the-art (SoTA) hand-crafted, and learning-based approaches for underwater single-image depth estimation (USIDE) and enhancement. [ABSTRACT FROM AUTHOR]

Glioblastoma (GBM) tumor is the most common primary brain malignant tumor. The precise identification of GBM tumors is very important for diagnosis and treatment. Hyperspectral imaging is a fast, noncontact, accurate, and safe modern medical detection technology, which is expected to be a new tool of intraoperative diagnosis. In order to make full use of the spectral and spatial information of hyperspectral images (HSIs) to achieve accurate GBM tumor identification, a method based on the fusion of multiple deep models is proposed for in vivo human brain HSI classification. The proposed method includes the following major steps: 1) spectral phasor analysis and data oversampling; 2) 1-D deep neural network (1D-DNN)-based spectral HSI feature extraction and classification; 3) 2-D convolution neural network (2D-CNN)-based spectral–spatial HSI feature extraction and classification; 4) edge-preserving filtering-based classification result fusion and optimization; and 5) fully convolutional network (FCN)-based background segmentation. To verify the capabilities of the proposed method, experiments are performed on two real human brain hyperspectral datasets, including 36 in vivo HSIs captured from 16 different patients. The proposed method can achieve an overall accuracy of 96.69% for four-class classification and overall accuracy of 96.34% for GBM tumor identification. Experimental results demonstrate that the proposed method exhibits competitive classification performance and can generate satisfactory thematic maps of the location of the GBM tumor, which can provide the surgeon with guidance on successful and precise tumor resection. [ABSTRACT FROM AUTHOR]

Eddy current testing (ECT) is an effective technique for evaluating depth of metal surface defects. However, in practice, evaluation primarily relies on the experience of an operator and is often carried out by manual inspection. In this article, we address the challenges of automatic depth evaluation of metal surface defects by virtual of state-of-the-art deep learning (DL) techniques. The main contributions are threefold. First, a highly integrated portable ECT device is developed, taking the advantage of an advanced field-programmable gate array (Zynq-7020 system on chip) and provides fast data acquisition and in-phase/quadrature demodulation. Second, a dataset, termed metal defects of different depths by ECT (MDDECT), is constructed using the ECT device by human operators and made openly available. It contains 48000 scans from 18 defects of different depths and liftoffs. Third, the depth evaluation problem is formulated as a time series classification problem, and various state-of-the-art 1-D residual convolutional neural networks are trained and evaluated on the MDDECT dataset. A 38-layer 1-D ResNeXt achieves an accuracy of 93.58% in discriminating the surface defects in a stainless steel sheet with depths from 0.3 to 2.0 mm in the resolution of 0.1 mm. In addition, the results show that the trained ResNeXt1D-38 model is immune to liftoff signals. [ABSTRACT FROM AUTHOR]

Recent progress on intelligent fault diagnosis (IFD) has greatly depended on deep representation learning and plenty of labeled data. However, machines often operate with various working conditions, or the target task has different distributions with the collected data used for training (the domain shift problem). Besides, the newly collected test data in the target domain are usually unlabeled, leading to unsupervised deep transfer learning (UDTL)-based IFD problem. Although it has achieved huge development, a standard and open source code framework and a comparative study for UDTL-based IFD are not yet established. In this article, we construct a new taxonomy and perform a comprehensive review of UDTL-based IFD according to different tasks. Comparative analysis of some typical methods and datasets reveals some open and essential issues in UDTL-based IFD, which is rarely studied, including transferability of features, the influence of backbones, negative transfer, physical priors, and so on. To emphasize the importance and reproducibility of UDTL-based IFD, the whole test framework will be released to the research community to facilitate future research. In summary, the released framework and comparative study can serve as an extended interface and basic results to carry out new studies on UDTL-based IFD. The code framework is available at https://github.com/ZhaoZhibin/UDTL. [ABSTRACT FROM AUTHOR]

With the continuous development of rail transit fully automatic operation, the urgent need to improve train operation safety makes obstacle detection become the research focus. In this work, a flexible and efficient multiscale one-stage object detector FE-YOLO was proposed for image obstacle detection. The feature extraction network is composed of attention module, downsampling module, residual block, spatial pyramid pooling (SPP) module, and so on. A repeatable bidirectional cross-scale path aggregation module was designed as the core of the feature fusion network. The dataset RT2021 of rail transit obstacles was constructed based on the real scene. The mean of average precision (mAP), detection time, iteration time, parameters, and anti-interference ability were used to compare FE-YOLO with other classic object detectors. The results showed that FE-YOLO has the best comprehensive performance. The mAP can reach 92.57%, and the single-frame detection time on the onboard embedded device is up to 0.0989 s. The ablation experiment verified the effectiveness of each module of FE-YOLO and has the best generalization performance on the PASCAL VOC dataset. Finally, a rail transit obstacle detection system was developed, and multiple sensors were used to improve the detection accuracy. Experiments showed that the detection system can work normally in different environments. [ABSTRACT FROM AUTHOR]

This article proposes an automatic approach for segmenting inclusion defects in composite materials inspected by lock-in thermography (LT) in a heat source invariant way. In the proposed pipeline, temperature maps are preliminarily processed to enhance the contrast between sound and defective areas, reduce the acquisition noise, and mitigate the effect of nonuniform heat excitation. Then, the enhanced sequences are grouped in highly local sets, the input of a convolution residual neural network, which segments the defects focusing on the temporal evolution of the temperature. Experiments have been run on a dataset acquired from a CFRP specimen tested under five different heat source configurations. Here, a brief study of signal variations associated with the heat source configuration is presented. Then, an ablation study has been presented to compare different architectures of increasing complexity. The best results are achieved by using the proposed preliminary processing, the residual layers, and an input set of temperature signals of size $3 \times 3$. This network can reach a global intersection over union (IOU) of 0.78, while defects are segmented with an IOU value of 0.58. In addition, the balanced segmentation accuracy of the test reaches 94.78%, with a maximum recall of the defective class of 91.41%. Further cross validation changing the source configurations of the training and test sets produces comparable results of IOU and balanced accuracy, which are on average equal to 0.71 and 84.02%, respectively. A final evaluation of the trained network is performed on a new dataset augmented through a vertical stretch. Results are still comparable with an average IOU and balanced accuracy of 0.74 and 87.15%, respectively. These results prove the capability of the method to segment inclusion defects of any shape without being affected by the configuration of the heating sources used in the LT inspections. [ABSTRACT FROM AUTHOR]

Advanced image fusion frameworks have focused on constructing a unified system to deliver general fusion solutions for all the potential subtasks, including visible and infrared image fusion, multi-focus image fusion, and multi-exposure image fusion. However, the fusion performance of these unified solutions cannot be guaranteed as the involved subtasks exhibit essential diversity. Besides, the huge volume of these models also sacrifices their flexibility in practical applications. To this end, we propose a lightweight unified fusion network to balance the multi-level information featured across different channels and different layers. In particular, We construct a novel network architecture based on the Ghost module endowed with a new loss function. The depth of the designed network is increased, while the number of parameters is reduced by an order of magnitude compared to existing learning-based fusion approaches. In principle, our network belongs to the autoencoder paradigm, comprising three parts: encoder, fusion layer, and decoder, respectively. Considering that the efficacy of existing autoencoder-based approaches is impeded by the single fusion strategy, we use the guided filtering approach to decompose the source image into a base layer and a detail layer to extend the diversity of the inputs. Drawing on this, we can design different fusion strategies in these two layers for adapting different image fusion tasks. In the fusion layer, besides the three basic fusion strategies, i.e., average, max, and spatial attention, an additional gradient perception strategy is proposed to handle the multi-focus image fusion problem, improving the corresponding fusion performance. Qualitative and quantitative experiments show that our UNIFusion achieves promising results in these image fusion tasks, compared to the state of the art. [ABSTRACT FROM AUTHOR]

In the recent decade, high-throughput plant phenotyping techniques, which combine noninvasive image analysis and machine learning, have been successfully applied to identify and quantify plant health and diseases. However, these techniques usually do not consider the progressive nature of plant stress and often require images showing severe signs of stress to ensure high confidence detection, thereby reducing the feasibility for early detection and recovery of plants under stress. To overcome the problem mentioned above, we propose a deep learning pipeline for the temporal analysis of the visual changes induced in the plant due to stress and apply it to the specific water stress identification case in Chickpea plant shoot images. For this, we have considered an image dataset of two chickpea varieties JG-62 and Pusa-372, under three water stress conditions: control, young seedling, and before flowering, captured over five months. We have employed a variant of convolutional neural network-long short-term memory (CNN-LSTM) network to learn spatiotemporal patterns from the chickpea plant dataset and use them for water stress classification. Our model has achieved ceiling level classification performance of 98.52% on JG-62 and 97.78% on Pusa-372 chickpea plant data and has outperformed the best reported time-invariant technique by at least 14% for both JG-62 and Pusa-372 species to the best of our knowledge. Furthermore, our CNN-LSTM model has demonstrated robustness to noisy input, with a less than 2.5% dip in average model accuracy and a small standard deviation about the mean for both species. Finally, we have performed an ablation study to analyze the performance of the CNN-LSTM model by decreasing the number of temporal session data used for training. [ABSTRACT FROM AUTHOR]

Together with the rapid development of the sensors technology in recent years, sensor-based human activity recognition (HAR) has shown promising performance using well-known supervised deep learning methods. However, it remains challenging in a realistic scenario, i.e., limited number of labeled samples and sensors. This article proposes a novel deep learning method to achieve accurate and robust HAR with only a single inertial measurement unit (IMU) sensor. Our contributions are twofold. First, based on the skinned multiperson linear (SMPL) model, we build a large synthetic HAR dataset containing multimodal measurements: acceleration and angular velocity, which were generated according to the forward kinematics. Second, We propose a multiple-level domain adaptive learning model with information-theoretically stimulated constraints to simultaneously align the distributions of low- and high-level representations of virtual and real HAR data. The proposed mutual information constraints encourage the neural network to learn a disentangled representation for the multimodal sensing data. Comprehensive experimental results on three publicly available datasets demonstrate that the proposed method compares favorably with competing ones and has robust performance with variable labeled samples. [ABSTRACT FROM AUTHOR]

Remote photoplethysmography (rPPG) enables noncontact heart rate (HR) estimation using facial videos and exhibits significant convenience over traditional contact-based HR measurement approaches. However, its effectiveness depends mainly on the number of pixels of the facial region; thus, the estimation accuracy is easily degraded due to missing rPPG information when the resolution of facial images is low. Given that, in most real application scenarios, such as group-oriented and long-distance vital signs measurement, only low-resolution facial videos are available, finding a solution to recover rPPG information is a key to improve the performance of noncontact HR measurement. In this article, we propose a two-stage deep learning scheme to achieve accurate HR measurement from low-resolution facial videos. In the first stage, an rPPG information recovery network is proposed to recover the rPPG information during video super-resolution (SR) processing. The up-scaled images are then fed into a temporally aware HR measurement network in the second stage. An attention mechanism that reassigns the weights of the temporal information in each feature channel is designed to improve the measurement accuracy. The mean absolute percentage error achieves 3.320 and 4.231, while Pearson’s correlation coefficient reaches 0.930 and 0.892 on two datasets, respectively. Experimental results show that rPPG information could be effectively recovered through SR processing. Compared with the state-of-the-art approaches that treat high-resolution images as the input, the proposed method achieved closely comparable measurement performance using merely low-resolution images. [ABSTRACT FROM AUTHOR]

Nowadays, the usage of deep learning-based approaches for salient object detection (SOD) is increasing exponentially to detect and localize visually distinct regions in static images. However, the variability in scales of salient objects requires further attention given the abstract nature of the multilayer feature hierarchy of convolution neural networks (CNNs). First, feature maps of different layers in CNNs embed abstract information about objects that changes with the object’s scale. Second, the progressive feature fusion in models, such as the feature pyramid network, loses its effectiveness in detecting sharp boundaries due to the late fusion of detailed features. This work proposes two modules namely, adjacent layer attention block and partial encoder–decoder (PED) block to handle the aforementioned issues. The proposed adjacent layer attention block facilitates communication among the layers of closest abstraction to mine abundant scale features at the current resolution. The resultant integrated feature at a resolution contains detailed and semantic information from interaction among the adjacent layers useful to extract scale information of complex objects. A PED module utilizes the resolution-specific integrated features from the adjacent layer attention block of its corresponding encoder to generate multiscale features, and fuse them in a top-down manner. This level-specific distribution of aggregated features within a PED helps coarser layers of the network to acquire boundary information. Experimental results on five broadly used SOD datasets are compared with recent 20 state-of-the-art SOD models. The proposed method performs favorably against its competitors without any preprocessing or postprocessing. [ABSTRACT FROM AUTHOR]

As an image sensing instrument, light field images can supply extra angular information compared with monocular images and have facilitated a wide range of measurement applications. Light field image capturing devices usually suffer from the inherent tradeoff between the angular and spatial resolutions. To tackle this problem, several methods, such as light field reconstruction and light field super-resolution, have been proposed but leaving two problems unaddressed, namely domain asymmetry and efficient information flow. In this article, we propose an end-to-end spatio-angular dense network (SADenseNet) for light field reconstruction with two novel components, namely correlation blocks and spatio-angular dense skip connections to address them. The former performs effective modeling of the correlation information in a way that conforms with the domain asymmetry. Also, the latter consists of three kinds of connections enhancing the information flow within two domains. Extensive experiments on both real-world and synthetic datasets have been conducted to demonstrate that the proposed SADenseNet’s state-of-the-art performance at significantly reduced costs in memory and computation. The qualitative results show that the reconstructed light field images are sharp with correct details and can serve as preprocessing to improve the accuracy of related measurement applications. [ABSTRACT FROM AUTHOR]

Deep-learning algorithms have produced promising results, however, domain adaptation remains a challenge. In addition, excessive training time and computing resource requirements need to be addressed. Deep-learning algorithms face a domain adaptation issue when the data distribution of a target domain differs from that of the source domain. The emerging concept of broad learning shows potential in addressing the domain adaptation and training time issues. An adaptive unsupervised broad transfer learning (AUBTL) algorithm is proposed to tackle the cross-domain problems. The proposed algorithm utilizes a sparse autoencoder and random orthogonal mapping to extract and augment the feature space. Then, it initializes the weights of a classifier by solving a ridge regression problem. The logit ranking strategy is applied to develop a transfer estimator to evaluate and sample data in the target domain for an adaptive transfer. Based on the sampled data, AUBTL optimizes the hyperparameter space. The performance of the AUBTL algorithm is validated with three benchmark datasets, including 20 transfer tasks. The computational results demonstrated the efficiency and accuracy of the proposed algorithm over other deep-learning algorithms considered in this research. [ABSTRACT FROM AUTHOR]

Brace sleeve (BS) fasteners, i.e., nut and bolt, are small components but play essential roles in fixing BS and cantilever in railway catenary system. They are commonly inspected by onboard cameras using computer vision to ensure the safety of railway operation. However, most BS fasteners cannot be directly localized because they are too small in the inspection images. Instead, the BS is first localized for detecting the BS fastener. This leads to a new problem that the localized BS boxes may not contain the complete BS fasteners due to low localization accuracy, making it infeasible to further diagnose the fastener conditions. To tackle this problem, this article proposes a novel pipeline for BS fastener looseness diagnosis. First, the competitive deep learning model Faster RCNN ResNet101 is used to coarsely localize BSs. Second, an action-driven reinforcement learning agent is adopted to refine the coarse-localized boxes through a dynamic position searching process. Then, BS fasteners are extracted from the refined localized BS image by the deep segmentation model YOLACT++, which is fast and interpretable. Finally, a looseness diagnosis criterion based on segmented information are proposed. We evaluate the performance of submodels independently and the overall performance of the whole model on a real-life catenary image dataset collected from a high-speed line in China. The test results show that the proposed method is effective for BS looseness detection in railway catenary. [ABSTRACT FROM AUTHOR]

Comparing with X-ray, ultrasound imaging has the advantages of no radiation and easy operation, etc., which is expected to be an effective alternative in clinical assessment for the human spine. At present, freehand scanning is mainly used in ultrasound imaging field, and recently robot-assisted scanning has emerged as a promising way to gradually reduce human intervention. However, the automatic recognition and scanning, as well as the control strategy of robots that guarantee the tight coupling between a probe and human skin, are still challenges. Moreover, the current measurement method of the spine is usually conducted based on plain radiographs, which lacks spatial structure information of spine. In this article, a new radiation-free robotic ultrasound system for 3-D imaging and a measurement method of the spine based on 3-D imaging are presented. A fully convolutional network (FCN) named fuse-Unet based on RGB and depth images captured by an RGB-depth (RGB-D) sensor Kinect, is proposed to realize the automatic recognition of human spine area and scanning path pre-planning. Then a six-degree-of-freedom robotic arm plays the role of a doctor to complete the automatic scanning along the pre-planning path, during which a normal-vector-based method and two force sensors are used to ensure that the probe well fits the spine area. Finally, 3-D ultrasound reconstruction and visualization of the spine are realized, based on which cobb angles are calculated for assessing the morphological structure of the spine. Phantom and in vivo experiments are conducted for evaluating the performance of the proposed system. Experimental results validate the practicability and accuracy of the system, indicating its potential for clinical application. [ABSTRACT FROM AUTHOR]

Real-time identification of threats has been the focus of attention in the security field in recent years. Researchers have aimed to build a fast, efficient, and accurate model to address risks in threat event identification. This study proposes a fast, effective deep learning model—the squeeze and excitation WaveNet (SE-WaveNet). WaveNet is a 1-D convolutional neural network (CNN) model with a large receptive field. It is small, fast, and highly accurate. Thus, it has a great advantage in processing time series data. On this basis, the squeeze and excitation (SE) structure of the attention mechanism is introduced to adapt to matrix data with complete information. To ensure the integrity of information in the experimental verification, the distributed optical fiber vibration sensing (DVS) system is used to collect the data near the vibration point and form the time–space matrix as the input of the model. SE-WaveNet is compared with WaveNet and related models. Results show that the accuracy of SE-WaveNet based on WaveNet is increased by approximately 3.5%, the model size is reduced by 39%, and the speed is increased by 32%. Compared with the five other related models, the average performance is 3%–9% higher, and the final test accuracy rate can reach approximately 97.73%. [ABSTRACT FROM AUTHOR]

Gesture recognition as a natural, convenient and recognizable way has been received more and more attention on human–machine interaction (HMI) recently. However, visual-based gesture recognition methods are often restricted by environments and classical wearable device-based strategies are suffered from relatively low accuracy or the complicated structures. In this study, we first design a low-cost and efficient data glove with simple hardware structure to capture finger movement and bending simultaneously. Second, a novel dynamic hand gesture recognition algorithm (DGDL-GR) is proposed to recognize human dynamic sign language, in which a fusion model of convolutional neural network (fCNN) and generic temporal convolutional network (TCN) is fully utilized. The fCNN (fusion of 1-D CNN and 2-D CNN) is proposed to extract time-domain features of finger resistance movement and spatial domain features of finger resistance bending simultaneously. Moreover, due to the superiorities of TCN in sequence modeling task, this work proposes a novel hand gesture recognition method based on the TCN, which includes causal convolution, dilation convolution, and a residual network with appropriate layers. Both long- and short-time dependencies of the hand gesture features are deeply mined and classified in the end. Results of extensive experiments have demonstrated that the proposed DGDL-GR algorithm outperforms many state-of-the-art algorithms on the measure of accuracy, F1 score, precision score, and recall score with the real-world dataset. Moreover, the number of residual blocks and some key hyperparameters of the proposed DGDL-GR algorithm has been studied thoroughly in this work. [ABSTRACT FROM AUTHOR]

Intelligent fault diagnosis is essential for downtime reduction in modern industries. However, domain (i.e., working condition) variants are unavoidable for fault diagnosis under changing environment. Though one can obtain a customized deep learning model for a given domain, it is not practical to design deep neural networks manually for multiple domains. Differentiable architecture search (DARTS), as an automatic machine learning technique, can automate the network design process for a specific domain efficiently by using hypernetwork and differentiable search strategy. Nevertheless, the representation of hypernetwork is restricted by several unfair factors, and the searched architecture of DARTS is overconfident. To address these issues, a one-shot neural architecture search approach, which involves two-stage learning, is proposed for efficient domain matching fault diagnosis. In the first training stage, the warmup and path-dropout strategies are taken to enhance the competitiveness of the parametric operators and alleviate the coadaptation problem to obtain an intuitively fair hypernetwork. In the second matching stage, variational inference is introduced into a differentiable search strategy to estimate the uncertainty of model matching, and a scale mixture prior is used to softly constrain the matching stage. Then, a candidate architectures set can be sampled and ordered from the posterior. Multidomain experiment is implemented by adding noise to the raw signal, and the proposed method outperforms four commonly used deep neural networks for aeroengine bevel gear fault diagnosis. [ABSTRACT FROM AUTHOR]

In this article, we propose an infrared and visible image fusion network based on the salient target detection, termed STDFusionNet, which can preserve the thermal targets in infrared images and the texture structures in visible images. First, a salient target mask is dedicated to annotating regions of the infrared image that humans or machines pay more attention to, so as to provide spatial guidance for the integration of different information. Second, we combine this salient target mask to design a specific loss function to guide the extraction and reconstruction of features. Specifically, the feature extraction network can selectively extract salient target features from infrared images and background texture features from visible images, while the feature reconstruction network can effectively fuse these features and reconstruct the desired results. It is worth noting that the salient target mask is only required in the training phase, which enables the proposed STDFusionNet to be an end-to-end model. In other words, our STDFusionNet can fulfill salient target detection and key information fusion in an implicit manner. Extensive qualitative and quantitative experiments demonstrate the superiority of our fusion algorithm over the current state of the arts, where our algorithm is much faster and the fusion results look like high-quality visible images with clear highlighted infrared targets. Moreover, the experimental results on the public datasets reveal that our algorithm can improve the entropy (EN), mutual information (MI), visual information fidelity (VIF), and spatial frequency (SF) metrics with about 1.25%, 22.65%, 4.3%, and 0.89% gains, respectively. Our code is publicly available at https://github.com/jiayi-ma/STDFusionNet. [ABSTRACT FROM AUTHOR]

Electrocardiogram (ECG) is a tool to help judge heart activity. In recent years, the convolutional neural network (CNN) and various deep learning algorithms have been widely used in ECG diagnosis. CNN only considers the local feature. However, the ECG signal is susceptible to noise, and the waveform is complex, making it difficult for existing methods to get a good result. This article presents a novel neural network architecture for ECG diagnosis based on deformable CNN (Deform-CNN). The architecture makes good use of the feature-learning capability of deformable convolution to learn the time-domain and lead characteristics of multilead ECG signals. The proposed end-to-end method can achieve an overall diagnostic accuracy of 86.3% in the 12-lead ECG data of CPSC-2018, with good antinoise ability, which makes the method have a more competitive performance than other deep learning algorithms. The source code is publicly available at https://github.com/HeartbeatAI/Deform-CNN . [ABSTRACT FROM AUTHOR]

Remaining useful life (RUL) prediction methods for rotating machines have been successfully developed in recent decades. More attention should be paid to predictions with inconsistent data distributions under different conditions. To solve this problem, this article proposes a new RUL prediction method that includes two phases. In the first phase, degradation features are extracted from both the training and testing data sets using probabilistic principal component analysis (PPCA). In the second phase, additive white Gaussian noise (AWGN) is intentionally injected into the degradation features; thereafter, the features that are mixed with manually injected noise are imported into a bidirectional long short-term memory (BLSTM) network. The AWGN can enhance the robustness of the RUL prediction method and achieve prediction for machines under different conditions. In contrast to most deep learning-based RUL prediction methods, the training samples are intentionally “polluted” by manually injected noise. The effectiveness of the proposed method is validated using the C-MAPSS lifetime data set for aeroengines and compared with the effectiveness of state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This article proposes a new loss function termed arccosine center loss, which can learn interclass and intraclass information simultaneously, to improve the discriminative ability of convolutional neural networks for finger vein verification. Specifically, the purpose of arccosine center loss is to reduce intraclass distance and increase the interclass distance through network model training. With the combination of softmax loss and arccosine center loss, the proposed network model can extract features with the interclass dispensation and intraclass compactness, which improves the discriminative ability of learned features. Experimental studies have confirmed the effectiveness and efficiency of the proposed method for finger vein verification. [ABSTRACT FROM AUTHOR]

Detection of navigable regions for the unmanned surface vehicles (USVs) sailing on the narrow rivers is very important. Existing detection methods mostly depend on cameras, which is sensitive to the environment and cannot provide reliable navigable regions for sailing. In this article, we propose a scheme to directly process 3-D LiDAR data to achieve accurate and stable navigable regions’ detection. In our scheme, a deep learning-based method is used to semantically segment the river objects, and then, a Kalman Filter (KF)-based multiobjects tracking method is deployed to accurately track the bank objects. Finally, the navigable regions are modeled by a wave frontier detection (WFD)-based method. In order to validate the performance of the proposed detection scheme, field experiments are carried out in a narrow and complicated river to prove the performance of the proposed scheme, and comparison experiments and ablation studies are conducted to show the impacts of deep segmentation and KF-based tracking methods. Experimental results show that our scheme outperforms other methods in accuracy up to 10% and smoothness up to 47%. [ABSTRACT FROM AUTHOR]

In this article, we propose a novel decomposition method by applying disentangled representation for visible and infrared image fusion (DRF). According to the imaging principle, we perform the decomposition depending on the source of information in the visible and infrared images. More concretely, we disentangle the images into the scene- and sensor modality (attribute)-related representations through the corresponding encoders, respectively. In this way, the unique information defined by the attribute-related representation is closer to the information captured by each type of sensor individually. Thus, the problem of inappropriate extraction of unique information can be alleviated. Then, different strategies are applied for the fusion of these different types of representations. Finally, the fused representations are fed into the pretrained generator to generate the fusion result. The qualitative and quantitative experiments on the publicly available TNO and RoadScene data sets demonstrate the comparable performance of our DRF over the state of the art in terms of both visual effect and quantitative metrics. [ABSTRACT FROM AUTHOR]

Playing a vital role in the safe operation of equipment, intelligent fault detection has been extensively applied in mechanical systems. However, the complicated mapping of signal symptoms and multicomponent failure modes under sharp speed variation makes it hard. The classification error is large due to insufficient samples in the fault condition. Aiming at these problems, a novel multitask redundant lifting adversarial network (MRLAN) is proposed. First, inspired by redundant second-generation wavelet transform, a reconstruction network is established to generate high-quality time–frequency representation of vibration signal, thus eliminating speed fluctuations and expanding training samples. The reconstruction network consists of transposed convolution layer, resume update layer, resume predict layer, and merge layer. Then, multitask category discriminator, which forms dual discriminator together with distance discriminator, is constructed to separate the features of multiple components, thereby realizing multiple tasks (multicomponent fault location and single-component health assessment). Finally, the model is alternately optimized through adversarial process between the generation network and the double discriminators. The proposed method is validated by two cases and other methods are also established to show its superiority. The results show satisfactory performance of MRLAN in multicomponent fault diagnosis under sharp speed fluctuation and little training data. [ABSTRACT FROM AUTHOR]

The work state of catenary support components (CSCs) is essential for trains’ powering safety under high-speed operation. However, because of the difficulty of positioning the CSCs whose sizes are much different, some detection methods based on classical deep learning technologies only position one specific component, and the others that can position all categories CSCs have poor performance. A new position method for 12 categories of CSCs called CSC structure inference net (CSCSIN) is proposed in this article. This method is composed of three sections, including the feature extraction with a multilevel feature, feature inference with structure prior information, regression, and classification. In the feature extraction section, to extract small-scale CSCs’ features, a modified VGG16, which is used to generate a legible feature map based on the fusion of low and high layers’ features, is proposed, and multiscales anchors for RoIs prediction are applied to suit multiscale CSCs. In the feature inference section, considering application scenarios and fixed position relationship between CSCs, the scene visual feature and position information between targets are used to iteratively update each target state in structure inference net, and the final state of each target is used for its classification in regression and classification section. Based on the modification in the feature extraction section and the combination of relevant information of CSCs, CSCSIN is efficient and sensitive to all 12 CSCs whose scales are much different. The experimental results demonstrate that the position precision of CSCSIN is superior to other classical models and relevant methods. [ABSTRACT FROM AUTHOR]

The input data of photometric stereo have three dimensions: one photometric dimension of different lights and two spatial dimensions, i.e., rows and columns in the image coordinate. Recent deep-learning-based photometric stereo algorithms usually use 2-D/3-D convolutions to process the input with three dimensions. Assumptions that violate the natural characters of photometric stereo problem, e.g., spatial pixel interdependence and light permutation invariance, have to be made due to the dimension mismatch. In this article, we propose a self-attention photometric stereo network (SPS-Net), which can exploit the information in all three dimensions without violating these natural characters. In SPS-Net, the spatial information is extracted by convolutional layers and the photometric information is aggregated by the proposed photometric fusion blocks based on the self-attention mechanism. Extensive experiments on both synthetic and real-world data sets are conducted. The proposed SPS-Net achieved higher performance than the state-of-the-art algorithms photometric stereo task with dense lightings. Without any changes, the proposed algorithm also outperformed the benchmarks in sparse and light-information-robust photometric stereo tasks. [ABSTRACT FROM AUTHOR]

Visible images contain rich texture information, whereas infrared images have significant contrast. It is advantageous to combine these two kinds of information into a single image so that it not only has good contrast but also contains rich texture details. In general, previous fusion methods cannot achieve this goal well, where the fused results are inclined to either a visible or an infrared image. To address this challenge, a new fusion framework called generative adversarial network with multiclassification constraints (GANMcC) is proposed, which transforms image fusion into a multidistribution simultaneous estimation problem to fuse infrared and visible images in a more reasonable way. We adopt a generative adversarial network with multiclassification to estimate the distributions of visible light and infrared domains at the same time, in which the game of multiclassification discrimination will make the fused result to have these two distributions in a more balanced manner, so as to have significant contrast and rich texture details. In addition, we design a specific content loss to constrain the generator, which introduces the idea of main and auxiliary into the extraction of gradient and intensity information, which will enable the generator to extract more sufficient information from source images in a complementary manner. Extensive experiments demonstrate the advantages of our GANMcC over the state-of-the-art methods in terms of both qualitative effect and quantitative metric. Moreover, our method can achieve good fused results even the visible image is overexposed. Our code is publicly available at https://github.com/jiayi-ma/GANMcC. [ABSTRACT FROM AUTHOR]

Active learning can reduce the human effort required for labeling training samples while preserving the performance of visual classifiers. However, existing active learning frameworks cannot be used to perform visual classification of industrial product surface defects because they still require intensive manual annotation efforts. In this study, we propose a cost-efficient autolabeling-enhanced active learning (ALEAL) framework to reduce the human annotation effort required for surface defect visual classification. The proposed ALEAL framework employs a deep convolutional neural network (CNN) as a visual classifier trained from an initial set of human-labeled training samples. Then, the collected unlabeled training samples are input into the classifier for category confidence estimation. Next, a novel diverse cost-effective query strategy (DCEQS) is proposed to select some high-confidence samples for autolabeling and some informative samples for sample proposals that need labeling. Subsequently, to further reduce the human annotation effort, a novel autolabeling module is proposed and introduced in ALEAL that can automatically label a portion of the informative unlabeled training samples selected by the DCEQS. In this study, a novel attention-based similarity measurement network (ASMN) is proposed as an implementation of this autolabeling module by measuring the similarity between unlabeled and labeled samples. Finally, the remaining unlabeled samples are annotated by human experts, and all the newly labeled samples are used to retrain the classifier. Through the autolabeling process from the DCEQS and ASMN, ALEAL can automatically label additional training samples and achieve a competitive performance while requiring few human-labeled training samples, which is highly important in industrial applications. Extensive experimental results show that, compared with popular active learning methods, ALEAL can dramatically reduce the effort involved in human annotation and achieve state-of-the-art cost efficiency for the visual classification of industrial product surface defects. [ABSTRACT FROM AUTHOR]

Automatic surface defect inspection (ASDI) is a crucial and challenging problem in industry because it affects the quality and efficiency of production greatly. Deep learning-based methods achieve promising improvements for surface detection, but they rely on a massive training data set that is impractical in industry. In this study, we propose a generic method that works effectively even on the small size training data set. Especially, we introduce a ResMask generative adversarial network (GAN) framework that is a residual GAN to expand the insufficient defect data sets. Meanwhile, the existing inspection data sets are so much easier to detect than the data sets in the real industrial scenarios that new industrial surface defect data sets containing more diverse and challenging images are established. Then, a coarse-to-fine module (CFM) that consists of a coarse detection subnetwork and a fine segmentation subnetwork is proposed for the needs of fast detection for high-resolution images. In the coarse detection stage, spatial pyramid pooling (SPP) is utilized to increase the receptive field of the network, reduce the false detection rate, and determine the approximate location of defects. In the fine segmentation stage, the receptive field of the network is enlarged by atrous SPP (ASPP), and skip links that incorporate low-level with high-level features achieve pixelwise precision on defect segmentation. Finally, our algorithm has achieved state-of-the-art results in DAGM, HR, and WB data sets (0.859, 0.761, and 0.805, respectively) according to MIoU. It achieves an average processing time of 44.4 ms on the test images with a resolution of $512 \,\,\times \,\, 512$ and 131.6 ms for $2048 \times 2048$ images. On the DAGM data set, the detection accuracy of mean intersection over union (MIoU) reaches 0.859. [ABSTRACT FROM AUTHOR]