Search

In order to accurately estimate the period of X-ray pulsars to improve navigation performance, this paper proposes a pulsar period estimation method based on photon energy using reverse Hilbert coding and double Convolutional Neural Networks (CNNs) discrimination. This method uses photon energy information to perform epoch folding. It converts one-dimensional contour information into two-dimensional image signals through reverse Hilbert curve and recognizes contour image features through CNN models to determine the optimal period. According to the simulation results, the period estimation accuracy of this method is 0.5350 ns, which is 65.71% higher than the χ 2 epoch folding method. The contour phase accuracy is 0.0004767, which is 41.93% higher than the χ 2 epoch folding method. At the same time, the observation time and noise interference that may affect the accuracy of period estimation are also quantitatively analyzed. In addition, in order to simulate real navigation scenarios, this paper also conducts dynamic period estimation experiments on PSR B0531+21 pulsar in a specific period of time. The method proposed in this paper has the advantages of high precision, fast calculation speed, strong anti-interference ability and accurate dynamic period estimation, which can significantly improve the navigation performance of X-ray pulsars. • The proposed period estimation method uses photon energy information for contour weighted folding. • The proposed method uses reverse Hilbert curve to convert one-dimensional contour information into two-dimensional image information. • Convolutional Neural Network models are used to optimize pulsar period estimation. [Display omitted] [ABSTRACT FROM AUTHOR]

Autonomous spacecraft relative navigation via monocular images became a hot topic in the past few years and, recently, received a further push thanks to the constantly growing field of artificial neural networks and the publication of several spaceborne image datasets. Despite the proliferation of spacecraft relative-state initialization algorithms developed, most architectures adopt computationally expensive solutions relying on convolutional neural networks (CNNs) that provide accurate output at the cost of a high computational burden that seems unfeasible for current spaceborne hardware. The paper addresses this issue by proposing a novel pose initialization algorithm based on lightweight CNNs. Inspired by previous state-of-the-art algorithms, the developed architecture leverages a fast and accurate target detection CNN followed by a line segment detection CNN capable of running with low inference time on mobile devices. The line segments and their junctions are grouped into complex geometrical groups, reducing the solution search space, and subsequently, they are adopted to extract the final pose estimate. As a main outcome, the analyses demonstrate that the lightweight architecture developed scores high accuracy in the pose estimation task, with a mean estimation error of less than 10 cm in translation and 2.5°in rotation. The baseline algorithm scores a mean SLAB error of 0.04552 with a standard deviation of 0.22972 in the test dataset. Detailed analyses demonstrate that the uncertainties on the overall pose score are driven mainly by errors in the relative attitude, which gives the highest contribution to the pose error metric adopted. The analyses on the error distributions point out that the uncertainties on the estimated relative position are higher in the camera boresight axis direction. Concerning the relative attitude, the algorithm proposed has higher uncertainties in estimating directions of the target x and y axes due to ambiguities related to the target geometry. Notably, the target detection CNN trained in this work outperforms the previous top scores in the benchmark dataset. The performances of the proposed algorithm have been investigated further by analyzing the effects on the accuracy due to the relative distance and the presence of background in the images. Lastly, the paper delves into the possibility of adopting a sub-portion of the 2D-to-3D match matrix made by the most complex perceptual groups identified that positively affects the overall run-time, pointing out the performances in terms of accuracy of the estimates and providing a comparison of both the baseline and the reduced match matrix versions against state-of-the-art algorithms concerning relative position and attitude errors and solution availability, highlighting the high accuracy and solution availability of the proposed architectures. • New relative pose initialization algorithm based on the detection of line segments. • Detailed literature review and description of available spaceborne image datasets. • Improvement of state of the art in spacecraft detection using lightweight CNN. • Assessment of accuracy of the proposed architecture and error sources analyses. • Proven feasibility of the approach via comparison with other architectures. [ABSTRACT FROM AUTHOR]

• AI improves infectious waste management. • Ensemble model exceeds 97.81% accuracy. • Reinforcement learning optimizes classification. • Geometric augmentation boosts model robustness. • Study addresses urgent urban waste challenges. This research paper focuses on effective infectious municipal waste management in urban settings, highlighting a dearth of dedicated research in this domain. Unlike general or specific waste types, infectious waste poses distinct health and environmental risks. Leveraging advanced artificial intelligence techniques, we prioritize infectious waste categorization and optimization, integrating metaheuristics into optimization methods to create a robust dual-ensemble framework. Our model, the "Enhanced Artificial Intelligence for Infectious Municipal Waste Classification System," combines ensemble image segmentation methods and diverse convolutional neural network models. Innovative geometric image augmentation enhances model robustness, diversifies training data, and improves accuracy across waste types. A pivotal aspect is the integration of a reinforcement learning-differential evolution algorithm as a decision fusion strategy, optimizing classification by harmonizing outputs from ensemble methods and convolutional neural network models. Computational results, using a newly collected dataset, demonstrate our model's accuracy exceeding 96.54% while using the existing solid waste dataset we achieve the accuracy of 97.81%, outperforming advanced approaches by margins ranging from 2.02% to 8.80%. This research significantly advances sustainable waste management, showcasing artificial intelligence's transformative potential in addressing intricate waste challenges. It establishes a foundational framework prioritizing efficiency, effectiveness, and sustainability for future waste management solutions. Acknowledging the importance of diverse datasets, customization for urban contexts, and practical integration into existing infrastructures, our work contributes to the broader discourse on the role of artificial intelligence in evolving waste management practices. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved outstanding performance in computer vision tasks. Convolutional Neural Networks (CNNs) typically operate in the spatial domain with raw images, but in practice, images are usually stored and transmitted in their compressed representation where JPEG is one of the most widely used encoder. Also, these networks are computationally intensive and slow. This paper proposes performing the learning and inference processes in the compressed domain in order to reduce the computational complexity and improve the speed of popular CNNs. For this purpose, a novel graph-based frequency channel selection method is proposed to identify and select the most important frequency channels. The computational complexity is reduced by retaining the important frequency components and discarding the insignificant ones as well as eliminating the unnecessary layers of the network. Experimental results show that the modified ResNet-50 operating in the compressed domain is up to 70% faster than the spatial-based traditional ResNet-50 while resulting in similar classification accuracy. Moreover, this paper proposes a preprocessing step with partial encoding to improve the resilience to distortions caused by low-quality encoded images. Finally, we show that training a network with highly compressed data can achieve a good classification accuracy with up to 93% reduction in the storage requirements of the training data. [ABSTRACT FROM AUTHOR]

Neurons are the fundamental units of neural networks. In this paper, we propose a method for explaining neural networks by visualizing the learning process of neurons. For a trained neural network, the proposed method obtains the features learned by each neuron and displays the features in a human-understandable form. The features learned by different neurons are combined to analyze the working mechanism of different neural network models. The method is applicable to neural networks without requiring any changes to the architectures of the models. In this study, we apply the proposed method to both Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) trained using the backpropagation learning algorithm. We conduct experiments on models for image classification tasks to demonstrate the effectiveness of the method. Through these experiments, we gain insights into the working mechanisms of various neural network architectures and evaluate neural network interpretability from diverse perspectives. [ABSTRACT FROM AUTHOR]

Generative Adversarial Networks (GANs) are well-known tools for data generation and semi-supervised classification. GANs, with less labeled data, outperform Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) in classification tasks. The success of GANs in classification tasks motivates the development of GAN-based techniques for semi-supervised regression tasks. However, developing GANs for regression introduces two major challenges: (1) inherent instability in the GAN formulation and (2) performing regression and achieving stability simultaneously. This paper introduces techniques that show improvement in the GANs' regression capability. We bake a differentiable fuzzy logic system at multiple locations in a GAN. The fuzzy logic takes the output of either the generator or the discriminator to predict the output, y , and evaluate the generator's performance. We outline the results of applying the fuzzy logic system across multiple GANs and summarize each approach's efficacy. This paper shows that adding a fuzzy logic layer can enhance GAN's ability to perform regression; the most desirable injection location is problem-specific, and we show this through experiments over various datasets. Besides, we demonstrate empirically that the fuzzy-infused GANs are competitive with the DNNs. [ABSTRACT FROM AUTHOR]

To capture the real-world luminance values, High Dynamic Range (HDR) image processing has been developed. HDR images have a richer content than the widely-used Standard Dynamic Range (SDR) images, and are used in a number of situations, e.g. in film industry. As HDR displays are more and more commercially available, we need to be able to process HDR images as well as SDR ones (for example, devising denoising algorithms, inpainting or anti-aliasing). The most powerful methods to process images are now deep neural networks. However, the training of such networks calls for a lot of images, and HDR images datasets are relatively small. One way to generate HDR images is inverse tone mapping operators (iTMOs). They are algorithms that expand the dynamic range of SDR images. In this paper, we propose HDR-LFNet, a novel iTMO, and its HDR training dataset. Our method merges several existing handcrafted iTMOs, combined in a supervised neural network to produce an HDR output. Our lightweight network requires less training images than state-of-the-art methods, and has faster training phase. Besides, the quality of the generated images is similar to the state-of-the-art. We present the architecture as well as the subjective and experimental evaluations of our method. [Display omitted] • We propose a new Inverse Tone Mapping Operator to create High Dynamic Range images. • Convolutional Neural Network are used to fuse together existing methods. • We performed a subjective study to evaluate our method. • The network's weights and training set are available online. [ABSTRACT FROM AUTHOR]

• Investigate pre-trained deep neural networks and random forest to detect driver distraction. • Propose a framework to automate the data reduction and hence save effort and money. • Examine the interpretability of the two fine-tuned convolutional neural network. • The framework was trained using some ANDS trips and tested using another set of trips. • The proposed framework can be used routinely in many other data reduction tasks. Naturalistic driving data (NDS), collected from fitted vehicles with multiple sensors, including cameras, is a rich source to understand how drivers become distracted. However, detecting the distraction events traditionally required watching NDS videos which is expensive and time-consuming. Adopting state-of-the-art machine learning techniques to automate distraction detection using videos will save effort and money. Reducing/annotating a random sample of the videos and using it to train a machine learning model to carry out the data reduction for the rest of the dataset is possible. This paper investigates the feasibility of using pre-trained deep neural networks and random forest to detect driver distraction in the Australian Naturalistic Driving Study (ANDS) video data. The pre-trained model was fine-tuned using transfer learning; therefore, it can be used to classify (i.e. distracted/non-distracted) successive frames independently. To capture the time dependency between the consecutive images/frame during the training phase, we converted the 1-D output probability of the fine-tuned model into a higher dimension space. The distraction probability of each frame at a time is transformed into a vector by concatenating the distraction output probabilities of all frames within a time window centred at time. Moreover, the vector centred at the time was assigned the actual label of the frame at this time. Then we used Random Forest (RF) classifier to model the non-linear relationship between the probabilities in this new space and the distracted/non-distracted labels. This proposed framework was trained using some ANDS trips and tested using another set of trips. When applied to the dashboard camera, the proposed framework achieved 0.609, 0.218 and 0.325 for the true positive rate (TPR), the false positive rate (FPR), and the precision, respectively. Moreover, the face camera achieved 0.748, 0.344 and 0.651 for TPR, the FPR and the precision, respectively. These promising results suggest the proposed framework can be used routinely in many other data reduction tasks. [ABSTRACT FROM AUTHOR]

In this paper, a three-step model based on the integration of Deep Neural Networks (DNN) and Decision Models is introduced for image classification which is inspired by the human visual system. To make a decision about an object, many actions should be done in a hierarchical process in the brain. First, the retina receives visual stimuli and transfers them to the visual cortex in the brain. The information extracted in the visual cortex, is accumulated over time to select an appropriate response. Many of the current decision-making models do not show how each image is converted into useful information for the decision model. Some models have used neural networks to convert each image into the information needed in the decision-making model; however, the role of the retina is ignored among these models. In this paper, a combination of retina inspired filters, CNN-based description and accumulator-based decision model is used to classify images. This model's structure resembles the human brain due to the usage of the DoG filter bank as retina inspired filter in the first stage of it. This model shows a significant improvement in accuracy in comparison to other models; furthermore, its performance is acceptable even with the small sample training set. [ABSTRACT FROM AUTHOR]

• An overview of deep learning algorithms used in medical image segmentation is presented. • More than 150 papers applying deep learning to different medical applications are summarised. • Challenges and future directions in medical image segmentation are discussed. Deep learning (DL) algorithms have rapidly become a robust tool for analyzing medical images. They have been used extensively for medical image segmentation as the first and significant components of the diagnosis and treatment pipeline. Medical image segmentation is efficiently addressed by many types of deep neural networks, such as convolutional neural networks, fully convolutional network recurrent networks, adversarial networks, and U-shaped networks. This paper reviews the major DL models and applications pertinent to medical image segmentation and summarizes over 150 contributions to the field. Brief overviews of articles are provided by application area: anatomical structures such as organs, bones, and vessels, and abnormalities such as lesions and calcification. Moreover, we discuss current challenges and suggest directions for future research. [ABSTRACT FROM AUTHOR]

Pipe wall-thinning is a phenomenon whereby the thickness of pipes in a nuclear power plant decreases over time owing to extended operational years. This thickness reduction is caused by various long-term thermal aging mechanisms such as Flow-Accelerated Corrosion, Liquid Droplet Impingement Erosion, cavitation, and flashing. Reducing the thickness of the secondary system pipes to the point of rupture can lead to severe human casualties and significant economic losses. Consequently, domestic power plant operators regularly manage the power plant pipes. Ongoing research focuses on the identification and management of pipe wall-thinning. However, previous studies have encountered problems in accurately judging the decrease in pipe wall-thinning in the presence of noise. To overcome this, Convolutional Neural Network (CNN) models based on image feature analysis have been used. This approach allows for the differentiation of pipe wall-thinning feature from small-sized noise in a single image. However, there were difficulties in making accurate judgments for large-sized noise that resembled the feature of pipe wall-thinning. This paper aims to analyze the limitations of the current pipe wall-thinning evaluation methods and to achieve accurate pipe wall-thinning discrimination through time-series analysis of continuous pipe wall-thinning data. The proposed method employs a Convolutional Recurrent Neural Network (CRNN) model, integrating the Recurrent Neural Network(RNN) model with the CNN model. The image feature of the pipes, extracted using CNN, are utilized as inputs for the RNN. This enables the observation of how the image features of the pipes change over time. This feature differentiates from the time-series feature of noise that occurs suddenly. Through this method, the paper proposes a new approach for effectively identifying the gradual decrease in pipe wall-thinning, enabling precise assessment of pipe wall-thinning progression. [ABSTRACT FROM AUTHOR]

Over the past decades, deep learning techniques have attracted increased attention from various research and industrial domains aligned with the development of Industry Internet-of-Things(IIoT). Specifically, with the advantage of data-driven methods, industrial organizations are seeking novel proactive strategies supported by analytic models to guarantee the quality of their production by observing degradation or predicting failure ahead of the occurrence of the component or asset. Predictive strategies are expected to promise the influence of unnecessary maintenance interruptions and mitigate the consequence of that, hence, extending the remaining useful life of products. This paper conducts a survey of the utilization of deep learning technologies on engineering applications where they provide satisfactory solutions with respect to specific data types or input signals. 106 primary papers are reviewed on deep learning–driven approaches which mainly explore five of the most popular architectures in the application of predictive maintenance. The main content of this paper summarizes the common advantages of each architecture and, accordingly, points out their limitations, as well as describes the application scopes of fully connected deep neural networks, convolutional neural networks, stacked autoencoders, deep belief networks, and deep recurrent neural networks. Based on the technique discussion for each of them, we intend to provide a comprehensive understanding and guidance of the appropriate usage of deep learning architectures to devise an effective predictive maintenance strategy for the scientific and industrial developers whose expertise lies in the prior domain knowledge of multi-source isomerization data. Moreover, the main content demonstrated the summarization of the decisive factor by which the incremental stages of the approaches were determined, fundamentally including the dataset specification, feature extraction, and the integration of deep learning approaches. [ABSTRACT FROM AUTHOR]

Electricity load exhibits daily and weekly cyclical patterns as well as random characteristics. At present, prevailing deep learning models cannot learn electricity load cyclical and stochastic features adequately. This results in insufficient prediction accuracy and the scalability of current methods. To tackle these difficulties, this paper proposes a framework for electrical load prediction based on an Attention Mechanism Time Series Depthwise Separable Convolutional Neural Network (ELPF-ATDSCN). The framework starts by using the Maximum Information Coefficient for exogenous variable selection. It then incorporates a seasonal decomposition algorithm with manual feature engineering to extract the cyclical and stochastic features of the electrical load. Subsequently, the framework employs the ATDSCN to learn the cyclical and stochastic features of the electrical load. In addition, the Bayesian algorithm optimizes model hyperparameters for optimal model performance. Experimental results of point and interval load prediction on datasets from the US and Nordic power markets reveal that the ATDSCN model proposed in this paper enhances load prediction accuracy compared with other models. It can provide more reliable predictions for power system operation and dispatch. • A new attention mechanism is designed to improve the model memory capability. • The ATDSCN model is proposed for learning periodic and stochastic features of electricity loads. • A framework for electricity load prediction is proposed to improve the accuracy of electricity load forecasting. • The effectiveness of the proposed method is validated on the real world dataset. [ABSTRACT FROM AUTHOR]

Pipe wall-thinning is a phenomenon whereby the thickness of pipes in a nuclear power plant decreases over time owing to extended operational years. This thickness reduction is caused by various long-term thermal aging mechanisms such as Flow-Accelerated Corrosion, Liquid Droplet Impingement Erosion, cavitation, and flashing. Reducing the thickness of the secondary system pipes to the point of rupture can lead to severe human casualties and significant economic losses. Consequently, domestic power plant operators regularly manage the power plant pipes. Ongoing research focuses on the identification and management of pipe wall-thinning. However, previous studies have encountered problems in accurately judging the decrease in pipe wall-thinning in the presence of noise. To overcome this, Convolutional Neural Network (CNN) models based on image feature analysis have been used. This approach allows for the differentiation of pipe wall-thinning feature from small-sized noise in a single image. However, there were difficulties in making accurate judgments for large-sized noise that resembled the feature of pipe wall-thinning. This paper aims to analyze the limitations of the current pipe wall-thinning evaluation methods and to achieve accurate pipe wall-thinning discrimination through time-series analysis of continuous pipe wall-thinning data. The proposed method employs a Convolutional Recurrent Neural Network (CRNN) model, integrating the Recurrent Neural Network(RNN) model with the CNN model. The image feature of the pipes, extracted using CNN, are utilized as inputs for the RNN. This enables the observation of how the image features of the pipes change over time. This feature differentiates from the time-series feature of noise that occurs suddenly. Through this method, the paper proposes a new approach for effectively identifying the gradual decrease in pipe wall-thinning, enabling precise assessment of pipe wall-thinning progression. [ABSTRACT FROM AUTHOR]

• A multi-physics 3D FEM is established for exciting electrodes of ECS installed around pipeline subjected to LTCTMF. • The LTCTMF behavior is studied at different levels of LTCTMF. • A theoretical model results of LTCTMF compliance over creep time are analyzed using the Modulus degradation Approach. • A deep learning model is designed to predict compliance for various LTCTMF levels not included in previous FEM evaluation. • The proposed method performance is estimated by computing the accuracy rate, regression rate, and F-score indexes. The composite pipeline is a relatively new and viable alternative pipeline to the more commonly used traditional one due to its good mechanical and fatigue properties and lower production cost. For this purpose, it is critical to assess the mechanical and fatigue performance of composite pipeline material under various working conditions, particularly for monitoring long-term creep thermo-mechanical fatigue behavior. In this paper, a long-term creep thermo-mechanical fatigue behavior in a basalt fiber reinforced polymer laminated composite pipeline is detected through an integrated expert system consisting of the electrical capacitance sensors and a deep learning algorithm. First, a multi-physics finite element model is established for the simulation of a long-term creep thermo-mechanical fatigue behavior in basalt fiber reinforced polymer composite pipelines subjected to long-term fatigue loading of internal pressure and thermal effect. Second, theoretical model results of long-term creep thermo-mechanical fatigue compliance (S f (t)) over the time of creep are analyzed in pipeline material using the modulus degradation approach. Finally, an electrical potential change between electrical capacitance sensors electrodes corresponding to S f (t) over the time of creep for some levels of long-term creep thermo-mechanical fatigue (R f %) is recorded and then used in these datasets for training of the novel deep neural network based on one of the most widely used of the deep neural network families is the convolutional neural network, to predict S f (t) in pipeline for various R f % not included in the previous finite element model evaluation (i.e. electrical capacitance sensors technique). In this paper first is detected the long-term creep thermo-mechanical fatigue behavior for R f % = 25 % , 50 % , and 75 % from a finite element model and modulus degradation approach, and then is predicted the long-term creep thermo-mechanical fatigue behavior for R f % = 15 % , 35 % , 60 % , and 85 % respectively via deep neural network. The proposed method results are in good agreement with the experimental results available in the literature, thus verifying the accuracy and reliability of the proposed technique and its applicability to other different composite structures. [ABSTRACT FROM AUTHOR]

To tune the wave energy converter (WEC) controller parameters such as damping to reduce the line force during extreme wave conditions, future knowledge of the line force is required. To achieve this, the incoming wave and system state should be predicted for a few seconds in the future. It is rather an arduous task to predict the future knowledge of waves and the system's dynamic when dealing with breaking and steep waves, and the system is subject to various nonlinear forces. The classical model-based control strategies often rely on linear assumptions to estimate the WEC dynamics for the sake of simplicity. Unlike the model-based, the data-driven approaches are free from modeling errors and the algorithms are trained over the true and noisy data to predict non-linear system behaviors. Using data-driven approaches, we are able to model nonlinear dynamics. However, new questions emerge on the accuracy of the future wave and system state predictions, and how this uncertainty propagates into the final prediction of the line force. As incorrect damping may lead to excessive line force and detrimental damage to the system, these are the knowledge gaps that need to be addressed. The main purpose of this paper is to answer these questions through a survivability strategy for wave energy converters by providing a realistic perspective on the implementation of the neural network approaches by accounting for the errors in the input data. For this purpose, a series of neural networks is designed that first predicts the surface elevation for 0.36 s ahead, i.e. corresponding to 2 s in the full-scale WEC. This future knowledge of the wave elevation is then used to predict the system state (i.e. power take-off (PTO) translator position) for the same prediction horizon based on the PTO damping. This information is then fed to a convolutional neural network (CNN) that predicts the peak line force 0.36 s ahead. This paper also evaluates the predictive capabilities of neural network (NN) models, comparing them to classical autoregressive (AR) and Kalman filter methods. While the AR model exhibits slightly higher accuracy in fixed PTO system configurations, it falters in providing real-time predictions when system parameters undergo variations. In contrast, the NN prediction model excels by establishing a robust relationship between system configuration and output signals. Especially noteworthy is its ability to outperform classical methods with minimal training on a selective set of system configurations. Then, the sensitivity of the peak line force prediction to the uncertainties in the input data from NN models and the prediction horizon is analyzed. The neural network models are trained over the experimental data subjected to extreme sea states for a point absorber wave energy converter. The results suggest that the accuracy of the surface elevation prediction has an insignificant direct effect on the peak force prediction model. However, these uncertainties are reflected in the PTO translator position prediction, and the model is considerably sensitive to the accuracy of this prediction. This sensitivity nonetheless is less notable for higher PTO damping values. • Predicting wave elevation, PTO position, and mooring force via neural networks. • Finding optimum damping to reduce mooring force in extreme wave conditions. • Predicting the mooring force is sensitive to the PTO position prediction. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

Recent studies show that gradient descent can train a deep neural network (DNN) to achieve small training and test errors when the DNN is sufficiently wide. This result applies to various over-parameterized neural network models including fully-connected neural networks and convolutional neural networks. However, existing theory does not apply to graph convolutional networks (GCNs), as GCNs is built according to the topological structures of the data. It has been empirically observed that GCNs can outperform vanilla neural networks when the underlying graph captures geometric information of the data. However, there is few theoretical justification of such observation. In this paper, we establish theoretical guarantees of the high-probability convergence of gradient descent for training over-parameterized GCNs. Specifically, we introduce a novel measurement of the relation between the graph and the data, called the "graph disparity coefficient", and show that the convergence of GCN is faster when the graph disparity coefficient is smaller. Our analysis provides novel insights into how the graph convolution operation in a GCN helps training, and provides useful guidance for GCN training in practice. [ABSTRACT FROM AUTHOR]

As the need for sensor fusion systems has grown, developing methods to find correspondences between images with different spectral ranges has become increasingly important. Since most images do not share low-level information, such as textures and edges, existing matching approaches fail even with convolutional neural networks (CNNs). In this paper, we propose an end-to-end metric learning method, called SPIMNet (SPectral-Invariant Matching Network) for robust cross- and multi-spectral image patch matching. While existing methods based on CNNs learn matching features directly from cross- and multi-spectral image patches, SPIMNet transforms across spectral bands and discriminates for similarity in three steps. First, (1) SPIMNet is adjusted for a feature domain by introducing a domain translation network; then (2) two Siamese networks learn to match the adjusted features with the same spectral domain; and (3) the matching features are fed to fully-connected layers to determine the identity of the patches as a classification task. By effectively incorporating each step, SPIMNet achieved competitive results on a variety of challenging datasets, including both VIS–NIR and VIS–Thermal image pairs. Our code is available at https://github.com/koyeongmin/SPIMNet. [Display omitted] • The method is based on domain conversion for matching between VIS and NIR images. • The method uses metric learning to integrate information from the two domains. • This work studies the effect of the domain conversion method and loss function. • This work achieves noteworthy performance in three public datasets. • The result can be adapted to various applications such as depth estimation. [ABSTRACT FROM AUTHOR]

The state-of-the-art performance for several real-world problems is currently reached by deep and, in particular, convolutional neural networks (CNN). Such learning models exploit recent results in the field of deep learning, leading to highly performing, yet very large neural networks with typically millions to billions of parameters. As a result, such models are often redundant and excessively oversized, with a detrimental effect on the environment in terms of unnecessary energy consumption and a limitation to their deployment on low-resource devices. The necessity for compression techniques able to reduce the number of model parameters and their resource demand is thereby increasingly felt by the research community. In this paper we propose the first extensive comparison, to the best of our knowledge, of the main lossy and structure-preserving approaches to compress pre-trained CNNs, applicable in principle to any existing model. Our study is intended to provide a first and preliminary guidance to choose the most suitable compression technique when there is the need to reduce the occupancy of pre-trained models. Both convolutional and fully-connected layers are included in the analysis. Our experiments involved two pre-trained state-of-the-art CNNs (proposed to solve classification or regression problems) and five benchmarks, and gave rise to important insights about the applicability and performance of such techniques w.r.t. the type of layer to be compressed and the category of problem tackled. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs) have been widely adopted as general computational tools both in computer science as well as many other engineering fields. Stochastic gradient descent (SGD) and adaptive methods such as Adam are popular as robust optimization algorithms used to train the ANNs. However, the effectiveness of these algorithms is limited because they calculate a search direction based on a first-order gradient. Although higher-order gradient methods such as Newton's method have been proposed, they require the Hessian matrix to be semi-definite, and its inversion incurs a high computational cost. Therefore, in this paper, we propose a variable three-term conjugate gradient (VTTCG) method that approximates the Hessian matrix to enhance search direction and uses a variable step size to achieve improved convergence stability. To evaluate the performance of the VTTCG method, we train different ANNs on benchmark image classification and generation datasets. We also conduct a similar experiment in which a grasp generation and selection convolutional neural network (GGS-CNN) is trained to perform intelligent robotic grasping. After considering a simulated environment, we also test the GGS-CNN with a physical grasping robot. The experimental results show that the performance of the VTTCG method is superior to that of four conventional methods, including SGD, Adam, AMSGrad, and AdaBelief. [ABSTRACT FROM AUTHOR]

Marine ecosystems and their fish habitats are becoming increasingly important due to their integral role in providing a valuable food source and conservation outcomes. Due to their remote and difficult to access nature, marine environments and fish habitats are often monitored using underwater cameras to record videos and images for understanding fish life and ecology, as well as for preserve the environment. There are currently many permanent underwater camera systems deployed at different places around the globe. In addition, there exists numerous studies that use temporary cameras to survey fish habitats. These cameras generate a massive volume of digital data, which cannot be efficiently analysed by current manual processing methods, which involve a human observer. Deep Learning (DL) is a cutting-edge Artificial Intelligence (AI) technology that has demonstrated unprecedented performance in analysing visual data. Despite its application to a myriad of domains, its use in underwater fish habitat monitoring remains under explored. In this paper, we provide a tutorial that covers the key concepts of DL, which help the reader grasp a high-level understanding of how DL works. The tutorial also explains a step-by-step procedure on how DL algorithms should be developed for challenging applications such as underwater fish monitoring. In addition, we provide a comprehensive survey of key deep learning techniques for fish habitat monitoring including classification, counting, localisation, and segmentation. Furthermore, we survey publicly available underwater fish datasets, and compare various DL techniques in the underwater fish monitoring domains. We also discuss some challenges and opportunities in the emerging field of deep learning for fish habitat processing. This paper is written to serve as a tutorial for marine scientists who would like to grasp a high-level understanding of DL, develop it for their applications by following our step-by-step tutorial, and see how it is evolving to facilitate their research efforts. At the same time, it is suitable for computer scientists who would like to survey state-of-the-art DL-based methodologies for fish habitat monitoring. • Marine environments and fish habitats are widely surveyed using underwater cameras. • Deep learning (DL) offers an effective solution for processing underwater videos. • We write a tutorial on the key concepts and steps of DL for underwater monitoring. • We provide a survey of key DL-based visual fish monitoring methods/datasets. • We shed light on challenges/opportunities of DL for underwater fish monitoring. [ABSTRACT FROM AUTHOR]

This paper investigates a novel CNN-based architecture for image resizing forensics in the presence of Double-JPEG compression. Two sub-problems are addressed as part of this paper: first, the detection of resizing in DJPEG images, and second, determining the factor used to resize the image before the second JPEG compression. The image resizing technique used in this paper is image scaling. The proposed network is based on utilizing specially chosen high pass filters that constitute the pre-processing layer, followed by four convolutional layers, two fully connected layers, and a final softmax layer. The high-pass filters preserve the resizing clues in the form of residuals applied at the pre-processing layer, followed by utilizing the residuals as inputs to a CNN to detect the presence of resizing and also for estimating the resizing factor. The proposed network is fully end-to-end and does not rely on any hand-crafting. Experimental validation is carried out utilizing images from the RAISE dataset. Comparative analysis with four state-of-the-art methods demonstrates the proposed CNN-based architecture's superiority for resizing detection and resize factor determination sub-problems. [ABSTRACT FROM AUTHOR]

Fine-grained visual categorization is an important task in computer vision. Prior works on fine-grained visual categorization have paid much attention to addressing intra-class variation and inter-class similarity. However, they rarely study that task from the perspective of probability distribution. In this paper, we propose a novel refined probability distribution module based on deep convolutional neural network. Our module computes the probability of an image by fully utilizing the similarity information between images. Firstly, we use deep neural networks to obtain the initial probability distribution and extract features. Then, we build a network whose inputs are features for calculating image-to-image similarity scores. Finally, our module refines the initial probability distribution based on an effective batch random walk operation with similarity scores. Our module can be plugged into many deep convolutional neural networks. Experimental results show that our approach outperforms state-of-the-art methods on the CUB-200–2011, FGVC-Aircraft and Stanford Cars datasets respectively. [ABSTRACT FROM AUTHOR]

The combination of convolutional and recurrent neural networks is a promising framework. This arrangement allows the extraction of high-quality spatio-temporal features together with their temporal dependencies. This fact is key for time series prediction problems such as forecasting, classification or anomaly detection, amongst others. In this paper, the TSFE DL library is introduced. It compiles 22 state-of-the-art methods for both time series feature extraction and prediction, employing convolutional and recurrent deep neural networks for its use in several data mining tasks. The library is built upon a set of Tensorflow+Keras and PyTorch modules under the AGPLv3 license. The performance validation of the architectures included in this proposal confirms the usefulness of this Python package. [ABSTRACT FROM AUTHOR]

• New subspace-based method for image set recognition. • Theoretically-grounded integration of subspace representation into DNNs while keeping end-to-end trainability. • Our new method generalizes the classic learning subspace method. This paper addresses the problem of object recognition given a set of images as input (e.g., multiple camera sources and video frames). Convolutional neural network (CNN)-based frameworks do not exploit these sets effectively, processing a pattern as observed, not capturing the underlying feature distribution as it does not consider the variance of images in the set. To address this issue, we propose the Grassmannian learning mutual subspace method (G-LMSM), a NN layer embedded on top of CNNs that can process image sets more effectively and can be trained in an end-to-end manner. The image set is first represented by a low-dimensional input subspace and then this input subspace is matched with dictionary subspaces by a similarity of their canonical angles, an interpretable and easy to compute metric. The key idea of G-LMSM is that the dictionary subspaces are learned as points on the Grassmann manifold, optimized with Riemannian stochastic gradient descent. This learning is stable, efficient and theoretically well-grounded. We demonstrate the effectiveness of our proposed method on hand shape recognition, face identification, and facial emotion recognition. [ABSTRACT FROM AUTHOR]

Interpretability and explainability of machine learning systems have received ever-increasing attention, especially for deep neural networks. In the case of convolutional neural networks (CNNs), their properties are usually explained by generating local explanation maps (e.g., visualizing the contribution of individual pixels to a given prediction). In this paper, we propose a new framework that analyzes the inner workings of CNNs in terms of neural activations. To be precise, we consider a forward-pass as sequential activations of neurons and develop its inverse process, so that the inverse preserves the physical meaning of neuron activations. Our inverse process is formulated as a constrained optimization problem, and we solve the problem with the gradient projection algorithm. The proposed approach can provide equivalent visualization results to several conventional methods, and thus can be a reference tool for CNN visualization. Also, the attributions generated by our inverse method yield the state-of-the-art deletion scores and visualize the contribution of colors as well as shape features. [ABSTRACT FROM AUTHOR]

• Thispaperfirstlyproposed a supervised-unsupervised combinednetwork for high-fidelitypansharpening. • Thispaperproposed a supervised networkby multiscale mechanisms, dilated convolution and skip connection. • This paperproposeda universalunsupervisedspatial-spectral joint compensationnetwork. Deep learning for pansharpening method has become a hot research topic in recent years due to the impressive performance, and the convolutional neural networks (CNN)-based pansharpening methods on Wald's protocol (i.e., the general adoption of the network learned at a coarser reduced resolution scale to the finer full resolution) have been dominating for a long time in this research area. However, the scale-invariant assumption may not be accurate enough to make full use of the spatial and spectral information of original panchromatic (PAN) and multispectral (MS) images at full resolution. In this paper, a Supervised-Unsupervised combined Fusion Network (SUFNet) for high-fidelity pansharpening is proposed to alleviate this problem. First, by comprehensively considering the robustness of the network with reference label images, a novel supervised network based on Wald's protocol is proposed by integrating the multiscale mechanisms, dilated convolution, and skip connection, termed SMDSNet. Then, an interesting Unsupervised Spatial-Spectral Compensation Network (USSCNet) without real high-spatial-resolution (HR) MS label image is proposed to enhance the spatial and spectral fidelity of the SMDSNet. The qualitative and quantitative results in reduced resolution and full resolution experiments on different satellite datasets demonstrate the competitive performance of the proposed method. Furthermore, the proposed USSCNet can be employed as a universal spatial-spectral compensation framework for other pansharpening methods. [ABSTRACT FROM AUTHOR]

Producing high-resolution urban land cover maps is essential for decision-making and urban management. In this regard, Synthetic Aperture Radar (SAR), especially Polarimetric SAR (PolSAR), has served as a valuable data source to fulfill this task. Direct conversion of low-level features into high-level Land Cover (LC) concept may reduce the final classification accuracy. Therefore, mid-level representation models, such as Bag of Visual Words (BOVW), were employed to resolve the existing semantic gap challenge. In this paper, a Segment-based BOVW (Seg-BOVW) model was developed for urban land cover classification using PolSAR data. To this end, two PolSAR data over San Francisco Bay (SFB) and Flevoland (FL) acquired by RADARSAT-2 were employed to comprehensively evaluate the Seg-BOVW model's performance. First, to exploit the full potential of PolSAR data, 169 low-level features in four categories: (1) original, (2) polarimetric, (3) texture, and (4) decomposition features were extracted. Afterward, a Multi-Objective Genetic Algorithm (MOGA) was implemented to investigate the importance of low-level features for urban land cover mapping. This step resulted in selecting 14 and 6 low-level features, as the high contributing features, for SFB and FL datasets, respectively. The Seg-BOVW model achieved significant overall accuracies of 96.02% and 98.82% for SFB and FL, respectively, indicating the high potential of the proposed method for urban land cover classification. Furthermore, a comparison with other well-known algorithms of Support Vector Machine (SVM), Artificial Neural Networks (ANN), and Convolutional Neural Networks (CNN) was made, suggesting the capability of the Seg-BOVW model to improve the urban land cover classification results. Finally, the Seg-BOVW model was tested with two other PolSAR datasets acquired with different sensors over SFB to examine its applicability with different datasets. [ABSTRACT FROM AUTHOR]

Accurate relative pose estimation of a spacecraft during space landing operation is critical to ensure a safe and successful landing. This paper presents a 3D Light Detection and Ranging (LiDAR) based AI relative navigation architecture solution for autonomous space landing. The proposed architecture is based on a hybrid Deep Recurrent Convolutional Neural Network (DRCNN) combining a Convolutional Neural Network (CNN) with an Recurrent Neural Network (RNN) based on a Long–Short Term Memory (LSTM) network. The acquired 3D LiDAR data is converted into a multi-projected images and feed the DRCNN with depth and other multi-projected imagery. The CNN module of the architecture allows an efficient representation of features, and the RNN module, as an LSTM, provides robust navigation motion estimates. A variety of landing scenarios are considered, simulated and experimented to evaluate the efficiency of the proposed architecture. A LiDAR based imagery data (Range, Slope, and Elevation) is initially created using PANGU (Planet and Asteroid Natural Scene Generation Utility) software and an evaluation of the proposed solution using this data is conducted. Tests using an instrumented Aerial Robot in Gazebo software to simulate landing scenarios on a synthetic but representative lunar terrain (3D digital elevation model) is proposed. Finally, real experiments using a real flying drone equipped with a Velodyne VLP16 3D LiDAR sensor to generate real 3D scene point clouds while landing on a designed down scaled lunar moon landing surface are conducted. All the test results achieved show that the suggested architecture is capable of delivering good 6 Degree of Freedom (DoF) pose precision with a good and reasonable computation. • Effective feature representation due to the adoption of the CNN module. • The RNN (LSTM) module provides a robust and efficient modelling of the navigation kinematics. • The hybrid network is flexible to train and test on different data modalities. [ABSTRACT FROM AUTHOR]

Artificial neural networks (ANNs) experience catastrophic forgetting (CF) during sequential learning. In contrast, the brain can learn continuously without any signs of catastrophic forgetting. Spiking neural networks (SNNs) are the next generation of ANNs with many features borrowed from biological neural networks. Thus, SNNs potentially promise better resilience to CF. In this paper, we study the susceptibility of SNNs to CF and test several biologically inspired methods for mitigating catastrophic forgetting. SNNs are trained with biologically plausible local training rules based on spike-timing-dependent plasticity (STDP). Local training prohibits the direct use of CF prevention methods based on gradients of a global loss function. We developed and tested the method to determine the importance of synapses (weights) based on stochastic Langevin dynamics without the need for the gradients. Several other methods of catastrophic forgetting prevention adapted from analog neural networks were tested as well. The experiments were performed on freely available datasets in the SpykeTorch environment. • Coding information in spikes does not prevent catastrophic forgetting during continuous learning. • Pseudo-rehearsal with random samples provoke fast unlearning of previous knowledge in convolutional spiking neural networks. • Saving a small number of samples from previous datasets can effectively decrease catastrophic forgetting. • A stochastic process of weights diffusion can establish the importance of the weights in spiking networks trained with local rules. [ABSTRACT FROM AUTHOR]

• This paper proposed a novel two-stage spatiotemporal features learning method based on a Siamese 3D CNN for MEs recognition. • The focal loss was introduced to address the problem of inefficient model training caused by the class imbalance in the MEs datasets. • An adaptive construction method based on adaptive CNN was proposed to construct the key-frames sequence to summarize the original MEs. Micro-expressions (MEs) are spontaneous and involuntary facial subtle reactions which often reveal the genuine emotions within human beings. Recognizing MEs automatically is becoming increasingly crucial for many areas such as diagnosis and security. However, the short time duration and low spatial intensity of MEs pose great challenges for accurately recognizing them. Additionally, the lack of sufficient and balanced spontaneous MEs data also makes this problem even harder to solve, and some adaptive modeling strategies have been quite urgent recently. To this end, this paper draws inspirations from few-shot learning to propose a novel two-stage learning (i.e., prior learning and target learning) method based on a siamese 3D convolutional neural network for MEs recognition (MERSiamC3D). Specifically, in the prior learning stage, the proposed MERSiamC3D is used to extract the generic features of MEs. In the target learning stage, the structure and parameters of the MERSiamC3D will be carefully adjusted and the Focal Loss is adopted for high-level features learning. Afterwards, in order to effectively retain the spatiotemporal information of the original MEs video, an adaptive construction method based on adaptive convolutional neural network is proposed to construct the key-frames sequence to summarize the original MEs video, which is able to help drop the redundant frames and relatively highlight the movement of the apex frame. Then, the new key-frames are taken as the input of the two-stage learning method. Finally, through extensive evaluations and experiments on three publically available MEs datasets, the proposed method in this work could outperform traditional methods and other deep learning baselines, which provides a novel insight on how to leverage scarce data for MEs recognition. [ABSTRACT FROM AUTHOR]

Urban ride-hailing demand prediction is a crucial but challenging task for intelligent transportation system construction. Predictable ride-hailing demand can facilitate more reasonable vehicle scheduling and online car-hailing platform dispatch. Conventional deep learning methods with no external structured data can be accomplished via hybrid models of Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) by meshing plentiful pixel-level labeled data, but data sparsity of high grid granularity in spatial perspective and limited learning capabilities of long-term dependencies in temporal perspective are still two striking bottlenecks. To address these problems, we propose a novel virtual graph modeling approach to focus on significant demand regions and a novel Deep Multi-View Spatio-temporal Virtual Graph Neural Network (DMVST-VGNN) to strengthen the learning capabilities of spatial dynamics and long-term temporal dependencies. Specifically, DMVST-VGNN integrates structures of 1D CNN, Multi-Graph Attention Neural Network and Transformer Network, which correspond to short-term temporal dynamics view, spatial dynamics view and long-term temporal dynamics view respectively. In this paper, multiple experiments are conducted on two large-scale New York City datasets in higher granularity prediction scenes. And the experimental results demonstrate the effectiveness of DMVST-VGNN framework in ride-hailing demand prediction, no matter in spatial scale or the temporal scale. [ABSTRACT FROM AUTHOR]

Lithium-ion batteries have been used in all aspects of life for environmental and resource reasons. The accurate estimation of the state of charge (SOC) ensures the proper operation of the battery. However, few methods have focused on the problem of SOC estimation at low temperatures. A hybrid method based on the CNN-BWGRU network is proposed in this paper. The method optimizes the influence of battery information on the results through a "multi-moment input" structure and the bidirectional network. The convolutional neural network (CNN) is used to learn the feature parameters in the input. The bidirectional weighted gated recurrent unit (BWGRU) can improve the fitting performance of the network at low temperatures by changing the weights. The proposed network has a strong generalization capability, estimation accuracy, and robustness. SOC estimation is performed under different conditions to verify the plausibility of the network. The experimental results show that the method has higher accuracy and stability than other networks. In addition, the proposed method can overcome the effect of different initial SOC on the estimation results. Therefore, the CNN-BWGRU network provides a new method in battery SOC estimation while providing helpful guidance for safe and stable battery operation in natural environments. [Display omitted] [ABSTRACT FROM AUTHOR]

• We propose a rotation invariant Gabor convolutional neural network (RIGCN). • We learn Gabor-guided convolutional features in the Siamese network architecture. • We explore a convolutional fusion operator to obtain rotation invariant features. • We show the effectiveness of RIGCN for rotation invariant image classification. Gabor filters have been recently integrated with deep convolutional neural networks to learn better features with fewer model parameters. However, during feature learning the rotation invariance is not well addressed. In this paper, we propose a rotation invariant Gabor convolutional neural network (RIGCN) for image classification. First, we transform each input image to generate multiple rotated image instances and feed them into a weight-sharing Siamese network architecture to learn Gabor-guided deep convolutional features. Then, we compute the maximum and average feature responses from all the rotated instances of the same input image and send them into a convolutional fusion module to obtain a rotation invariant image representation. Finally, we use the cross-entropy loss for image classification. In our method, the use of Siamese architecture enables us to obtain rotation invariant features from rotated image instances. The use of convolutional fusion operator enables us to obtain richer statistical features with high efficiency. Experimental results on several benchmark datasets demonstrate the effectiveness of RIGCN for rotation invariant image classification. [ABSTRACT FROM AUTHOR]

We address the pixel-wise mapping problem that commonly exists in the fine-grained computer vision tasks, such as estimating keypoint heatmaps and segmentation masks. These tasks require, at low computation overheads, modeling the long-range dependencies among high-resolution inputs and estimating the highly nonlinear pixel-wise outputs. While the attention mechanism added to Deep Convolutional Neural Networks (DCNNs) can boost long-range dependencies, the element-specific attention, such as the Nonlocal block, is highly complex and noise-sensitive to learn, and most of the simplified attention blocks are designed for image-wise classification purposes and simply applied to pixel-wise tasks. In this paper, we present the Polarized Self-Attention (PSA) block targeting the high-quality pixel-wise mapping with: (1) Polarized filtering: keeping high internal resolution in both channel and spatial attention computation while completely collapsing input tensors along their counterpart dimensions. (2) Enhancement: composing non-linearity that directly fits the output distribution of typical pixel-wise mappings, such as the 2D Gaussian distribution (keypoint heatmaps), or the 2D Binormial distribution (binary segmentation masks). Experimental results show that PSA boosts standard baselines by 2–4 points, and boosts state-of-the-arts by 1–2 points on 2D pose estimation and semantic segmentation benchmarks. Codes are available at (https://github.com/DeLightCMU/PSA). [ABSTRACT FROM AUTHOR]

Long short-term memory (LSTM) networks are widely used to handle temporal or sequential data, and have great potential for video recognition. Existing LSTM-based video recognition methods either insert LSTM modules at the end of 2D convolutional neural networks (CNNs), called global LSTM methods, or build networks solely by stacking multiple LSTM modules. Unfortunately, these LSTM-based video recognition methods are not competitive, compared to state-of-the-art 3D CNNs or two-stream CNNs. In order to fully explore the potential of LSTM, this paper rethinks its role in video recognition network architectures and proposes a novel Temporal Grafter Network (TGN). Specifically, we develop an efficient and effective variant of convolutional LSTM module, which is grafted between different stages of very deep 2D CNNs for temporal modeling and delivery. Our TGN can capture local motion patterns of varying scales inherent in feature maps from high to low resolutions, while attending to the spatial context information and modeling global temporal dependency across the whole video. The proposed TGN can capture and transmit temporal information throughout very deep 2D CNNs, overcoming the downsides of existing LSTM-based methods and being able to make full use of the potential of LSTM for effective video recognition and early action recognition. We perform extensive ablation study to verify the effectiveness of our proposed methods, and experiments on three widely used video benchmarks show that our methods can achieve performance matching or better than the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.com/shivram1987/ActivationFunctions. [ABSTRACT FROM AUTHOR]

Interpretation of large-scale aerial imagery is an essential challenge due to the dense distribution of objects and complex intra-class backgrounds. Non-local relational modeling has been adopted as the mainstream solution to better interpret such large-scale scenes. Existing non-local relations modeling methods are based on building deep convolution neural networks, attention mechanisms, or deep graph neural networks for all positions of the whole image. However, the existing methods are computationally expensive and inefficient. In this paper, we dig into the main causes of such inefficiency and find two main disadvantages of existing methods, i.e., (1) semantic relations are blurry and homogeneous, (2) insufficient construction of high-order relations. To overcome the above issues, we analyze the inadequacy of traditional hypergraph definition and propose a lightweight hypergraph construction strategy to learn the non-local relations. The proposed method can model high-order relations more effectively and lead to an explicit representation of semantic information. We apply this strategy in the spatial dimension and construct a fully-weighted hypergraph neural network (HGNN) to capture short- and long-range dependencies in such a large-scale aerial image. Furthermore, we design a hypergraph convolutional feature pyramid network (Hyper-FPN), which learns the non-local relations in multi-scale features and then aggregates hierarchical global contexts. Extensive experiments on geospatial visual recognition demonstrate that Hyper-FPN significantly improves the performance under our strategy. Moreover, our approach can be easily embedded into state-of-the-art (SOTA) architectures to achieve higher performance. [ABSTRACT FROM AUTHOR]

• Propose a feature fusion method based on the higher-order spectral analysis (HOSA) of bi-spectral features (BSFs) and non-linear bi-coherent features (BCFs). • The HOSA of speech-related features not only has a high accuracy in speech emotion recognition but is also accurate for depression recognition. • Fused features were better than the speech-related features obtained from HOSA in terms of accuracy, and the latter were better than the COVAREP-extracted speech-related features. Approximately 300 million people worldwide suffer from depression, and more than 60% of psychiatric patients do not have access to mental health services due to the shortage of psychiatrists and the high costs associated with clinical diagnosis and treatment. Correct and efficient diagnosis of depression can help overcome these straits. Automatic detection of depressive symptoms can help improve the accuracy and availability of diagnosis. In this paper, a fusion feature for Bispectral Features and Bicoherent Features by using higher-order spectral analysis. Experiments were performed on the Depression Sub-Challenge Dataset of the Audio/Visual Emotion Challenge 2017. The fusion feature fuses higher-order spectral features and traditional speech features with classification weights greater than 100 extracted by using A Collaborative Voice Analysis Repository. The support vector machine and k-nearest neighbor classification algorithms were used as the traditional machine learning models, and the convolutional neural network was used as the deep learning model to verify the proposed features. The experimental results show that under the support vector machine algorithm, the accuracies of extraction of speech-related features by using a collaborative voice analysis repository, The higher-order spectral analysis, and their fusion features were 63.15%, 68.42%, and 73.68%, respectively. Under the k-nearest neighbor classification algorithms model algorithm, the corresponding accuracies were 68.18%, 72.73%, and 77.27%, respectively. For the convolutional neural network model, the corresponding accuracies were 70%, 77%, and 85%, respectively. The results demonstrate that the fusion feature recognition accuracy is high and can be employed to improve the accuracy of depression identification by using traditional machine learning and deep learning models. [ABSTRACT FROM AUTHOR]

Accurate traffic flow prediction can effectively improve traffic efficiency and safety. This has become a trending topic in intelligent transportation systems. However, the occurrence of traffic accidents will cause great fluctuations of traffic flow in a short time, thus affecting the accuracy of flow prediction. This paper analyses the influence of traffic accident information on traffic flow, and proposes a grey convolutional neural network called G-CNN for traffic flow prediction. First, considering the small sample size of traffic accidents and the incomplete information description, the traffic flow and traffic speed change rate are defined to represent traffic accident grey information, and a grey fixed-weight clustering method for extracting the characteristics of traffic accident grey information is established. Second, the spatiotemporal characteristics of traffic flow are analysed, and a third-order tensor is developed to fuse them with the accident characteristics and serve as the input of the G-CNN. Then the G-CNN model is built to predict traffic flow in a traffic accident environment. The Canadian Whitemud Drive experiment on traffic flow without traffic accident information reveals that the developed G-CNN model has a better ability to predict future traffic flow with better accuracy and stability. Meanwhile, it is verified that the fluctuation of the traffic change rate will cause traffic accidents, and the probability of traffic accidents at a certain time point will be obtained through feature extraction of traffic accidents. Finally, the model is verified for the traffic flow of Hangzhou Viaduct in China, which demonstrates that the G-CNN model outperforms all of the compared models. [ABSTRACT FROM AUTHOR]

When GPS signal is interfered or lost, the visual geo-localization method is particularly important for Unmanned Aerial Vehicle (UAV). Since matching UAV images with satellite maps is a multi-source and multi-view problem, visual geo-localization is very challenging. Most existing methods use Convolutional Neural Network (CNN), which extract the final output of the backbone Network to predict the similarity between UAV images and satellite maps. Due to continuous stacked convolution and pooling, rich local information is gradually lost while semantic information is acquired. To solve this problem, a dual attention and dual fusion (DADF) scene matching algorithm is proposed. The contributions of this paper are as follows: 1) In order to achieve accurate matching between UAV and satellite images, a visual geo-localization algorithm based on siamese network is designed. 2) In order to improve the ability of semantic feature extraction, a dual-attention model is constructed. The network pays more attention to the parts that are useful for similarity metric. 3) A dual fusion model is established. According to the feature fusion method and multi-level matching result fusion algorithm, the confidence of matching is improved. To verify the performance of the proposed approach, LA850 and NWPU-ChangAn datasets were collected and enhanced. The experimental results show that the proposed algorithm is more efficient than comparison algorithms. [ABSTRACT FROM AUTHOR]

Multimodal registration is a challenging problem in visual computing, commonly faced during medical image-guided interventions, data fusion and 3D object retrieval. The main challenge of multimodal registration is finding accurate correspondence between modalities, since different modalities do not exhibit the same characteristics. This paper explores how the coherence of different modalities can be utilized for the challenging task of 3D multimodal registration. A novel deep learning multimodal registration framework is proposed by introducing a siamese deep learning architecture, especially designed for aligning and fusing modalities of different structural and physical principles. The cross-modal attention blocks lead the network to establish correspondences between features of different modalities. The proposed framework focuses on the alignment of 3D point clouds and the micro-CT 3D volumes of the same object. A multimodal dataset consisting of real micro-CT scans and their synthetically generated 3D models (point clouds) is presented and utilized for evaluating our methodology. [Display omitted] • The problem of multimodal 3D registration of CT volumes and 3D point clouds is formally defined. • A framework for multimodal 3D registration of CT volumes and 3D point clouds is proposed. • A DLN that combines regular CNNs suited for data with standard grid structure and GDL suited for unstructured data. • The proposed network employs siamese architecture for effective multimodality fusion. • A multimodal dataset for evaluating algorithms for aligning CT volumes and point clouds is created. [ABSTRACT FROM AUTHOR]

Integrating the special electromagnetic characteristics of Synthetic Aperture Radar (SAR) in deep neural networks is essential in order to enhance the explainability and physics awareness of deep learning. In this paper, we first propose a novel physically explainable convolutional neural network for SAR image classification, namely physics guided and injected learning (PGIL). It comprises three parts: (1) explainable models (XM) to provide prior physics knowledge, (2) physics guided network (PGN) to encode the knowledge into physics-aware features, and (3) physics injected network (PIN) to adaptively introduce the physics-aware features into classification pipeline for label prediction. A hybrid Image-Physics SAR dataset format is proposed for evaluation, with both Sentinel-1 and Gaofen-3 SAR data being experimented. The results show that the proposed PGIL substantially improve the classification performance in case of limited labeled data compared with the counterpart data-driven CNN and other pre-training methods. Additionally, the physics explanations are discussed to indicate the interpretability and the physical consistency preserved in the predictions. We deem the proposed method would promote the development of physically explainable deep learning in SAR image interpretation field. [ABSTRACT FROM AUTHOR]

Skin cancer is one of the most serious threats to human health among skin lesions. Computer-aided diagnosis methods can assist patients in identifying and detecting skin lesion types early, thereby enabling corresponding treatments. In this paper, we propose a dual-branch neural network model Conformer with Residual Cosine Similarity Attention and Bidirectional Convolutional fusion strategy, named RCSABC-Conformer. The core of this network structure comprises three parts: a Convolutional Neural Network (CNN) branch with Residual Cosine Similarity Attention (RCSA), a Transformer branch, and a Feature Couple Unit with Bidirectional Convolutional strategy (BC-FCU). The RCSA module calculates the cosine similarity value between the feature map generated by the convolutional operation and the feature map of the residual edge to assess whether their semantic information is similar. The semantic information of similar parts is weighted by exponential normalization to enhance the network's memory of similar features of the same type of skin lesion. The BC-FCU module interactively fuses local features and global representations of skin lesion images with different resolutions in the two branches. Specifically, when the global representations is integrated into local features, we introduce a new bidirectional convolution strategy to extract the feature map from both forward and backward directions, and then select the element with the smaller feature value from the two directions to fuse into local features. In this way, we can minimize the interference of the artifact features extracted by the Transformer branch on the CNN branch. In addition, taking advantage of the Transformer branch's capacity to construct global representations, our model can learn contextual semantic information of normal skin and lesion areas to enhance model robustness. We conducted experiments on three datasets, consisting of clinical and dermoscopic skin lesion images, as well as a hybrid of both. The experimental results show that RCSABC-Conformer outperforms both advanced and classical classification methods in terms of classification accuracy across all three datasets, without requiring an increase in the number of parameters and computational complexity. Compared with the baseline model, the classification accuracy of our proposed method improves by 2.40%, 5.39%, and 4.44% on the three datasets, respectively. To the best of our knowledge, this is the first study to apply an interactive fusion dual-branch network for multi-disease classification on different modalities of skin lesion databases. Code will be available at https://github.com/AlenLi817/RCSABC-Conformer. [ABSTRACT FROM AUTHOR]

Skin diseases are reported to contribute 1.79% of the global burden of disease. The accurate diagnosis of specific skin diseases is known to be a challenging task due, in part, to variations in skin tone, texture, body hair, etc. Classification of skin lesions using machine learning is a demanding task, due to the varying shapes, sizes, colors, and vague boundaries of some lesions. The use of deep learning for the classification of skin lesion images has been shown to help diagnose the disease at its early stages. Recent studies have demonstrated that these models perform well in skin detection tasks, with high accuracy and efficiency. Our paper proposes an end-to-end framework for skin lesion classification, and our contributions are two-fold. Firstly, two fundamentally different algorithms are proposed for segmenting and extracting features from images during image preprocessing. Secondly, we present a deep convolutional neural network model, S-MobileNet that aims to classify 7 different types of skin lesions. We used the HAM10000 dataset, which consists of 10000 dermatoscopic images from different populations and is publicly available through the International Skin Imaging Collaboration (ISIC) Archive. The image data was preprocessed to make it suitable for modeling. Exploratory data analysis (EDA) was performed to understand various attributes and their relationships within the dataset. A modified version of a Gaussian filtering algorithm and SFTA was applied for image segmentation and feature extraction. The processed dataset was then fed into the S-MobileNet model. This model was designed to be lightweight and was analyzed in three dimensions: using the Relu Activation function, the Mish activation function, and applying compression at intermediary layers. In addition, an alternative approach for compressing layers in the S-MobileNet architecture was applied to ensure a lightweight model that does not compromise on performance. The model was trained using several experiments and assessed using various performance measures, including, loss, accuracy, precision, and the F1-score. Our results demonstrate an improvement in model performance when applying a preprocessing technique. The Mish activation function was shown to outperform Relu. Further, the classification accuracy of the compressed S-MobileNet was shown to outperform S-MobileNet. To conclude, our findings have shown that our proposed deep learning-based S-MobileNet model is the optimal approach for classifying skin lesion images in the HAM10000 dataset. In the future, our approach could be adapted and applied to other datasets, and validated to develop a skin lesion framework that can be utilized in real-time. [ABSTRACT FROM AUTHOR]

Skin cancer is one of the most serious threats to human health among skin lesions. Computer-aided diagnosis methods can assist patients in identifying and detecting skin lesion types early, thereby enabling corresponding treatments. In this paper, we propose a dual-branch neural network model Conformer with Residual Cosine Similarity Attention and Bidirectional Convolutional fusion strategy, named RCSABC-Conformer. The core of this network structure comprises three parts: a Convolutional Neural Network (CNN) branch with Residual Cosine Similarity Attention (RCSA), a Transformer branch, and a Feature Couple Unit with Bidirectional Convolutional strategy (BC-FCU). The RCSA module calculates the cosine similarity value between the feature map generated by the convolutional operation and the feature map of the residual edge to assess whether their semantic information is similar. The semantic information of similar parts is weighted by exponential normalization to enhance the network's memory of similar features of the same type of skin lesion. The BC-FCU module interactively fuses local features and global representations of skin lesion images with different resolutions in the two branches. Specifically, when the global representations is integrated into local features, we introduce a new bidirectional convolution strategy to extract the feature map from both forward and backward directions, and then select the element with the smaller feature value from the two directions to fuse into local features. In this way, we can minimize the interference of the artifact features extracted by the Transformer branch on the CNN branch. In addition, taking advantage of the Transformer branch's capacity to construct global representations, our model can learn contextual semantic information of normal skin and lesion areas to enhance model robustness. We conducted experiments on three datasets, consisting of clinical and dermoscopic skin lesion images, as well as a hybrid of both. The experimental results show that RCSABC-Conformer outperforms both advanced and classical classification methods in terms of classification accuracy across all three datasets, without requiring an increase in the number of parameters and computational complexity. Compared with the baseline model, the classification accuracy of our proposed method improves by 2.40%, 5.39%, and 4.44% on the three datasets, respectively. To the best of our knowledge, this is the first study to apply an interactive fusion dual-branch network for multi-disease classification on different modalities of skin lesion databases. Code will be available at https://github.com/AlenLi817/RCSABC-Conformer. [ABSTRACT FROM AUTHOR]

Segmentation is an important task in a wide range of computer vision applications, including medical image analysis. Recent years have seen an increase in the complexity of medical image segmentation approaches based on sophisticated convolutional neural network architectures. This progress has led to incremental enhancements in performance on widely recognised benchmark datasets. However, most of the existing approaches are computationally demanding, which limits their practical applicability. This paper presents an expand-squeeze dual multiscale residual network (ESDMR-Net), which is a full y convolutional network that is particularly well-suited for resource-constrained computing hardware such as mobile devices. ESDMR-Net focusses on extracting multiscale features, enabling the learning of contextual dependencies among semantically distinct features. The ESDMR-Net architecture allows dual-stream information flow within encoder–decoder pairs. The expansion operation (depthwise separable convolution) makes all of the rich features with multiscale information available to the squeeze operation (bottleneck layer), which then extracts the necessary information for the segmentation task. The Expand-Squeeze (ES) block helps the network pay more attention to under-represented classes, which contributes to improved segmentation accuracy. To enhance the flow of information across multiple resolutions or scales, we integrated dual multiscale residual (DMR) blocks into the skip connection. This integration enables the decoder to access features from various levels of abstraction, ultimately resulting in more comprehensive feature representations. We present experiments on seven datasets from five distinct examples of applications: segmentation of retinal vessels (2 ×), skin lesions (2 ×), digestive tract polyps, lung regions, and cells. Our model demonstrates strong performance, with an F1 score of 0.8287%, 0.8211%, 0.9034%, 0.9451%, 0.9543%, 0.9840%, and 0.8424% on the DRIVE, CHASE, ISIC2017, ISIC2016, CVC-ClinicDB, MC and MoNuSeg datasets, respectively. Remarkably, our model achieves these results despite having significantly fewer trainable parameters, with a reduction of two or even three orders of magnitude. • Multiscale feature extraction to understand contextual dependencies. • The expand-squeeze operation facilitates multiscale information extraction. • Dual-stream information flow within encoder–decoder pairs. • The use of DMR blocks in skip connections improves the information flow. • The deep structure of the network allows it to learn more features. [ABSTRACT FROM AUTHOR]

[Display omitted] • Aerial thermographic inspection is done by thermal cameras embedded in UAVs. • A new approach based on image analysis with two consecutive Neural Networks. • Detect thermal patterns associated with superficial faults in an IoT platform. • A real case study is proposed with thermograms from three PV solar plants. • Accuracy of 99 % for PV panel detection, and 96 % for hot spot detection. • False positives are reduced in comparison with other studies. Aerial thermographic inspection is performed with thermal cameras embedded in unmanned aerial vehicles, being one of the most relevant monitoring techniques for photovoltaic panels. This technique allows the detection of thermal patterns associated with faults in the photovoltaic panels, although image analysis and fault detection require novel processing methodologies. The volume and variety of aerial thermal data together with the need for faster and more efficient analysis for early maintenance operations require efficient Internet of Things architectures to provide fast and effective diagnostics with easy access. This paper presents a new approach based on image analysis with two consecutive convolutional neural networks to detect hot spots in an Internet of Things platform and reduce the number of false positives. The architecture of the platform is designed to automatically process the received data through the implementation of different Convolutional Neural Networks. A real case study is proposed with thermograms from three photovoltaic solar plants with different sizes, shapes of panels and a wide temperature range. The results show an accuracy of 99 % for panel detection and 96 % for hot spot detection with a reduction of false positives compared to other studies, such as Support Vector Machine or different Artificial Neural Networks, demonstrating the robustness of the method. [ABSTRACT FROM AUTHOR]