Search

Remote sensing scene (RSS) image classification plays a vital role in various fields such as urban planning and environmental protection. However, due to higher inter-class similarity and intra-class variability, achieving accurate classification for RSS images poses a considerable challenge for current convolutional neural networks (CNNs)-based and visual transformer (ViT)-based methods. To address these issues, this paper proposes a novel dual-encoding method named master–slave encoding network (MSE-Net) from two perspectives of feature extraction and fusion. The master encoder, based on ViT, extracts higher-level semantic features, while the slave encoder, based on CNN, captures relative lower-level spatial structure information. Secondly, to integrate feature information from the two encoders effectively, this paper further develop two fusion strategies. The first strategy involves the auxiliary enhancement units (AEUs), which eliminates semantic divergence between the two encoders, enhances spatial context awareness of the slave encoder and promotes effective feature learning. The interactive perception unit (IPU), as the second strategy, facilitates interaction and integration of the two encoders' representations to extract more discriminative feature information. In addition, we conducted comparative experiments on four widely-used RSS datasets, including RSSCN7, SIRI-WHU, the aerial image dataset (AID) and NWPU-RESISC45 (NWPU45), to verify the effectiveness of MSE-Net. The experimental results demonstrate that MSE-Net achieved state-of-the-art (SOTA) performance across all the datasets. [Display omitted] [ABSTRACT FROM AUTHOR]

The core of data series classification and clustering is how to evaluate the similarity between time series. However, for one-dimensional time series, the vast majority of similarity measurement methods currently relied on one-dimensional information. Although some classifications of time series were achieved by the images transformed from time series, they mostly relied on a single imaging mechanism (time domain or frequency domain). In order to give a new idea for time series similarity measurement, this paper proposed a similarity measurement method of time series based on the fused images. According to recurrence plots (RP) and wavelet scalogram (WS), the original time series was first transformed into images. Then, RP and WS were fused into an image. Next, the feature in the image was extracted by ResNet-18. Finally, the similarity of the time series can be obtained by the Euclidean distance between the features extracted by ResNet-18. The proposed method was verified by the time series from UCR Time Series Classification/Clustering Homepage. The results showed that the similarity measurement of time series can be effectively improved by fusing the images obtained by different imaging mechanisms in the time domain and frequency domain. • This paper proposes a new similarity measurement method for time series based on the image fusion of different forms. • The time domain and frequency domain information of time series are represented by recurrence plots and wavelet scalogram. • The characteristics of the image fused by recurrence plots and wavelet scalogram are extracted by Convolutional Neural Network to measure the similarity of time series. • 30 sets of time series show the validity and superiority of the proposed method. • The reasons for the effectiveness of the proposed method are analyzed in detail. [ABSTRACT FROM AUTHOR]

The fault management process is facilitated by equipping power distribution systems with automated devices, especially remote-controlled switches (RCSs). Although RCS plays a key role in improving the system's reliability, it imposes significant investment, installation, and maintenance costs. Hence, RCSs should be optimally located in distribution feeders for the highest profit. In previous works, reliability-oriented mathematical optimization models have been formulated to reach this goal. However, the number of test solutions exponentially grows with the problem size to find the globally optimal solution. This paper uses machine learning to propose a scalable and easy-to-implement model for optimal switch placement in real power distribution systems. At first, the features of candidate points for installing RCSs are introduced. Then, a learning model is applied to deeply explore the relationship between these features and optimal locations for installing RCSs. After training, the learning-based surrogate model directly determines the optimal RCS placement strategy in real power distribution systems by leveraging knowledge gained from past experiences. Simulation results demonstrate that the proposed surrogate model is approximately 29 times faster than the integer programming-based mathematical model, without a significant loss of accuracy, when implemented on a modified 11 kV network connected to Bus 4 of the Roy Billiton test system. This paper also employs explainable artificial intelligence (XAI) tools to select the most important features, where the Hamming loss is decreased by approximately 5%. [ABSTRACT FROM AUTHOR]

The aging of transformer oil–paper insulation (OPI) contributes significantly to transformer failure and reduced service life. Therefore, reliable and precise evaluation of the aging status of transformer insulation condition is crucial in averting transformer failure. This article proposes a novel deep learning-based aging condition assessment technique of transformers by combining FDS (Frequency domain spectroscopy) and squeeze-and-excitation-enabled convolutional neural network (SE-CNN). An intrinsic product-based transformation using a Gramian angular field is employed to dimensionally expand the 1-D FDS signals acquired from OPI samples into 2D images. Each feature extraction channel in the SE-CNN-based aging diagnosis framework is assigned a specific weight, enabling the model to concentrate on the most relevant features while discarding the rest. Furthermore, the effect of moisture variation on the model is also considered under three different conditions to ensure the consequence of compensation for the effect of moisture on FDS. A comparison with traditional machine learning models has also been made to ensure the viability of the proposed methodology. Additionally, the model is interpreted by creating explanation weight plots of the feature coefficients using a local interpretable model-agnostic explanations (LIME) method in Python. The findings demonstrate that the novel aging diagnosis framework visually assesses the transformer's OPI system with the highest accuracy of 99.12% in the test set. Furthermore, it facilitates the automatic extraction of deep aging features, thus providing better generalization and convergence capability. [ABSTRACT FROM AUTHOR]

Pavement crack detection continues to be a stubborn problem given the interference of various factors in the actual pavement and the complex topological structure of asphalt pavement. Among all the obstacles, the bottleneck of pavement crack detection lies in the difficulty of segmenting the cracks in the pavement images whose edges are blurred. This paper proposes a multi-scale context feature and cross-attention based on convolutional neural network for accurate and robust pavement crack segmentation. The multi-scale context feature module is built in different deep networks to extract rich crack feature information. Subsequently, in order to effectively promote the seamless integration of features at different levels, we deploy cross-attention modules to each branch. After that, we add deep supervision to each branch to accelerate training. Finally, we integrate the outputs of each branch to obtain the final output diagram. The comparative experiments on various pavement datasets show that the method has better robustness. At the same time, this paper designs a complete system of pavement crack detection (PCD) and develops corresponding engineering application software. The PCD system can record the real-time pavement image data to the edge server, and the client can also monitor the real-time pavement images from the edge server through the HTTP protocol. • A multi-scale context feature module is built in the deep layer for robustly describing the size of the cracks. • The cross-attention module enables each pixel to select individual contextual information from multi-layer features. • In-depth supervision is added to each branch to accelerate the optimization of the network model. • An intelligent detection application software is designed and integrated with multiple modules. [ABSTRACT FROM AUTHOR]

This paper proposes a real-time state of health (SOH) estimation model based on a deep learning (DL) framework. The proposed model is a combination of two different architectures; a one-dimensional convolutional neural network (1D-CNN) and a bidirectional gated recurrent unit (BiGRU). The hybrid CNN-BiGRU uses the 1D CNN layers to extract pertinent features from input data and then relies on the BiGRU layers for sequence learning in both directions. To account for all SOH indicators, the proposed approach uses the current, voltage, and temperature measurements, which are readily obtainable from the electric vehicle's battery management system (BMS). This prevents the complex and time-consuming feature extraction used in most related papers. Since the hyperparameters have a significant impact on the performance of neural network models, a Bayesian Optimization (BO) technique based on the Gaussian Process (GP) was considered to tune the CNN-BiGRU model hyperparameters. Accordingly, the objective function was able to converge to a low Mean Squared Error (MSE) of 1.2 × 10 − 5 in just 19 iterations. Afterward, to verify the accuracy of the optimized model, a Lithium-ion battery dataset with several discharge profiles provided by the National Aeronautics and Space Administration (NASA) was used. The obtained results demonstrated the accuracy and robustness of the proposed method compared to other commonly used models. The CNN-BiGRU model yielded a Mean Absolute Error (MAE) of 2.080% and a root-mean-square error (RMSE) of 2.516% in the case of the battery set #C, referring to a set with 70 cycles already used at 24 °C. Additionally, the End of life (EOL) indicator error of zero cycles for the same data. [ABSTRACT FROM AUTHOR]

Fruit ripeness estimation models have for decades depended on spectral index features or colour-based features, such as mean, standard deviation, skewness, colour moments, and/or histograms for learning traits of fruit ripeness. Recently, few studies have explored the use of deep learning techniques to extract features from images of fruits with visible ripeness cues. However, the blackberry (Rubus fruticosus) fruit does not show obvious and reliable visible traits of ripeness when mature and therefore poses great difficulty to fruit pickers. The mature blackberry, to the human eye, is black before, during, and post-ripening. To address this problem, this paper proposes a novel multi-input convolutional neural network (CNN) ensemble classifier (MCE) for detecting subtle traits of ripeness in blackberry fruits. The multi-input CNN was created from a pre-trained visual geometry group 16-layer deep convolutional network (VGG16) model trained on the ImageNet dataset. The fully connected layers were optimized for learning traits of ripeness of mature blackberry fruits. The resulting model of about 600 K trainable parameters served as the base for building homogeneous ensemble learners t ensembled using the stack generalization ensemble (SGE) framework. The input to the network are images acquired with a stereo sensor using visible and near-infrared (Vis-NIR) spectral filters at wavelengths of 700 nm and 770 nm. Through experiments, the proposed model achieved 95.1 % accuracy on unseen sets and 90.2 % accuracy with in-field conditions. Further experiments reveal that machine sensory is highly and positively correlated to human sensory over blackberry fruit skin texture • Fruits that do not show obvious and reliable visible traits of ripeness when mature pose great difficulty in fruit ripeness estimation. • Hyperspectral imaging was investigated for evaluating blackberry fruit ripeness and determining the spectral wavelengths that optimally correspond to ripeness in mature blackberry fruit. • A novel multi-input convolutional neural network (CNN) ensemble classifier was proposed for detecting subtle traits of ripeness in images of blackberry fruits. • The proposed model with blackberry fruit images captured at spectral wavelengths of 700 nm and 770 nm achieved an 11.3 % and 16.2 % increase in test accuracies compared to the RGB-based model. • Bi-spectral imagery significantly improves the ripeness-sensing capabilities of deep learning techniques. • Machine sensory has a ripeness sensing capability that is highly and positively correlated to human sensory over blackberry fruit skin texture. [ABSTRACT FROM AUTHOR]

Advances in deep learning and convolutional neural networks (ConvNets) have driven remarkable face recognition (FR) progress recently. However, the black-box nature of modern ConvNet-based face recognition models makes it challenging to interpret their decision-making process, to understand the reasoning behind specific success and failure cases, or to predict their responses to unseen data characteristics. It is, therefore, critical to design mechanisms that explain the inner workings of contemporary FR models and offer insight into their behavior. To address this challenge, we present in this paper a novel template-inversion approach capable of reconstructing high-fidelity face images from the embeddings (templates, feature-space representations) produced by modern FR techniques. Our approach is based on a novel Deep Face Decoder (DFD) trained in a regression setting to visualize the information encoded in the embedding space with the goal of fostering explainability. We utilize the developed DFD model in comprehensive experiments on multiple unconstrained face datasets, namely Visual Geometry Group Face dataset 2 (VGGFace2), Labeled Faces in the Wild (LFW), and Celebrity Faces Attributes Dataset High Quality (CelebA-HQ). Our analysis focuses on the embedding spaces of two distinct face recognition models with backbones based on the Visual Geometry Group 16-layer model (VGG-16) and the 50-layer Residual Network (ResNet-50). The results reveal how information is encoded in the two considered models and how perturbations in image appearance due to rotations, translations, scaling, occlusion, or adversarial attacks, are propagated into the embedding space. Our study offers researchers a deeper comprehension of the underlying mechanisms of ConvNet-based FR models, ultimately promoting advancements in model design and explainability. [Display omitted] Above are example DFD reconstructions of selected sample images from a single subject in the LFW dataset. The rows display: original images (1st row), reconstructions from the VGG–VGG DFD model (2nd row), reconstructions from the VGG–ResNet DFD model (3rd row), reconstructions from the ResNet–VGG DFD model (4th row), and reconstructions from the ResNet–ResNet DFD model (5th row). • We present the Deep Face Decoder (DFD), a novel SOTA template inversion technique. • We use DFD to analyze the embedding space of ConvNet face recognition (FR) models. • Using DFD we explore the impact of image perturbations, occlusions, and attacks. • Through DFD we investigate various template construction strategies. [ABSTRACT FROM AUTHOR]

The high-power direct current (DC) charging method for electric vehicles (EVs) can easily lead to overheating during the charging process, potentially resulting in thermal runaway accidents. Addressing this safety issue, accurately and reliably predicting and providing multi-level warnings for thermal accidents during the charging of EVs becomes an urgent problem to be solved. Therefore, this paper proposes the construction of a composite prediction model called QCNB (Q learning- CNN- BiNLSTM- BiGRU) by incorporating Convolutional Neural Networks (CNN), Bidirectional Nested Long Short-Term Memory Networks (BiNLSTM), Bidirectional Gated Recurrent Units (BiGRU), and Q learning algorithm. First, taking into account the influence of ambient temperature, three sets of charging historical data from spring/autumn, summer, and winter are selected for model training. Second, the sliding window analysis method is employed to establish the multi-level warning thresholds and rules using the historical charging data, enabling safety multi-level warnings. Lastly, the effectiveness of the QCNB model is validated using actual operating data from EVs. The experimental results demonstrate that the QCNB's prediction accuracy outperforms other models, and the abnormal characteristics of temperature and voltage can be used to identify the charging thermal runaway accident in advance and adopt the multi-level warning for effective protection. This achievement emphasizes the importance of employing a combination of predictive models to address the complex characteristics of charging data and implementing multi-level warnings for protection. It effectively mitigates the occurrence of thermal accidents, offering promising possibilities for practical applications. [ABSTRACT FROM AUTHOR]

Implementing key engineering solutions to optimise the operation of energy industries requires daily electricity demand forecasting and including uncertainty, to promote markets insight analysis as part of their strategic planning, regulating and supplying electricity to consumers. This paper proposes hybrid artificial intelligence models combining convolutional neural networks (CNN) as a feature extraction algorithm with extreme learning machines (ELM) as a framework to predict electricity demand with confidence intervals generated by Kernel Density Estimation (KDE) approaches. In order to develop CELM-KDE model, time-lagged series of daily electricity demand with local climate variables based on the air temperature, atmospheric vapour pressure, evaporation, solar radiation, humidity and sea level pressures are used to train the proposed CELM-KDE hybrid model. In order to fully evaluate the newly developed model from a point-based, as well as a probabilistic prediction strategy, the observed and predicted electricity demand as well as the probability distribution of errors are analysed using KDE method that operates without prior data distribution assumptions. Based on observed and predicted electricity demand and the relevant probabilistic confidence intervals generated by the CELM-KDE model, the final results show that the proposed method attains significantly better probability interval predictions than traditionally-used point-based models. The proposed CELM-KDE model is demonstrated to be highly effective in providing a comprehensive coverage of predicted errors, as well as providing greater insights into the average bandwidth and detailed predicted electricity demand in the testing stage. The results also indicate that the proposed hybrid model is a reliable decision support tool to develop engineering solutions in area of energy modelling, monitoring and forecasting, which could potentially be useful to the industry policymakers. We show that the point-and probabilistic-based electricity demand predictive models can be employed as an effective tool to improve accuracy of forecasting and provision of insights for national electricity markets and key energy industry stakeholder application tools. [Display omitted] • A hybrid deep learning model for daily electricity demand prediction is proposed. • A Kernel Density Estimation is applied to derive probabilistic prediction intervals. • The algorithm quantifies uncertainties in point-based electricity demand predictions. • The proposed model outperforms deep learning models using statistical uncertainty indicators. [ABSTRACT FROM AUTHOR]

3D face reconstruction and dense alignment play a key role in digital humans. Learning-based approaches have been developed by employing cascaded convolutional neural networks for feature extraction and spatial-semantic relationship construction. However, these networks often struggle with accurately capturing the complex relationships among various facial components and attributes, particularly in unconstrained environments. Furthermore, the mappings modeled by convolutional weights are at low levels, which are usually implicit and local, and lack global discrimination. In this paper, we propose a global and instance-level perceived relationship matrices-based network (PRMNet) to recover high-fidelity 3D faces and perform accurate dense alignment in unconstrained environments. Specifically, a Key Information Extraction Module (KIEM) extracts crucial features from global and local feature banks. This reduces reasoning costs while maintaining high perceptual quality. Learnable global and instance-level perceived feature relationship matrices are then integrated into the Feature Relationship Reasoning Module (FRRM). This calibration combines key information from both the global macroscopic and sample-specific microscopic views, allowing accurate construction of spatial-semantic relationships to harvest both global discrimination and local relevance. Finally, we introduce a Spatial Reasoning and Guidance Module (SRGM), designed to recalibrate the joint weight responses of feature extraction and its enhancement paths using various attention mechanisms, thereby further enhancing global discrimination ability. Extensive quantitative and qualitative experiments on the benchmark datasets show that our PRMNet outperforms the state-of-the-art. Codes and all resources will be publicly available at https://github.com/Ray-tju/PRMNet. [ABSTRACT FROM AUTHOR]

Several studies have focused on generating seismic vulnerability maps for earthquake-prone areas, particularly in Indonesia. Building typologies are a key factor in determining vulnerability to earthquakes. However, conducting large-scale field surveys to determine the spatial distribution of building typologies in a city is uneconomical. This paper explores the use of a convolutional neural network (CNN) to automatically detect building typologies from diverse regions in Indonesia, utilizing both conventional and automated building image acquisition processes. In this study, datasets from three distinct image acquisition methods are trained with four unique CNN architectures to identify the best-performing model to classify building typologies. The sample size effect on CNN performance is also investigated. The results showed that randomly sampled Google Street View (GSV) images are the most effective dataset for the CNN model, achieving an f1-score of 84.33%. Among the network architectures tested, MobileNet demonstrated superior performance on the majority of evaluated datasets. As the sample size increases by about 350% in the dataset, there is a positive correlation with up to 2.3% f1-score improvement. Using the best-performing CNN model, two building vulnerability models were employed to assess the spatial distribution of building damage in the urban area of Bandung, considering a hypothetical scenario of an M7 earthquake. Incorporating local construction data, one of the generated maps estimated that approximately 55% of buildings in Bandung would experience moderate to severe structural damage. This study showcases the potential of CNN models in automating regional seismic assessments and providing valuable insights for comprehensive seismic mitigation strategies. • A decision tree to label building typology objectively in Indonesia was developed. • CNN model trained with randomly sampled GSV dataset outperforms other models. • The CNN and probability model generated a city-scale building vulnerability map. [ABSTRACT FROM AUTHOR]

This paper addresses the problem for fine-grained video super-resolution (FGVSR) to suppress temporal flickering caused by separately processed consecutive frames and enhance the quality of restored video frame details when upsizing videos. Some existing video SR methods fail to sufficiently utilize spatial-temporal information from input low-resolution (LR) videos, while others may generate undesirable artifacts or cannot well reconstruct image details. To overcome these problems, we present a novel deep learning framework for FGVSR, which takes a set of consecutive LR video frames and generate the corresponding super-resolved frames. Our deep FGVSR framework focuses on reconstructing missing information from the LR sources based on the proposed multi-frame alignment and refinement strategies. More specifically, we propose an alignment module, where multiple frames are aligned at feature level, to prevent the output videos from flickering. Then, we introduce a feature fusion module, where aligned features generated from our alignment module are fused and refined in a multi-scale manner. Finally, the proposed refinement module is used to reconstruct missing information based on the fused features. In addition, we also embed an image enhancement module on the skip connection from the input layer to the output layer of our network for further enhancing the SR results. Experimental results show that the proposed deep FGVSR, compared with existing deep learning-based VSR methods, achieves state-of-the-art performances on the three well-known benchmarks, including REDS, Vid4, and Vimeo90k. More specifically, compared with the state-of-the-art VSR methods in our experiments, our FGVSR achieves quantitative improvements from 0.70 dB to 9.54 dB in PSNR. On the other hand, our method has also been shown to be efficient to other image restoration tasks, such as image inpainting. • End-to-end trainable deep learning-based fine grained video super-resolution model is proposed. • The temporal dependency is implicitly obtained without performing explicit motion compensation. • Image details are reconstructed and further sharpened by our refinement and enhancement modules. • Adaptively integrated into other deep image restoration model for further boosting performances. [ABSTRACT FROM AUTHOR]

Multi-view stereo (MVS) reconstruction is a key task of image-based 3D reconstruction, and deep learning-based methods can achieve better results than traditional algorithms. However, most of the current deep learning-based MVS methods use convolutional neural networks (CNNs) to extract image features, which cannot achieve the aggregation of long-distance context information and capture robust global information. In addition, in the process of fusing depth maps into point clouds, the confidence filters will filter out the depth values with low confidence in weak texture areas. These problems will lead to the low completeness of 3D reconstruction of weak texture and texture-less areas. To address the above problems, this paper proposes SA-MVSNet based on the PatchmatchNet with a self-attentive mechanism. First, we design a coarse-to-fine network framework to advance depth map estimation. In the feature extraction network, a module with a pyramid structure based on Swin Transformer Block is used to replace the original Feature Pyramid Network (FPN), and the self-correlation between weak texture areas is enhanced by applying a global self-attention mechanism. Then, we also propose a self-attention-based adaptive propagation module (SA-AP), which applies a self-attention calculation within depth value propagation window to obtain the relative weight values of current pixel and others, and then adaptively samples the depth values of neighbors on the same surface for propagation. Experiments show that SA-MVSNet has significantly improved the completeness of 3D reconstruction for the images with weak texture on DTU (provided by Danish Technical University), BlendedMVS, and Tanks and Temple datasets. Our code is available at https://github.com/miaowang525/SA-MVSNet. • SA-MVSNet proposed a multi-scale feature extraction module based on Swin Transformer. • A global self-attention mechanism is applied to enhance the self-correlation between weak textured areas. • SA-MVSNet proposed a self-attention-based adaptive propagation module (SA-AP). • Apply window self-attention mechanism to enhance the confidence of depth value. • SA-MVSNet has significantly improved the completeness of 3D reconstruction on three datasets. [ABSTRACT FROM AUTHOR]

Deep learning methods have achieved a lot of success in various applications involving converting wearable sensor data to actionable health insights. A common application areas is activity recognition, where deep-learning methods still suffer from limitations such as sensitivity to signal quality, sensor characteristic variations, and variability between subjects. To mitigate these issues, robust features obtained by topological data analysis (TDA) have been suggested as a potential solution. However, there are two significant obstacles to using topological features in deep learning: (1) large computational load to extract topological features using TDA, and (2) different signal representations obtained from deep learning and TDA which makes fusion difficult. In this paper, to enable integration of the strengths of topological methods in deep-learning for time-series data, we propose to use two teacher networks — one trained on the raw time-series data, and another trained on persistence images generated by TDA methods. These two teachers are jointly used to distill a single student model, which utilizes only the raw time-series data at test-time. This approach addresses both issues. The use of KD with multiple teachers utilizes complementary information, and results in a compact model with strong supervisory features and an integrated richer representation. To assimilate desirable information from different modalities, we design new constraints, including orthogonality imposed on feature correlation maps for improving feature expressiveness and allowing the student to easily learn from the teacher. Also, we apply an annealing strategy in KD for fast saturation and better accommodation from different features, while the knowledge gap between the teachers and student is reduced. Finally, a robust student model is distilled, which can at test-time uses only the time-series data as an input, while implicitly preserving topological features. The experimental results demonstrate the effectiveness of the proposed method on wearable sensor data. The proposed method shows 71.74% in classification accuracy on GENEActiv with WRN16-1 (1D CNNs) student, which outperforms baselines and takes much less processing time (less than 17 sec) than teachers on 6k testing samples. [ABSTRACT FROM AUTHOR]

The aim of this paper is to investigate emotion recognition using a multimodal approach that exploits convolutional neural networks (CNNs) with multiple input. Multimodal approaches allow different modalities to cooperate in order to achieve generally better performances because different features are extracted from different pieces of information. In this work, the facial frames, the optical flow computed from consecutive facial frames, and the Mel Spectrograms (from the word melody) are extracted from videos and combined together in different ways to understand which modality combination works better. Several experiments are run on the models by first considering one modality at a time so that good accuracy results are found on each modality. Afterward, the models are concatenated to create a final model that allows multiple inputs. For the experiments the datasets used are BAUM-1 ((Bahçeşehir University Multimodal Affective Database - 1) and RAVDESS (Ryerson Audio–Visual Database of Emotional Speech and Song), which both collect two distinguished sets of videos based on the different intensity of the expression, that is acted/strong or spontaneous/normal, providing the representations of the following emotional states that will be taken into consideration: angry, disgust, fearful, happy and sad. The performances of the proposed models are shown through accuracy results and some confusion matrices, demonstrating better accuracy than the compared proposals in the literature. The best accuracy achieved on BAUM-1 dataset is about 95%, while on RAVDESS it is about 95.5%. [Display omitted] • Emotion recognition through multimodal architectures. • Comparison of early to late fusion of 1-input models. • Synchronization of video and audio channels. [ABSTRACT FROM AUTHOR]

Understanding thermal dynamics and obtaining the computational model of residential buildings enable its scaled application in energy retrofits, control optimization and decarbonization. In this paper, we present a deep learning approach to model building thermal dynamics with smart thermostat data collected from residential buildings, with the goal to investigate model generalizability. In the first stage, we developed and compared different Deep Learning architectures including Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) models and CNN-LSTM to predict indoor air temperature in a multi-step time horizon. In the second stage, we implemented a Transfer Learning (TL) process, which aims to improve the prediction performance on a new set of buildings (targets), exploiting the knowledge of related or similar buildings (sources). Different TL strategies and source model identification methods were investigated. The study showed that the CNN-LSTM performed the best among the architectures compared, with an average Mean Absolute Error (MAE) of 0.26 °C for one-hour-ahead (twelve 5-min future steps) predictions. Furthermore, the results showed that freezing the LSTM layer and fine-tuning the other layers of the CNN-LSTM achieved the best performance among four TL strategies, which further improved the performance with respect to a machine learning approach by 10%, and proving the effectiveness and generalizability of the proposed approach. A comparison of three different source model identification methods showed that randomly selecting source models constrained by similar building characteristics can provide good TL performance while retaining simplicity comparing with other quantitative source identification methods. • Investigated deep transfer learning techniques for residential thermal dynamics using smart thermostat data. • Compared diverse neural network architectures for indoor temperature prediction. • Explored transfer learning to enhance model accuracy. • Evaluated four transfer learning methods and three source model identification approaches. • Achieved high accuracy and generalizability with the proposed deep transfer learning approach. [ABSTRACT FROM AUTHOR]

Data augmentation is a powerful and widely used technique to improve the generalization of convolutional neural networks. The majority of data augmentation methods emphasize transforming samples to bolster network robustness. However, these methods overlook learning the invariance between the samples augmented by different transformations, impairing the quality of learned representations and generalization of the model. In this paper, we introduce a straightforward yet highly effective method called Consistent Augmentation (CA) to compel networks to learn explicitly the invariance of transformations. Specifically, CA minimizes the bidirectional KL divergence between the predicted distributions of different variants generated from the same samples using different transformations, explicitly enforcing consistency among various semantics-preserving transformations. Implementing CA is intuitive, uncomplicated, and does not introduce additional computational costs during inference. The validation experiments performed on conventional and fine-grained classification tasks demonstrate that CA can be employed across various network architectures and consistently improve their generalization ability. Notably, convolutional neural networks benefit significantly on fine-grained classification tasks, with an improvement of 5.18% and 6.61% compared with other augmentation methods in accuracy observed for the ShuffleNetV2 model on the Stanford Dogs and CUB-200-2011 datasets. Compatibility experiments conducted on the CIFAR-100 dataset reveal that CA improves the accuracy of the Mixup and CutMix methods by 1.25% and 2.11%, respectively. These results underscore the effectiveness of our approach when employed alongside mix-based augmentation methods, further enhancing the capabilities of the model. [ABSTRACT FROM AUTHOR]

Induction motors are widely used in various industrial applications due to their simplicity, robustness, and high efficiency. In recent years, deep learning approaches have shown great potential for detecting and diagnosing faults in induction motors. However, it was observed that the convolutional neural network (CNN) might miss the crucial long-range temporal dependencies of the significant features. At the same time, the recursive neural network (RNN) may face issues such as vanishing gradient and less computational efficiency if the data is too long with complex patterns. This paper proposes a novel deep-learning model that combines a one-dimensional CNN and an RNN into two independent pipelines. The spatial, temporal, long-term, and short-term features can be extracted from the raw input signal. A multi-head mechanism, a state-of-the-art deep learning technique in natural language processing (NLP), enables the model to focus more on relevant features and prevent overfitting. Specifically, our method simplifies the conventional fault diagnosis workflow while maintaining competitive accuracy compared to several state-of-the-art methods. By leveraging the ability to detect faults directly from raw, unprocessed electrical signals, our approach offers a practical, cost-effective solution for real-time fault detection in industrial applications. This innovation has the potential to significantly reduce downtime and the associated operational costs, thereby providing immediate practical value in various industrial settings. [ABSTRACT FROM AUTHOR]

Urban scene image segmentation is an important research area in high-resolution remote sensing image processing. However, due to its complex three-dimensional structure, interference factors such as occlusion, shadow, intra-class inconsistency, and inter-class indistinction affect segmentation performance. Many methods have combined local and global information using CNNs and Transformers to achieve high performance in remote sensing image segmentation tasks. However, these methods are not stable when dealing with these interference factors. Recent studies have found that semantic segmentation is highly sensitive to frequency information, so we introduced frequency information to make the model learn more comprehensively about different categories of targets from multiple dimensions. By modeling the target with local features, global information, and frequency information, the target features can be learned in multiple dimensions to reduce the impact of interference factors on the model and improve its robustness. In this paper, we consider frequency information in addition to combining CNNs and Transformers for modeling and propose a Multidimensional Information Fusion Network (MIFNet) for high-resolution remote sensing image segmentation of urban scenes. Specifically, we design an information fusion Transformer module that can adaptively associate local features, global semantic information, and frequency information and a relevant semantic aggregation module for aggregating features at different scales to construct the decoder. By aggregating image features at different depths, the specific representation of the target and the correlation between targets can be modeled in multiple dimensions, allowing the network to better recognize and understand the features of each class of targets to resist various interference factors that affect segmentation performance. We conducted extensive ablation experiments and comparative experiments on the ISPRS Vaihingen and ISPRS Potsdam benchmarks to verify our proposed method. In a large number of experiments, our method achieved the best results, with 84.53% and 87.3% mIoU scores on the Vaihingen and Potsdam datasets, respectively, proving the superiority of our method. The source code will be available at https://github.com/JunyuFan/MIFNet. [ABSTRACT FROM AUTHOR]

Most pavement crack detection methods based on deep learning rely too much on pixel-wise labels, and are facing with sample imbalance problem. This paper proposes a three-stage pavement crack detection framework with positive sample augmentation by an autoencoder-deep convolutional generative adversarial network (E-DCGAN). Positive samples are the images labeled as crack in the training datasets. Firstly, crack regions are located using graph based on visual saliency (GBVS). Secondly, a convolutional neural network (CNN) model is designed to recognize crack pixels in the located crack regions. The E-DCGAN is explored to generate crack images, which are then added as augmented positive samples in the training dataset of the CNN model. The trained CNN model takes the located crack regions as inputs, outputs the recognized crack pixels. Finally, fine crack detection is realized through region growing. Crack contours are completely depicted pixel-wise based on the recognized crack pixels and the Breadth-first search. Experiments are carried out on two benchmark datasets and in practical applications. The results showed that the framework could detect cracks at the pixel level. Its performance is improved compared with standard fully convolutional network (FCN)-based and U-shaped CNN (U-Net)-based crack detection methods. In addition, the standard FCN-based and U-Net-based methods require accurate pixel-wise annotations, while the proposed framework only requires image-wise annotations. It can be concluded that the proposed framework not only effectively avoids the dependence on pixel-level annotations, improves the sample imbalance issues, but also achieves competitive detection performance over the FCN and U-Net based methods. • Proposed a crack image augmentation method based on a novel GAN model. • Designed a three-stage detection framework to locate cracks in pixel from coarse to fine. • Verified necessity of the augmentation and feasibility of the proposed framework. [ABSTRACT FROM AUTHOR]

Traditional machine learning methods require the assumption that training and test data are drawn from the same distribution, which proves challenging in real-world applications. Moreover, deep learning models require a substantial amount of labeled data for training in classification tasks and limited samples may lead to overfitting. In many real-world scenarios, there is an insufficient supply of labeled samples within the target domain for learning. Transfer learning offers an effective solution, allowing knowledge from a source domain to be transferred to a target domain. Additionally, data augmentation enhances model generalization by increasing data samples, particularly beneficial when dealing with limited target domain data. In this paper, we synergistically enhance the model's performance on classification tasks by integrating transfer learning techniques with a data augmentation strategy. By conducting numerous experiments across various datasets, we verified the effectiveness of our proposed approach. [ABSTRACT FROM AUTHOR]

In recent years, learned image compression has witnessed significant advancements. However, many existing learned image compression models predominantly rely on Convolutional Neural Networks (CNNs) and predicting the distribution of latent features to use ever more expensive entropy models, which cannot fully capture detail redundancy and lead to prohibitively slow decoding. To address this challenge, this paper introduces novel image compression models, MGIC and RMGIC, which consist of the Residual Neighborhood-based Attention Module (RNAM), Multi-Scale Guide Entropy Model (MGEM) and Refined Multi-Scale Guide Entropy Model (RMGEM). Specifically, Neighborhood Attention (NA) is combined with global feature learning to construct the RNAM, which can accurately capture the correlation of spatial neighborhoods and is more suitable for image compression. The proposed RNAM is flexible and could work as a plug-and-play component to enhance CNN-based models. Additionally, MGEM employs two mask convolutions with two kernel sizes to leverage multi-scale information for context modeling and eliminating the effect of parallel decoding on model performance. Building upon MGEM, RMGEM incorporates a refined two-stage spatial context modeling technique to enhance the utilization of spatial neighborhoods and achieve more precise entropy prediction. Furthermore, MGIC and RMGIC are derived by integrating RNAM, MGEM, and RMGEM into the variational autoencoder framework. The proposed methods can save more bandwidth and improve the quality of information storage. Extensive experimental evaluations have shown that the proposed method is effective and achieves state-of-the-art performance. • Achieving a balance between file size reduction and image quality preservation. • Neighborhood-based Attention is more suitable for image compression. • Improving entropy model by multi-scale guiding. • Accurate feature capture provides solid coding foundation for entropy model. [ABSTRACT FROM AUTHOR]

This paper introduces a novel integrated microstructure design methodology that replaces the common existing design approaches for multifunctional composites: 1) reconstruction of microstructures, 2) analyzing and quantifying material properties, and 3) inverse design of materials. The problem of microstructure reconstruction is addressed using the diffusion-based generative model (DGM), which is a state-of-the-art generative model formulated with a Markovian diffusion process. Then, the conditional formulation of DGM is introduced for guidance to the embedded desired material properties with a transformer-based attention mechanism, which enables the inverse design of multifunctional composites. Furthermore, a convolutional neural network (CNN)-based surrogate model is utilized to facilitate the prediction of linear/nonlinear material properties for building microstructure-property linkages. Combined, the proposed artificial intelligence-based design framework enables large data processing and database construction that is often not affordable with resource-intensive finite element method (FEM)-based direct numerical simulation (DNS) or iterative reconstruction methods. It is worth noting that the proposed DGM-based methodology is not susceptible to unstable training or mode collapse, which are common issues in generative models that are often difficult to address even with extensive hyperparameter tuning. An example case is presented to demonstrate the effectiveness of the proposed approach, which is designing mechanoluminescence (ML) particulate composites. The results show that the designed ML microstructure samples with the proposed generative and surrogate models meet the multiple design requirements (e.g., volume fraction, elastic constant, and light sensitivity). This assessment demonstrates that the proposed integrated methodology provides an end-to-end solution for practical material design applications. [Display omitted] • An integrated microstructure design framework utilizing the diffusion-based generative model (DGM) is proposed. • The problem of microstructure reconstruction is addressed using the DGM with forward/reverse Markovian diffusion processes. • The conditional formulation of DGM is introduced to enable the inverse design of multifunctional composites. [ABSTRACT FROM AUTHOR]

In a large skewed dataset, the data imbalance is severe and the classifier's accuracy is biased towards the majority class. Insufficient data makes it challenging for the classifier to learn the feature of the minority classes. Moreover, some existing techniques of binary classification cannot directly apply to multi-classification. Plain oversampling may generate redundant and irrelevant data for the minority classes while undersampling may eliminate too many features for the majority class. This paper proposes a combination of constrained undersampling, oversampling, noise cleaning, and a weighted majority voting classifier (WMVC) to detect multi-classification obfuscated malware via memory and network traffic anomalies. As a constraint on undersampling and oversampling, the proposed framework divides the total sample number by the class number to obtain the average class size. According to the constraint, one or more majority classes are down-sampled using Random Undersampling, while all the minority classes are up-sampled using Adaptive Synthetic Sampling Approach, which is followed by Tomek Link to remove the noisy data. Then a weighted majority voting classifier aggregated tree-based ensemble algorithms is designed and compared to the XGBoost and Convolutional Neural Network (CNN) classifiers and six state-of-the-art ensemble algorithms. The comparison results show that the performance of the classifiers on balanced data outperforms those on imbalanced data, and the WMVC outperforms XGBoost, CNN, and the six other ensemble algorithms. Our approach can alleviate the classifier's bias towards the majority class while improving its performance for the difficult minority class. [ABSTRACT FROM AUTHOR]

Remaining useful life (RUL) prediction of aircraft engines is significant in the health monitoring, operation, and maintenance of aircraft. Capturing more comprehensive device degradation trends at different time scales and extracting long-term dependencies effectively among elements in long time series are two challenges in the field of aircraft engine RUL estimation. To address the aforementioned challenges, this paper proposes a novel multiscale Hourglass-Transformer (MHT) aircraft engine RUL prognostics. Specifically, an hourglass-shaped multiscale feature extractor (HME) is designed based on one-dimensional convolutional neural network, which can scale the time sequence into multi-time scales for feature fusion. Then, a transformer network is employed to further extract features from the fused feature map and output the RUL. To enhance inter-scale data attention, a pyramid self-attention mechanism is employed in both the encoder and decoder. Finally, the superiority and effectiveness of this approach are verified on the Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) dataset. Furthermore, the robustness and generalization capability of this method are further validated on New Commercial Modular Aero-Propulsion System Simulation (N-CMAPSS) dataset. • A novel multiscale Hourglass-Transformer is developed for aircraft engine RUL prognostics. • An hourglass-shaped multiscale feature extractor is designed based on one-dimensional convolutional neural network. • A transformer network is employed to further extract features. • Pyramid self-attention mechanism is adopted to reduce space complexity. [ABSTRACT FROM AUTHOR]

Detecting malicious attacks in Industrial Internet of Things (IIoT) is crucial to minimize downtime and financial losses. However, existing deep learning (DL) research faces limitations in reliability and generalization due to reliance on a single dataset. This single dataset approach hinders the effectiveness of the models when applied to new and unseen datasets. This paper addresses the issue using multiple-domain learning, creating a generalized DL framework for IIoT traffic classification. It combines three datasets: Edge-IIoTSet, WUSTL-IIoT-2021, and X-IIoTID. The proposed framework has two stages. The first stage involves the fusion of different datasets into a common feature space. To achieve this, an autoencoder architecture is proposed to match the dimensionality of the datasets into a common feature space. Subsequently, a modified locally linear embedding is used as a manifold alignment to ensure statistical matching of multiple datasets such that the maximum mean discrepancy between multiple datasets is minimized to compare and combine datasets effectively. The second stage presents a hybrid DL model with a convolutional neural network (CNN) and gated recurrent unit (GRU) for IIoT traffic classification. Bayesian optimization fine-tunes hyperparameters. Experiments on combined, single, and cross-datasets show the superiority of the framework. The CNN-GRU model achieved 97.68% accuracy, 97.70% recall, 97.67% precision, and 97.68% F1-score for binary classification on the combined dataset. Transfer learning improved accuracy from 50.95% to 97.80% and F1-score from 48.52% to 97.79% when trained on X-IIoTID and tested on Edge-IIoTSet. Multi-classification accuracy and weighted averages of recall, precision, and F1-score were approximately 98.85%. [ABSTRACT FROM AUTHOR]

The use of machine vision methods has rapidly developed for detecting product defects, but it remains challenging to identify shape and color defects as well as the assembly relationship of parts due to occlusion between components and changes in assembly structures. This paper proposed a method for detecting saw chain assembly defects based on residual networks and knowledge coding to achieve parts segmentation and identification of assembly defects. Firstly, we design an adaptive segmentation algorithm for saw chain parts, which uses the Hough transform to locate rivets and completes the image segmentation based on the inherent assembly features between parts. Secondly, A trained ResNet (Residual Network) based deep learning model was used to identify the individual defects of saw chain parts, in which the transfer learning method was used to improve the detection speed of individual defects. Finally, a parallel inference method based on a knowledge encoding matrix is proposed for detecting part defects and constructing a sparse matrix to locate assembly defects. Evaluating the proposed method in building a saw chain testing platform. Results show that the proposed method can achieve real-time and high-precision detection requirements in different application scenarios. The detection accuracy on individual and assembly defects reaches 94.2% and 89.2%, respectively, which is better than the current state-of-the-art object detection models. Assembly defect detection methods are an important reference for inspecting other types of simple assemblies in the industry. [ABSTRACT FROM AUTHOR]

Visual behavior intention inference is crucial for enabling escort robots to interact naturally with humans, which is very challenging due to the big inner-class similarity and the small intra-class distinguishability of successive actions in the assistive scenario. To attain a reliable behavior intention inference, not only the current state of behaviors is concerned, but also the semantic information in both spatial and temporal domains plays an important role. This paper presents a segmentation–detection–recognition hierarchical system to represent the spatio-temporal semantic features for formulating descriptions of body parts, trajectories and deep relationships of sub-behaviors. Specifically, a dense trajectory matching scheme based on temporal sampling and Binarized Normed Gradients (BING) algorithm is formulated to segment the 3-Dimensional (3D) behavior cubes, based on which, local trajectories are obtained by clustering dense trajectories according to the distance similarity, and the body parts are then detected by multi-kernel learning of the encoded local features. Moreover, a global three-stream context Convolutional Neural Networks (CNN) is proposed for behavior classification by designing a texture module using expansion, connection and 1D convolution implementations. Based on transfer learning, scene information is also recognized efficiently. Finally, the semantic descriptors are modeled by two cascaded And-Or Graphs (AoGs) constraining the spatial scenarios and temporal sequences. Our unified approach is demonstrated on two public benchmarks containing long-term activities and on an escort robot for real-world applications. • A segmentation–detection–recognition hierarchical framework is proposed to infer behavior intentions for natural human–robot visual interaction. • We design an efficient body part detection scheme by segmenting 3D behavior cubes and fusing local features. • We present a new three-stream context CNN capturing global spatio-temporal relationships of sub-behaviors. • An inference model with two cascaded AoGs constraining spatial scenarios and temporal sequences is constructed. [ABSTRACT FROM AUTHOR]

Skin cancer is the most prevalent type of cancer worldwide. Early detection is essential as it could be fatal at later stages. The classification of skin lesions is challenging since there are many variations, including changes in color, shape, size, high intra-class variation, and high inter-class similarity. In this paper, a unique class-wise attention method is proposed that considers each class equally while extracting additional discriminative information of skin lesions. The proposed attention mechanism is employed in a progressive manner to incorporate discriminative feature information from multiple scales. The proposed approach obtained competitive performance against more than 15 state-of-the-art methods including HAM1000 and ISIC 2019 leaderboard winners. The proposed method achieved 97.40% accuracy on the HAM10000 and 94.9% accuracy on the ISIC 2019 dataset. • A novel attention based mechanism is proposed for skin lesions classification. • The problem of significant intra-class variance, high inter-class similarity, and class imbalance is addressed by a class-wise attention mechanism. • The important features for the classification are extracted without the burden of additional parameters using a progressive class-wise attention mechanism. • The final three layers of the baseline model are discarded and the global average pooling (GAP) and classification output layers are added to reduce the number of parameters, while maintaining performance. • The proposed network is robust, end-to-end trainable and has good generalization on unseen data. [ABSTRACT FROM AUTHOR]

The antinuclear antibody (ANA) test plays the role of screening, diagnosis, treatment and prognosis for autoimmune diseases. However, the ANA test is time-consuming, and strongly depends on the interpretation of experienced medical examiners. With the rise of artificial intelligence, a lot of convolutional neural networks (CNN) have been used to recognize the features of ANA images, such as GoogLeNet, ResNet, and MobileNet. By extracting the disease-related features from the entire ANA image, these models achieve acceptable accuracy, but none of them can replace the manual ANA test completely. This is because they ignore the key features of ANA images. Accordingly, this paper aims to design an object detection model that can directly identify the key features of ANA images, i.e., the Metaphase cells. This process is critical in the manual ANA test performed by the medical examiners. Based on the YOLO model, our design first applies various image preprocessing and data augmentation techniques to enhance the critical features in the training data. We modify the original YOLO model, in which the residual block is replaced with SENet module to improve the classification performance by emphasizing critical features and reducing the influence of low-level features. Moreover, we use the different activation function to prevent the modified model from partial judgment. The experimental results show that our design can effectively improve the performance of ANA metaphase identification. Compared to the related CNN models, our design can improve F1-score and mAP by range 5%–14% and 11%–13%, respectively. In particular, the Kappa value of our design is 0.82 that is almost perfect agreement with the ANA experts. [ABSTRACT FROM AUTHOR]

The Internet of Things (IoT) has experienced phenomenal expansion over the past few years and has emerged as one of the most dynamic sectors of the international market. Android has become the driving force behind the rapid development of the IoT, and malware is one of the most severe concerns. Android is an open-source platform with a significant number of users. The identification of Android malware has become more critical in cybersecurity due to the meteoric rise of Android malware over the past few years and the severity of the damage it inflicts on smartphone users. Existing traditional Android malware detection methods based on machine learning have achieved encouraging performance. Still, the solution requires a massive workforce of feature engineering to build dynamic or static features and continuous learning through pre-extracted features to maintain high performance in identifying malware. Therefore, technologies for end-to-end malware detection without human interaction are necessary. In response to the rapid evolution of Android malware, a convolutional neural network that can acquire knowledge without requiring feature extraction is ideally suited. This paper offers a unique classification approach for malware based on a convolutional neural network, with feature extraction selected from the current pre-trained models DenseNet169, Xception, InceptionV3, ResNet50, and VGG16. The proposed methods have an advantage over existing detection methods due to their end-to-end learning process. Our suggested approaches convert Android APK files to binary codes and RGB images for usage as inputs to deep learning models. These models are trained and evaluated using the CICInvesAndMal2019 dataset, which consists of 333 benign and 428 malicious applications. Experiments demonstrate that the proposed approaches achieve a classification accuracy of 95.24%, 95.24%, and 95.83% for DenseNet169, InceptionV3, and VGG16, respectively, on a 761-sample dataset of malware, which is superior to other compared algorithms. Compared to previous methods, our suggested methods are more applicable to Android IoT devices since they do not require manual feature engineering and consume fewer resources. [ABSTRACT FROM AUTHOR]

The Tarim Basin region harbors abundant carbonate reservoirs in which fractures throughout the structures play a crucial role in facilitating the transportation of oil and gas. The degree and distribution of fracture development play a crucial role in achieving a high yield and stable production. Seismic attribute analysis is a widely used and efficacious approach for examining fractures in reservoirs. However, different seismic attributes characterize fractures at different scales, and any single seismic attribute may be of poor quality, making it difficult to describe reservoir channel media comprehensively. In the present study, the seismic attribute volume was sliced from three perspectives to obtain complete geological spatial structural information of the seismic volume. This was accomplished using a shifted window (Swin) transformer module guided by a t-distribution before extracting and capturing the prior information and global contextual cues of the seismic attributes. Additionally, a dual-branch fusion (DBFusion) network that combined a convolutional neural network and a Swin transformer to extract the spatial geological structural features of seismic attributes was used to facilitate a comprehensive integration of local and global contextual information. Through the DBFusion network modeling, the network comprehensively extracted local information and integrated complementary global information. Taking three-dimensional seismic data from a unit in the Tarim Basin as an example, this study explores the fusion of three attributes: coherence, ant tracking, and curvature. When compared to individual attributes, the fused attributes can comprehensively leverage the advantages of all three seismic attributes in reflecting fractures. This results in a more precise representation of fractures of varying sizes in the fused output. It was discovered through comparative experiments that the DBFusion network proposed in this paper exhibits the best performance compared to other models. • The DBFusion network is proposed for fracture attribute fusion. • The geological spatial structure information is extracted from multiple angles. • The T-distribution prior is proposed to extract and capture the prior information. • CNN and Swin Transformer are combined to extract the spatial geological structural features. [ABSTRACT FROM AUTHOR]

Negative emotion detection is challenging during the peak time of the COVID-19 period. Most earlier studies contain individuals' physical health recognition rather than mental health detection, which is a significant concern in facing the COVID-19 situation. Identifying mental health in advance is essential to understand individuals' psychological condition. This paper considers the texts from social media during the pandemic of COVID-19. We propose a novel context-based auto-regressive transformer with bidirectional long short-term memory and a convolutional neural network (Context-ABT-BiLSTM-CNN) model to detect emotions such as abuse, anger, anxiety, depression, disgust, fear, guilt, sadness, and shame on social media. The existing works do not suggest relevant terms to detect suitable context; as a result, there is no scope for detecting emotions. We introduce a novel topic-based text (TBT) with a rule-based permutation (RBP) procedure to extract the relevant text from social media to identify emotions. Random search is suggested to store each input's correlated information and the order of each sequence. We recommend various transformer components to maintain the text sequence, avoid discrepancies during model training, capture the long-distance semantics in bidirectional contexts, and adopt both the permutation and factorization processes to build the model. Moreover, a comparative study is introduced to detect the most dominant emotions on social media during the pandemic and non-pandemic periods. The proposed model with XLNet embeddings surpasses state-of-the-art models for detecting text emotions. The ablation study is conducted to understand the essential components needed for the proposed model. [ABSTRACT FROM AUTHOR]

While image segmentation is crucial in various computer vision applications, such as autonomous driving, grasping, and robot navigation, annotating all objects at the pixel-level for training is nearly impossible. Therefore, the study of unsupervised image segmentation methods is essential. In this paper, we present a pixel-level clustering framework for segmenting images into regions without using ground truth annotations. The proposed framework includes feature embedding modules with an attention mechanism, a feature statistics computing module, image reconstruction, and superpixel segmentation to achieve accurate unsupervised segmentation. Additionally, we propose a training strategy that utilizes intra-consistency within each superpixel, inter-similarity/dissimilarity between neighboring superpixels, and structural similarity between images. To avoid potential over-segmentation caused by superpixel-based losses, we also propose a post-processing method. Furthermore, we present an extension of the proposed method for unsupervised semantic segmentation. We conducted experiments on three publicly available datasets (Berkeley segmentation dataset, PASCAL VOC 2012 dataset, and COCO-Stuff dataset) to demonstrate the effectiveness of the proposed framework. The experimental results show that the proposed framework outperforms previous state-of-the-art methods. [Display omitted] [ABSTRACT FROM AUTHOR]

Ship classification based on machine learning (ML) has proven to be a significant underwater acoustic research direction. One of the critical challenges rests with how to embed domain signal knowledge into ML models to obtain suitable features that highly correlate with the classification and create better predictors. In this paper, a novel ML-based ship classification model, Hierarchical Underwater Acoustic Transformer (HUAT), is proposed to improve the classification performance. Firstly, the Detection of Envelope Modulation on Noise (DEMON) spectra of ship-radiated noise signals are estimated by cyclostationary analysis. The motivation for using a DEMON-based preprocessing scheme is that valuable propeller information can be revealed by exploiting the second-order cyclostationarity of ship-radiated noise signals. Secondly, the useful features of DEMON spectra are enhanced using a multi-head self-attention module, and the potential features of the Mel spectrograms are extracted employing a Convolutional Neural Network (CNN) module. The two kinds of features are fused to provide ship classification patterns. The challenge of feature learning in the deep classification model is reduced by leveraging domain-related classification knowledge. Finally, the Swin Transformer, based on shifted window self-attention mechanism, is used to learn high-level feature representations and conduct ship classification. Experimental results show that the HUAT model achieves excellent classification performance on ship-radiated noise datasets, ShipsEar and DeepShip. And its classification efficiency is better than the model based on traditional Transformer architecture. In addition, the proposed method provides technical support for the underwater intelligent system capable of automatically sensing sailing vessels and recognizing vessel types. • The Detection of Envelope Modulation on Noise (DEMON) spectra of ship-radiated noise signals are estimated by cyclostationary analysis. • A novel feature fusion strategy is proposed for embedding domain signal knowledge into ML models to obtain suitable features highly correlated with the classification. • The valuable features of DEMON spectra and the Mel spectrogram of ship signals are extracted using two different feature extractors. • The Swin Transformer is used to learn high-level feature representations and conduct ship classification. [ABSTRACT FROM AUTHOR]

Sub-synchronous oscillation (SSO) caused by large-scale renewable energy generations can threaten the safe and reliable operation of power systems. In this paper, a dissipating energy-based quaternion feature set convolutional neural network (QFS-CNN) method is proposed to locate the SSO sources quickly and accurately, providing the foundation for isolation of the SSO sources from power systems in time. Firstly, aiming at the difficulty in feature extraction of SSO sources due to the poor observability of power system, a temporal and spatial feature extraction method based on oscillation energy is proposed to characterize the SSO sources information in the form of images with limited measurement data. The temporal and spatial feature images are extracted based on the variation of oscillation energy in time and the direction of energy flow power in space, respectively. Then, the correlation between the oscillation energy and the SSO sources location can be revealed by the feature images with less calculation and storage space. Secondly, a QFS-CNN method based on temporal and spatial feature images for SSO sources location is proposed to address the insufficient labeled data of SSO. The QFS data augmentation technique increases the feature set via meaningful recombination of the existing labeled feature images. Then, the augmentative feature set is used to ensure the model training accuracy of QFS-CNN to improve the location accuracy of SSO sources. Finally, the proposed method is demonstrated and evaluated by a modified IEEE 39-bus wind power generation system. Simulation results show that the method has high accuracy and stronger anti-noise ability in the case of poor system observability and few sample data. [ABSTRACT FROM AUTHOR]

One of the most important features of a reliable power system is its capability to supply the demand continuously. This continuous supply has been maintained by the transient stability of the system against large disturbances. The study of this type of stability is examined through different indicators. One of the common indicators to evaluate the transient stability of the power system is the well-known Critical Clearing Time (CCT) index. Conventional methods for calculating CCT have presented good accuracy, however, their computational cost is very high which makes them not suitable for real-time applications and real large-scale networks. Considering their ability to feature extraction of big data, deep neural networks can be utilized as reliable tools to cover these deficiencies. In this regard, to cover the shortcomings of the conventional methods, this paper proposes a method based on deep Convolutional Neural Networks (CNN) to estimate the CCT index in real-time power system applications. Moreover, to analyze a realistic case, nonlinear limiters of the excitation systems which have a considerable effect on transient stability index are considered in this study. Thanks to the using of several deep layers and the comprehensive established database, the accuracy of proposed method is appropriately high. Numerical studies on IEEE standard networks as well as a real case in Iran Power Grid (IPG) represents the advantages of the proposed method. • A deep-learning-based convolutional neural network is designed to estimate CCT index of power system. • Nonlinear limiters of the excitation system are taken into account. • A real case study in Iran grid is considered for simulation. • As a result, the transient stability of system is accurately determined in on-line manner. [ABSTRACT FROM AUTHOR]

In order to predict the transient emission characteristics from diesel engine accurately and quickly, a novel prediction model, based on temporal convolutional networks (TCN) that incorporates the dilated convolutions and residual connections, was presented in the paper. Firstly, 1800 samples from the World Harmonized Transient Cycle (WHTC) were employed to train and validate the model. A Random Forest algorithm was used to select six top important variables as inputs to reduce the data dimensionality. Then the effect of model hyperparameters on the prediction performance was discussed and the optimal hyperparameter combination was obtained by a particle swarm optimization (PSO) algorithm. The optimized TCN model showed a coefficient of determination value (R2) above 0.972 for training dataset and 0.941 for validation dataset, respectively. The root mean squared error (RMSE) and the mean absolute error (MAE) were relatively low. Finally, the measured data from World Harmonized Steady Cycle (WHSC) was used to test model, and the average R2 value of 0.936 demonstrated that TCN model has excellent robustness and generalization. Moreover, a comparative investigation between TCN model and other advanced algorithms, including BP, GBRT, XGBoost, RNN, LSTM and Transformer, was also conducted. The result showed that TCN model has not only higher accuracy, but also has less computing time. This demonstrates that it is a promising method to predict the emission characteristics of diesel engine. • A novel emission prediction model based on temporal convolutional networks was proposed. • Multiple algorithms including Random Forest and particle swarm optimization were applied to improve model performance. • The comparison study with other algorithms demonstrated TCN model has higher accuracy and less computing time. [ABSTRACT FROM AUTHOR]