Search

Strip steel serves as a crucial raw material in numerous industries, including aircraft and automobile manufacturing. Surface defects in strip steel can degrade the performance, quality, and appearance of industrial steel products. Detecting surface defects in steel strip products is challenging due to the low contrast between defects and background, small defect targets, as well as significant variations in defect sizes. To address these challenges, a two-stage attention-based feature-enhancement network (TAFENet) is proposed, wherein the first-stage feature-enhancement procedure utilizes an attentional convolutional fusion module with convolution to combine all four-level features and then strengthens the features of different levels via a residual spatial-channel attention connection module (RSC). The second-stage feature-enhancement procedure combines three-level features using an attentional self-attention fusion module and then strengthens the features using a RSC attention module. Experiments on the NEU-DET and GC10-DET datasets demonstrated that the proposed method significantly improved detection accuracy, thereby confirming the effectiveness and generalization capability of the proposed method. [ABSTRACT FROM AUTHOR]

Copyright of Systems Engineering & Electronics is the property of Journal of Systems Engineering & Electronics Editorial Department and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Low‐light image enhancement (LLIE) aims to recover high‐ quality images from low‐quality images acquired in dimly illuminated scenes. However, deep learning algorithms often struggle with uneven exposure and obscured texture features. To address these obstacles, we propose a simple but novel Transformer structure for LLIE, called Double Collapse Transformer (DCTFormer), which has the advantage of modeling non‐local self‐attention and capturing long‐range dependencies easily. The core of DCTFormer is multiple CT blocks composed of pixel‐level spaces and channel self‐attention, which can effectively aggregate features in both the space and channel dimensions. Furthermore, we design the Local Processing Unit (LPU) and an Inverted Residual Feed‐Forward Module (IRFFN) to further enhance the model's ability to learn effective features from current local information. DCTFormer adopts a high‐resolution preservation mechanism overall and gradually integrates deep features to achieve intra‐block feature aggregation. Experimental results on existing benchmark datasets demonstrate the superiority of the proposed DCTFormer. [ABSTRACT FROM AUTHOR]

Stress has been identified as one of major causes of health issues. To detect the stress levels with higher accuracy, fusion of multimodal physiological signals is a promising technique. However, there is an asynchrony between physiological signals observed from different perspectives. Exploring the temporal alignment relationship between modalities is helpful to improve the quality of multimodal fusion. This paper proposes an end-to-end multimodal stress detection model based on Bidirectional Cross- and Self-modal Attention (BCSA) mechanism. Specifically, we first construct different feature extractors based on the characteristics of Blood Volume Pulse (BVP) and Electrodermal Activity (EDA) to complete automated temporal feature extraction. Second, cross-modal attention is used to seek the alignment relationship between the two modalities and fully fuse cross-modal information. The self-modal attention is used to attenuate noise and redundant information, highlight important information and obtain salient stress representations. Finally, the stress representations of the two modalities are processed separately, and the mean square error (MSE) is used to narrow the gap between them. Experimental results on the UBFC-Phys dataset and WESAD dataset show that the proposed model can effectively improve the accuracy of stress recognition, and outperforms several state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Registration of deformable objects is a fundamental prerequisite for many modern virtual reality and computer vision applications. However, due to the difficulties of acquiring labeled datasets and the inherent irregular deformation, non-rigid registration for 3D scanner-captured data remains challenging. This paper proposes an unsupervised non-rigid 3D point cloud registration network based on the self-attention mechanism. Specifically, considering the registration as the result of point drifts between the source and target shapes, a Transformer-based encoder-decoder module is utilized to estimate the point displacements. Additionally, a symmetric registration procedure is adopted with regularization loss to manage the regular deformation of points, ultimately producing reasonable registration results for real-world deformable objects. Experiments are conducted on public and synthesized datasets which simulate diversiform non-rigid 2D or 3D deformations. Numerical and qualitative experimental results demonstrate that the proposed network achieves outstanding performance and is robust in scenes with multiple interferences. [ABSTRACT FROM AUTHOR]

Aiming at the problems of overfitting and low classification accuracy due to the scarcity of ore samples in industrial applications of deep learning models for online ore classification, a method combining X ray transmission imaging technology for ore data augmentation and classification is proposed. The method is based on an improved auxiliary classifier generative adversarial network (Enhance based Classification ACGAN gp, EC ACGAN gp), which uses convolutional and continuous residual blocks to construct the network structures of the discriminator and generator. An attention mechanism is introduced to capture detailed ore features and generate high quality samples to expand the original dataset. Simultaneously, the Wasserstein distance function with gradient penalty is used to reconstruct the classification loss function, achieving improved stability of adversarial training and avoiding mode collapse. Finally, an auxiliary classifier is utilized to reconstruct label information for ore sample category prediction. The research results show that the proposed method can accurately predict the ore grade classification, with an accuracy of up to 89. 62%, which is 3. 98% higher than traditional methods. The model generated ore samples demonstrate good generalization performance, significantly improving the generalization of small sample datasets* When tested on SVM, LeNet5, VGGNet, and ResNet, the accuracy is increased by 2. 83%, 2. 36%89%, and 3. 74%, respectively. This method can further be used to enhance the performance of other classification models in ore grade prediction* [ABSTRACT FROM AUTHOR]

Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets.

With the continuous evolution of novel network attacks, traditional Intrusion Detection Systems (IDSs) have commonly employed Deep Neural Networks (DNNs) for intrusion detection. However, the effectiveness of a DNN in this respect is closely related to the quality of the training data set, and large-scale network traffic data are difficult to label accurately. Therefore, some challenges still need to be addressed to detect network attacks. In this paper, we introduce a Time-Space Separable Attention Network (TSSAN) for intrusion detection. TSSAN utilizes depth wise separable convolution and a time-space self-attention mechanism to effectively extract temporal and spatial features. By extracting the common features from the unlabeled data, TSSAN significantly enhanced the detection performance for rare attack types. Experimental evaluations were conducted using UNSW-NB15 and CICIDS-2017 datasets. Meticulous experiments for evaluating the individual components of the model were rigorously carried out using the CICIDS-2017 dataset. In the unsupervised learning experiment, our method achieved 0.86 and 0.92 f1score in the two datasets. In semi-supervised learning, the experiment showed that our method performed significantly better than the traditional deep learning method when the labelled data were gradually reduced.

As global population growth and the use of household appliances increase, residential electricity consumption has surged, leading to challenges in maintaining a balanced electrical load. This surge often results in localized and intermittent power outages, adversely affecting residential electricity reliability and the profitability of power supply companies. Addressing this, we propose a novel CNN-BiLSTM-SA model, combining Convolutional Neural Network (CNN), Bidirectional Long Short-Term Memory (BiLSTM), and Self-Attention (SA) mechanisms, to accurately predict residential electricity consumption. The model integrates temporal feature extraction through the CNN module, correlation capture via BiLSTM, and intrinsic feature analysis using SA mechanisms, enhancing predictive accuracy. Our experimental results demonstrate the model’s precision, outperforming existing methods in key performance metrics. Additionally, ablation studies affirm the synergistic design of CNN-BiLSTM-SA’s network components, contributing to its overall efficacy.

Active learning is a strategy that involves the deliberate selection of training samples with significant discriminative features, to formulate a more efficient learning paradigm. While active learning has seen success in image classification, its application in object detection remains complex due to the multiple candidate object instances in an image and the interference of background regions on these instances. To tackle the challenge of sample selection in active learning for large object recognition datasets, this paper introduces a active learning method based on local feature saliency and posterior probability. Initially, the method selects active learning samples with discriminative saliency by clustering the multi-scale feature maps of the processed samples by self-attention weights. Subsequently, it calculates the posterior entropy based on the posterior probability of the current model’s candidate instances. This is combined with the entropy encoding of feature points corresponding to instances in the cluster to further select uncertain samples in the object set. On the COCO dataset, our method demonstrates superior overall performance in the learning curve compared to existing object detection baselines. Moreover, under the same training configuration, the final detection accuracy shows a notable improvement of at least 2.36%, thus confirming the effectiveness of our method in enhancing learning and detection performance.

Pest identification is a challenging task in the agricultural sector, as accurate and timely detection of pests is essential for effective pest control and crop protection. Conventional approaches to pest detection, such as entomological knowledge and manual examination, take a lot of time and are prone to human mistakes. The advent of Deep Learning (DL) techniques has revolutionized the field of computer vision, enabling automated and efficient pest recognition systems.In this research, we compared the effectiveness of many deep learning models and suggested an enhanced approach for more effective feature extraction. In the proposed approach, we have incorporated two parallel attention mechanisms in ResNet architectures and it has a significant improvement in performance. Experimental result shows that the performance accuracy obtained in ResNet50-SA, ResNet101-SA, and ResNet152-SA is 99.80%, 88.48% and 96.68%, respectively. The performance of ResNet50-SA outperforms the other state of art deep learning by a large margin. The result shows that ResNet with self-attention (SA) has a better ability to extract features and focus on the important features which increases the performance.

Single-cell omics sequencing has rapidly advanced, enabling the quantification of diverse omics profiles at a single-cell resolution. To facilitate comprehensive biological insights, such as cellular differentiation trajectories, precise annotation of cell subtypes is essential. Conventional methods involve clustering cells and manually assigning subtypes based on canonical markers, a labor-intensive and expert-dependent process. Hence, an automated computational prediction framework is crucial. While several classification frameworks for predicting cell subtypes from single-cell RNA sequencing datasets exist, these methods solely rely on single-omics data, offering insights at a single molecular level. They often miss inter-omic correlations and a holistic understanding of cellular processes. To address this, the integration of multi-omics datasets from individual cells is essential for accurate subtype annotation. This article introduces moSCminer, a novel framework for classifying cell subtypes that harnesses the power of single-cell multi-omics sequencing datasets through an attention-based neural network operating at the omics level. By integrating three distinct omics datasets--gene expression, DNA methylation, and DNA accessibility--while accounting for their biological relationships, moSCminer excels at learning the relative significance of each omics feature. It then transforms this knowledge into a novel representation for cell subtype classification. Comparative evaluations against standard machine learning-based classifiers demonstrate moSCminer's superior performance, consistently achieving the highest average performance on real datasets. The efficacy of multi-omics integration is further corroborated through an in-depth analysis of the omics-level attention module, which identifies potential markers for cell subtype annotation. To enhance accessibility and scalability, moSCminer is accessible as a user-friendly web-based platform seamlessly connected to a cloud system, publicly accessible at http://203.252.206.118:5568. Notably, this study marks the pioneering integration of three single-cell multi-omics datasets for cell subtype identification. [ABSTRACT FROM AUTHOR]

The evolution of technology has raised concerns regarding cybersecurity for intelligent connected vehicles (ICVs). In-vehicle network in ICVs lacks robust protection mechanisms, making it vulnerable to cyber threats. In response, intrusion detection systems (IDSs) for ICVs have been developed to protect vehicles from malicious cyber attacks. However, current IDS methods solely rely on independent features, limiting their learning capabilities and increasing the number of false detections. Moreover, many IDSs require the knowledge of mapping between network messages and contents, which restricts their application. To address these limitations, we propose the Multi-order Feature Interaction-aware Intrusion Detection (MIFI) scheme for ICVs. Feature attention cross network is designed to address higher-order feature interactions, while factorization machine is used for second-order interactions. Then a discriminator is utilized to detect the attacks. MIFI expands the feature space through features interaction, thereby enhancing its ability to detect attacks. Moreover, it perceives the relationships of vehicle messages, facilitating intrusion detection without knowing the corresponding rules of vehicle messages. The performance of the proposed method is evaluated on two real-vehicle datasets, affirming its effectiveness and robustness. MIFI achieves an accuracy of over 99% in detecting different attacks. The proposed method can improve the accuracy of traditional IDS to a maximum of 99.99%, and increase the highest F1-score to 97.18%, demonstrating the model's ability of achieving multi-order feature interactions. Ultimately, MIFI is suitable for intrusion detection in different types of ICV networks, significantly contributing to the cybersecurity of ICVs. • We reveal a robust similarity between different real-vehicle attacks data. • Feature attention cross network explores interactions between different features. • No need for prior knowledge of vehicle message rules for intrusion detection. • Proposed scheme detects intrusion with 99% accuracy in real vehicle data. [ABSTRACT FROM AUTHOR]

Background: Emotion recognition using electroencephalogram (EEG) has become a research hotspot in the field of human-computer interaction, how to sufficiently learn complex spatial-temporal representations of emotional EEG data and obtain explainable model prediction results are still great challenges., New Method: In this study, a novel hierarchical and explainable attention network ST-SHAP which combines the Swin Transformer (ST) and SHapley Additive exPlanations (SHAP) technique is proposed for automatic emotional EEG classification. Firstly, a 3D spatial-temporal feature of emotional EEG data is generated via frequency band filtering, temporal segmentation, spatial mapping, and interpolation to fully preserve important spatial-temporal-frequency characteristics. Secondly, a hierarchical attention network is devised to sufficiently learn an abstract spatial-temporal representation of emotional EEG and perform classification. Concretely, in this decoding model, the W-MSA module is used for modeling correlations within local windows, the SW-MSA module allows for information interactions between different local windows, and the patch merging module further facilitates local-to-global multiscale modeling. Finally, the SHAP method is utilized to discover important brain regions for emotion processing and improve the explainability of the Swin Transformer model., Results: Two benchmark datasets, namely SEED and DREAMER, are used for classification performance evaluation. In the subject-dependent experiments, for SEED dataset, ST-SHAP achieves an average accuracy of 97.18%, while for DREAMER dataset, the average accuracy is 96.06% and 95.98% on arousal and valance dimension respectively. In addition, import brain regions that conform to prior knowledge of neurophysiology are discovered via a data-driven approach for both datasets., Comparison With Existing Methods: In terms of subject-dependent and subject-independent emotional EEG decoding accuracies, our method outperforms several closely related existing methods., Conclusion: These experimental results fully prove the effectiveness and superiority of our proposed algorithm., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024. Published by Elsevier B.V.)

In recent years, Transformer has been widely used in the crossing task of computer vision (CV) and natural language processing (NLP), e.g., image captioning. The prior works of image captioning based on Transformer have achieved remarkable progress. In order to further improve the model's ability to describe detail features and generate high-quality sentences, in the paper, we first construct a Multi-stage Transformer Feature Enhancement Network (MT-FEN), which obtains more semantics by fusing features at different scales. Furthermore, we crucially propose a novel second-attention (SA) that can focus more on valuable features and filter out noises in MT-FEN. Besides that, to generate more sensible sentences, we use multiple decoding layers to choose the best-described word via the Max-out module. The overall model is dubbed as SAMT-Generator. After conducting extensive experiments on two well-known datasets: MS COCO and Flickr8k, the results demonstrate the effectiveness of our proposed model, and it is comparable with SOTA methods. [Display omitted] [ABSTRACT FROM AUTHOR]

Many diverse phenomena in nature often inherently encode both short- and long-term temporal dependencies, which especially result from the direction of the flow of time. In this respect, we discovered experimental evidence suggesting that interrelations of these events are higher for closer time stamps. However, to be able for attention-based models to learn these regularities in short-term dependencies, it requires large amounts of data, which are often infeasible. This is because, while they are good at learning piece-wise temporal dependencies, attention-based models lack structures that encode biases in time series. As a resolution, we propose a simple and efficient method that enables attention layers to better encode the short-term temporal bias of these data sets by applying learnable, adaptive kernels directly to the attention matrices. We chose various prediction tasks for the experiments using Electronic Health Records (EHR) data sets since they are great examples with underlying long- and short-term temporal dependencies. Our experiments show exceptional classification results compared to best-performing models on most tasks and data sets., Competing Interests: KK was employed by VUNO Inc. BL was employed by Medical AI Co., Ltd., (Copyright © 2024 Kim and Lee.)

Transformer-based methods have achieved impressive performance in image super-resolution (SR). To reduce the computational cost and redundancy of global attention, most transformer-based methods adopt a localized attention mechanism, which diminishes the desirable characteristics of self-attention (SA), such as the effective modeling of long-range dependencies and the ability to capture a global receptive field. To alleviate this problem, we propose a dilated neighborhood attention transformer for image SR (DiNAT-SR) to improve SwinIR for image SR; in it, we replace SA with DiNA to capture more global data and allow the receptive field to grow exponentially. In addition, we also introduce a convolutional modulation block into the transformer to enhance the visual representation and facilitate smoother convergence during training. Our research has, for the first time, confirmed the feasibility of DiNA in the field of image SR. Experimental results have demonstrated the effectiveness of DiNAT-SR with better results compared with SwinIR on most benchmarks in terms of both quantitatively and visually. We also provide a comparison of light-weight image SR models, and our model performs better than SwinIR-light on all benchmarks, with similar total numbers of parameters and floating-point operations. The effectiveness of each introduced component is also validated by an ablation study. [ABSTRACT FROM AUTHOR]

Single-cell omics sequencing has rapidly advanced, enabling the quantification of diverse omics profiles at a single-cell resolution. To facilitate comprehensive biological insights, such as cellular differentiation trajectories, precise annotation of cell subtypes is essential. Conventional methods involve clustering cells and manually assigning subtypes based on canonical markers, a labor-intensive and expert-dependent process. Hence, an automated computational prediction framework is crucial. While several classification frameworks for predicting cell subtypes from single-cell RNA sequencing datasets exist, these methods solely rely on single-omics data, offering insights at a single molecular level. They often miss inter-omic correlations and a holistic understanding of cellular processes. To address this, the integration of multi-omics datasets from individual cells is essential for accurate subtype annotation. This article introduces moSCminer, a novel framework for classifying cell subtypes that harnesses the power of single-cell multi-omics sequencing datasets through an attention-based neural network operating at the omics level. By integrating three distinct omics datasets-gene expression, DNA methylation, and DNA accessibility-while accounting for their biological relationships, moSCminer excels at learning the relative significance of each omics feature. It then transforms this knowledge into a novel representation for cell subtype classification. Comparative evaluations against standard machine learning-based classifiers demonstrate moSCminer's superior performance, consistently achieving the highest average performance on real datasets. The efficacy of multi-omics integration is further corroborated through an in-depth analysis of the omics-level attention module, which identifies potential markers for cell subtype annotation. To enhance accessibility and scalability, moSCminer is accessible as a user-friendly web-based platform seamlessly connected to a cloud system, publicly accessible at http://203.252.206.118:5568. Notably, this study marks the pioneering integration of three single-cell multi-omics datasets for cell subtype identification., Competing Interests: The authors declare that they have no competing interests., (© 2024 Choi et al.)