Search

Breast cancer is ranked as the second most common cancer among women globally, highlighting the critical need for precise and early detection methods. Our research introduces a novel approach for classifying benign and malignant breast ultrasound images. We leverage advanced deep learning methodologies, mainly focusing on the vision transformer (ViT) model. Our method distinctively features progressive fine‐tuning, a tailored process that incrementally adapts the model to the nuances of breast tissue classification. Ultrasound imaging was chosen for its distinct benefits in medical diagnostics. This modality is noninvasive and cost‐effective and demonstrates enhanced specificity, especially in dense breast tissues where traditional methods may struggle. Such characteristics make it an ideal choice for the sensitive task of breast cancer detection. Our extensive experiments utilized the breast ultrasound images dataset, comprising 780 images of both benign and malignant breast tissues. The dataset underwent a comprehensive analysis using several pretrained deep learning models, including VGG16, VGG19, DenseNet121, Inception, ResNet152V2, DenseNet169, DenseNet201, and the ViT. The results presented were achieved without employing data augmentation techniques. The ViT model demonstrated robust accuracy and generalization capabilities with the original dataset size, which consisted of 637 images. Each model's performance was meticulously evaluated through a robust 10‐fold cross‐validation technique, ensuring a thorough and unbiased comparison. Our findings are significant, demonstrating that the progressive fine‐tuning substantially enhances the ViT model's capability. This resulted in a remarkable accuracy of 94.49% and an AUC score of 0.921, significantly higher than models without fine‐tuning. These results affirm the efficacy of the ViT model and highlight the transformative potential of integrating progressive fine‐tuning with transformer models in medical image classification tasks. The study solidifies the role of such advanced methodologies in improving early breast cancer detection and diagnosis, especially when coupled with the unique advantages of ultrasound imaging. [ABSTRACT FROM AUTHOR]

This study introduces Hybrid-DC, a hybrid deep-learning model integrating ResNet-50 and Vision Transformer (ViT) for high-accuracy steel surface defect classification. Hybrid-DC leverages ResNet-50 for efficient feature extraction at both low and high levels and utilizes ViT's global context learning to enhance classification precision. A unique hybrid attention layer and an attention fusion mechanism enable Hybrid-DC to adapt to the complex, variable patterns typical of steel surface defects. Experimental evaluations demonstrate that Hybrid-DC achieves substantial accuracy improvements and significantly reduced loss compared to traditional models like MobileNetV2 and ResNet, with a validation accuracy reaching 0.9944. The results suggest that this model, characterized by rapid convergence and stable learning, can be applied for real-time quality control in steel manufacturing and other high-precision industries, enhancing automated defect detection efficiency. [ABSTRACT FROM AUTHOR]

Fire detection is a critical task in environmental monitoring and disaster prevention, with traditional methods often limited in their ability to detect fire and smoke in real time over large areas. The rapid identification of fire and smoke in both indoor and outdoor environments is essential for minimizing damage and ensuring timely intervention. In this paper, we propose a novel approach to fire and smoke detection by integrating a vision transformer (ViT) with the YOLOv5s object detection model. Our modified model leverages the attention-based feature extraction capabilities of ViTs to improve detection accuracy, particularly in complex environments where fires may be occluded or distributed across large regions. By replacing the CSPDarknet53 backbone of YOLOv5s with ViT, the model is able to capture both local and global dependencies in images, resulting in more accurate detection of fire and smoke under challenging conditions. We evaluate the performance of the proposed model using a comprehensive Fire and Smoke Detection Dataset, which includes diverse real-world scenarios. The results demonstrate that our model outperforms baseline YOLOv5 variants in terms of precision, recall, and mean average precision (mAP), achieving a mAP@0.5 of 0.664 and a recall of 0.657. The modified YOLOv5s with ViT shows significant improvements in detecting fire and smoke, particularly in scenes with complex backgrounds and varying object scales. Our findings suggest that the integration of ViT as the backbone of YOLOv5s offers a promising approach for real-time fire detection in both urban and natural environments. [ABSTRACT FROM AUTHOR]

Pinelliae Rhizoma is a key ingredient in botanical supplements and is often adulterated by Rhizoma Pinelliae Pedatisectae, which is similar in appearance but less expensive. Accurate identification of these materials is crucial for both scientific and commercial purposes. Traditional morphological identification relies heavily on expert experience and is subjective, while chemical analysis and molecular biological identification are typically time consuming and labor intensive. This study aims to employ a simpler, faster, and non‐invasive image recognition technique to distinguish between these two highly similar plant materials. In the realm of image recognition, we aimed to utilize the vision transformer (ViT) algorithm, a cutting‐edge image recognition technology, to differentiate these materials. All samples were verified using DNA molecular identification before image analysis. The result demonstrates that the ViT algorithm achieves a classification accuracy exceeding 94%, significantly outperforming the convolutional neural network model's 60%–70% accuracy. This highlights the efficiency of this technology in identifying plant materials with similar appearances. This study marks the pioneer work of the ViT algorithm to such a challenging task, showcasing its potential for precise botanical material identification and setting the stage for future advancements in the field. [ABSTRACT FROM AUTHOR]

In the digital era, social media has become a cornerstone for educational institutions, driving public engagement and enhancing institutional communication. This study utilizes AI-driven image processing and Web-enhanced Deep Learning (DL) techniques to investigate the effectiveness of King Faisal University's (KFU's) social media strategy as a case study, particularly on Twitter. By categorizing images into five primary event management categories and subcategories, this research provides a robust framework for assessing the social media content generated by KFU's administrative units. Seven advanced models were developed, including an innovative integration of Vision Transformers (ViTs) with Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, VGG16, and ResNet. The AI-driven ViT-CNN hybrid model achieved perfect classification accuracy (100%), while the "Development and Partnerships" category demonstrated notable accuracy (98.8%), underscoring the model's unparalleled efficacy in strategic content classification. This study offers actionable insights for the optimization of AI-driven digital communication strategies and Web-enhanced data collection processes, aligning them with national development goals and Saudi Arabia's Vision 2030, thereby showcasing the transformative power of DL in event-centric management and the broader higher education landscape. [ABSTRACT FROM AUTHOR]

The three-dimensional reconstruction of forests is crucial in remote sensing technology, ecological monitoring, and forestry management, as it yields precise forest structure and tree parameters, providing essential data support for forest resource management, evaluation, and sustainable development. Nevertheless, forest 3D reconstruction now encounters obstacles including higher equipment costs, reduced data collection efficiency, and complex data processing. This work introduces a unique deep learning model, CPH-Fmnet, designed to enhance the accuracy and efficiency of 3D reconstruction in intricate forest environments. CPH-Fmnet enhances the FPN Encoder-Decoder Architecture by meticulously incorporating the Channel Attention Mechanism (CA), Path Aggregation Module (PA), and High-Level Feature Selection Module (HFS), alongside the integration of the pre-trained Vision Transformer (ViT), thereby significantly improving the model's global feature extraction and local detail reconstruction abilities. We selected three representative sample plots in Haidian District, Beijing, China, as the study area and took forest stand sequence photos with an iPhone for the research. Comparative experiments with the conventional SfM + MVS and MVSFormer models, along with comprehensive parameter extraction and ablation studies, substantiated the enhanced efficacy of the proposed CPH-Fmnet model in addressing difficult circumstances such as intricate occlusions, poorly textured areas, and variations in lighting. The test results show that the model does better on a number of evaluation criteria. It has an RMSE of 1.353, an MAE of only 5.1%, an r value of 1.190, and a forest reconstruction rate of 100%, all of which are better than current methods. Furthermore, the model produced a more compact and precise 3D point cloud while accurately determining the properties of the forest trees. The findings indicate that CPH-Fmnet offers an innovative approach for forest resource management and ecological monitoring, characterized by cheap cost, high accuracy, and high efficiency. [ABSTRACT FROM AUTHOR]

Glacial landforms, created by the continuous movements of glaciers over millennia, are crucial topics in geomorphological research. Their systematic analysis affords invaluable insights into past climatic oscillations and augments understanding of long-term climate change dynamics. The classification of these types of terrain traditionally depends on labor-intensive manual or semi-automated methods. However, the emergence of automated techniques driven by deep learning and neural networks holds promise for enhancing efficiency of terrain classification workflows. This study evaluated the effectiveness of Convolutional Neural Network (CNN) architectures, particularly Residual Neural Network (ResNet) and VGG in comparison with Vision Transformer (ViT) architecture in the glacial landform classification task. By using preprocessed input data from Digital Elevation Model (DEM) which covers regions such as the Lubawa Upland and Gardno-Leba Plain in Poland, as well as the Elise Glacier in Svalbard, Norway, comprehensive assessments of those methods were conducted. The final results highlight the unique ability of deep learning methods to accurately classify glacial landforms. Classification process presented in this study can be the efficient, repeatable and fast solution for automatic terrain classification.

Abstract To ensure the reliability of machining quality, it is crucial to predict tool wear accurately. In this paper, a novel deep learning-based model is proposed, which synthesizes the advantages of power spectral density (PSD), convolutional neural networks (CNN), and vision transformer model (ViT), namely PSD-CVT. PSD maps can provide a comprehensive understanding of the spectral characteristics of the signals. It makes the spectral characteristics more obvious and makes it easy to analyze and compare different signals. CNN focuses on local feature extraction, which can capture local information such as the texture, edge, and shape of the image, while the attention mechanism in ViT can effectively capture the global structure and long-range dependencies present in the image. Two fully connected layers with a ReLU function are used to obtain the predicted tool wear values. The experimental results on the PHM 2010 dataset demonstrate that the proposed model has higher prediction accuracy than the CNN model or ViT model alone, as well as outperforms several existing methods in accurately predicting tool wear. The proposed prediction method can also be applied to predict tool wear in other machining fields.

Bridges are crucial components of infrastructure networks that facilitate national connectivity and development. According to the National Bridge Inventory (NBI) and the Federal Highway Administration (FHWA), the cost to repair U.S. bridges was recently estimated at approximately USD 164 billion. Traditionally, bridge inspections are performed manually, which poses several challenges in terms of safety, efficiency, and accessibility. To address these issues, this research study introduces a method using Unmanned Aerial Systems (UASs) to help automate the inspection process. This methodology employs UASs to capture visual images of a concrete bridge deck, which are then analyzed using advanced machine learning techniques of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to detect damage and delamination. A case study on the Beyer Road Concrete Bridge in Michigan is used to demonstrate the developed methodology. The findings demonstrate that the ViT model outperforms the CNN in detecting bridge deck damage, with an accuracy of 97%, compared to 92% for the CNN. Additionally, the ViT model showed a precision of 96% and a recall of 97%, while the CNN model achieved a precision of 93% and a recall of 61%. This technology not only enhances the maintenance of bridges but also significantly reduces the risks associated with traditional inspection methods.

• The increase of decoding precision of motor imaging EEG signal. • The hybrid TCN and ViT method achieves good results. • A shared convolution strategy and a dual-branching strategy. Brain-computer interface (BCI) is a technology that directly connects signals between the human brain and a computer or other external device. Motor imagery electroencephalographic (MI-EEG) signals are considered a promising paradigm for BCI systems, with a wide range of potential applications in medical rehabilitation, human–computer interaction, and virtual reality. Accurate decoding of MI-EEG signals poses a significant challenge due to issues related to the quality of the collected EEG data and subject variability. Therefore, developing an efficient MI-EEG decoding network is crucial and warrants research. This paper proposes a loss joint training model based on the vision transformer (VIT) and the temporal convolutional network (EEG-VTTCNet) to classify MI-EEG signals. To take advantage of multiple modules together, the EEG-VTTCNet adopts a shared convolution strategy and a dual-branching strategy. The dual-branching modules perform complementary learning and jointly train shared convolutional modules with better performance. We conducted experiments on the BCI Competition IV-2a and IV-2b datasets, and the proposed network outperformed the current state-of-the-art techniques with an accuracy of 84.58% and 90.94%, respectively, for the subject-dependent mode. In addition, we used t-SNE to visualize the features extracted by the proposed network, further demonstrating the effectiveness of the feature extraction framework. We also conducted extensive ablation and hyperparameter tuning experiments to construct a robust network architecture that can be well generalized. [ABSTRACT FROM AUTHOR]

The Vision Transformer (ViT) has demonstrated impressive performance in various visual tasks, but its high computational requirements limit its applicability on edge devices. Conversely, convolutional neural networks (CNNs) are commonly used in mobile applications, but their static and weak global properties hinder their performance. In this work, we propose a lightweight, high-density predictive classification hybrid-based model called CSFNet, which combines good local inductive bias capability with long-distance modeling property. To establish local-global information association, we introduce two layered structures. Firstly, we use the Local-Attention Block (LAB) with adaptive kernels and channel expansion ratio to aggregate n × n local information layer by layer, capturing multi-stage detail features and inducing efficient local inductive properties. Secondly, we introduce a linear complexity Channel-Spatial Fusion Attention (CSFA) that projects the attention matrix from both channel and tokens dimensions. The relationships between tokens are aggregated stage by stage to encode efficient contextual association information using low-rank matrix and element-by-element operations to reduce computational complexity. Experimental results demonstrate that our proposed CSFNet-XXS/XS/S models with 1.4M/2.4M/5.6M parameters and 0.3G/0.5G/1.1G multiply-adds (MAdds) achieve 70.23%/74.91%/78.82% top-1 accuracy on ImageNet-1k with competitive performance compared to recent mainstream methods. Furthermore, CSFNet performs well on small-scale datasets, MS-COCO2017 and ADE-20K. [ABSTRACT FROM AUTHOR]

Bridges are crucial components of infrastructure networks that facilitate national connectivity and development. According to the National Bridge Inventory (NBI) and the Federal Highway Administration (FHWA), the cost to repair U.S. bridges was recently estimated at approximately USD 164 billion. Traditionally, bridge inspections are performed manually, which poses several challenges in terms of safety, efficiency, and accessibility. To address these issues, this research study introduces a method using Unmanned Aerial Systems (UASs) to help automate the inspection process. This methodology employs UASs to capture visual images of a concrete bridge deck, which are then analyzed using advanced machine learning techniques of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to detect damage and delamination. A case study on the Beyer Road Concrete Bridge in Michigan is used to demonstrate the developed methodology. The findings demonstrate that the ViT model outperforms the CNN in detecting bridge deck damage, with an accuracy of 97%, compared to 92% for the CNN. Additionally, the ViT model showed a precision of 96% and a recall of 97%, while the CNN model achieved a precision of 93% and a recall of 61%. This technology not only enhances the maintenance of bridges but also significantly reduces the risks associated with traditional inspection methods. [ABSTRACT FROM AUTHOR]

Background: Skin cancer is one of the highly occurring diseases in human life. Early detection and treatment are the prime and necessary points to reduce the malignancy of infections. Deep learning techniques are supplementary tools to assist clinical experts in detecting and localizing skin lesions. Vision transformers (ViT) based on image segmentation classification using multiple classes provide fairly accurate detection and are gaining more popularity due to legitimate multiclass prediction capabilities. Materials and methods: In this research, we propose a new ViT Gradient‐Weighted Class Activation Mapping (GradCAM) based architecture named ViT‐GradCAM for detecting and classifying skin lesions by spreading ratio on the lesion's surface area. The proposed system is trained and validated using a HAM 10000 dataset by studying seven skin lesions. The database comprises 10 015 dermatoscopic images of varied sizes. The data preprocessing and data augmentation techniques are applied to overcome the class imbalance issues and improve the model's performance. Result: The proposed algorithm is based on ViT models that classify the dermatoscopic images into seven classes with an accuracy of 97.28%, precision of 98.51, recall of 95.2%, and an F1 score of 94.6, respectively. The proposed ViT‐GradCAM obtains better and more accurate detection and classification than other state‐of‐the‐art deep learning‐based skin lesion detection models. The architecture of ViT‐GradCAM is extensively visualized to highlight the actual pixels in essential regions associated with skin‐specific pathologies. Conclusion: This research proposes an alternate solution to overcome the challenges of detecting and classifying skin lesions using ViTs and GradCAM, which play a significant role in detecting and classifying skin lesions accurately rather than relying solely on deep learning models. [ABSTRACT FROM AUTHOR]

Polarimetric high-resolution range profile (HRRP), with its rich polarimetric and spatial information, has become increasingly important in radar automatic target recognition (RATR). This study proposes an interpretable target-aware vision Transformer (ITAViT) for polarimetric HRRP target recognition with a novel attention loss. In ITAViT, we initially fuse the polarimetric features and the amplitude of polarimetric HRRP with a polarimetric preprocessing layer (PPL) to obtain the feature map as the input of the subsequent network. The vision Transformer (ViT) is then used as the backbone to automatically extract both local and global features. Most importantly, we introduce a novel attention loss to optimize the alignment between the attention map and the HRRP span. Thus, it can improve the difference between the target and the background, and enable the model to more effectively focus on real target areas. Experiments on a simulated X-band dataset demonstrate that our proposed ITAViT outperforms comparative models under various experimental conditions. Ablation studies highlight the effectiveness of polarimetric preprocessing and attention loss. Furthermore, the visualization of the self-attention mechanism suggests that attention loss enhances the interpretability of the network. [ABSTRACT FROM AUTHOR]

Arabic handwriting recognition and conversion are crucial for financial operations, particularly for processing handwritten amounts on cheques and financial documents. Compared to other languages, research in this area is relatively limited, especially concerning Arabic. This study introduces an innovative AI-driven method for simultaneously recognizing and converting Arabic handwritten legal amounts into numerical courtesy forms. The framework consists of four key stages. First, a new dataset of Arabic legal amounts in handwritten form (".png" image format) is collected and labeled by natives. Second, a YOLO-based AI detector extracts individual legal amount words from the entire input sentence images. Third, a robust hybrid classification model is developed, sequentially combining ensemble Convolutional Neural Networks (CNNs) with a Vision Transformer (ViT) to improve the prediction accuracy of single Arabic words. Finally, a novel conversion algorithm transforms the predicted Arabic legal amounts into digital courtesy forms. The framework's performance is fine-tuned and assessed using 5-fold cross-validation tests on the proposed novel dataset, achieving a word level detection accuracy of 98.6% and a recognition accuracy of 99.02% at the classification stage. The conversion process yields an overall accuracy of 90%, with an inference time of 4.5 s per sentence image. These results demonstrate promising potential for practical implementation in diverse Arabic financial systems. [ABSTRACT FROM AUTHOR]

Wetland mapping is a critical component of environmental monitoring, requiring advanced techniques to accurately represent the complex land cover patterns and subtle class differences innate in these ecosystems. This study aims to address these challenges by proposing CVTNet, a novel deep learning (DL) model that integrates convolutional neural networks (CNNs) and vision transformer (ViT) architectures. CVTNet uses channel attention (CA) and spatial attention (SA) mechanisms to enhance feature extraction from Sentinel-1 (S1) and Sentinel-2 (S2) satellite data. The primary goal of this model is to achieve a balanced trade-off between Precision and Recall, which is essential for accurate wetland mapping. The class-specific analysis demonstrated CVTNet's proficiency across diverse classes, including pasture, shrubland, urban, bog, fen, and water. Comparative analysis showed that CVTNet outperforms contemporary algorithms such as Random Forest (RF), ViT, multi-layer perceptron mixer (MLP-mixer), and hybrid spectral net (HybridSN) classifiers. Additionally, the attention mechanism (AM) analysis and sensitivity analysis highlighted the crucial role of CA, SA, and ViT in focusing the model's attention on critical regions, thereby improving the mapping of wetland regions. Despite challenges at class boundaries, particularly between bog and fen, and misclassifications of swamp pixels, CVTNet presents a solution for wetland mapping. [ABSTRACT FROM AUTHOR]

Image denoising is a fundamental problem in computer vision (CV). Vision Transformer (ViT) is an improvement in CV after convolutional neural networks (CNNs). Recently, it has been demonstrated that the RepVGG network of the structural re-parameterization performs well in image tasks. Experiments indicate that the ViT-based uniformer transformer network has successfully balanced local and global information. As image denoising tasks require more local and global information about the image, a novel image denoising model named structural Re-parameterization Uniformer Transformer-UNet (Rep-UUNet) is proposed in this paper. The model is structural and re-parameterized the structure of the Uniformer Transformer network and uses the UNet skip connections to reconstruct the output image. To complete image downstream tasks, the RepVGG model which utilizes the image local information is used. Indicators such as PSNR, SSIM and others are used to assess the performance of image denoising. Experimental results demonstrate that our Rep-UUNet network model outperforms other five models. [ABSTRACT FROM AUTHOR]

In the realm of autonomous vehicle technology, the multimodal vehicle detection network (MVDNet) represents a significant leap forward, particularly in the challenging context of weather conditions. This paper focuses on the enhancement of MVDNet through the integration of a multi-head attention layer, aimed at refining its performance. The integrated multi-head attention layer in the MVDNet model is a pivotal modification, advancing the network's ability to process and fuse multimodal sensor information more efficiently. The paper validates the improved performance of MVDNet with multi-head attention through comprehensive testing, which includes a training dataset derived from the Oxford Radar RobotCar. The results clearly demonstrate that the multi-head MVDNet outperforms the other related conventional models, particularly in the average precision (AP) of estimation, under challenging environmental conditions. The proposed multi-head MVDNet not only contributes significantly to the field of autonomous vehicle detection but also underscores the potential of sophisticated sensor fusion techniques in overcoming environmental limitations. [ABSTRACT FROM AUTHOR]

Current unsupervised industrial product surface anomaly detection methods suffer from poor reconstructed image quality and difficulty in detecting low-contrast anomalies, resulting in low anomaly detection accuracy. To address the above problems, we propose an unsupervised masked hybrid convolutional Transformer anomaly detection model, which forces the model to predict missing or edited regions based on unmasked information by introducing a mask reconstruction strategy, and utilises convolutional blocks and Transformer self-attention mechanism to extract the local features and global context of the image at different resolutions, enhancing the model’s ability to understand the interrelationships among image parts and the overall structure. information to enhance the model’s ability to understand the interrelationships between image parts and the overall structure, and to improve the reconstruction ability of the model; then a method based on Gaussian difference significance is proposed, which is combined with gradient magnitude similarity and colour difference to compare the differences between reconstructed and original images from multiple perspectives, and to improve the anomaly localisation performance of the model. We conducted extensive experiments on the industrial datasets MVTec AD and MTD to validate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Cervical cancer is a major health concern worldwide, highlighting the urgent need for better early detection methods to improve outcomes for patients. In this study, we present a novel digital pathology classification approach that combines Low-Rank Adaptation (LoRA) with the Vision Transformer (ViT) model. This method is aimed at making cervix type classification more efficient through a deep learning classifier that does not require as much data. The key innovation is the use of LoRA, which allows for the effective training of the model with smaller datasets, making the most of the ability of ViT to represent visual information. This approach performs better than traditional Convolutional Neural Network (CNN) models, including Residual Networks (ResNets), especially when it comes to performance and the ability to generalize in situations where data are limited. Through thorough experiments and analysis on various dataset sizes, we found that our more streamlined classifier is highly accurate in spotting various cervical anomalies across several cases. This work advances the development of sophisticated computer-aided diagnostic systems, facilitating more rapid and accurate detection of cervical cancer, thereby significantly enhancing patient care outcomes. [ABSTRACT FROM AUTHOR]

Natural hazards are considered to have a strong link with climate change and human activities. With the rapid advancements in remote sensing technology, real-time monitoring and high-resolution remote-sensing images have become increasingly available, which provide precise details about the Earth's surface and enable prompt updates to support risk identification and management. This paper proposes a new network framework with Transformer architecture and a Residual network for detecting the changes in high-resolution remote-sensing images. The proposed model is trained using remote-sensing images from Shandong and Anhui Provinces of China in 2021 and 2022 while one district in 2023 is used to test the prediction accuracy. The performance of the proposed model is evaluated by using five matrices and further compared to both convention-based and attention-based models. The results demonstrated that the proposed structure integrates the great capability of conventional neural networks for image feature extraction with the ability to obtain global context from the attention mechanism, resulting in significant improvements in balancing positive sample identification while avoiding false positives in complex image change detection. Additionally, a toolkit supporting image preprocessing is developed for practical applications. [ABSTRACT FROM AUTHOR]

Background: In response to the escalating global concerns surrounding skin cancer, this study aims to address the imperative for precise and efficient diagnostic methodologies. Focusing on the intricate task of eight-class skin cancer classification, the research delves into the limitations of conventional diagnostic approaches, often hindered by subjectivity and resource constraints. The transformative potential of Artificial Intelligence (AI) in revolutionizing diagnostic paradigms is underscored, emphasizing significant improvements in accuracy and accessibility. Methods: Utilizing cutting-edge deep learning models on the ISIC2019 dataset, a comprehensive analysis is conducted, employing a diverse array of pre-trained ImageNet architectures and Vision Transformer models. To counteract the inherent class imbalance in skin cancer datasets, a pioneering “Naturalize” augmentation technique is introduced. This technique leads to the creation of two indispensable datasets—the Naturalized 2.4K ISIC2019 and groundbreaking Naturalized 7.2K ISIC2019 datasets—catalyzing advancements in classification accuracy. The “Naturalize” augmentation technique involves the segmentation of skin cancer images using the Segment Anything Model (SAM) and the systematic addition of segmented cancer images to a background image to generate new composite images. Results: The research showcases the pivotal role of AI in mitigating the risks of misdiagnosis and under-diagnosis in skin cancer. The proficiency of AI in analyzing vast datasets and discerning subtle patterns significantly augments the diagnostic prowess of dermatologists. Quantitative measures such as confusion matrices, classification reports, and visual analyses using Score-CAM across diverse dataset variations are meticulously evaluated. The culmination of these endeavors resulted in an unprecedented achievement—100% average accuracy, precision, recall, and F1-score—within the groundbreaking Naturalized 7.2K ISIC2019 dataset. Conclusion: This groundbreaking exploration highlights the transformative capabilities of AI-driven methodologies in reshaping the landscape of skin cancer diagnosis and patient care. The research represents a pivotal stride towards redefining dermatological diagnosis, showcasing the remarkable impact of AI-powered solutions in surmounting the challenges inherent in skin cancer diagnosis. The attainment of 100% across crucial metrics within the Naturalized 7.2K ISIC2019 dataset serves as a testament to the transformative capabilities of AI-driven approaches in reshaping the trajectory of skin cancer diagnosis and patient care. This pioneering work paves the way for a new era in dermatological diagnostics, heralding the dawn of unprecedented precision and efficacy in the identification and classification of skin cancers.

High resolution remote sensing imagery plays a crucial role in monitoring coastal wetlands. Coastal wetland landscapes exhibit diverse features, ranging from fragmented patches to expansive areas. Mainstream convolutional neural networks cannot effectively analyze spatial relationships among consecutive image elements. This limitation impedes their performance in accurately classifying coastal wetlands. In order to tackle the above issues, we propose a Vision Transformer based UNet (ViT-UNet) model. This model extracts wetland features from high resolution remote sensing images by sensing and optimizing multiscale features. To establish global dependencies, the Vision Transformer (ViT) is introduced to replace the convolutional layer in the UNet encoder. Simultaneously, the model incorporates a convolutional block attention module and a multiple hierarchies attention module to restore attentional features and reduce feature loss. In addition, a skip connection is added to the single-skip structure of the original UNet model. This connection simultaneously links the output of the entire transformer and internal attention features to the corresponding decoder level. This enhancement aims to furnish the decoder with comprehensive global information guidance. Finally, all the extracted feature information is fused using Bilinear Polymerization Pooling (BPP). The BPP assists the network in obtaining a more comprehensive and detailed feature representation. Experimental results on the Gaofen-1 dataset demonstrate that the proposed ViT-UNet method achieves a Precision score of 93.50$\%$, outperforming the original UNet model by 4.10$\%$. Compared with other state-of-the-art networks, ViT-UNet performs more accurately and finer in the extraction of wetland information in the Yellow River Delta.

Emotional health significantly impacts physical and psychological well-being, with emotional imbalances and cognitive disorders leading to various health issues. Timely diagnosis of mental illnesses is crucial for preventing severe disorders and enhancing medical care quality. Physiological signals, such as Electrocardiograms (ECG) and Electroencephalograms (EEG), which reflect cardiac and neuronal activities, are reliable for emotion recognition as they are less susceptible to manipulation than physical signals. Galvanic Skin Response (GSR) is also closely linked to emotional states. Researchers have developed various methods for classifying signals to detect emotions. However, these signals are susceptible to noise and are inherently non-stationary, meaning they change constantly over time. Consequently, emotions can vary rapidly. Traditional techniques for analyzing physiological signals may not be adequate to study the dynamic changes in emotional states. This research introduces a deep learning approach using a combination of advanced signal processing and machine learning to analyze physiological signals for emotion recognition. We propose a CNN-Vision Transformer (CVT) based method with ensemble classification. The process involves decomposing signals into segments, removing noise, and extracting features using 1D CNN and Vision Transformers. These features are integrated into a single vector for classification by an ensemble of LSTM, ELM, and SVM classifiers. The outputs are then synthesized using Model Agnostic Meta Learning (MAML) to improve prediction accuracy. Validated on AMIGOS and DEAP datasets with 10-fold cross-validation, our method achieved accuracies up to 98.2%, sensitivity of 99.15%, and specificity of 99.53%, outperforming existing emotion charting techniques. This novel method provides significant improvements 3 to 4% in the accuracy, sensitivity, and specificity of emotion detection, leveraging physiological signals for comprehensive emotional assessments.

In recent decades, weakly supervised object localization (WSOL) has gained increasing attention in remote sensing. However, unlike optical images, remote sensing images (RSIs) often contain more complex scenes, which poses challenges for WSOL. Traditional convolutional neural network (CNN)-based WSOL methods are often limited by a small receptive field and yield unsatisfactory results. Transformer-based methods can obtain global perception, addressing the limitations of receptive fields in CNN-based methods, yet it may also introduce attention diffusion. To address the aforementioned problems, this article proposes a novel WSOL method based on an interpretable vision transformer (ViT), RPGV. We introduce a feature fusion enhancement module to obtain the saliency map that captures global information. Simultaneously, we solve the problem of discrete attention in the traditional ViT and eliminate local distortion in the feature map by introducing a global semantic screening module. We conduct comprehensive experiments on DIOR and HRRSD datasets, demonstrating the superior performance of our method compared to current state-of-the-art methods.

In recent years, collaborative classification of multimodal data, e.g., hyperspectral image (HSI) and light detection and ranging (LiDAR), has been widely used to improve remote sensing image classification accuracy. However, existing fusion approaches for HSI and LiDAR suffer from limitations. Fusing the heterogeneous features of HSI and LiDAR proved to be challenging, leading to incomplete utilization of information for category representation. In addition, during the extraction of spatial features from HSI, the spectral and spatial information are often disjointed. It leads to the difficulty of fully exploiting the rich spectral information in hyperspectral data. To address these issues, we proposed a multimodal data fusion framework specifically designed for HSI and LiDAR fusion classification, called modality fusion vision transformer. We have designed a stackable modality fusion block as the core of our model. Specifically, these blocks mainly consist of multimodal cross-attention modules and spectral self-attention modules. The proposed novel multimodal cross-attention module for feature fusion addresses the issue of insufficient fusion of heterogeneous features from HSI and LiDAR for category representation. Compared to other cross-attention methods, it reduces the alignment requirements between modal feature spaces during cross-modal fusion. The spectral self-attention module can preserve spatial features while exploiting the rich spectral information and participating in the process of extracting spatial features from HSI. Ultimately, we achieve overall classification accuracies of 99.91%, 99.59%, and 96.98% on three benchmark datasets respectively, surpassing all state-of-the-art methods, demonstrating the stability and effectiveness of our model.

By improving image quality and resolution, Single Image Super-Resolution (SISR) models help advance understanding of underwater environments. Super-resolution techniques hold promise in addressing these issues by enhancing image details. The transformer architecture has recently gained considerable popularity in low-level vision tasks, including image super-resolution (SR). In this paper, we present a novel multi-distillation algorithm based on the swin transformer and wavelet transform (KDSWT) for underwater image super-resolution. A wavelet block is added to the swin transformer to alleviate the loss of detailed features because wavelet transforms can preserve the high-frequency components, subtle variations, and localized features that contribute to the overall richness in images. Based on a large dataset of high-resolution underwater images, a ViT-based student model is trained to learn the intricate features of underwater images from the teacher model. Using knowledge distillation, the student model learns to upscale input images and capture the domain-specific features essential to understanding underwater scenes. To address the computational cost issue in ViT, we introduce a lightweight network training strategy that combines multi-anchor distillation and progressive learning, enabling the lightweight network to achieve exceptional performance and the fastest inference time compared to recent lightweight state-of-art methods. Our experimental results show that KDSWT lightweight (student) significantly improves peak signal-to-noise ratio (PSNR) over the (original) teacher model while decreasing the number of parameters by 54% and the calculation cost by 22%. Concurrently, KDSWT-SR excels beyond state-of-the-art SR techniques on three benchmark underwater datasets, leading to noteworthy improvements in both PSNR and structural similarity index (SSIM).

In this paper, we propose a method to improve the accuracy of speech emotion recognition (SER) by using vision transformer (ViT) to attend to the correlation of frequency (y-axis) with time (x-axis) in spectrogram and transferring positional information between ViT through knowledge transfer. The proposed method has the following originality i) We use vertically segmented patches of log-Mel spectrogram to analyze the correlation of frequencies over time. This type of patch allows us to correlate the most relevant frequencies for a particular emotion with the time they were uttered. ii) We propose the use of image coordinate encoding, an absolute positional encoding suitable for ViT. By normalizing the x, y coordinates of the image to −1 to 1 and concatenating them to the image, we can effectively provide valid absolute positional information for ViT. iii) Through feature map matching, the locality and location information of the teacher network is effectively transmitted to the student network. Teacher network is a ViT that contains locality of convolutional stem and absolute position information through image coordinate encoding, and student network is a structure that lacks positional encoding in the basic ViT structure. In feature map matching stage, we train through the mean absolute error (L1 loss) to minimize the difference between the feature maps of the two networks. To validate the proposed method, three emotion datasets (SAVEE, EmoDB, and CREMA-D) consisting of speech were converted into log-Mel spectrograms for comparison experiments. As a result of the experiment, the proposed method achieved 99.47%, 99.76%, and 95.24% accuracy, significantly exceeding the state-of-the-art methods, with much less computational complexity on three emotion datasets.

Pansharpening is a fundamental and crucial image processing task for many remote sensing applications, which generates a high-resolution multispectral image by fusing a low-resolution multispectral image and a high-resolution panchromatic image. Recently, vision transformers have been introduced into the pansharpening task for utilizing global contextual information. However, long-range and local dependencies modeling and multiscale feature learning are all essential to the pansharpening task. Learning and exploiting these various information raises a big challenge and limits the performance and efficiency of existing pansharpening methods. To solve this issue, we propose a pansharpening network based on multiscale embedding and dual attention transformers (MDPNet). Specifically, a multiscale embedding block is proposed to embed multiscale information of the images into vectors. Thus, transformers only need to process a multispectral embedding sequence and a panchromatic embedding sequence to efficiently use multiscale information. Furthermore, an additive hybrid attention transformer is proposed to fuse the embedding sequences in an additive injection manner. Finally, a channel self-attention transformer is proposed to utilize channel correlations for high-quality detail generation. Experiments over QuickBird and WorldView-3 datasets demonstrate the proposed MDPNet outperforms state-of-the-art methods visually and quantitatively with low running time. Ablation studies further verify the effectiveness of the proposed multiscale embedding and transformers in pansharpening.

The scattering properties of targets in polarimetric synthetic aperture radar (PolSAR) images are directly influenced by the targets' orientations, as the scattering properties from the same target with different orientations can be very different. This interpretation diversity caused by the target orientations is one of the primary technical bottlenecks in PolSAR image interpretation. In this article, a 3-D convolutional vision transformer (3-D-Conv-ViT) is proposed to describe the relationship between polarimetric coherent matrices with different polarization orientation angles (POAs) for PolSAR image classification and change detection. First, 3-D convolutional neural networks are used to capture the high-level feature representations of the polarimetric coherent matrix sequence. Second, a new Rotation-3-D-ViT block is proposed to learn the local and global representations of the high-level feature maps. The self-attention mechanism in the ViT can express the regularity of polarimetric coherent matrices with different POAs and improve the PolSAR image interpretation performance. Third, combined with different classifiers, the proposed 3-D-Conv-ViT can be applied to both PolSAR image classification and change detection. Experiments on real PolSAR image datasets demonstrate that the proposed method can overcome the problem of the interpretation ambiguity caused by the target orientation. The classification accuracies of the proposed method can reach 94.01%–99.48%, and the change detection accuracies can reach 93.84%–96.86%.

Ensuring the security and safety of passengers and cargo through effective baggage screening presents a critical challenge in high-traffic environments like airports, where traditional manual processes are affected by high error rates, fatigue among security personnel, and privacy concerns. These issues underscore the urgent need for sophisticated automated systems capable of accurately detecting concealed threats. Advancements in deep learning have introduced more refined approaches to baggage screening, yet they struggle with capturing the global context of threats and challenges presented by clutter, concealment, and occlusion. Addressing these issues, this paper presents the Multi-Scale Hierarchical Transformer for X-ray Detection (MHT-X), a novel framework that surpasses conventional limitations by integrating a multi-scale contour mapping technique with a hierarchical feature extraction process using visual Transformers. MHT-X is distinctive in its ability to generate multi-level contour maps that unveil complex features of threat objects, facilitating their accurate identification. It employs a top-down approach for hierarchical feature processing, capturing vital spatial characteristics essential for precise threat localization through an efficient encoder-decoder mechanism. MHT-X represents a significant advancement in the detection of concealed and obstructed threat objects in X-ray baggage scans. By combining multi-scale contour maps with a hierarchical feature extraction process, it significantly enhances detection accuracy and outperforms traditional Convolutional Neural Network (CNN) models by effectively capturing the global context of illegal items with advanced feature representation. The proposed framework is rigorously validated on two public datasets, dubbed CLCXray, and PIDray, where it outperforms the state-of-the-art (SOTA) methods by achieving the mean average precision score of 65.26%, and 78.19%, respectively.

The multitemporal polarimetric SAR (PolSAR) data contains the scattering change information during the growth of crops. However, the current classification methods usually directly use the addition of features extracted at single-temporal or use the temporal and spatial variations of certain features, not really exploring the complete scattering variation information. The specific data representation models for multitemporal PolSAR data should combine time with polarimetry to characterize the scattering variations. However, the characterization and utilization of such kind of models are inadequate. In this article, we construct data representation model based on the power form of coherence matrix to comprehensively represent all kinds of scattering mechanism variation, which is full-rank positive semidefinite Hermitian matrix. We extract new time-variant scattering features and design vision transformer classifier accordingly for crop classification. Experiment results on RADARSAT-2 datasets show that the proposed power representation model outperforms other models.

With the development of deep learning technology, the application of convolutional neural network (CNN) and vision transformer (ViT) for polarimetric synthetic aperture radar (PolSAR) image classification has been deepened. However, the PolSAR image has very rich information due to its special data form, which makes it difficult for the existing single network structure to comprehensively extract such effective information. In addition, deep learning methods require a large amount of data for training, whereas PolSAR labeled data is scarce and difficult to obtain. Therefore, a multigranularity hybrid CNN-ViT model based on external tokens and cross-attention is proposed for PolSAR image classification. First of all, CNN is able to learn local features very well. Thus, a CNN-based external feature extractor is designed to extract local features from the PolSAR image. Then, ViT can focus on global features. So, a multigranularity attention structure is constructed for extracting global information at multiple scales. With these two modules, the model can fully access the feature information contained in PolSAR images, which is more advantageous than a single network structure. Next, to further utilize these features, a cross-attention feature fusion module is built for fusing global–local information of different granularities. Finally, by connecting with the softmax classifier, the network outputs the final prediction results. Experimental results on three benchmark datasets show that the present method using a small amount of labeled data for training also achieves the highest level of classification among the compared methods.

The Vision Transformer (ViT) model serves as a powerful model to capture and comprehend global information, particularly when trained on extensive datasets. Conversely, the Convolutional Neural Network (CNN) model is beneficial to training with small datasets for retaining essential local information. Inspired by these properties of ViT and CNN models, we introduce a hybrid framework that smoothly increases the cross-domain generalization in Semi-supervised Domain Adaptation (SSDA). To achieve this goal, we first train the ViT model on abundant labeled source data, while the CNN model is trained on a few labeled target samples. Then, these models dynamically exchange their knowledge for potential generalization to unlabeled target data via the proposed method, named Dual Dynamic Consistency Regularization (D2CR). Specifically, the ViT model provides its pseudo labels to update the global perspective for the CNN model. Similarly, the CNN model offers pseudo labels to complement the local perspective for the ViT model. The previous methods use a fixed threshold algorithm for the pseudo-labeling process. However, we utilize the dynamic threshold strategy to create pseudo labels for the bi-directional consistency regularization learning between the ViT and CNN models. We verify our approach across several SSDA benchmark datasets. The outstanding experimental results provide strong evidence of the effectiveness and superiority of our approach over previous state-of-the-art SSDA methods.

Clouds in remote sensing images inevitably affect information extraction, which hinders the following analysis of satellite images. Hence, cloud detection is a necessary preprocessing procedure. However, most existing methods have numerous calculations and parameters. In this article, a lightweight convolutional neural network (CNN)-Transformer network, CD-CTFM, is proposed to solve the problem, which is based on encoder–decoder architecture and incorporates the attention mechanism. In the encoder part, we utilize a lightweight network combing CNN and Transformer as backbone, which is conducive to extracting local and global features simultaneously. The backbone of CD-CTFM also incorporates attention gate based on dark channel extraction module. Moreover, a lightweight feature pyramid module is designed to fuse multiscale features with contextual information. In the decoder part, a lightweight channel-spatial attention module is integrated into each skip connection between encoder and decoder to extract low-level features while suppressing irrelevant information without introducing many parameters. Finally, the proposed model is evaluated on two cloud datasets, 38-Cloud and MODIS. The results demonstrate that CD-CTFM achieves comparable accuracy as the state-of-art methods and outperforms in terms of efficiency.

Seizure prediction are necessary for epileptic patients. The global spatial interactions among channels, and long-range temporal dependencies play a crucial role in seizure onset prediction. In addition, it is necessary to search for seizure prediction features in a vast space to learn new generalized feature representations. Many previous deep learning algorithms have achieved some results in automatic seizure prediction. However, most of them do not consider global spatial interactions among channels and long-range temporal dependencies together, and only learn the feature representation in the deep space. To tackle these issues, in this study, an novel bi-level programming seizure prediction model, B2-ViT Net, is proposed for learning the new generalized spatio-temporal long-range correlation features, which can characterize the global interactions among channels in spatial, and long-range dependencies in temporal required for seizure prediction. In addition, the proposed model can comprehensively learn generalized seizure prediction features in a vast space due to its strong deep and broad feature search capabilities. Sufficient experiments are conducted on two public datasets, CHB-MIT and Kaggle datasets. Compared with other existing methods, our proposed model has shown promising results in automatic seizure prediction tasks, and provides a certain degree of interpretability.

Currently, convolutional neural networks (CNNs) and vision transformers (ViTs) are widely adopted as the predominant neural network architectures for remote sensing image scene classification. Although CNNs have lower computational complexity, ViTs have a higher performance ceiling, making both suitable as backbone networks for remote sensing scene classification tasks. However, remote sensing imagery has high intraclass variation and interclass similarity, which poses a challenge for existing methods. To address this issue, we propose the class-aware self-distillation (CASD) framework. This framework uses an end-to-end distillation mechanism to mine class-aware knowledge, effectively reducing the impact of significant intraclass variation and interclass similarity in remote sensing imagery. Specifically, our approach involves constructing pairs of images: similar pairs consisting of images belonging to the same class, and dissimilar pairs consisting of images from different classes. We then apply a distillation loss that we designed, which distills the corresponding probability distributions to ensure that the distributions of similar pairs become more consistent, and those of dissimilar pairs become more distinct. In addition, the enforced learnable $\alpha$ added to the distillation loss further amplifies the network's ability to comprehend class-aware knowledge. The experiment section demonstrates that our method CASD outperforms other methods on four publicly available datasets. And the ablation experiments demonstrate the effectiveness of the method.

In the digital era, social media has become a cornerstone for educational institutions, driving public engagement and enhancing institutional communication. This study utilizes AI-driven image processing and Web-enhanced Deep Learning (DL) techniques to investigate the effectiveness of King Faisal University’s (KFU’s) social media strategy as a case study, particularly on Twitter. By categorizing images into five primary event management categories and subcategories, this research provides a robust framework for assessing the social media content generated by KFU’s administrative units. Seven advanced models were developed, including an innovative integration of Vision Transformers (ViTs) with Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, VGG16, and ResNet. The AI-driven ViT-CNN hybrid model achieved perfect classification accuracy (100%), while the “Development and Partnerships” category demonstrated notable accuracy (98.8%), underscoring the model’s unparalleled efficacy in strategic content classification. This study offers actionable insights for the optimization of AI-driven digital communication strategies and Web-enhanced data collection processes, aligning them with national development goals and Saudi Arabia’s Vision 2030, thereby showcasing the transformative power of DL in event-centric management and the broader higher education landscape.

Fire detection is a critical task in environmental monitoring and disaster prevention, with traditional methods often limited in their ability to detect fire and smoke in real time over large areas. The rapid identification of fire and smoke in both indoor and outdoor environments is essential for minimizing damage and ensuring timely intervention. In this paper, we propose a novel approach to fire and smoke detection by integrating a vision transformer (ViT) with the YOLOv5s object detection model. Our modified model leverages the attention-based feature extraction capabilities of ViTs to improve detection accuracy, particularly in complex environments where fires may be occluded or distributed across large regions. By replacing the CSPDarknet53 backbone of YOLOv5s with ViT, the model is able to capture both local and global dependencies in images, resulting in more accurate detection of fire and smoke under challenging conditions. We evaluate the performance of the proposed model using a comprehensive Fire and Smoke Detection Dataset, which includes diverse real-world scenarios. The results demonstrate that our model outperforms baseline YOLOv5 variants in terms of precision, recall, and mean average precision (mAP), achieving a mAP@0.5 of 0.664 and a recall of 0.657. The modified YOLOv5s with ViT shows significant improvements in detecting fire and smoke, particularly in scenes with complex backgrounds and varying object scales. Our findings suggest that the integration of ViT as the backbone of YOLOv5s offers a promising approach for real-time fire detection in both urban and natural environments.

The balance of brain functional segregation (i.e., the process in specialized local subsystems) and integration (i.e., the process in global cooperation of the subsystems) is crucial for cognition in human beings, and many deep learning models have been used to evaluate the spatial information during EEG-based affective computing. However, acquiring the intrinsic spatial representation in the topology of EEG channels is still challenging. To further address the issue, we propose the FAformer to enhance spatial information in EEG signals with parallel-branch architectures based on a vision transformer (ViT). In the encoder, there is a branch that utilizes Adaptive Neural Fourier Operators (AFNO) to model global spatial patterns using the Fourier transform in the electrode channel dimension. The other branch utilizes multi-head self-attention (MSA) to explore the dependence of emotion on different channels, which is conducive to building key local networks. Additionally, a self-supervised learning (SSL) task of adaptive feature dissociation (AdaptiveFD) is developed to improve the distinctiveness of spatial features generated from the parallel branches and guarantee robustness in different subjects. FAformer achieves superior performance over the competitive models on the DREAMER and DEAP. Moreover, the rationality and hyperparameters analysis are conducted to demonstrate the effectiveness of the FAformer. Finally, the visualization of features reveals the spatial global connections and key local patterns during the deep learning process in FAformer, which benefits EEG-based affective computing. [ABSTRACT FROM AUTHOR]

Audio-visual speech synthesis (AVSS) has garnered attention in recent years for its utility in the realm of audio-visual learning. AVSS transforms one speaker's speech into another's audio-visual stream while retaining linguistic content. This approach extends existing AVSS methods by first modifying vocal features from the source to the target speaker, akin to voice conversion (VC), and then synthesizing the audio-visual stream for the target speaker, termed audio-visual synthesis (AVS). In this work, a novel AVSS approach is proposed using vision transformer (ViT)-based Autoencoders (AEs), enriched with a combination of cycle consistency and reconstruction loss functions, with the aim of enhancing synthesis quality. Leveraging ViT's attention mechanism, this method effectively captures spectral and temporal features from input speech. The combination of cycle consistency and reconstruction loss improves synthesis quality and aids in preserving essential information. The proposed framework is trained and tested on benchmark datasets, and compared extensively with state-of-the-art (SOTA) methods. The experimental results demonstrate the superiority of the proposed approach over existing SOTA models, in terms of quality and intelligibility for AVSS, indicating the potential for real-world applications. [ABSTRACT FROM AUTHOR]