Search

Facial Emotion Recognition is gaining interest from researchers in various fields due to its numerous applications. The Vision Transformer (ViT) outperforms convolutional neural network (CNN)-based systems significantly in terms of performance when compared to contemporary image categorisation algorithms. In this context, the primary objective of this paper is to present an innovative and efficient deep learning model for facial emotion recognition. This hybrid intelligent model combines the Vision Transformer with Particle Swarm Optimization (PSO). The proposed model accurately identified the majority of emotions in images, achieving an accuracy of 95.93% for the Fer2013plus dataset and 100% for the CK+ dataset, as demonstrated through experiments conducted on both datasets. According to the results, the proposed facial expression recognition approach has either outperformed or closely matched the performance of state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Recently, convolutional neural network (CNN) and Transformer have been successfully applied in hyperspectral image (HSI) classification. Despite their success, some challenges persist. Popular HSI classification methods typically employ only a single patch as the network input, which may not provide robust information for various datasets or within different scenarios of a specific dataset. Large-scale patches are usually conducive to classifying large-scale scenes, while small-scale patches excel at classifying boundary locations. In addition, the majority of methods do not distinguish between the central pixel and its adjacent pixels, although the information from the central position is crucial. In this paper, a novel double-branch hierarchical partial convolution network (DBHP-Net) is proposed to address the aforementioned challenges. Concretely, DBHP-Net employs a new convolution, named TConv, to emphasize mining the spatial-channel features of the central position. Then, the TConvFormer Block is built by adopting the advanced engineering techniques of Transformer. Furthermore, we construct our model in a hierarchical style. The double-branch structure with large-scale and small-scale patches provides complementary information from the dataset. Experimental results based on randomly selected samples and spatially disjoint samples of two benchmark datasets demonstrate that our method outperforms other state-of-the-art (SOTA) methods. [ABSTRACT FROM AUTHOR]

Vision Transformers (ViTs) have become one of the main architectures in deep learning with the self-attention mechanism, and are becoming an alternative to Convolutional Neural Networks (CNNs) for remote sensing scene classification tasks. However, the earlier self-attention layer of ViTs focuses on local features rather than global features, and the deeper self-attention layer focuses on global features but ignores the different impact of different frequency information. This will greatly increase the training and computational cost due to the quadratic complexity of the self-attention mechanism on the long sequence representation. In this paper, we propose a hybrid CNN – vision transformer structure (HCVNet), which uses convolutional layers to replace the earlier self-attention layers, and a novel Frequency Multi-head Self Attention (F-MSA) mechanism to replace the deeper self-attention layers. Specifically, F-MSA is a dual-stream structure that reduces the computational cost and improves the classification performance by encoding the high/low frequency information separately. In addition, a Semantic-aware Localization (SaL) module is introduced, which can guide the selection of crop by learning prior knowledge, avoiding the issue of pure background sampling. Our method performed an accuracy value of 97.20 ± 0.02% on the Aerial Image Dataset and 93.89 ± 0.03% on the NWPU-RESISC45 Dataset, with low complexity costs. [ABSTRACT FROM AUTHOR]

Automated brain tumor detection and classification systems have gained popularity in recent years because traditional diagnosis procedures are time-consuming and costly in nature. Deep learning(DL) methods, specifically pre-trained convolutional neural networks (CNNs), have shown promising results in accurately and rapidly classifying brain tumors. However, the lack of diverse magnetic resonance imaging (MRI) datasets has hindered the ability of DL algorithms to generalize effectively. To address this issue, this paper proposes a DL model using generative adversarial networks (GANs) in conjunction with data augmentation strategies and a structural similarity loss function employed for generating annotated images. A novel DL model inspired from Vision Transformer, Shrinking Linear Time Vision Transformer (SL(t)-ViT) network model is proposed for disease data classification. The model underwent extensive evaluation across multiple datasets, employing standard performance metrics to assess its efficacy in brain tumor identification. The proposed model achieved remarkable testing accuracies of 0.995, 0.996, 0.9954, 0.998, and 0.997 for binary classification tasks across multiple datasets, and 0.986, 0.982, 0.985, and 0.993 for multi-class classification tasks. These results underscore the superior performance of our proposed model, showcasing its capability to outperform state-of-the-art techniques. Specifically, it demonstrated a substantial margin of improvement, ranging from 1-2% for binary classification and 9-10% for multi-class classification, solidifying its position as a leading approach in brain tumor identification. Results demonstrate the efficacy of the proposed model, outperforming other models. Overall, this study highlights the potential of SL(t)-ViT and GANs in improving the accuracy, resource consumption, and efficiency of brain tumor diagnosis. [ABSTRACT FROM AUTHOR]

Accurately segmenting roads is challenging due to substantial intra-class variations, indistinct inter-class distinctions, and occlusions caused by shadows, trees, and buildings. To address these challenges, it is crucial to simultaneously pay attention to important texture details and perceive global geometric context information. Recent research has shown that CNN-Transformer hybrid structures outperform using CNN or Transformer alone. CNN excels at extracting local detail features and Transformer naturally has the ability to perceive global context information. In this paper, we propose a dual-branch network module called ConSwin, which combines the advantages of ResNet and Swin Transformer for better road information extraction. Taking ConSwin as the basic encoding module, we build an hourglass-shaped network with an encoder-decoder structure. In order to better transfer texture and structural detail information between the encoder and decoder, we also design two novel connection blocks. We conduct comparative experiments on the Massachusetts and CHN6-CUG datasets, and the proposed method outperforms state-of-the-art methods in terms of overall accuracy, IOU, and F1 metrics. Other experiments verify the effectiveness of our method, while visualization results demonstrate its ability to obtain better road representations. [ABSTRACT FROM AUTHOR]

Most existing remote sensing disaster assessment methods rely on convolutional neural networks (CNNs). Although CNNs can extract effective semantic features, determining global spatial relationships remains limited due to the locality of convolutional operations. The recently developed transformer-based method can extract global information from images effectively by encoding image tokens. However, its consumption of computational resources varies markedly with increasing image resolution. In this paper, we propose a novel transformer-based neural network for disaster assessment problems. The network uses a multihead neighbourhood attention (MNA) transformer as the base layer of the encoder to achieve more efficient self-attention computation. In addition, the bitemporal feature fusion module (BFFM) performs differential enhancement and injects the change information to the decoder via skip connections. The multiscale tokenizer generates multiscale image tokens to mitigate the loss of detail during encoding. Experimental results on three datasets show that the proposed method outperforms existing methods. [ABSTRACT FROM AUTHOR]

The intelligent segmentation of high-resolution remote sensing (HRS) image, also called as dense prediction task for HRS image, has been and will continue to be important research in the remote sensing community. In recent years, the growing wave of artificial intelligence (AI) technology has introduced innovative paradigms to this domain, yielding outstanding results and overcoming many challenges with conventional segmentation techniques. This paper provides a comprehensive review of these intelligent segmentation methodologies, including traditional pattern recognition, convolution neural network (CNN)-based, and Transformer-based techniques. However, the explosive but incomplete development of intelligent segmentation techniques also poses more challenges for earth observation experts, the most of which is the technical interpretability. Consequently, we consider these segmentation techniques in the aspect of explainable artificial intelligence (XAI). Data-centric XAI thinks the practical applications of the segmentation model while model-centric XAI will facilitate the understanding of decision-making processes and the adjustment of structural features. Moreover, this review identifies novel research questions and provides constructive insights and recommendations to HRS image segmentation tasks, which may shed new light on the intelligent segmentation methods within the remote sensing image understanding community. [ABSTRACT FROM AUTHOR]

Crop field boundary extraction is crucial to remote sensing images attained to support agricultural production and planning. In recent years, deep convolutional neural networks (CNNs) have gained significant attention for edge detection tasks. Moreover, transformers have shown superior feature extraction and classification capabilities compared to CNNs due to their self-attention mechanism. We proposed a novel structure that combines full edge extraction with CNNs and enhances connectivity with transformers, consisting of three stages: a) preprocessing the training data; b) training the semantic edge and spatial structure graph models; and c) vectorizing the fusion of semantic edge and spatial structure graph outputs. To cater specifically to high-resolution remote sensing image crop-field boundary extraction, we developed a CNN model called Densification D-LinkNet. Its fullscale skip connections and edge-guided module adapted well to different crop-field boundary features. Additionally, we employed a spatial graph structure generator (Relationformer) based on object detection that directly outputs the structural graph of the crop field boundary. This method relies on good connectivity to repair fragmented edges that may appear in semantic edge detection. Through multiple experiments and comparisons with other edge-detection methods, such as BDCN, DexiNed, PidiNet, and EDTER, we demonstrated that our proposed method can achieve at least 9.77% improvement in boundary intersection over union (IoU) and 2.07% improvement in polygon IoU on two customized datasets. These results indicate the effectiveness and robustness of our approach. [ABSTRACT FROM AUTHOR]

Automatic building detection from remote sensing images holds significant applications in various domains, including cartography, land-use change detection, urban planning, and 3D intelligent city construction. Driven by the innovation of deep learning technology, convolutional neural networks (CNNs) have emerged as a powerful framework for automatic interpretation, exhibiting hierarchical feature generation and robust representation for building extraction. However, the inherent computational limitations of convolutional operators hinder the acquisition of an effective global receptive field. Recent advancements in deep learning have witnessed the adoption of the multi-head self-attention (MHSA) mechanism by Transformer, presenting the superiority of global context construction. Nonetheless, this network structure encounters difficulties in effectively modelling two-dimensional spatial structure features with scale variation in semantic segmentation and often has huge parameters with high algorithm complexity. To address the above problems, this paper proposed a hybrid deep neural network synergistically harnessing the superiority of CNN and Transformer to extract buildings from remote sensing images. Meanwhile, an improved MHSA mechanism is developed to facilitate the construction of multiscale global representations, concurrently reducing computational complexity. Furthermore, the cross-level MHSA modules the semantic correlation between the deep and shallow-level features to refine spatial fine-grained information. The proposed method was comprehensively evaluated on two building datasets via quantitative analysis and visualization. The experimental results confirmed that the developed network and modules outperform other state-of-the-art methods and can effectively improve the accuracy (93.4% F1-score on the WHU dataset, 89.38% F1-score on the CAMB dataset) with high efficiency for building extraction. [ABSTRACT FROM AUTHOR]

Building detection and change detection using remote sensing images can help urban and rescue planning. Moreover, they can be used for building damage assessment after natural disasters. Currently, most of the existing models for building detection use only one image (pre-disaster image) to detect buildings. This is based on the idea that post-disaster images reduce the model's performance because of the presence of destroyed buildings. In this paper, we propose a siamese model, called SiamixFormer, which uses pre- and post-disaster images as input. Our model has two encoders and has a hierarchical transformer architecture. The output of each stage in both encoders is given to a temporal transformer for feature fusion in a way that query is generated from pre-disaster images and (key, value) is generated from post-disaster images. To this end, temporal features are also considered in feature fusion. Another advantage of using temporal transformers in feature fusion is that they can better maintain large receptive fields generated by transformer encoders compared with CNNs. Finally, the output of the temporal transformer is given to a simple MLP decoder at each stage. The SiamixFormer model is evaluated on xBD, and WHU datasets, for building detection and on LEVIR-CD and CDD datasets for change detection and could outperform the state-of-the-art. [ABSTRACT FROM AUTHOR]

The purpose of pan-sharpening is to generate high-resolution multispectral (HRMS) images by combining multispectral images (MS) and panchromatic images (PAN), which have become an important part of remote sensing image processing. Therefore, how to better extract complete feature information from MS images and PAN images has become the focus of our attention. In this paper, we propose a new pansharpening method, called Transformer-based Dual-path cross fusion network (TDF) for Pan-sharpening remote sensing images, which aims to extract the spatial details of PAN images while maintaining the spectral fidelity of MS images. The whole network structure can be divided into two parts: in the encoder part, we adopt the Swin-Transformer module for the downsampling operation, which expands the sensory field of the network to the feature map, and then extracts the global information. However, since the Swin-Transformer module is not good at extracting pixel-level details, we introduce the Base Feature Extraction (BFE) and the Invertible Neural Network Block (INNB) modules for the interaction between the local and the global feature information. we also introduce the Edge-Enhancement Block (EEB) to further enhance the feature extraction at multi-scales during the image fusion process. In the decoder section, we once again employ the Swin Transformer module for downsampling. After the convolution operation and activation function, we utilize the Sub-Pixel Convolutional Neural Network for upsampling to generate the ultimate high-resolution multispectral images. Simulation experiments and real experiments are conducted on QuickBird (QB) and WorldView2 (WV2) datasets, which demonstrated our method are superior to the current methods. [ABSTRACT FROM AUTHOR]

Hyperspectral images carry numerous spectral bands, and their wealth of band data is a valuable source of information for the accurate classification of ground objects. Three-dimensional (3D) convolution, although an excellent spectral information extraction method, is limited by its huge number of parameters and long model training time. To allow better integration of 3D convolution with the most popular transformer models currently available, a new architecture called mobile 3D convolutional vision transformer (MDvT) is proposed. The MDvT introduces inverted residual structure to reduce the number of model parameters and balance the data mining efficiency of low-dimensional data input. Simultaneously, a square patch is used to cut the sequence of tokens to accelerate the model operation. Through extensive experiments, we evaluated the classification overall performance of the proposed MDvT on the WHU-Hi and Pavia University datasets, and demonstrated significant improvements in classification accuracy and model runtime compared with classical deep learning models. It is worth noting that compared with directly integrating 3D convolution into the transformer model, the MDvT architecture improves the accuracy while reducing the time to train an epoch by approximately 58.54%. To facilitate the reproduction of the work in this paper, the model code is available at . [ABSTRACT FROM AUTHOR]

Wildfires pose a serious threat to the environment, ecosystems, property, biodiversity, and human life. Early detection of wildland fires is crucial for effective firefighting and mitigation. In this paper, we propose an ensemble learning method, called CT-Fire, which combines the deep CNN RegNetY and the vision transformer EfficientFormer v2 to recognize and detect forest fires on ground and aerial images. Testing results showed that CT-Fire achieved excellent performance with fast speed and accuracy of 99.62% and 87.77% using ground and aerial images, respectively. CT-Fire also outperformed benchmark CNNs and vision transformer methods, showing its accurate reliability in detecting wildfires. It also surpassed various challenges, including the detection of very small wildfires, background complexity, image quality, and wildland fire variability in terms of intensity, size, and shape. [ABSTRACT FROM AUTHOR]

Remote sensing Change Detection (CD) involves identifying changing regions of interest in bi-temporal remote sensing images. CD technology has rapidly developed in recent years through the powerful learning ability of Convolutional Neural Networks (CNN), affording complex feature extraction. However, the local receptive fields in the CNN limit modeling long-range contextual relationships in semantic changes. Therefore, this work explores the great potential of Siamese Transformers in CD tasks and proposes a general CD model entitled STCD that relies on Swin Transformers. In the encoding process, pure Transformers without CNN are used to model the long-range context of semantic tokens, reducing computational overhead and improving model efficiency compared to current methods. During the decoding process, the 3D convolution block obtains the changing features in the time series and generates the predicted change map in the deconvolution layer with axial attention. Extensive experiments on three binary CD datasets and one semantic CD dataset demonstrate that the proposed STCD model outperforms several popular benchmark methods considering performance and the required parameters. Among the STCD variants, the F1-Score of the Base-STCD on the three binary CD datasets LEVIR, DSIFN, and SVCD reached 89.85%, 54.72%, and 93.75%, respectively, and the mF1-Score and mIoU on the semantic CD dataset SECOND were 75.60% and 66.19%. [ABSTRACT FROM AUTHOR]

Hyperspectral unmixing (HU) is considered one of the most important ways to improve hyperspectral image analysis. HU aims to break down the mixed pixel into a set of spectral signatures, often commonly referred to as endmembers, and determine the fractional abundance of those endmembers. Deep learning (DL) approaches have recently received great attention regarding HU. In particular, convolutional neural networks (CNNs)-based methods have performed exceptionally well in such tasks. However, the ability of CNNs to learn deep semantic features is limited, and computing cost increase dramatically with the number of layers. The appearance of the transformer addresses these issues by effectively representing high-level semantic features well. In this article, we present a novel approach for HU that utilizes a deep convolutional transformer network. Firstly, the CNN-based autoencoder (AE) is used to extract low-level features from the input image. Secondly, the concept of tokenizer is applied for feature transformation. Thirdly, the transformer module is used to capture the deep semantic features derived from the tokenizer. Finally, a convolutional decoder is utilized to reconstruct the input image. The experimental results on synthetic and real datasets demonstrate the effectiveness and superiority of the proposed method compared with other unmixing methods. [ABSTRACT FROM AUTHOR]

Deep learning instantiated by convolutional neural networks has achieved great success in high-resolution remote-sensing image change detection. However, such networks have a limited receptive field, being unable to extract long-range dependencies in a scene. As the transformer model with self-attention can better describe long-range dependencies, we introduce a hierarchical transformer model to improve the precision of change detection in high-resolution remote sensing images. First, the hierarchical transformer extracts abstract features from multitemporal remote sensing images. To effectively minimize the model's complexity and enhance the feature representation, we limit the self-attention calculation of each transformer layer to local windows with different sizes. Then, we combine the features extracted by the hierarchical transformer and input them into a nested U-Net to obtain the change detection results. Furthermore, a simple but effective model fusion strategy is adopted to improve the change detection accuracy. Extensive experiments are carried out on two large-scale data sets for change detection, LEVIR-CD and SYSU-CD. The quantitative and qualitative experimental results suggest that the proposed method outperforms the advanced methods in terms of detection performance. [ABSTRACT FROM AUTHOR]

Convolutional Neural Network (CNN) has developed hyperspectral image (HSI) classification effectively. Although many CNN-based models can extract local features in HSI, it is difficult for them to extract global features. With its ability to capture long-range dependencies, Transformer is gradually gaining prominence in HSI classification, but it may overlook some local details when extracting features. To address these issues, we proposed a CNN and transformer interaction network (CTIN) for HSI classification. Firstly, A dual-branch structure was constructed in which CNN and Transformer are arranged in parallel to simultaneously extract global features and local features in HSI. Secondly, a feature interaction module has been imported between the two branches, thus facilitating a bi-directional flow of information between the global and local feature spaces. In this way, the network structure combines the advantages of CNN and Transformer in extracting features as much as possible. In addition, a token generation method is designed to harness abundant contextual information that is relevant to the centre pixel, and improve the accuracy of the final classification. Experiments were conducted on four hyperspectral datasets (two classical datasets – Indian Pines, Salinas Valley, a new satellite dataset – Yellow River, and an self-made UAV dataset-Yellow River Willow). Experimental results show that the proposed method outperforms the other state-of-the-art methods, with overall accuracies of 99.21%, 99.61%, 92.40%, and 98.17%, respectively. [ABSTRACT FROM AUTHOR]

Nowadays ranked 3rd, pancreatic cancer is predicted to become second highest mortality cancer in next decade. Early diagnosis is the only way to significantly improve survival rate for which endoscopic ultrasound (EUS) stands as the only viable medical imaging option. One challenging aspect is the complex interpretation of images during examination. It is not rare for non-experts to miss the screening of parts of the pancreas, leaving tumours undetected. Here, we propose an automated method to support non-expert clinicians in their practice by providing a deep-learning based tool able to detect the anatomical parts seen under EUS. We have collected 41 EUS videos and annotated the anatomy in each video frame. Considering the challenging and novel nature of EUS data, we propose a systematic analysis of feature extractors and temporal modules. We extend popular models with LSTM modules and compare their performance to the newly introduced vision transformers, yielding an overall comparison of 35 models. The results highlight the benefits of transformers for their ability to capture more anatomical context thanks to the division into patches and positional embeddings. As a result, our study paves the way to AI-assisted pancreas examination for early cancer detection. [ABSTRACT FROM AUTHOR]

The urbanization worldwide leads to the rapid increase of solid waste, posing a threat to environment and people's wellbeing. However, it is challenging to detect solid waste sites with high accuracy due to complex landscape, and very few studies considered solid waste mapping across multi-cities and in large areas. To tackle this issue, this study proposes a novel deep learning model for solid waste mapping from very high resolution remote sensing imagery. By integrating a multi-scale dilated convolutional neural network (CNN) and a Swin-Transformer, both local and global features are aggregated. Experiments in China, India and Mexico indicate that the proposed model achieves high performance with an average accuracy of 90.62%. The novelty lies in the fusion of CNN and Transformer for solid waste mapping in multi-cities without the need for pixel-wise labelled data. Future work would consider more sophisticated methods such as semantic segmentation for fine-grained solid waste classification. [ABSTRACT FROM AUTHOR]

With the remarkable success of change detection (CD) in remote sensing images in the context of deep learning, many convolutional neural network (CNN) based methods have been proposed. In the current research, to obtain a better context modeling method for remote sensing images and to capture more spatiotemporal characteristics, several attention-based methods and transformer (TR)-based methods have been proposed. Recent research has also continued to innovate on TR-based methods, and many new methods have been proposed. Most of them require a huge number of calculation to achieve good results. Therefore, using the TR-based mehtod while maintaining the overhead low is a problem to be solved. Here, we propose a GNN-based multi-scale transformer siamese network for remote sensing image change detection (GMTS) that maintains a low network overhead while effectively modeling context in the spatiotemporal domain. We also design a novel hybrid backbone to extract features. Compared with the current CNN backbone, our backbone network has a lower overhead and achieves better results. Further, we use high/low frequency (HiLo) attention to extract more detailed local features and the multi-scale pooling pyramid transformer (MPPT) module to focus on more global features respectively. Finally, we leverage the context modeling capabilities of TR in the spatiotemporal domain to optimize the extracted features. We have a relatively low number of parameters compared to that required by current TR-based methods and achieve a good effect improvement, which provides a good balance between efficiency and performance. [ABSTRACT FROM AUTHOR]