Search

The purpose of this study is to establish a deep learning automatic assistance diagnosis system for benign and malignant classification of mediastinal lesions in endobronchial ultrasound (EBUS) images. EBUS images are in the form of video and contain multiple imaging modes. Different imaging modes and different frames can reflect the different characteristics of lesions. Compared with previous studies, the proposed model can efficiently extract and integrate the spatiotemporal relationships between different modes and does not require manual selection of representative frames. In recent years, Vision Transformer has received much attention in the field of computer vision. Combined with convolutional neural networks, hybrid transformers can also perform well on small datasets. This study designed a novel deep learning architecture based on hybrid transformer called TransEBUS. By adding learnable parameters in the temporal dimension, TransEBUS was able to extract spatiotemporal features from insufficient data. In addition, we designed a two-stream module to integrate information from three different imaging modes of EBUS. Furthermore, we applied contrastive learning when training TransEBUS, enabling it to learn discriminative representation of benign and malignant mediastinal lesions. The results show that TransEBUS achieved a diagnostic accuracy of 82% and an area under the curve of 0.8812 in the test dataset, outperforming other methods. It also shows that several models can improve performance by incorporating two-stream module. Our proposed system has shown its potential to help physicians distinguishing benign and malignant mediastinal lesions, thereby ensuring the accuracy of EBUS examination. [ABSTRACT FROM AUTHOR]

Wheat yellow rust disease poses a significant threat to global wheat yield and grain quality. Early detection of this disease will help to minimize the loss caused by its effects. Existing models work well on images taken in a controlled environment, whereas a uniform background is placed behind the leaf, but these models fail to produce good results in natural settings. Previous research also involves manual interventions in the pipeline to achieve good classification results such as cropping the images, using uniform backgrounds, etc. These systems are not practical to use in natural environments where there will be a lot of background noise to the image and manual cropping becomes an extra step for the farmer. Moreover, the unavailability of the dataset in which images of leaves are taken in a natural setting became another challenge. In this research, a dataset is curated and leaves are annotated for object detection, object segmentation further the leaves are classified into 3 classes ie healthy, resistant, and susceptible. A novel unsupervised image rotation algorithm is proposed that takes input from YOLOv8 to align the leave in such a way that maximum background can be removed by a rectangular bounding box. Then the comparison between multiple state-of-the-art segmentation models ie. UNET, Segment-Anything (SAM), Segnet, LinkNet, PSPNet, FPN, Deep-Labv3+ (Xception), and DeepLabv3+ (Mo-bileNet) has shown that UNET has outperformed all the other segmentation models with an IOU score of 0.9563. Lastly for classification, the performance of multiple convolution neural networks ie. VGG16, Resnet 101(v2), Xception, Mo-bileNetV2, and Transformer-based models ie. Swin trans-former and MobileVit have been compared. Swin transformer has outperformed the state-of-the-art CNN models with an accuracy of 95.8%. This paper proposes a complete robust pipeline that can be deployed in natural environment and does not need any manual intervention to produce good results. This research shows that good localization of leaves and removal of unwanted background noise at the earliest stage of the pipeline will assist the segmentation model to effectively segment the leaf from the background which will enable classification models to achieve high classification accuracy, even when dealing with very small datasets. [ABSTRACT FROM AUTHOR]

We developed a deep learning-based bubble detector with a Shifted window Transformer (Swin Transformer) to detect and segment individual bubbles among overlapping bubbles. To verify the performance of the detector, we calculated its average precision (AP) with different number of training images. The mask AP increased with the increase in the number of training images when there were less than 50 images. It was observed that the AP for the Swin Transformer and ResNet were almost the same when there were more than 50 images; however, when few training images were used, the AP of the Swin Transformer were higher than that of the ResNet. Furthermore, for the increase in void fraction, the AP of the Swin Transformer showed a decrease similar to that in the case of the ResNet; however, for few training images, the AP of the Swin Transformer was higher than that of the ResNet in all void fractions. Moreover, we confirmed the detector trained with experimental and synthetic bubble images was able to segment overlapping bubbles and deformed bubbles in a bubbly flow experiment. Thus, we verified that the new bubble detector with Swin Transformer provided higher AP than the detector with ResNet for fewer training images. [ABSTRACT FROM AUTHOR]

Recent advances of artificial intelligence (AI) in retinal imaging found its application in two major categories: discriminative and generative AI. For discriminative tasks, conventional convolutional neural networks (CNNs) are still major AI techniques. Vision transformers (ViT), inspired by the transformer architecture in natural language processing, has emerged as useful techniques for discriminating retinal images. ViT can attain excellent results when pretrained at sufficient scale and transferred to specific tasks with fewer images, compared to conventional CNN. Many studies found better performance of ViT, compared to CNN, for common tasks such as diabetic retinopathy screening on color fundus photographs (CFP) and segmentation of retinal fluid on optical coherence tomography (OCT) images. Generative Adversarial Network (GAN) is the main AI technique in generative AI in retinal imaging. Novel images generated by GAN can be applied for training AI models in imbalanced or inadequate datasets. Foundation models are also recent advances in retinal imaging. They are pretrained with huge datasets, such as millions of CFP and OCT images and fine-tuned for downstream tasks with much smaller datasets. A foundation model, RETFound, which was self-supervised and found to discriminate many eye and systemic diseases better than supervised models. Large language models are foundation models that may be applied for text-related tasks, like reports of retinal angiography. Whereas AI technology moves forward fast, real-world use of AI models moves slowly, making the gap between development and deployment even wider. Strong evidence showing AI models can prevent visual loss may be required to close this gap. [ABSTRACT FROM AUTHOR]

This research aims to explore key issues in the field of walking robots and exoskeletons. By using a depth camera to record human walking data in different terrains and employing RGB and depth information for image classification, our approach utilizes both early fusion and late fusion methods to explore their impact on terrain classification. This approach achieved fast and accurate performance in recognizing terrain. We captured data for walking upstairs, downstairs, upward ramps, downward ramps and level ground and used the vision transformer for data fusion. This work compares the performance of unimodal RGB model, unimodal deep model, Pre-Fusion model and Post-Fusion model in terms of accuracy, recall and processing frequency, with accuracies of 0.954, 0.934, 0.976, and 0.976, respectively. The results show that the Post-Fusion model slightly outperforms the Pre-Fusion model in terms of accuracy and recall with more reliable classification performance. To understand the attention mechanism better, we've been visualizing the weights of the model, observing how different attention heads may capture different features of the data from various dimensional perspectives. The visualization helps in interpreting the model's focus and how it distinguishes between features of different nature, such as color texture and spatial depth. These findings can inform the ongoing refinement of terrain classification systems for applications in robotics and autonomous navigation. [ABSTRACT FROM AUTHOR]

In order to predict phase distributions within complex pore structures during two-phase capillary-dominated drainage, we select subsamples from computerized tomography (CT) images of rocks and simulated porous media, and develop a pore morphology-based simulator (PMS) to create a diverse dataset of phase distributions. With pixel size, interfacial tension, contact angle, and pressure as input parameters, convolutional neural network (CNN), recurrent neural network (RNN) and vision transformer (ViT) are transformed, trained and evaluated to select the optimal model for predicting phase distribution. It is found that commonly used CNN and RNN have deficiencies in capturing phase connectivity. Subsequently, we develop a higher-dimensional vision transformer (HD-ViT) that drains pores solely based on their size, regardless of their spatial location, with phase connectivity enforced as a post-processing step. This approach enables inference for images of varying sizes and resolutions with inlet-outlet setup at any coordinate directions. We demonstrate that HD-ViT maintains its effectiveness, accuracy and speed advantage on larger sandstone and carbonate images, compared with the microfluidic-based displacement experiment. In the end, we train and validate a 3D version of the model. [ABSTRACT FROM AUTHOR]

The fashion industry is often cited as the world's second-largest polluting sector, necessitating the development of systems for clothing reuse. This study aims to construct a system capable of determining the reusability of clothing items based on images. The judgment system utilizes classification models from machine learning, with parameters set through 5-fold cross-validation. Experimental results demonstrated that a model fine-tuned using Vision Transformer (ViT) achieved an F1 score of 0.98. Consequently, this establishes a method for straightforwardly determining clothing reusability from pictures. [ABSTRACT FROM AUTHOR]

This paper introduces Knowledge Distillation of Vision Transformer (KDViT), a novel approach for medical image classification. The Vision Transformer architecture incorporates a self-attention mechanism to autonomously learn image structure. The input medical image is segmented into patches and transformed into low-dimensional linear embeddings. Position information is integrated into each patch, and a learnable classification token is appended for classification, thereby preserving spatial relationships within the image. The output vectors are then fed into a Transformer encoder to extract both local and global features, leveraging the inherent attention mechanism for robust feature extraction across diverse medical imaging scenarios. Furthermore, knowledge distillation is employed to enhance performance by transferring insights from a large teacher model to a small student model. This approach reduces the computational requirements of the larger model and improves overall effectiveness. Integrating knowledge distillation with two Vision Transformer models not only showcases the novelty of the proposed solution for medical image classification but also enhances model interpretability, reduces computational complexity, and improves generalization capabilities. The proposed KDViT model achieved high accuracy rates of 98.39%, 88.57%, and 99.15% on the SARS-CoV-2-CT, COVID-CT, and iCTCF datasets respectively, surpassing the performance of other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

The domain of deep learning has seen significant advancements, particularly in the context of detecting macular edema from images of the retina, in recent times. This study introduces an innovative model for identifying macular edema, employing two deep learning models: Deeplabv3 + and VGG with a vision transformer. The Deeplabv3 + model is used to segment the macula region in the retinal images. The segmented macula region is then fed into the VGG for feature extraction with a vision transformer model for detection. This approach leverages the strengths of both models in detecting accurately and efficiently. The Deeplabv3 + model can accurately segment the macula region, which is crucial for accurate detection. The VGG combined with a vision transformer model proves highly efficient in detecting even subtle changes in the macular region, signifying the existence of macular edema. The results of our experiments with the dataset show that the proposed method outperforms current cutting-edge techniques. With an outstanding precision rate of 99.53%, the suggested approach firmly solidifies its superiority. The results highlight the effectiveness of the proposed technique in precisely and effectively detecting pathological fluid accumulation in retina images. This ability can have a substantial influence on the early detection and management of eye disorders. [ABSTRACT FROM AUTHOR]

Microservice Architectures (MSA) provide flexibility and scalability in software development. However, accurately measuring the level of interdependence among Microservices continues to be a difficult task. Precisely evaluating this connection is essential for efficient MSA design, maintenance, and future development. Conventional techniques for assessing Microservice coupling are frequently done by hand, require a significant amount of time, and are susceptible to mistakes. This impedes the capacity to make well-informed judgments regarding the integration and adjustment of services. This study introduces a new method for automating the computation of the Microservice Coupling Index (MCI) by utilizing the You Only Look at One Sequence (YOLOS) object identification technique in combination with Vision Transformer (ViTs) technology. YOLOS is utilized for identifying constituents within Unified Modeling Language (UML) Component Diagrams, facilitating precise classification and effective assessment of coupling. The model exhibits varying performance over multiple Intersection over Union (IoU) thresholds and object sizes, with an average precision (AP) of 0.406 over IoU values ranging from 0.50 to 0.95. The maximum precision is achieved at an IoU of 0.50, with an AP of 0.709. The model demonstrates good performance in identifying smaller components, especially when evaluated at a 0.75 IoU threshold. However, it faces challenges in detecting small items, suggesting potential areas for improvement in future iterations. Initial results indicate that this automation greatly decreases the need for manual, labor-intensive tasks and enhances the precision of measuring coupling in MSA, hence facilitating effective decision-making in service integration and modification. Automating the computation of the coupling index has the potential to significantly influence the design and management of durable and readily controllable microservice architectures. [ABSTRACT FROM AUTHOR]

With the increase of interest in Facial Expression Recognition (FER) in the past few decades. Several challenges surfaced with the invention of many different FER models which are often based on Convolution Neural Network (CNN) architectures. Recently, an attention-based transformer model has been presented to address FER. One of the major issues with Transformers is the need for a large data quantity for training. Therefore, in this paper, we propose to learn how to fine-tune a vision transformer-based (ViT) model using a limited dataset. We will be using the JAFFE Dataset, which consists of only 213 images containing seven different emotions. The proposed method is evaluated using several fine-tuning methods, such as adding dropout, data augmentation, and layer freezing. We compared the models implemented with 5% dropout regularization, augmented dataset (up to 5000 images), and freezing the initial model's layers, fine-tuning around a fourth of the last layers. The best model was achieved by fine-tuning ViT L-16 with 96.06% accuracy, trained with 5% dropout in the augmented dataset, and freezing the initial 21st layers. We also compared our model to the other previous work model and the results showed that our model reached the state-of-the art for the JAFFE dataset. [ABSTRACT FROM AUTHOR]

Plant disease is one prevalent factor that has a significant impact worldwide on food security and production, with plants contributing over 80% of food that is consumed by human, due to the importance of plants for worldwide food security and a projected food demand rise of 59-102% by 2050, a method for early detection of plant disease can be a significant solution in the prevention of food shortage around the world, in the present day advancement in technology has made possible for Artificial Intelligence techniques such as Convolutional Neural Network (CNN) and Vision Transformer (ViT) to automatically detect and classify plant disease with success, this research compared 20 efficient and speed-focused Pretrained ViT model in plant disease classification with the top 7 performing models ensembled together, the results show that ensemble of ViT model beats all other models with a 7 model ensemble using average voting method resulted in an accuracy of Accuracy of 0.9816, Precision of 0.9737, Recall of 0.9723, and F1 score of 0.9723, although it's recommended to use lower model ensemble such as the 3 model ensemble with max voting that has an Accuracy of 0.9804, Precision of 0.9708, Recall of 0.9703, and F1 score of 0.9705 due to negligible difference between model ensemble and lower cost of the ensemble. [ABSTRACT FROM AUTHOR]

The widespread implementation of surveillance systems on construction sites has led to the accumulation of vast amounts of visual data, highlighting the need for an effective semantic analysis methodology. Natural language, as the most intuitive mode of expression, can significantly enhance the interpretability of such data. The adoption of multi-modality models promotes the interaction between surveillance video and textual data, thereby enabling managers to swiftly comprehend on-site dynamics. This study introduces a Visual Question Answering (VQA) approach for the construction industry and presents a specialized dataset to address the unique requirements of on-site management. Utilizing a Vision Transformer (ViT) architecture, the proposed model conducts feature extraction, fusion and interaction between visual and textual features. An additional projection layer is added to establish a transfer learning strategy that is optimized for construction site data. This novel approach facilitates rapid alignment of visual and language features in the model and is validated through ablation studies. The proposed approach achieves a testing accuracy of 83.8%, effectively converting image data from construction sites into natural language descriptions that enhance the analysis of construction processes. Compared to existing methods, this approach does not rely on object detection and allows for the direct extraction of deep-level semantic information from the on-site images. This study further discusses the feasibility of applying VQA within the architecture, engineering and construction (AEC) industry, examines its limitations, and offers suggestions for viable future directions of development. [ABSTRACT FROM AUTHOR]

Lung disease ranked as the third leading global cause of mortality, underscores the critical importance of accurately detecting respiratory disease abnormalities to reduce mortality rates. This study applies the Vision Transformer (Vit) with attention mechanisms to classify respiratory sounds into fve respiratory cases (Normal, Asthma, Pneumonia, COPD, and Lung fbrosis). The approach involves utilizing a dataset of lung audio signals represented as spectrogram images to train the Vit transformer model. Multiple data augmentation techniques enhance model generalization and diversify the dataset. Furthermore, the experimentation phase explores two normalization techniques, RMS and EBU. The results reveal that the highest achieved accuracy is 91.04%, achieved through a combination of data augmentation and EBU normalization, surpassing the state-of-the-art methods such as ResNet101, VGG-16, VGG-19, and others. These findings offer valuable insights into medical diagnostics, suggesting that the Vision Transformer model holds promise for creating more precise and reliable systems for classifying lung sounds. [ABSTRACT FROM AUTHOR]

The proposed method presents an innovative solution that addresses the challenges of unlabeled data, robust performance in unknown environments, and foreground-background differentiation in object segmentation. The objective of this work is to improve object segmentation tasks with an unannotated pool of data, derive model structure to demystify and educate large AI models by using knowledge distillation method, implementing framework with self-supervised learning analyse impacting and driving factors. Proposed method handles unlabeled data effectively, improving accuracy and generalization in diverse scenarios. It demonstrates remarkable reliability in unknown environments, ensuring consistent and accurate object segmentation in complex visual contexts. By incorporating novel techniques, the approach overcomes the longstanding problem of achieving consistent and non-trivial solutions for object segmentation, offering a comprehensive and effective solution for computer vision and image analysis. The literature review reveals that the DINO model using a ViT-B/8 backbone achieves an impressive 80.1% accuracy in linear regression, while the DINO model with a ViT-B/16 backbone attains the highest k-NN accuracy at 78.3%. The approach integrates deep neural networks for object segmentation with Kalman filtering for non-linear state estimation. Specifically, depth information obtained from point clouds is directly incorporated into the Unscented Kalman filtering, allowing for efficient and accurate performance. A one-parameter procedure is employed to identify and eliminate irregularities in point clouds, significantly improving the performance of segmenting objects. By incorporating self-supervised learning with distillation, the method effectively predicts and addresses the challenge of refining trivial solutions, leading to improved performance in viewpoint and illumination scenarios. Decision-making is deferred until a global view of the entire frame can be established, enabling accurate and consistent object segmentation in subsequent frames. The proposed method, SSL-DM+UKF, outperforms existing approaches in various challenges, including occlusion, viewpoint changes, and illumination variations. It achieves statistical superiority over the baseline SSL-Distillation Model by up to 2.50% in different scenarios. Incorporating depth observations through the Unscented Kalman Filter and using self-supervised learning with knowledge distillation are key factors contributing to its success. SSL-DM+UKF offers a valuable and efficient solution for real-time object segmentation in dynamic environments. [ABSTRACT FROM AUTHOR]

Interstitial Lung Disease (ILD) is a lung illness characterized by inflammation and scarring. Identifying and categorizing ILD patterns using chest Computed Tomography (CT) images is crucial for diagnosis and treatment planning. Deep learning and computer vision advancements offer the potential for automating medical image examination, such as the transformer model, which identifies intricate dependencies and relationships in data. Chest CT scans provide valuable information for ILD pattern classification and diagnosis. The Vision Transformer (ViT) based Multi-Head Self Attention (MHSA) architecture detects local and global spatial dependencies, focusing on relevant regions and considering contextual interactions. The ViT-based model architecture aims to categorize ILD patterns using MHSA mechanisms. The proposed ViT-ILD model improves the performance by modifying hyperparameters, attention heads, and hidden units. It also utilises techniques of residual connections, layer normalization, and positional encoding for improvement. The proposed method ViT-ILD has achieved comparable training, validation and test accuracy of 100%, 98%, and 82.75% respectively for the 4-class classification with a healthy lung, hypersensitivity pneumonitis, pulmonary fibrosis, and tuberculosis from the MedGift CT dataset. It is observed that the proposed ViT-ILD model has achieved test accuracy, recall, precision, and f1-score of 82.75%, 74.15%, 100%, and 82.35%. [ABSTRACT FROM AUTHOR]

Much like humans focus solely on object movement to understand actions, directing a deep learning model's attention to the core contexts within videos is crucial for improving video comprehension. In the recent study, VideoMasked Auto-Encoder (VideoMAE) employs a pre-training approach with a high ratio of tube masking and reconstruction, effectively mitigating spatial bias due to temporal redundancy in full video frames. This steers the model's focus toward detailed temporal contexts. However, as the VideoMAE still relies on full video frames during the action recognition stage, it may exhibit a progressive shift in attention towards spatial contexts, deteriorating its ability to capture the main spatio-temporal contexts. To address this issue, we propose an attention-directing module named Transformer Encoder Attention Module (TEAM). This proposed module effectively directs the model's attention to the core characteristics within each video, inherently mitigating spatial bias. The TEAM first figures out the core features among the overall extracted features fromeach video. After that, it discerns the specific parts of the video where those features are located, encouraging themodel to focusmore on these informative parts. Consequently, during the action recognition stage, the proposed TEAM effectively shifts the VideoMAE's attention from spatial contexts towards the core spatio-temporal contexts. This attention-shift manner alleviates the spatial bias in the model and simultaneously enhances its ability to capture precise video contexts. We conduct extensive experiments to explore the optimal configuration that enables the TEAM to fulfill its intended design purpose and facilitates its seamless integrationwith theVideoMAE framework. The integratedmodel, i.e., VideoMAE+TEAM, outperforms the existing VideoMAE by a significant margin on Something-Something-V2 (71.3% vs. 70.3%). Moreover, the qualitative comparisons demonstrate that theTEAMencourages themodel to disregard insignificant features and focus more on the essential video features, capturing more detailed spatio-temporal contexts within the video. [ABSTRACT FROM AUTHOR]

In this study, we address the issue of limited generalization capabilities in intelligent diagnosis models caused by the lack of high-quality fault data samples for aero-engine rolling bearings. We provide a fault anomaly detection technique based on distillation learning to address this issue. Two Vision Transformer (ViT) models are specifically used in the distillation learning process, one of which serves as the teacher network and the other as the student network. By using a small-scale student network model, the computational efficiency of the model is increased without sacrificing model accuracy. For feature-centered representation, new loss and anomaly score functions are created, and an enhanced Transformer encoder with the residual block is proposed. Then, a rolling bearing dynamics simulation method is used to obtain rich fault sample data, and the pre-training of the teacher network is completed. For anomaly detection, the training of the student network is completed based on the proposed loss function and the pre-trained teacher network, using only the vibration acceleration samples obtained from the normal state. Finally, the trained completed network and the designed anomaly score function are used to achieve the anomaly detection of rolling bearing faults. The experimental validation was carried out on two sets of test data and one set of real vibration data of a whole aero-engine, and the detection accuracy reached 100 %. The results show that the proposed method has a high capability of rolling bearing fault anomaly detection. • The dynamic numerical simulation method is used to obtain fault sample data sets to train the T-Net. • By using a small-scale S-Net, the computational efficiency of the model is increased without sacrificing model accuracy. • A new loss function and anomaly score function of feature center representation are proposed. • A new enhanced Transformer encoder that fuses residual blocks is proposed to solve the problem of attention collapse. • The method in this paper has been applied to the real aero-engine rolling bearing fault detection. [ABSTRACT FROM AUTHOR]

Fabric defect detection is a pivotal step in quality control in the textile manufacturing industry. Due to the diversity and complexity of defects, manual visual inspection and traditional fabric defect detection methods suffer from low efficiency and accuracy. To address the issues, a saliency model capable of mining local and global information from CNN and vision Transformer is proposed for fabric defect detection in this paper, named ACCTNet. Specifically, to enhance the feature interaction of different scales, an adjacent context coordination module composed of one local branch and two adjacent branches is proposed. Meanwhile, a contrast-aggregation module is proposed to highlight the defects from low contrast background using pooling and subtraction operations. In addition, vision Transformer is adopted to capture global contextual information with long-range dependencies, which can guide local information to further refines the defect detection results. Experimental results demonstrate that the proposed method can accurately inspect the defects from plain and patterned fabric surfaces, achieving E m values of 78.49% and 97.19% respectively, which significantly surpasses the existing state-of-the-art fabric defect detection methods. [ABSTRACT FROM AUTHOR]

The physical and morphological characteristics of wool and cashmere fibers exhibit notable similarities, making distinguishing them challenging. In this study, we propose a method based on a lightweight hybrid model called MobileViT, which combines a vision transformer and convolutional neural network, for the real-time identification of fiber categories. After training on a large sample dataset, the model was validated on a test set of 61,095 fiber images belonging to six categories; it took 26.2 s to achieve a recognition accuracy of 97.19%. This paper presents the first attempt to use a hybrid model of Transformer and Convolutional Neural Network (CNN) for the recognition of fiber images. Experimental results demonstrate that the model is capable of effectively extracting features from fibers, and it outperforms pure CNN models in terms of both speed and accuracy. [ABSTRACT FROM AUTHOR]

To develop a deep learning model for the automated classification of breast ultrasound images as benign or malignant. More specifically, the application of vision transformers, ensemble learning, and knowledge distillation is explored for breast ultrasound classification. Single view, B-mode ultrasound images were curated from the publicly available Breast Ultrasound Image (BUSI) dataset, which has categorical ground truth labels (benign vs malignant) assigned by radiologists and malignant cases confirmed by biopsy. The performance of vision transformers (ViT) is compared to convolutional neural networks (CNN), followed by a comparison between supervised, self-supervised, and randomly initialized ViT. Subsequently, the ensemble of 10 independently trained ViT, where the ensemble model is the unweighted average of the output of each individual model is compared to the performance of each ViT alone. Finally, we train a single ViT to emulate the ensembled ViT using knowledge distillation. On this dataset that was trained using five-fold cross validation, ViT outperforms CNN, while self-supervised ViT outperform supervised and randomly initialized ViT. The ensemble model achieves an area under the receiver operating characteristics curve (AuROC) and area under the precision recall curve (AuPRC) of 0.977 and 0.965 on the test set, outperforming the average AuROC and AuPRC of the independently trained ViTs (0.958 ± 0.05 and 0.931 ± 0.016). The distilled ViT achieves an AuROC and AuPRC of 0.972 and 0.960. Both transfer learning and ensemble learning can each offer increased performance independently and can be sequentially combined to collectively improve the performance of the final model. Furthermore, a single vision transformer can be trained to match the performance of an ensemble of a set of vision transformers using knowledge distillation. [ABSTRACT FROM AUTHOR]

In regions with limited resources and moderate incomes, the relentless spectre of cardiovascular diseases (CVDs) continues to loom large. Amidst this challenge, the precise classification of heart sounds is emerging as a pivotal linchpin in the realm of early CVD diagnosis and intervention. Manual heart sound auscultation efficacy remains tethered to the expertise of physicians, but the tides are shifting. With deep learning algorithms, heart sound classification reaches new heights. In this paradigm-shifting article, we unveil an ingenious model fortified by bispectrum-inspired feature extraction and the cutting-edge prowess of the Vision Transformer (ViT) model. This model spearheads the binary classification of heart sounds, labelling them as either 'normal' or 'abnormal.' Our model uses data from the PhysioNet Challenge 2022 database, which contains 3163 data points from 942 patients. The model showcases an adept classification process with a remarkable consistency, notably holding its own when distinguishing between heart sounds of pregnant and nonpregnant patients. Moreover, we dare to challenge the status quo. This article boldly pits the performance of our model against that of seasoned cardiologists. Our model emerges as the triumphant frontrunner, eclipsing the proficiency of even the most seasoned cardiologists. In a world where health resources are unevenly distributed, this pioneering work offers a beacon of hope, unlocking a future where advanced algorithms not only match but surpass human expertise. Step into the forefront of transformative cardiac care—where artificial intelligence becomes the key to unravelling the secrets of heart sounds. [Display omitted] ● We used bispectrum for feature extraction and Vision Transformer model for binary classification of heart sounds. ● Our model obtained an accuracy of 0.91 and an AUC of 0.98. ● The model classified pregnant and non-pregnant patients with AUC of 0.97 and 0.98, respectively. ● In a comparison of the classification ability of the cardiologists versus the model, the model outperformed the cardiologists (AUC of 0.9 and 0.55, respectively). [ABSTRACT FROM AUTHOR]

Due to the lack of long-range association and spatial location information, fine details and accurate boundaries of complex clothing images cannot always be obtained by using the existing deep learning-based methods. This paper presents a convolutional structure with multi-scale fusion to optimize the step of clothing feature extraction and a self-attention module to capture long-range association information. The structure enables the self-attention mechanism to directly participate in the process of information exchange through the down-scaling projection operation of the multi-scale framework. In addition, the improved self-attention module introduces the extraction of 2-dimensional relative position information to make up for its lack of ability to extract spatial position features from clothing images. The experimental results based on the colorful fashion parsing dataset (CFPD) show that the proposed network structure achieves 53.68% mean intersection over union (mIoU) and has better performance on the clothing parsing task. [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (DCNNs ) are widely used in content-based image retrieval (CBIR) because of the advantages in image feature extraction. However, the training of deep neural networks requires a large number of labeled data, which limits the application. Self-supervised learning is a more general approach in unlabeled scenarios. A method of fine-tuning feature extraction networks based on masked learning is proposed. Masked autoencoders (MAE) are used in the fine-tune vision transformer (ViT) model. In addition, the scheme of extracting image descriptors is discussed. The encoder of the MAE uses the ViT to extract global features and performs self-supervised fine-tuning by reconstructing masked area pixels. The method works well on category-level image retrieval datasets with marked improvements in instance-level datasets. For the instance-level datasets Oxford5k and Paris6k, the retrieval accuracy of the base model is improved by 7% and 17% compared to that of the original model, respectively. [ABSTRACT FROM AUTHOR]

Automatic diagnosis of various ophthalmic diseases from ocular medical images is vital to support clinical decisions. Most current methods employ a single imaging modality, especially 2D fundus images. Considering that the diagnosis of ophthalmic diseases can greatly benefit from multiple imaging modalities, this paper further improves the accuracy of diagnosis by effectively utilizing cross-modal data. In this paper, we propose Transformer-based cross-modal multi-contrast network for efficiently fusing color fundus photograph (CFP) and optical coherence tomography (OCT) modality to diagnose ophthalmic diseases. We design multi-contrast learning strategy to extract discriminate features from cross-modal data for diagnosis. Then channel fusion head captures the semantically shared information across different modalities and the similarity features between patients of the same category. Meanwhile, we use a class-balanced training strategy to cope with the situation that medical datasets are usually class-imbalanced. Our method is evaluated on public benchmark datasets for cross-modal ophthalmic disease diagnosis. The experimental results demonstrate that our method outperforms other approaches. The codes and models are available at https://github.com/ecustyy/tcmn. [ABSTRACT FROM AUTHOR]

Automatic diagnosis of Covid-19 lung complications from Computerized Tomography (CT) scans is an increasingly important research topic. In this rapidly developing area of Covid detection from medical image sequences, it is noted that most prior literature has focused on binary classification to detect diseased versus healthy cases from single X-ray or CT image. In this paper, we advance a step further by presenting a comprehensive framework for automated classification of the severity of lung infection (mild, moderate and severe) from CT sequences of confirmed Covid cases. We consider the sequence information for automation because in practice, the medical experts look at the CT sequence to score the severity of infection. We have collected a new lung CT sequence dataset at various stages of Covid infection from Indian patients. This dataset has been scored in terms of the severity of each lung lobe by experts in the field. We present a novel application of space-time transformers for CT sequences and achieve 93.3% accuracy for sequence level and 99% accuracy for patient-level, for multi- class classification of severity classes. [ABSTRACT FROM AUTHOR]

Defect object detection for printed circuit boards (PCBs) is one of the most important tasks in the field of industrial inspections. For this task, deep learning-based object detection algorithms are generally used. Regardless of whether one-stage or two-stage object detection algorithms are used, image features are extracted using multi-layer convolutional neural networks, which makes it hard to focus on global features. Few methods use Transformer-based models to extract features because of their computational complexity, which is proportional to the square of the length of the input, and they also lack the ability to extract multi-scale features. In our paper, we design a variant of the Vision Transformer (ViT) to extract features at multiple scales. We use a multi-stage ViT to extract image features, and then 1 × 1 convolution is used to connect the adjacent stages to enhance the feature dimensionality. Although the conventional ViT does not have the ability to extract multi-scale features, we use convolution to downsample the output of each stage of the ViT to obtain multi-scale features of the image. The size and stride of the convolution kernel used to downsample also change as the feature dimensionality changes. In addition, to reduce the computational complexity, we introduce window attention into each stage of the ViT, and as the feature dimensionality changes, the window size also changes. We construct a feature pyramid network and a path aggregation network based on the multi-scale features obtained as described above, and we adopt the detection head from YOLOx to implement the final multi-scale ViT-based object detection algorithm. Experimental results show that our improved model achieves a mean average precision of 69.3% on the MSCOCO 2017 test dataset, 86.0% on the PASCAL VOC 2007 test dataset, and 99.9% on our PCB ultrasonic image dataset. [ABSTRACT FROM AUTHOR]

The standard approach for automated clinical image diagnosis is being held with the use of Convolutional Neural Networks (CNN) for a decade. Vision Transformers (ViT) are new in this domain and yield similar levels of performance to that of CNN making them a competitive alternative to CNNs. This paper proposes an alternative off-the-shelf ViT-based approach to detecting lung diseases. This approach has been compared with a CNN-based hybrid deep learning approach that outperforms existing different deep learning techniques. The hybrid deep learning model used for comparison is called Visual Geometric Group Data Spatial Transformer with CNN (VDSNet) and the experimental results are computed by using the open-source NIH chest X-rays dataset from Kaggle. In this study, we observe vision transformers that are pre-trained outperform CNN-based VDSNet in several metrics on full as well as different subsets of the dataset. Vision Transformers also show an increase in accuracy with the addition of internal layers and reduction in patch size at the expense of slightly higher training time making them a potential alternative to Convolutional Neural Networks. [ABSTRACT FROM AUTHOR]

The increasing demand for the Internet of Things (IoT) and various distributed devices has highlighted the need for reliable and effective intrusion detection systems to safeguard physical devices and data from cyber threats. However, managing large volumes of data with diverse dimensions and security features poses significant challenges, including reduced recognition accuracy, increased computational time, a large number of false positives, and overfitting issues. Integrating Artificial Intelligence (AI) into intrusion detection systems has recently gained attention to enable intelligent recognition and protection against cyber threats. This research introduces a novel architecture called Automated Separate Guided Attention Federated Graph Neural Network (ASGAFGNN) for predicting and detecting cyberattacks. Initially, cyberattack data is gathered and pre-processed to enhance its quality. The pre-processed data then undergoes feature extraction to obtain global, local, and temporal features using a hybrid vision transformer with bidirectional long short-term memory. The extracted features are further processed using Batched Sparse Principal Component Analysis (BSPCA) and Intuitionistic Fuzzy C-Means (IFCM) for feature reduction. Finally, ASGAFGNN is deployed to detect and categorize network traffic as benign or malicious. An Enhanced Osprey Optimization Algorithm (EOOA) minimizes the network's loss function. The proposed technique is executed on four datasets including CICIDS2017, UNR-IDD, NSL-KDD, and NF-UQ-NIDS-v2, achieving detection accuracies of 99.98 %, 99.44 %, 99.74 %, and 99.95 %, respectively and it outperforms the existing schemes. [Display omitted] • A novel ASGAFGNN method is introduced in this research work. • Hybrid vision transformer with bidirectional long short term memory is introduced. • BSPCA with IFCM algorithm is applied for feature reduction. • The introduced model is tested on three data sets. • It attains detection accuracy that is higher than that of existing techniques. [ABSTRACT FROM AUTHOR]

In recent years, the medical image segmentation method based on hybrid convolutional neural network (CNN) and Vision Transformer (ViT) has made great progress, but it still faces the challenge of unbalanced global and local modeling, and excessive parameters. In addition, ViT repeatedly uses the whole feature map to model the global information, thus generating irrelevant and weakly related information, which will weaken the performance of the model when facing small datasets and segmentation targets. Therefore, this paper proposes a feature screening network based on similarity, named Screening Feature (SF)-MixedNet. Specifically, this paper first proposes a new feature extractor, namely Correlation based Similarity Transformer (CSimFormer). On the basis of parameter pruning, it uses the Screening Feature Multi-head Self Attention (SF-MSA) to establish the remote dependency, and calculates the similarity between local elements through the Location-Sensitive Mechanism (LsM) to obtain the weight matrix. Then, the correlation between regional elements is mined by Region Matching and Selection (RMS) mechanism, and the obtained information is filtered according to the corresponding rules to reduce the side effects of redundant information. Extensive experiments on Synapse dataset, ACDC dataset and SegPC-2021 dataset show that the segmentation accuracy reaches 83.51%, 92.20% and 81.27% respectively. Especially in the Synapse dataset, our method is 6.31% higher than the baseline. The method proposed in this paper effectively improves the segmentation accuracy, provides more detailed information for medical diagnosis and promotes the development of medical artificial intelligence technology. [Display omitted] • Balancing local modeling and global modeling. • Exploring inter-local similarities and eliminating redundant information. • A lightweight segmentation network is proposed. [ABSTRACT FROM AUTHOR]

Detection and segmentation of cell nuclei in hematoxylin and eosin-stained tissue images pose critical clinical challenges, including intricate backgrounds, variable nuclear appearances, overlapping nuclei, and unclear boundaries. These complexities render automated instance segmentation a difficult research area. Thus, this paper proposes a deep aggregation transformer network for automatic nucleus instance segmentation. The method employs a robust Vision Transformer-based backbone to extract informative features from images. To fuse multi-level information, we propose the group aggregation multi-level feature fusion module, which merges low-level features for shape representation and high-level features for pixel edge segmentation, enhancing overall model performance. The depthwise integrated multilayer perceptron module enhances feature learning by combining the global context-dependent representation of MLP and the local representation of convolution through channel separation. This fosters feature learning across different modes, thereby improving network representation. The bidirectional path interaction feature processing module is introduced for multi-scale information extraction and fusion. This module enhances the entire feature pyramid hierarchy by transferring deep semantic information to shallow layers with robust spatial information via bidirectional paths. The Large kernel Attention module at each layer extensively improves feature extraction through a large receptive field. Our approach outperforms state-of-the-art models, including StarDIST, CPP-Net, TSFD-Net and TransNuSeg, by 7.8%, 6.3%, 1.5% and 3.5% in average panoptic quality for different cell nuclei segmentation categories across tissues of the Pannuke dataset. Furthermore, DAT-Net outperforms all other state-of-the-art models in terms of the F1 score for the five classes of cell nuclei in the PanNuke dataset. • We innovatively employ the Vision Transformer for cell nucleus instance segmentation tasks. • We design a new feature fusion module, group aggregation multi-level feature fusion module. • We develop an integrated learning module, deepwise integrate multilayer perceptron module. • We construct a bidirectional path structure with large kernel attention mechanism for efficient feature extraction. • DAT-Net has notably enhanced segmentation performance and efficiency across two widely used datasets. [ABSTRACT FROM AUTHOR]

Currently, gesture recognition primarily utilizes Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs) among deep learning methods. However, the unique spatial and temporal features of surface Electromyography (sEMG) signals render these methods insufficient for effective feature extraction. To tackle this challenge, this paper proposes a novel model named CNN-VIT, which integrates the architecture of CNN and Vision Transformers (VIT) with a weighted mechanism. This innovative model combines the strengths of CNN and VIT to comprehensively exploit the motion information encoded in sEMG signals, subsequently improving the accuracy and reliability of gesture recognition. To validate the algorithm's practical efficacy, we conducted experiments on the Ninapro DB2 Exercise B dataset, followed by tests on the DB-MYO dataset, which was collected using a myoelectric data bracelet. Additionally, we performed real-time prediction experiments to further assess the model's performance. Results demonstrate a classification accuracy of 83.05%, 90.40%, and 85.00%, affirming the superior classification performance of CNN-VIT. [Display omitted] • We propose a novel model named CNN-VIT, which integrates the architecture of CNN and Vision Transformers (VIT) with a weighted mechanism. • Our model combines the strengths of CNN and VIT to comprehensively exploit the motion information encoded in sEMG signals, achieving excellent performance in gesture recognition. • We propose a weighting mechanism named ECSA. This mechanism enhances the representational power of the features by assigning weights to each channel of the feature maps. • The ECSA mechanism not only captures the attention along the width of the image data but also maps the positional information of various sEMG signal channels onto the feature maps. [ABSTRACT FROM AUTHOR]

U-shaped structures are widely employed in medical image segmentation. However, in existing methods, the skip connection component primarily employs straightforward addition or concatenation, which can result in a reduced complementarity between features at hierarchical levels. These approaches can result in problems like imprecise identification of organs and unclear boundaries. In this paper, we propose a Hybrid Multi-scale Cross-order Fusion Network (HM-Net) for medical image segmentation tasks. Specifically, we first design a hybrid pyramid attention module (HPAM) to adaptively deepen shallow semantic features from both the spatial and channel dimensions through multi-scale feature fusion to alleviate the semantic interval between the decoder and encoder in the skip connection. In addition, we propose a cross-order multi-scale fusion decoder, which effectively captures the layered features produced by the decoder for fusion, mitigating information loss during the up-sampling process using a feature enhancement module and substantially improving the edge blurring problem. Through extensive experimentation on both the Synapse and ACDC datasets, our method has demonstrated superior performance compared to previous state-of-the-art methods. [Display omitted] [ABSTRACT FROM AUTHOR]

Automating cardiac image analysis poses formidable challenges due to deformations during respiratory and cardiac cycles and the unique features of imaging modalities like CT and MRI. Transformer-based methods integrated with U-Net often miss essential local spatial details and incur high computational costs. To overcome these hurdles, we present the Pyramid Shifted-window-based Vision Transformer (PSVT), an innovative backbone for cardiac segmentation. Our model is meticulously designed to maintain a robust representation of both global and local features while operating on lower-dimensional inputs. By leveraging Continuous Position Bias in the Attention mechanism, we seamlessly integrate the attention module of Swin-Transformer-v2 with CNNs, yielding enhanced segmentation performance and computational efficiency. Additionally, we amplify receptive field understanding by incorporating Depth-Wise Convolution into the Feed-Forward module. Inspired by various adaptations of U-Net, we have restructured the Patch Merging and Patch Expanding modules, utilizing transposed convolution techniques to elegantly fuse multi-scale features into the classification head. Our experimental results compellingly demonstrate the superiority of PSVT, outperforming state-of-the-art cardiac segmentation models across three small-sample cardiac datasets: ACDC, MMWHS-CT, and LASC-2013. Our code is available in this URL. [Display omitted] • PSVT outperforms SOTA methods on ACDC, MMWHS-CT, and LASC-2013 datasets. • PSB fuses CNNs with Swin-Transformer-v2, boosting feature extraction and accuracy. • Patch Merging/Expanding leverages CNNs for multi-scale cardiac feature fusion. • PSVT achieves DSCs of 94.67% (LV) and 89.94% (RV), with strong clinical potential. [ABSTRACT FROM AUTHOR]

• To present a TPDMF approach for pre-process the histopathological images to reduce the noises. • To introduce a CAR2N model for extracting the relevant features from the pre-processed image. • To implement an optimized Bidirectional Vision transformer (OBVT) model to detect Colorectal Cancer appropriately. • To tune the hyperparameter in classifier model using chaotic sequence-based snake optimization algorithm (Ch-SOA). The high death rate of colorectal cancer has a long-term impact on human life around the world. Early detection of colorectal cancer leads to an increase in the survival rate of the patients. Overfitting issues occurred due to the presence of imbalanced datasets. To resolve the challenges, a novel deep learning-based approach based on an effective optimization strategy with a bidirectional vision transformer model for colorectal cancer detection. Disease classification is performed using two datasets: colorectal histology images and the NCT-CRC-HE-100 K dataset. Initially, the histopathological images from the dataset are trained and pre-processed using a Trimmed Pixel density-based median filter (TPDMF), which removes noise from the input images and enhances the quality of the images. The features of the pre-processed image are extracted using the Capsule Assisted Res2Net (CAR2N) model, and an optimized Bidirectional Vision Transformer (OBVT) model is presented to identify colorectal cancer. Here, the Bi- Long short-term memory (Bi-LSTM) model is used in the Multi-layer perception module of the vision transformer to reduce the complexity of the detection process. The proposed methodology uses a chaotic sequence-based Snake optimization algorithm (Ch-SOA) to tune the hyperparameters in the proposed model. The proposed model can be analyzed using different performance metrics like accuracy, precision, sensitivity, specificity, False negative rate (FNR), and False positive rate (FPR). The proposed model can obtain an accuracy of 97.11458 % for the colorectal histology images dataset and 97.01235 % for the NCT-CRC-HE-100 K dataset. From this analysis, the proposed model can obtain better results than other existing models. [ABSTRACT FROM AUTHOR]

Medicinal plants, such as safflower (Carthamustinctorius), are essential in both conventional and modern healthcare. This study evaluatesthe reliability of leaf image classification for differentiating safflower germplasm accessions, despite the challenge posed by similar leaf structures across varieties. Traditional classification methods can be time-consuming and error prone, underscoring the need for fine-grained classification techniques. To address this, we introduce a novel comprehensive leaf database of safflower varieties that is meticulously curated by experts. This database can be used to support future research. We evaluate state-of-the-art deep learning methods for classifying safflower varieties and propose a novel approach using a Vision Transformer (ViT) model with an optimal leaf structure feature (OLSF). The OLSF, calculated as the average response of a multidirectional Gabor filter bank and optimized with the structural similarity index measure (SSIM), enhances complex leaf features, such as veins, texture, and frequency variations, to enhance the classification performance of deep learning models. The experimental results reveal that the OLSF-ViT model achieves excellent accuracy scores of 100 %, 99.05 %, and 89.65 % on the Folio, UCI Leaves, and JNUSafflower datasets, respectively. These findings demonstrate that leaf image analysis is an effective and affordable tool for investigating the phenotypic diversity among safflower cultivars. This study highlights the potential of OLSF-ViT in automatic plant classification, and the results can be used in plant science, herbal medicine, and biodiversity conservation. • Introducing an expert-curated safflower leaf database for cultivar characterization. • Evaluating phenotypic diversity in safflower varieties with deep learning. • OLSF averages Gabor filter bank responses in various directions, optimized with SSIM. • OLSF captures intricate details for enabling the ViT to fully leverage deep learning. • Attained high accuracy in leaf image classification across datasets. [ABSTRACT FROM AUTHOR]

• A RanCom-ViT is proposed for AD classification in brain MRI scans using a ViT as the backbone for better contextual representation learning. • A token compression block is proposed to improve the computational efficiency of the RanCom-ViT by dropping the inattentive tokens. • The head block of the RanCom-ViT is improved with a random enhanced feature projection to achieve better generalization performance. Alzheimer's disease (AD) is the most common dementia that is often seen among the elderly. AD can cause the loss of cognitive ability and memory, which can result in death as AD is progressive. The exact cause of AD is still in research, but it is believed to be related to genes, diet, and environment. One observation of AD is the shrinkage of the hippocampus and frontal lobe cortex. Magnetic resonance imaging (MRI) is often employed in the diagnosis of AD as it can produce clear images of the soft tissues. In this study, a new computer-aided diagnosis (CAD) method named RanCom-ViT, is proposed to interpret the brain MRI slices automatically and precisely for AD diagnosis with better global representation learning and efficiency. A pre-trained vision transformer (ViT) is chosen as the backbone because ViTs with attention modules can achieve better performance than convolutional neural networks on larger datasets. Then, a novel token compression block is proposed to improve the efficiency of the RanCom-ViT by removing the less important tokens. Further, the classification head of the RanCom-ViT is enhanced by a random vector functional-link structure to obtain better classification performance in AD diagnosis. A large public brain MRI dataset is utilized in the evaluation experiments of the proposed RanCom-ViT, and it achieved an overall accuracy of 99.54% with a double throughput than the benchmark model. The results reveal that the RanCom-ViT outperforms several existing state-of-the-art AD diagnosis methods in terms of accuracy, and the token compression method contributes to higher efficiency. [ABSTRACT FROM AUTHOR]

This paper introduces a systematic classification of the facial nerve grading system using a comprehensive methodology using a pioneering Multi-Path Heterogeneous Neural Network (MPHNN) method designed for the accurate classification of exercise. It integrates four distinct Convolutional Neural Networks (CNNs) and Custom Feedforward Neural Networks (CFNNs) to enhance the precision of the classification. The CNNs are specifically tailored to scrutinize changes in the coordinates of facial landmarks over time, enabling the capture of both spatial information and temporal patterns in facial expressions during exercise. The CFNNs incorporate patient-specific variables and exercise statistics, including factors such as their surgical history, the type of exercise, its duration, and synthetic features like cumulative movement for each landmark. By leveraging this comprehensive framework, the proposed method offers a nuanced representation of the patient's exercise performance, thereby facilitating more precise outcomes of a classification. • Multi-Path Heterogeneous Neural Network method designed for accurate classification of exercises. • Convolutional Neural Networks and Custom Feedforward Neural Networks to enhance classification precision. • Facilitates more precise classification outcomes. [ABSTRACT FROM AUTHOR]

• A coarse-to-fine inference based vision transformer network (CFI-ViT). • Combining the discrimination, global attention extraction, and differentiable modules for balancing the computational budget and accuracy. • Combining cross entropy loss and K-L divergence as the total loss for CFI-ViT training. • Using the perturbation maximum method as the optimization method for information region screening. • Evaluating the proposed method through a private dataset and two public datasets. Accurate detection of histopathological cancer subtypes is crucial for personalized treatment. Currently, deep learning methods based on histopathology images have become an effective solution to this problem. However, existing deep learning methods for histopathology image classification often suffer from high computational complexity, not considering the variability of different regions, and failing to synchronize the focus on local–global information effectively. To address these issues, we propose a coarse-to-fine inference based vision transformer (ViT) network (CFI-ViT) for pathological image detection of gastric cancer subtypes. CFI-ViT combines global attention and discriminative and differentiable modules to achieve two-stage inference. In the coarse inference stage, a ViT model with relative position embedding is employed to extract global information from the input images. If the critical information is not sufficiently identified, the differentiable module is adopted to extract local image regions with discrimination for fine-grained screening in the fine inference stage. The effectiveness and superiority of the proposed CFI-ViT method have been validated through three pathological image datasets of gastric cancer, including one private dataset clinically collected from Yunnan Cancer Hospital in China and two publicly available datasets, i.e., HE-GHI-DS and TCGA-STAD. The experimental results demonstrate that CFI-ViT achieves superior recognition accuracy and generalization performance compared to traditional methods, while using only 80 % of the computational resources required by the ViT model. [ABSTRACT FROM AUTHOR]

Semantic segmentation is a fundamental task in computer vision and it is a building block of many other vision applications. Nevertheless, semantic segmentation annotations are extremely expensive to collect, so using pre-training to alleviate the need for a large number of labeled samples is appealing. Recently, self-supervised learning (SSL) has shown effectiveness in extracting strong representations and has been widely applied to a variety of downstream tasks. However, most works perform sub-optimally in semantic segmentation because they ignore the specific properties of segmentation: (i) the need of pixel level fine-grained understanding; (ii) with the assistance of global context understanding; (iii) both of the above achieve with the dense self-supervisory signal. Based on these key factors, we introduce a systematic self-supervised pre-training framework for semantic segmentation, which consists of a hierarchical encoder–decoder architecture MEVT for generating high-resolution features with global contextual information propagation and a self-supervised training strategy for learning fine-grained semantic features. In our study, our framework shows competitive performance compared with other main self-supervised pre-training methods for semantic segmentation on COCO-Stuff, ADE20K, PASCAL VOC, and Cityscapes datasets. e.g., MEVT achieves the advantage in linear probing by +1.3 mIoU on PASCAL VOC. • The system-level design combines dense semantic understanding from scratch. • Hybrid attention combines window attention for high-res and global for low-res layers. • Integrate a multi-scale decoder into pre-training, jointly training encoder and decoder for fine-grained features. • Introduce spatially aligned feature maps with skip connections, via only image-level supervision. [ABSTRACT FROM AUTHOR]

Cognitive state identification based on Electroencephalography (EEG) not only helps to diagnose various types of cognitive dysfunctions as early as possible in the clinical field, but also enables timely detection of cognitive overload caused by subjects' participation in cognitive tasks, which could prevent many accidents from occurring. Deep learning (DL) plays a vital role in the recognition of cognitive states based on EEG. However, due to the significant individual differences of EEG, it remains a great challenge to design a robust deep learning model to identify the cognition state in time and effectively. In this study, a domain adaptive vision transformer framework DAEEGViT is proposed for cognitive state identification. In DAEEGViT, a MBConv module with a self-attention mechanism is introduced to extract the global and local features of EEG simultaneously, followed by a domain adaptation method to improve the transfer learning capability and a classifier to predict the cognitive state. Two self-collected datasets: FDD and ECED, with two public datasets: SEED and SEED-IV have been used in our experiments to identify the fatigue, cognitive impairment and emotion states. Compared with the normal ViT model, in the intra-subject experiments, the accuracy of DAEEGViT improved 25.28%, 54.70% and 43.70% on FDD, SEED, and SEED-IV, respectively. In the cross-subject experiments, it improved 1.23%, 1.85%, and 0.97% on three tasks of ECED (Happiness, neutral and sadness). On FDD, SEED, and SEED-IV, it improved 14.33%, 4.72% and 5.35%, respectively. The results prove that DAEEGViT has distinguished improvement in cognitive state recognition. • The encoder that integrates the self-attention mechanism and MBConv can effectively extract the local and global features of EEG data. • Combining local and global features enables comprehensive analysis of EEG-based cognitive data. • The inclusion of domain adaptive methods in the model is helpful for cross-subject learning and it improves the generalization of the model. [ABSTRACT FROM AUTHOR]

• We propose a new model, CTBViT, for the more efficient and accurate classification of TB in both CT and X-ray images. • We introduce a new Patch Reduction Block (PRB) to improve efficiency by removing unimportant tokens. • We propose a randomized classifier to avoid the overfitting problem encountered when applying large pre-trained models to TB datasets. • We evaluate our model on multiple datasets for testing to verify its generalization. The results show that our model achieves superior performance to existing state-of-the-art techniques. Despite the impressive achievements of Transformer models in tuberculosis (TB) analysis, there are still some shortcomings, such as the high computational cost and overfitting problem. To address these issues, this paper proposes a novel CTBViT model to detect TB in CT and X-ray images. In the traditional transformer block, all the tokens are taken into account for self-attention. However, many regions in the images are irrelevant to the classification task. Therefore, we propose the Patch Reduction Block (PRB) to reduce computation by evaluating each token then dropping less important tokens and retaining only important tokens for subsequent processing. Additionally, we propose the randomized classifier where all the parameters in the input layer are randomly set and do not change during training. This design effectively avoids the overfitting problem. To fairly validate our model, we conducted experiments on both private and public datasets and the CTBViT surpassed the classification performance of other state-of-the-art methods. The results reveal the effectiveness and efficiency of the proposed CTBViT in TB classification, which can be a useful tool in clinical diagnosis. [ABSTRACT FROM AUTHOR]

Chest X-ray is an essential diagnostic tool in the identification of chest diseases given its high sensitivity to pathological abnormalities in the lungs. However, image-driven diagnosis is still challenging due to heterogeneity in size and location of pathology, as well as visual similarities and co-occurrence of separate pathology. Since disease-related regions often occupy a relatively small portion of diagnostic images, classification models based on traditional convolutional neural networks (CNNs) are adversely affected given their locality bias. While CNNs were previously augmented with attention maps or spatial masks to guide focus on potentially critical regions, learning localization guidance under heterogeneity in the spatial distribution of pathology is challenging. To improve multi-label classification performance, here we propose a novel method, HydraViT, that synergistically combines a transformer backbone with a multi-branch output module with learned weighting. The transformer backbone enhances sensitivity to long-range context in X-ray images, while using the self-attention mechanism to adaptively focus on task-critical regions. The multi-branch output module dedicates an independent branch to each disease label to attain robust learning across separate disease classes, along with an aggregated branch across labels to maintain sensitivity to co-occurrence relationships among pathology. Experiments demonstrate that, on average, HydraViT outperforms competing attention-guided methods by 1.9% AUC and 5.3% MAE, region-guided methods by 2.1% AUC and 8.3% MAE, and semantic-guided methods by 2.0% AUC and 6.5% MAE in multi-label classification performance. • HydraViT is a hybrid model for reliable multi-label disease classification. • It uses a transformer to capture long-range context and pathology co-occurrence. • It uses distance-based adaptive weights to enhance performance across pathologies. [ABSTRACT FROM AUTHOR]

Weed control plays a crucial role in agricultural production. The utilization of advanced vision algorithms on intelligent weeding robots enables the autonomous and efficient resolution of weed-related issues. Vision transformers are highly sensitive to plant texture and shape, but their computational cost is too high. Consequently, we propose a novel hybrid CNN-transformer network for the semantic segmentation of crops and weeds on Resource-Constrained Embedded Devices. Our network follows an encoder–decoder structure, incorporating the proposed concat extended downsampling block in the encoder, which increases inference speed by reducing memory access time and improves the accuracy of feature extraction. For global semantic extraction, we introduce the proposed Parallel input transformer semantic enhancement module, which employs a shared transformer block to increase the computation rate. Additionally, global–local semantic fusion block mitigates the semantic gap problem well. To fully utilize the transformer's ability to process plant texture and shape, we employ the fusion enhancement block in the decoder, thus minimizing the loss of feature information. Segmentation results on three publicly benchmark datasets show that our network outperforms the commonly used CNN-based, transformer-based, and hybrid CNN-transformer-based methods in terms of segmentation accuracy. Moreover, our network comprises only 0.1887M parameters and 0.2145G floating-point operations. We also evaluate the inference speed on an NVIDIA Jetson Orin NX embedded system, which result for inference single image 28.28 msec, and achieving a detection speed of 35.36 FPS. The experimental results highlight that our network maintains the best inference speed and exhibits the strongest segmentation performance on resource-constrained embedded systems. • Proposed a hybrid CNN-transformer network for crop and weed semantic segmentation. • Reduces the memory access time and enhances the feature extraction. • Improves the computation rate and effectively mitigates the semantic gap. • Provides timely and accurate weed management solutions. [ABSTRACT FROM AUTHOR]

Automatic segmentation of two-dimensional (2D) echocardiogram is beneficial for heart disease diagnosis and assessment. Convolutional Neural Network (CNN) based U-shaped architectures such as UNet have shown remarkable success for medical images segmentation. UNet generally exhibits limitations for seizing long-range dependencies due to the intrinsic locality of the convolution operation. On the contrary, transformer models can capture global-level information using the multi-head attention mechanism. Taken separately these models exhibit limited localization abilities due to insufficient low-level details. To overcome these limitations, this paper proposes the novel vision transformer CoST-UNet (Convolution and Swin Transformer-based U-shaped Network) architecture that incorporates CNN to leverage spatial information from images in the upper layers and transformer to emphasize global contextual insight in the deeper levels. Unlike existing hybrid models like TransUNet and UNETR, the transformer block of the proposed model employs a Swin Transformer backbone, which ensures linear computational complexity relative to image size. Furthermore, the primary barrier to improving the performance of the transformers, which is the lack of medical images, is effectively addressed by incorporating two convolution layers at the network's uppermost level. The experimental results demonstrate that the model achieved state-of-the-art performance on the ultrasound-based CAMUS dataset (by achieving mean Dice Similarity Coefficients of 0.925, 0.851, and 0.895 for segmenting LV endo , LV epi , and LA, respectively, from apical 4CH echocardiograms), as well as competitive results for MRI-based ACDC datasets, due to its effective capture of local and global context. • Introduced CoST-UNet, a novel model integrating convolution and transformer to extract local–global spatial information. • Offered linear computational complexity with image size by leveraging the swin transformer architecture. • Alleviated the lack of medical images by employing a CNN layer before the transformer, increasing the number of input images (feature maps). • The proposed model exhibits state-of-the-art performance on both CAMUS (ultrasound) and ACDC (MRI) datasets. [ABSTRACT FROM AUTHOR]

Melanoma is a skin cancer that spreads quickly and has serious risks. Early diagnosis is essential, but since the symptoms of skin lesions in the early stages are vague and similar, they can be difficult for specialists to detect. Therefore, machine learning-based alternative diagnostic methods can be developed in addition to existing ones. This study proposes a new deep learning model, a modified lightweight vision transformer (ViT) architecture, and a hybrid framework developed with an integrated deep learning model and an Ensemble Learning (EL) model for the early-stage diagnosis of skin lesions. The proposed deep learning model was developed based on convolution layers and transformers. The model is called multi-head attention block depthwise separable convolution network (MABSCNET). The proposed hybrid framework was developed by combining modern deep learning and EL models pre-trained with the ImageNet dataset along with the MABSCNET model. In the experimental process, the effectiveness of the proposed methods was evaluated on the ISIC 2020 dataset. Additionally, additional experiments were conducted on ISIC 2018 and a Kaggle dataset to analyze the proposed hybrid framework's classification performance. Image enhancement techniques were used in the datasets. In the ISIC 2020 dataset, the MABSCNET model reached 78.63 % accuracy, the ViT model obtained 76.50 %, and the hybrid framework reached 92.74 % accuracy. Moreover, the proposed hybrid framework achieved 100 % on the ISIC 2018 dataset and 94.24 % on the Kaggle dataset. • MABSCNET, a new deep learning model based on CNN and transformer, was developed. • Hybrid methods have been developed with the MABSCNET model and traditional machine learning algorithms. • Metaheuristic algorithms were used to determine the most appropriate hyperparameters of traditional machine learning algorithms. • A hybrid framework with deep and ensemble learning models is proposed. • The datasets used in the experimental process were optimized using image enhancement techniques such as CLAHE and Hypercolumn. [ABSTRACT FROM AUTHOR]

Vision Transformers (ViTs) have exhibited exceptional performance across a broad spectrum of visual tasks. Nonetheless, their computational requirements often surpass those of prevailing CNN-based models. Token sparsity techniques have been employed as a means to alleviate this issue. Regrettably, these techniques often result in the loss of semantic information and subsequent deterioration in performance. In order to address these challenges, we propose the Adaptive Masked Autoencoder Transformer (AMAT), a masked image modeling-based method. AMAT integrates a novel adaptive masking mechanism and a training objective function for both pre-training and fine-tuning stages. Our primary objective is to reduce the complexity of Vision Transformer models while concurrently enhancing their final accuracy. Through experiments conducted on the ILSVRC-2012 dataset, our proposed method surpasses the original ViT by achieving up to 40% FLOPs savings. Moreover, AMAT outperforms the efficient DynamicViT model by 0.1% while saving 4% FLOPs. Furthermore, on the Places365 dataset, AMAT achieves a 0.3% accuracy loss while saving 21% FLOPs compared to MAE. These findings effectively demonstrate the efficacy of AMAT in mitigating computational complexity while maintaining a high level of accuracy. • Propose AMAT for pre-training ViT-based models in image classification. • Develop an adaptive masking mechanism in the pre-training and fine-tuning stages. • Propose a novel objective function to facilitate the training of the AMAT method. • AMAT gains 40% FLOPs reduction compared with the original ViT with higher accuracy. [ABSTRACT FROM AUTHOR]

The domain of Cloth-Changing Person Re-identification has garnered substantial attention recently owing to its pivotal role in security applications. The essence of Cloth-Changing Person Re-identification lies in acquiring information unaffected by clothing variations, such as gait and silhouette. Most studies focus on the Convolutional Neural Network models. However, main drawback of Convolutional Neural Network is the limitation of receptive field. Methods based on Convolutional Neural Network primarily focus on modeling local information, making it complex to extract the discriminative part of multiple spatial contexts. Therefore, we proposed a Vision Transformer-based Cloth-Changing Person Re-identification framework called TransRL-ReID. Specifically, we encoded an image as a sequence of patches and constructed a Vision Transformer-based framework. In addition, a module named DropAll was designed to reduce local bias by adaptive adjustment of attention weights for each patch. We conducted extensive experiments on the real-world benchmark PRCC dataset. TransRL-ReID achieves 66.7% on Rank-1 and 60.0% on mean Average Precision, outperforming previous Convolutional Neural Network-based methods. After visual analysis of the results, we found that TransReID could extract information unrelated to clothing, such as gait and silhouette, and paid less attention to clothing information. DropAll improves the model's ability to fit pedestrian distributions. • Proposed a Vision Transformer-based Cloth-Changing Person Re-identification framework. • Introduced an innovative approach for CC-ReID called TransRL-ReID. • Improved CNN-based Dropout for ViT-based model capability to capture global information. • Conducted extensive experiments on the real-world benchmark PRCC dataset. • TransRL-ReID model achieved 66.7% on Rank-1 and 60.0% on mean Average Precision. [ABSTRACT FROM AUTHOR]

• This work constructed a multi-channel Vision Transformer model, which is more suitable for multi-channel EEG data compared to traditional Vision Transformer models. • In the preprocessing stage, this work used Stockwell transform to extract the time-frequency features of EEG signals and compressed the time-frequency feature matrix to reduce computational load. • In this work, evaluations were conducted using two databases. In addition to the public CHB-MIT database, the SH-SDU database was collected from a clinical hospital to comprehensively assess the performance of the designed model in this study. • In terms of data augmentation, this work proposed a novel data augmentation method called random segmentation and random combination, which can expand the data for pre-ictal periods. • In this study, two evaluation criteria were used, including segment-based evaluation criteria and event-based evaluation criteria. The event-based evaluation criteria provide more valuable references for clinical applications. Automatic seizure prediction is crucial for developing a new therapy for patients suffering from medically intractable epilepsy, possessing important clinical application value. In order to solve the problems of overfitting due to the complexity of deep learning based seizure prediction networks and the susceptibility of EEG features to noise contamination, an automatic seizure prediction model based on Stockwell Transform (S-transform) and Multi-Channel Vision Transformer (MViT) is proposed in this work. The time-frequency representation of multi-channel electroencephalography (EEG) signals is obtained by using S-Transform. These time-frequency spectrograms are then compressed and sent into the MViT model for further spatial feature extraction and identification of preictal EEG state. The designed MViT model is lightweight, ensuring efficient feature discriminating ability even with a minimal number of network layers. In view of the persistence of EEG activities, K-of-N strategy is employed to further augment the predictive performance. Experiments on the CHB-MIT database and the SH-SDU clinical database yield segment-based accuracy of 97.57 % and 95.88 % respectively, demonstrating the superiority and potential of the proposed method in seizure prediction. [ABSTRACT FROM AUTHOR]