Search

Quantitative susceptibility mapping (QSM) is an MRI-based technique that estimates the underlying tissue magnetic susceptibility based on phase signal. Deep learning (DL)-based methods have shown promise in handling the challenging ill-posed inverse problem for QSM reconstruction. However, they require extensive paired training data that are typically unavailable and suffer from generalization problems. Recent model-incorporated DL approaches also overlook the non-local effect of the tissue phase in applying the source-to-field forward model due to patch-based training constraint, resulting in a discrepancy between the prediction and measurement and subsequently suboptimal QSM reconstruction. This study proposes an unsupervised and subject-specific DL method for QSM reconstruction based on implicit neural representation (INR), referred to as INR-QSM. INR has emerged as a powerful framework for learning a high-quality continuous representation of the signal (image) by exploiting its internal information without training labels. In INR-QSM, the desired susceptibility map is represented as a continuous function of the spatial coordinates, parameterized by a fully-connected neural network. The weights are learned by minimizing a loss function that includes a data fidelity term incorporated by the physical model and regularization terms. Additionally, a novel phase compensation strategy is proposed for the first time to account for the non-local effect of tissue phase in data consistency calculation to make the physical model more accurate. Our experiments show that INR-QSM outperforms traditional established QSM reconstruction methods and the compared unsupervised DL method both qualitatively and quantitatively, and is competitive against supervised DL methods under data perturbations., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier B.V. All rights reserved.)

The human gut microbiome encodes a large variety of antimicrobial peptides (AMPs), but the short lengths of AMPs pose a challenge for computational prediction. Here we combined multiple natural language processing neural network models, including LSTM, Attention and BERT, to form a unified pipeline for candidate AMP identification from human gut microbiome data. Of 2,349 sequences identified as candidate AMPs, 216 were chemically synthesized, with 181 showing antimicrobial activity (a positive rate of >83%). Most of these peptides have less than 40% sequence homology to AMPs in the training set. Further characterization of the 11 most potent AMPs showed high efficacy against antibiotic-resistant, Gram-negative pathogens and demonstrated significant efficacy in lowering bacterial load by more than tenfold against a mouse model of bacterial lung infection. Our study showcases the potential of machine learning approaches for mining functional peptides from metagenome data and accelerating the discovery of promising AMP candidate molecules for in-depth investigations., (© 2022. The Author(s), under exclusive licence to Springer Nature America, Inc.)

Quantitative susceptibility mapping (QSM) has demonstrated great potential in quantifying tissue susceptibility in various brain diseases. However, the intrinsic ill-posed inverse problem relating the tissue phase to the underlying susceptibility distribution affects the accuracy for quantifying tissue susceptibility. Recently, deep learning has shown promising results to improve accuracy by reducing the streaking artifacts. However, there exists a mismatch between the observed phase and the theoretical forward phase estimated by the susceptibility label. In this study, we proposed a model-based deep learning architecture that followed the STI (susceptibility tensor imaging) physical model, referred to as MoDL-QSM. Specifically, MoDL-QSM accounts for the relationship between STI-derived phase contrast induced by the susceptibility tensor terms (χ 13 , χ 23 and χ 33 ) and the acquired single-orientation phase. The convolutional neural networks are embedded into the physical model to learn a regularization term containing prior information. χ 33 and phase induced by χ 13 and χ 23 terms were used as the labels for network training. Quantitative evaluation metrics were compared with recently developed deep learning QSM methods. The results showed that MoDL-QSM achieved superior performance, demonstrating its potential for future applications., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

While many pre-defined computed tomographic (CT) measures have been utilized to characterize chronic obstructive pulmonary disease (COPD), it is still challenging to represent pathological alternations of multiple dimensions and highly spatial heterogeneity. Deep CNN transferred multiple instance learning (DCT-MIL) is proposed to identify COPD via CT images. After the lung is divided into eight sections along the axial direction, one random axial CT image is taken out from each section as one instance. With one instance as the input, the activations of neural layers of AlexNet trained by natural images are extracted as features. After dimension reduction through principle component analysis, features of all instances are input into three MIL methods: Citation k-Nearest-Neighbor (Citation-KNN), multiple instance support vector machine, and expectation-maximization diverse density. Moreover, the performance dependence of the resulted models on the depth of the neural layer where activations are extracted and the number of features is investigated. The proposed DCT-MIL achieves an exceptional performance with an accuracy of 99.29% and area under curve of 0.9826 while using 100 principle components of features extracted from the fourth convolutional layer and Citation-KNN. It outperforms not only DCT-MIL models using other settings and the pre-trained AlexNet with fine-tuning by montages of eight lung CT images, but also other state-of-art methods. Deep CNN transferred multiple instance learning is suited for identification of COPD using CT images. It can help finding subgroups with high risk of COPD from large populations through CT scans ordered doing lung cancer screening.

Deep learning technology has grown rapidly in recent years and achieved tremendous success in the field of computer vision. At present, many deep learning technologies have been applied in daily life, such as face recognition systems. However, as human life increasingly relies on deep neural networks, the potential harms of neural networks are being revealed, particularly in terms of deep neural network security. More and more studies have shown that existing deep learning-based face recognition models are vulnerable to attacks by adversarial samples, resulting in misjudgments that could have serious consequences. However, existing adversarial face images are rather easy to identify with the naked eye, so it is difficult for attackers to carry out attacks on face recognition systems in practice. This paper proposes a method for generating adversarial face images that are indistinguishable from the source images based on facial landmark detection and superpixel segmentation. First, the eyebrows, eyes, nose, and mouth regions are extracted from the face image using a facial landmark detection algorithm. Next, the superpixel segmentation algorithm is used to include the pixels neighboring the extracted facial landmarks with similar pixel values. Lastly, the segmented regions are used as masks to guide existing attack methods to insert adversarial noise within the masked areas. Experimental results show that our method can generate adversarial samples with high Structural Similarity Index Measure (SSIM) values at the cost of a small percentage of attack success rate. In addition, to simulate real-time physical attacks, printouts of the adversarial images generated by the proposed method are presented to the face recognition system via a camera and are still able to fool the face recognition model. Experimental results indicated that the proposed method can successfully perform adversarial attacks on face recognition systems in real-world scenarios.

Objective: Fetal anomaly screening via ultrasonography, which involves capturing and interpreting standard views, is highly challenging for inexperienced operators. We aimed to develop and validate a prenatal‐screening artificial intelligence system (PSAIS) for real‐time evaluation of the quality of anatomical images, indicating existing and missing structures. Methods: Still ultrasonographic images obtained from fetuses of 18–32 weeks of gestation between 2017 and 2018 were used to develop PSAIS based on YOLOv3 with global (anatomic site) and local (structures) feature extraction that could evaluate the image quality and indicate existing and missing structures in the fetal anatomical images. The performance of the PSAIS in recognizing 19 standard views was evaluated using retrospective real‐world fetal scan video validation datasets from four hospitals. We stratified sampled frames (standard, similar‐to‐standard, and background views at approximately 1:1:1) for experts to blindly verify the results. Results: The PSAIS was trained using 134 696 images and validated using 836 videos with 12 697 images. For internal and external validations, the multiclass macro‐average areas under the receiver operating characteristic curve were 0.943 (95% confidence interval [CI], 0.815–1.000) and 0.958 (0.864–1.000); the micro‐average areas were 0.974 (0.970–0.979) and 0.973 (0.965–0.981), respectively. For similar‐to‐standard views, the PSAIS accurately labeled 90.9% (90.0%–91.4%) with key structures and indicated missing structures. Conclusions: An artificial intelligence system developed to assist trainees in fetal anomaly screening demonstrated high agreement with experts in standard view identification. Synopsis: An artificial intelligence system developed for trainees showed high agreement with experts in identifying standard view, which may promote novice operators to obtain diagnostic images. [ABSTRACT FROM AUTHOR]

Deep learning methods have greatly promoted the ship detection performance in synthetic aperture radar (SAR) images. However, due to the different imaging mechanisms and target characteristics, detecting ships in SAR imagery precisely and efficiently is still challenging. To address this issue, a streamlined rotational ship detection network (SRSD-Net) is proposed in this letter. First, we construct a lightweight network using one-level feature and a receptive field block (RFB) to extract ship features with rich context. Second, a simple and effective representation for rotated ship bounding boxes using the minimum external horizontal box and two phase factors is proposed. Finally, to reduce the missed detections of small ships, k-nearest label assignment (KNLA) is designed to make all targets equally contribute to the network training. Extensive experiments are conducted on high-resolution SAR images dataset (HRSID) and the results show that the proposed method can achieve high detection accuracy with low computational cost. [ABSTRACT FROM AUTHOR]

Organic photovoltaics have attracted worldwide interest due to their unique advantages in developing low-cost, lightweight, and flexible power sources. Functional molecular design and synthesis have been put forward to accelerate the discovery of ideal organic semiconductors. However, it is extremely expensive to conduct experimental screening of the wide organic compound space. Here we develop a framework by combining a deep learning model (graph neural network) and an ensemble learning model (Light Gradient Boosting Machine), which enables rapid and accurate screening of organic photovoltaic molecules. This framework establishes the relationship between molecular structure, molecular properties, and device efficiency. Our framework evaluates the chemical structure of the organic photovoltaic molecules directly and accurately. Since it does not involve density functional theory calculations, it makes fast predictions. The reliability of our framework is verified with data from previous reports and our newly synthesized organic molecules. Our work provides an efficient method for developing new organic optoelectronic materials. [ABSTRACT FROM AUTHOR]

Due to a shortage of labeled examples, few-shot image classification frequently experiences noise interference and insufficient feature extraction. In this paper, we present a two-stage framework based on the distribution propagation graph neural network (DPGN) called the multilevel distribution propagation network (MDPN). An instance-segmentation-based object localization (ISOL) module and a graph-based multilevel distribution propagation (GMDP) module are both included in the MDPN. To create a clear and full object zone, the ISOL module generates a mask that eliminates background and pseudo-object noises. The GMDP module enriches the level of features. We carried out comprehensive experiments on the few-shot dataset CUB-200-2011 to show the usefulness of MDPN. The results demonstrate that MDPN indeed outperforms DPGN in terms of few-shot image classification accuracy. Under 5-way 1-shot and 5-way 5-shot settings, the classification accuracy of MDPN exceeds the baseline by 8.17% and 1.24%, respectively. MDPN also outperforms the majority of the existing few-shot classification methods in the same setting. [ABSTRACT FROM AUTHOR]

Change detection is a technique that can observe changes in the surface of the earth dynamically. It is one of the most significant tasks in remote sensing image processing. In the past few years, with the ability of extracting rich deep image features, the deep learning techniques have gained popularity in the field of change detection. In order to obtain obvious image change information, the attention mechanism is added in the decoder and output stage in many deep learning-based methods. Many of these approaches neglect to upgrade the ability of the encoders and the feature extractors to extract the representational features. To resolve this problem, this study proposes a deep multi-scale multi-attention siamese transformer network. A special contextual attention module combining a convolution and self-attention module is introduced into the siamese feature extractor to enhance the global representation ability. A lightly efficient channel attention block is added in the siamese feature extractor to obtain the information interaction among different channels. Furthermore, a multi-scale feature fusion module is proposed to fuse the features from different stages of the siamese feature extractor, and it can detect objects of different sizes and irregularities. To increase the accuracy of the proposed approach, the transformer module is utilized to model the long-range context in two-phase images. The experimental results on the LEVIR-CD and the CCD datasets show the effectiveness of the proposed network. [ABSTRACT FROM AUTHOR]

Obtaining crowd flow distribution with recognized human intention is extremely valuable for a series of applications for metropolitan cities. Previous solutions look at spatial correlation and temporal periodicity based on historical crowd flow information to calculate future crowd flow distribution. However, these mechanisms cannot recognize the intention behind crowd flow. We address this problem by leveraging a key insight – people's intention behind their movement is highly correlated with the point-of-interest (POI) distribution of the corresponding regions and adjacent regions. Therefore, we propose DeepFlowGen to model the complicated relationship between crowd flow, POI, check-ins, and time to generate intention-aware crowd flow. Specifically, we solve the conflict between dynamic crowd flow and static POI distribution by fusing the information in both time and POI domains. Besides, we employ a sequence of residual blocks in DeepFlowGen to address the challenges of modeling the diverse temporal rhythms and heterogeneous influence of POI. Furthermore, we examine the generated intention-aware crowd flow from two aspects to substantiate the reasonability of DeepFlowGen. Extensive experiments demonstrate that our model outperforms the state-of-the-art solutions by at most 30 percent in terms of NRMSE of total crowd flow. Moreover, the correlation between the generated intention-aware crowd flow and the check-in distribution across different categories of POIs is as high as 0.90 and 0.80 in Beijing and Shanghai. Combined with extensive case studies, we demonstrate the strong ability of our model in generating intention-aware crowd flow. [ABSTRACT FROM AUTHOR]

As a fundamental component in map service, map matching is of great importance for many trajectory-based applications, e.g., route optimization, traffic scheduling, and fleet management. In practice, Hidden Markov Model and its variants are widely used to provide accurate and efficient map matching service. However, HMM-based methods fail to utilize the knowledge (e.g., the mobility pattern) of enormous trajectory big data, which are useful for intelligent map matching. Furthermore, with many following-up works, they are still easily influenced by the common noisy and sparse records in the reality. In this paper, we revisit the map matching task from the data perspective and propose to utilize the great power of massive data and deep learning to solve these problems. Based on the seq2seq learning framework, we build a trajectory2road model with attention mechanism to map the sparse and noisy trajectory into the accurate road network. Different from previous algorithms, our deep learning based model complete the map matching in the latent space, which provides the high tolerance to the noisy trajectory and also enhances the matching with the knowledge of mobility pattern. Extensive experiments demonstrate that the proposed model outperforms the widely used HMM-based methods by more than 10 percent (absolute accuracy) in various situations especially the noisy and sparse settings. [ABSTRACT FROM AUTHOR]

As a powerful machine learning method, deep learning has attracted the attention of numerous researchers. While exploring a high-performance neural network model, the floating-point operations of a neural network model are also increasing. In recent years, many researchers have noticed that efficiency is also one of important indicators to measure the property of neural network models. Obviously, the efficient neural network model is more helpful to deploy on mobile and embedded devices. Therefore, the efficient neural network model becomes a hot research spot. In this paper, we review the methods related to the structural design of efficient convolution neural networks in recent years. According to the characteristics of these methods, we divide them into three kinds of methods: model pruning, efficient architecture, and neural architecture search. Detailed analyses of each method are presented to demonstrate their advantages and disadvantages. Then, we comprehensively compare them in detail and propose many suggestions about the design of the efficient convolution neural network model structure. Inspired by these suggestions, we built a new efficient neural network model, SharedNet. And the SharedNet obtains the best accuracy of manually-designed efficient CNN models on the ImageNet dataset. [ABSTRACT FROM AUTHOR]

Deep learning methods have gained rapid development in hyperspectral pansharpening (HP) due to powerful spatial–spectral feature extraction ability. However, most of these methods are optimized using a single reconstruction objective. It is difficult for these methods to find a balance between spectral preservation and spatial preservation. Furthermore, these methods adopt interpolation or convolution to upsample the hyperspectral images (HSIs), which tends to cause noticeable spectral distortion. To conquer these issues, a novel multiobjective guided divide-and-conquer network (MO-DCN) is proposed for HP. It consists of a deconvolution long short-term memories (LSTMs) network (DLSTM) and a divide-and-conquer network (DCN). DLSTM leverages bi-direction learning to upsample HSIs by considering 3-D spatiotemporal dependencies. Then, DCN designs a two-branch architecture to reconstruct spatial and spectral information from upsampled HSIs and panchromatic images (PANIs), respectively, where the spatial branch designs an attention-in-attention module (AIAM) to emphasize complementary attention in a coarse-to-fine way. Finally, co-improvement of spatial and spectral information is formulated as an Epsilon-constraint-based multiobjective optimization. The Epsilon constraint method transforms one objective into a constraint and regards it as a penalty bound to make an excellent tradeoff between different objectives. Experimental results demonstrated that the proposed method markedly improves pansharpening performance in both the spatial and spectral domains and has superior fusion performance than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Online services are playing critical roles in almost all aspects of users’ life. Users usually have multiple online identities (IDs) in different online services. In order to fuse the separated user data in multiple services for better business intelligence, it is critical for service providers to link online IDs belonging to the same user. On the other hand, the popularity of mobile networks and GPS-equipped smart devices have provided a generic way to link IDs, i.e., utilizing the mobility traces of IDs. However, linking IDs based on their mobility traces has been a challenging problem due to the highly heterogeneous, incomplete and noisy mobility data across services. In this paper, we propose DPLink, an end-to-end deep learning based framework, to complete the user identity linkage task for heterogeneous mobility data collected from different services with different properties. DPLink is made up by a feature extractor including a location encoder and a trajectory encoder to extract representative features from trajectory and a comparator to compare and decide whether to link two trajectories as the same user. Particularly, we propose a pre-training strategy with a simple task to train the DPLink model to overcome the training difficulties introduced by the highly heterogeneous nature of different source mobility data. Besides, we introduce a multi-modal embedding network and a co-attention mechanism in DPLink to deal with the low-quality problem of mobility data. By conducting extensive experiments on two real-life ground-truth mobility datasets with eight baselines, we demonstrate that DPLink outperforms the state-of-the-art solutions by more than 15 percent in terms of hit-precision. Moreover, it is expandable to add external geographical context data and works stably with heterogeneous noisy mobility traces. [ABSTRACT FROM AUTHOR]

Deep learning methods have achieved the state-of-the-art performance of object detection and tracking in natural images, such as keypoint-based detectors and appearance/motion-based trackers. However, for small and blurry moving vehicles in satellite videos, keypoint-based detectors cause the missed detection of keypoints and incorrect keypoint matching. In terms of multi-object tracking, it is difficult to track the crowded similar vehicles stably only by using the appearance or motion information. To address these problems, a novel deep learning framework is proposed for moving vehicle detection and tracking in the satellite videos. It is comprised of the cross-frame keypoint-based detection network (CKDNet) and spatial motion information-guided tracking network (SMTNet). In CKDNet, a customized cross-frame module is designed to assist the detection of keypoints by exploiting complementary information between frames. Furthermore, CKDNet improves keypoint matching by incorporating size prediction around the keypoints and defining the soft mismatch suppression for out-of-size keypoint pairs. Based on high-quality detection, SMTNet can track the densely-packed vehicles effectively by constructing two-branch long short-term memories. It extracts not only spatial information of the same frame by considering the relative spatial relationship of neighboring vehicles, but also motion information among consecutive frames by calculating the movement velocity. Especially, it regresses virtual positions for missed or occluded vehicles and keeps on tracking these vehicles while they reappear. Experimental results on Jilin-1 and SkySat satellite videos demonstrate the effectiveness of the proposed detection and tracking methods. [ABSTRACT FROM AUTHOR]

• The lymphoma aggressiveness prediction is formulated as a few-shot MIL problem. • Deep-learning-based and radiomics features from PET and CT are combined. • Our proposed bag-level few-shot learning framework is utilized for tackling the issue of limited NHL samples and incomplete annotations. Patients with aggressive non-Hodgkin lymphoma (NHL) undergo distinct therapy strategies compared with indolent NHL patients. However, it is challenging to estimate NHL aggressiveness based on visual inspection of positron emission tomography (PET) or computed tomography (CT) images. Since diffuse large B-cell lymphoma (DLBCL) and Follicular lymphoma (FL) are the most typical and dominant aggressive and indolent NHL, respectively, this study aims to develop an artificial-intelligence-enabled model to distinguish DLBCL from FL in PET/CT images as the first step to tackle this challenge. We propose a hybrid few-shot multiple-instance learning model to predict the aggressiveness of the NHL. First, rotation-based self-supervision learning (SSL) has been employed to train the encoder on a large-scale, publicly available CT image dataset. Second, hybrid instance-level features are obtained for each NHL lesion by combining deep features with the radiomics features from both PET and CT modalities. Third, instance-level features are transformed into bag-level (or patient-level) representations. Finally, bag-level representations are fed into a distance-based classifier through few-shot learning to predict NHL aggressiveness. Our model achieves an accuracy of 0.751 ± 0.008, a sensitivity of 0.787 ± 0.012, a specificity of 0.715 ± 0.013, an F1-score of 0.753 ± 0.009, and an area under the curve (AUC) of 0.795 ± 0.009 at the bag level. It outperforms the typical counterparts that use the radiomic features, random forest for feature selection, and support vector machines (SVMs) as classifiers. The three counterparts yield accuracies of 0.714 ± 0.023, 0.705 ± 0.008, and 0.698 ± 0.008, respectively. Moreover, settings of the SSL training dataset (Deep lesion) and task (rotation), hybrid CT and radiomic PET features, the pool-layer strategy of maximum, and distance-based classifier generate the best model. A hybrid few-shot multiple-instance learning model can predict lymphoma aggressiveness in PET/CT images and could be a potential tool for determining therapy strategies. Hybrid features and the combination of SSL, few-shot learning, and weakly supervised learning are the two powerful pillars of the model, and these can be expanded to other medical applications with limited samples and incomplete annotations. [ABSTRACT FROM AUTHOR]

Smartphone mobile application (App) usage prediction, i.e., which Apps will be used next, is beneficial for user experience improvement. Through an in-depth analysis on a real-world dataset, we find that App usage is highly spatio-temporally correlated and personalized. Given the ability to model complex spatio-temporal contexts, we aim to apply deep learning to achieve high prediction accuracy. However, the personalization yields a problem: training one network for each individual suffers from data scarcity, yet training one deep neural network for all users often fails to uncover user preference. In this article, we propose a novel App usage prediction framework, named DeepApp, to achieve context-aware prediction via multi-task learning. To tackle the challenge of data scarcity, we train one general network for multiple users to share common patterns. To better utilize the spatio-temporal contexts, we supplement a location prediction task in the multi-task learning framework to learn spatio-temporal relations. As for the personalization, we add a user identification task to capture user preference. We evaluate DeepApp on the large-scale dataset by extensive experiments. Results demonstrate that DeepApp outperforms the start-of-the-art baseline by 6.44%. [ABSTRACT FROM AUTHOR]

The extraction of joint spatial–spectral features has been proved to improve the classification performance of hyperspectral images (HSIs). Recently, utilizing convolutional neural networks (CNNs) to learn joint spatial–spectral features has become of great interest. However, the existing CNN models ignore complementary spatial–spectral information among the shallow and deep layers. Moreover, insufficient training samples in HSIs afflict these CNN models with overfitting problem. In order to address these problems, a novel CNN method for HSI classification is proposed. It considers multilayer spatial–spectral feature fusion and sample augmentation with local and nonlocal constraints, which is abbreviated as MSLN-CNN. In MSLN-CNN, a triple-architecture CNN is constructed to extract spatial–spectral features by cascading spectral features to dual-scale spatial features from shallow to deep layers. Then, multilayer spatial–spectral features are fused to learn complementary information among the shallow layers with detailed information and the deep layers with semantic information. Finally, the multilayer spatial–spectral feature fusion and classification are integrated into a unified network, and MSLN-CNN can be optimized in the end-to-end way. To alleviate the small sample size problem, the unlabeled samples having high confidences on local spatial constraint and nonlocal spectral constraint are selected and prelabeled. The nonlocal spectral constraint considers the structure information with spectrally similar samples in the nonlocal searching, while the local spatial constraint utilizes the contextual information with spatially adjacent samples. Experimental results on several hyperspectral datasets demonstrate that the proposed method achieves more encouraging classification performance than the current state-of-the-art classification methods, especially with the limited training samples. [ABSTRACT FROM AUTHOR]

• Morphological abnormalities of airway tree and lung field are indicative of COPD. • Different snapshots represent characteristics of airway tree and lung field. • Majority voting is used to integrate different views of 2D snapshots. • A deep CNN is developed for identification of COPD via CT images. Chronic obstructive pulmonary disease (COPD) is a complex and irreversible respiratory disease with potential morphological abnormalities of the airway and lung fields. To date, whether and how these abnormalities can be used to identify COPD is unknown. This study developed a deep convolutional neural network (CNN) integrating the airway tree and lung field morphologies to identify COPD. We represent 3D airway and lung fields through multi-view 2D snapshots and their integration via deep CNN, to estimate the possibility of COPD. We constructed two datasets named Dataset 1 including 380 participants (190 COPD and 190 healthy controls) for training and validation and Dataset 2 including 201 participants (101 COPD and 100 healthy controls) for testing. First, the 3D airway tree and lung field are automatically extracted from computed tomography (CT) images, and 2D snapshots in nine views are captured. Second, the proposed ResNet-26 is trained with each view of snapshots as input. Finally, majority voting of nine models is performed to identify COPD. The accuracy (ACC) of the single-view ResNet-26 model (ventral, dorsal, and isometric view of airway; front, rear, left, right, top, and bottom view of lung field) is 0.900, 0.873, 0.889, 0.868, 0.824, 0.876, 0.861, 0.839, and 0.884, respectively. For the multi-view ResNet-26 model of airway tree and lung field, the ACC is 0.913 and 0.895, respectively. For the model integrating all nine views, the ACC eventually reaches as high as 0.947. The deep CNN model identifies COPD through integrating morphology of the airway tree and lung field extracted from CT images. A different view of 2D snapshots represents various characteristics of the 3D airway tree and lung field. The integration of multiple views can improve the performance of COPD prediction. The CNN model provides a potential method of identifying COPD via CT scans. [ABSTRACT FROM AUTHOR]

Hyperspectral pansharpening (HP) can fuse different kinds of remote sensing images for the downstream classification or detection task. Deep learning methods have made rapid development in HP. However, most deep learning-based HP methods design the network architecture without considering the specific downstream task. When the pansharpening image is transferred to the downstream tasks, it may lead to poor performance on the downstream tasks. To solve this problem, a novel classification-oriented multi-task network (CMNet) is proposed for HP. CMNet combines the classification and HP networks in a multi-task learning way. In the classification network, class-specific and spatial-difference contrastive constraints are defined to enforce the extraction module to extract discriminative features for classification. In the HP network, a fusion module with an identical structure to the extraction module is constructed to extract fused features for reconstruction. Then, multi-gradient learning is designed to optimize the HP network under the guidance of the classification network. It adopts gradient mapping to consider the conflicting or collaborative situations between these different tasks. In this way, the fusion module in the HP network learns discriminative feature extraction ability from the classification network, which makes the HP network realize classification-oriented HP. Experimental results demonstrate that the proposed method markedly improves the pansharpening performance and the pansharpening is also able to elevate the classification performance. [ABSTRACT FROM AUTHOR]

With the introduction of spatial-spectral fusion and deep learning, the classification performance of hyperspectral imagery (HSI) has been promoted greatly. For some widely used datasets, the classification accuracy almost reaches 100%. However, for hyperspectral image classification, random sampling is still the most common strategy to collect the training and test samples. Because the training and test samples are randomly selected from the same image, so they have a high correlation and the classification results are overoptimistic. Besides, random sampling is not a good choice for practical applications because we cannot always collect training and test samples from the same region. Disjoint sampling selects training and testing samples from different local regions, which will provide a more objective performance evaluation for HSI classification models. In this paper, we first show the huge classification performance difference caused by different sampling strategies with a simple experiment, then we analyze the underlying reasons from the spectral information, spatial-spectral combination, sample overlapping and spatial distance, finally, a semi-supervised feature learning method is proposed for disjoint HSI classification, in which the spatial and spectral information are exploited effectively and reasonably. The experimental results based on three HSI datasets demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Band selection is a research hotspot in hyperspectral image processing. The continuity of the spectral bands causes the adjacent bands to be highly correlated, and correlation among long-range bands is possible with hundreds of spectral bands. Most existing deep learning methods fail to make full use of the inter-band correlation for band selection. In this paper, a novel dual-graph convolutional network based on band attention and a sparse constraint is proposed for band selection. The network consists of two branches. In the attention branch, band-based dual graphs are constructed to encode the contextual correlation of adjacent bands and the structural correlation of long-range bands into non-Euclidean space. Subsequently, the graph convolution-based band attention mechanism is devised to aggregate the band information in the band-based dual graphs and to generate the attention map for all bands. The band attention map is sparsely constrained and embedded as a mask into the trunk branch. In the trunk branch, sample-based dual graphs are constructed to represent the topological information of the samples in the spectral and spatial domains. Furthermore, a dense graph convolutional network is designed to extract and fuse the spatial–spectral and topological features from the shallow to deep layers for classification. A soft-shifting optimization strategy is implemented by defining a new loss from full bands and selected bands to solve the optimization problem caused by the sparse constraint. In this manner, band selection, feature extraction, and classification can be combined into an end-to-end trainable network. The experimental results on representative hyperspectral image datasets demonstrate the superiority of the proposed method over current state-of-the-art band selection methods. • An improved graph convolutional network is proposed for band selection. • Band-based and sample-based graphs are built with spatial and spectral information. • Design soft-shifting optimization strategy to overcome the optimization problem. • Mini batches of labeled and unlabeled samples are used for graph construction. [ABSTRACT FROM AUTHOR]

With the increasing number of underwater pipeline investigation activities, the research on automatic pipeline detection is of great significance. At this stage, object detection algorithms based on Deep Learning (DL) are widely used due to their abilities to deal with various complex scenarios. However, DL algorithms require massive representative samples, which are difficult to obtain for pipeline detection with sub-bottom profiler (SBP) data. In this paper, a zero-shot pipeline detection method is proposed. First, an efficient sample synthesis method based on SBP imaging principles is proposed to generate samples. Then, the generated samples are used to train the YOLOv5s network and a pipeline detection strategy is developed to meet the real-time requirements. Finally, the trained model is tested with the measured data. In the experiment, the trained model achieved a mAP@0.5 of 0.962, and the mean deviation of the predicted pipeline position is 0.23 pixels with a standard deviation of 1.94 pixels in the horizontal direction and 0.34 pixels with a standard deviation of 2.69 pixels in the vertical direction. In addition, the object detection speed also met the real-time requirements. The above results show that the proposed method has the potential to completely replace the manual interpretation and has very high application value. [ABSTRACT FROM AUTHOR]

Horizon picking from sub-bottom profiler (SBP) images has great significance in marine shallow strata studies. However, the mainstream automatic picking methods cannot handle multiples well, and there is a need to set a group of parameters manually. Considering the constant increase in the amount of SBP data and the high efficiency of deep learning (DL), we proposed a physicals-combined DL method to pick the horizons from SBP images. We adopted the DeeplabV3+ net to extract the horizons and multiples from SBP images. We generated a training dataset from the Jiaozhou Bay survey (Shandong, China) and the Zhujiang estuary survey (Guangzhou, China) to increase the applicability of the trained model. After the DL processing, we proposed a simulated Radon transform method to eliminate the surface-related multiples from the prediction by combining the designed pseudo-Radon transform and correlation analysis. We verified the proposed method using actual data (not involved in the training dataset) from Jiaozhou Bay and Zhujiang estuary. The positions of picked horizons are accurate, and multiples are suppressed. [ABSTRACT FROM AUTHOR]

• A classification neural networks and a regression neural network were trained on simulated CBED patterns to predict TEM sample thickness from 4D STEM convergent beam diffraction patterns. • Sub-unit cell lateral resolution, single unit cell thickness resolution, and 1 nm deviation from quantitative HAADF STEM thickness estimates were achieved. • Convolution neural networks were trained on simulation data with automatic labelling, and do not require quantitative matching between experiment and simulation. • Similar methodology might be extended to learn other physical properties of a sample from large 4D STEM datasets. Two types of convolutional neural network (CNN) models, a discrete classification network and a continuous regression network, were trained to determine local sample thickness from convergent beam diffraction (CBED) patterns of SrTiO 3 collected in a scanning transmission electron microscope (STEM) at atomic column resolution. Acquisition of atomic resolution CBED patterns for this purpose requires careful balancing of CBED feature size in pixels, acquisition speed, and detector dynamic range. The training datasets were derived from multislice simulations, which must be convolved with incoherent source broadening. Sample thicknesses were also determined using quantitative high-angle annular dark-field (HAADF) STEM images acquired simultaneously. The regression CNN performed well on sample thinner than 35 nm, with 70% of the CNN results within 1 nm of HAADF thickness, and 1.0 nm overall root mean square error between the two measurements. The classification CNN was trained for a thicknesses up to 100 nm and yielded 66% of CNN results within one classification increment of 2 nm of HAADF thickness. Our approach depends on methods from computer vision including transfer learning and image augmentation. [ABSTRACT FROM AUTHOR]

In smart home services, non-intrusive load monitoring (NILM) can reveal individual appliances' power consumption from the aggregate power and requires only one measurement point at the entrance by a smart meter. Most of the existing load disaggregation methods are based on deep and complex neural networks, and excessively long input sequences could increase the model disaggregation time. Meanwhile, constructing representative features and designing effective disaggregation model is becoming increasingly important. Therefore, we utilize a gramian summation difference angular field (GASDF) image, taking any two power sample points' temporal correlations as input to our baseline model, to better recognize different appliances from the aggregate power sequence. Then, since GASDF could not provide statistical characteristics, we further build the expert feature encoder (EFE) to realize the multi-dimensional representation of power by encoding both current aggregate power and statistical characteristics from historical data as prior knowledge. Afterwards, a batch-normalization (BN)-based normalization fusion (NF) method is proposed to lower the disaggregation error incurred by the distribution difference between GASDF and prior knowledge. Finally, to verify the proposed method's effectiveness, named ApplianceFilter, experiments are conducted on the UK-DALE and REDD data, showing that load disaggregation is improved using prior knowledge fusion, superior to the existing end-to-end neural network model. • A prior knowledge idea is introduced into the proposed baseline model to improve the effectiveness of the disaggregation model. • Proposed prior knowledge to provide spatial and correlation features different from those extracted from GASDF images. • Proposed a fusion method to lower disaggregation error caused by distribution difference between the GASDF feature map and expert feature. [ABSTRACT FROM AUTHOR]

• A novel end-to-end heterogeneous graph-based deep learning model. • Capturing the internal temporal pattern of single-dimensional time series and the rich spatial relations among variables. • Capturing both the static and dynamic relations of multivariate time series (MTS). • Combining graph neural networks with MTS to take full advantage of complex relations among variables of MTS. Multivariate time series forecasting, which analyzes historical time series to predict future trends, can effectively help decision-making. Complex relations among variables in MTS, including static, dynamic, predictable, and latent relations, have made it possible to mining more features of MTS. Modeling complex relations are not only essential in characterizing latent dependency as well as modeling temporal dependence, but also brings great challenges in the MTS forecasting task. However, existing methods mainly focus on modeling certain relations among MTS variables. In this paper, we propose a novel end-to-end deep learning model, termed M ultivariate T ime Series Forecasting via Het erogeneous G raph N eural N etworks (MTHetGNN). To characterize complex relations among variables, a relation embedding module is designed in MTHetGNN, where each variable is regarded as a graph node, and each type of edge represents a specific static or dynamic relationship. Meanwhile, a temporal embedding module is introduced for time series features extraction, where involving convolutional neural network (CNN) filters with different perception scales. Finally, a heterogeneous graph embedding module is adopted to handle the complex structural information generated by the two modules. Three benchmark datasets from the real world are used to evaluate the proposed MTHetGNN. The comprehensive experiments show that MTHetGNN achieves state-of-the-art results in the MTS forecasting task. [ABSTRACT FROM AUTHOR]

Graph similarity computation aims to predict a similarity score between one pair of graphs to facilitate downstream applications, such as finding the most similar chemical compounds similar to a query compound or Fewshot 3D Action Recognition. Recently, some graph similarity computation models based on neural networks have been proposed, which are either based on graph-level interaction or node-level comparison. However, when the number of nodes in the graph increases, it will inevitably bring about reduced representation ability or high computation cost. Motivated by this observation, we propose a graph partitioning and graph neural network-based model, called PSimGNN, to effectively resolve this issue. Specifically, each of the input graphs is partitioned into a set of subgraphs to extract the local structural features directly. Next, a novel graph neural network with an attention mechanism is designed to map each subgraph into an embedding vector. Some of these subgraph pairs are automatically selected for node-level comparison to supplement the subgraph-level embedding with fine-grained information. Finally, coarse-grained interaction information among subgraphs and fine-grained comparison information among nodes in different subgraphs are integrated to predict the final similarity score. Experimental results on graph datasets with different graph sizes demonstrate that PSimGNN outperforms state-of-the-art methods in graph similarity computation tasks using approximate Graph Edit Distance (GED) as the graph similarity metric. [ABSTRACT FROM AUTHOR]