Search

Background: The purpose of this article is to develop a deep learning automatic segmentation model for the segmentation of Crohn's disease (CD) lesions in computed tomography enterography (CTE) images. Additionally, the radiomics features extracted from the segmented CD lesions will be analyzed and multiple machine learning classifiers will be built to distinguish CD activity., Methods: This was a retrospective study with 2 sets of CTE image data. Segmentation datasets were used to establish nnU-Net neural network's automatic segmentation model. The classification dataset was processed using the automatic segmentation model to obtain segmentation results and extract radiomics features. The most optimal features were then selected to build 5 machine learning classifiers to distinguish CD activity. The performance of the automatic segmentation model was evaluated using the Dice similarity coefficient, while the performance of the machine learning classifier was evaluated using the area under the curve, sensitivity, specificity, and accuracy., Results: The segmentation dataset had 84 CTE examinations of CD patients (mean age 31 ± 13 years, 60 males), and the classification dataset had 193 (mean age 31 ± 12 years, 136 males). The deep learning segmentation model achieved a Dice similarity coefficient of 0.824 on the testing set. The logistic regression model showed the best performance among the 5 classifiers in the testing set, with an area under the curve, sensitivity, specificity, and accuracy of 0.862, 0.697, 0.840, and 0.759, respectively., Conclusion: The automated segmentation model accurately segments CD lesions, and machine learning classifier distinguishes CD activity well. This method can assist radiologists in promptly and precisely evaluating CD activity., (© The Author(s) 2023. Published by Oxford University Press on behalf of Crohn’s & Colitis Foundation. All rights reserved. For commercial re-use, please contact reprints@oup.com for reprints and translation rights for reprints. All other permissions can be obtained through our RightsLink service via the Permissions link on the article page on our site—for further information please contact journals.permissions@oup.com.)

Background and Purpose: To develop a computed tomography (CT)-based deep learning model to predict overall survival (OS) among small-cell lung cancer (SCLC) patients and identify patients who could benefit from prophylactic cranial irradiation (PCI) based on OS signature risk stratification., Materials and Methods: This study retrospectively included 556 SCLC patients from three medical centers. The training, internal validation, and external validation cohorts comprised 309, 133, and 114 patients, respectively. The OS signature was built using a unified fully connected neural network. A deep learning model was developed based on the OS signature. Clinical and combined models were developed and compared with a deep learning model. Additionally, the benefits of PCI were evaluated after stratification using an OS signature., Results: Within the internal and external validation cohorts, the deep learning model (concordance index [C-index] 0.745, 0.733) was far superior to the clinical model (C-index: 0.635, 0.630) in predicting OS, but slightly worse than the combined model (C-index: 0.771, 0.770). Additionally, the deep learning model had excellent calibration, clinical usefulness, and improved accuracy in classifying survival outcomes. Remarkably, patients at high risk had a survival benefit from PCI in both the limited and extensive stages (all P < 0.05), whereas no significant association was observed in patients at low risk., Conclusions: The CT-based deep learning model exhibited promising performance in predicting the OS of SCLC patients. The OS signature may aid in individualized treatment planning to select patients who may benefit from PCI., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Computational protein design has been demonstrated to be the most powerful tool in the last few years among protein designing and repacking tasks. In practice, these two tasks are strongly related but often treated separately. Besides, state-of-the-art deep-learning-based methods cannot provide interpretability from an energy perspective, affecting the accuracy of the design. Here we propose a new systematic approach, including both a posterior probability and a joint probability parts, to solve the two essential questions once for all. This approach takes the physicochemical property of amino acids into consideration and uses the joint probability model to ensure the convergence between structure and amino acid type. Our results demonstrated that this method could generate feasible, high-confidence sequences with low-energy side conformations. The designed sequences can fold into target structures with high confidence and maintain relatively stable biochemical properties. The side chain conformation has a significantly lower energy landscape without delegating to a rotamer library or performing the expensive conformational searches. Overall, we propose an end-to-end method that combines the advantages of both deep learning and energy-based methods. The design results of this model demonstrate high efficiency, and precision, as well as a low energy state and good interpretability., (© The Author(s) 2023. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com.)

The adaptive immune response to foreign antigens is initiated by T-cell receptor (TCR) recognition on the antigens. Recent experimental advances have enabled the generation of a large amount of TCR data and their cognate antigenic targets, allowing machine learning models to predict the binding specificity of TCRs. In this work, we present TEINet, a deep learning framework that utilizes transfer learning to address this prediction problem. TEINet employs two separately pretrained encoders to transform TCR and epitope sequences into numerical vectors, which are subsequently fed into a fully connected neural network to predict their binding specificities. A major challenge for binding specificity prediction is the lack of a unified approach to sampling negative data. Here, we first assess the current negative sampling approaches comprehensively and suggest that the Unified Epitope is the most suitable one. Subsequently, we compare TEINet with three baseline methods and observe that TEINet achieves an average AUROC of 0.760, which outperforms baseline methods by 6.4-26%. Furthermore, we investigate the impacts of the pretraining step and notice that excessive pretraining may lower its transferability to the final prediction task. Our results and analysis show that TEINet can make an accurate prediction using only the TCR sequence (CDR3$\beta $) and the epitope sequence, providing novel insights to understand the interactions between TCRs and epitopes., (© The Author(s) 2023. Published by Oxford University Press. All rights reserved. For Permissions, please email: journals.permissions@oup.com.)

Attention deficit hyperactivity disorder (ADHD) is a common neurodevelopmental disorder. Although genome-wide association studies (GWAS) identify the risk ADHD-associated variants and genes with significant P-values, they may neglect the combined effect of multiple variants with insignificant P-values. Here, we proposed a convolutional neural network (CNN) to classify 1033 individuals diagnosed with ADHD from 950 healthy controls according to their genomic data. The model takes the single nucleotide polymorphism (SNP) loci of P-values $\le{1\times 10^{-3}}$, i.e. 764 loci, as inputs, and achieved an accuracy of 0.9018, AUC of 0.9570, sensitivity of 0.8980 and specificity of 0.9055. By incorporating the saliency analysis for the deep learning network, a total of 96 candidate genes were found, of which 14 genes have been reported in previous ADHD-related studies. Furthermore, joint Gene Ontology enrichment and expression Quantitative Trait Loci analysis identified a potential risk gene for ADHD, EPHA5 with a variant of rs4860671. Overall, our CNN deep learning model exhibited a high accuracy for ADHD classification and demonstrated that the deep learning model could capture variants' combining effect with insignificant P-value, while GWAS fails. To our best knowledge, our model is the first deep learning method for the classification of ADHD with SNPs data., (© The Author(s) 2021. Published by Oxford University Press.)

Acquiring events massively from single-molecule force spectroscopy (SMFS) experiments, which is crucial for revealing important biophysical information, is usually not straightforward. A significant amount of human labor is usually required to identify events in the measured force spectrum during measuring or before performing further data analysis. This prevents the experiment from being done in a fully-automated manner or scaling with the throughput of the measuring setup. In this work, we attempt to tackle this problem with a deep learning approach. A deep neural network model is developed to infer the occurrence of the events using the data stream from the measuring setup. We demonstrated that the proposed method could achieve high accuracy with force spectrums of a variety of samples from both optical tweezers and AFMs by learning from user-given samples instead of complicated manual algorithm designing or parameter tuning. Furthermore, we found that the trained model can be used to perform event detection on datasets measured from a different optical tweezer setup, showing the potential of being leveraged in more complex deep learning schemes., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2021 Elsevier Inc. All rights reserved.)

Background: Horizontal Gene Transfer (HGT) refers to the sharing of genetic materials between distant species that are not in a parent-offspring relationship. The HGT insertion sites are important to understand the HGT mechanisms. Recent studies in main agents of HGT, such as transposon and plasmid, demonstrate that insertion sites usually hold specific sequence features. This motivates us to find a method to infer HGT insertion sites according to sequence features., Results: In this paper, we propose a deep residual network, DeepHGT, to recognize HGT insertion sites. To train DeepHGT, we extracted about 1.55 million sequence segments as training instances from 262 metagenomic samples, where the ratio between positive instances and negative instances is about 1:1. These segments are randomly partitioned into three subsets: 80% of them as the training set, 10% as the validation set, and the remaining 10% as the test set. The training loss of DeepHGT is 0.4163 and the validation loss is 0.423. On the test set, DeepHGT has achieved the area under curve (AUC) value of 0.8782. Furthermore, in order to further evaluate the generalization of DeepHGT, we constructed an independent test set containing 689,312 sequence segments from another 147 gut metagenomic samples. DeepHGT has achieved the AUC value of 0.8428, which approaches the previous test AUC value. As a comparison, the gradient boosting classifier model implemented in PyFeat achieve an AUC value of 0.694 and 0.686 on the above two test sets, respectively. Furthermore, DeepHGT could learn discriminant sequence features; for example, DeepHGT has learned a sequence pattern of palindromic subsequences as a significantly (P-value=0.0182) local feature. Hence, DeepHGT is a reliable model to recognize the HGT insertion site., Conclusion: DeepHGT is the first deep learning model that can accurately recognize HGT insertion sites on genomes according to the sequence pattern.

Background: Preoperative diagnoses of metastatic lymph nodes (LNs) by the most advanced deep learning technology of Faster Region-based Convolutional Neural Network (Faster R-CNN) have not yet been reported., Materials and Methods: In total, 545 patients with pathologically confirmed rectal cancer between January 2016 and March 2019 were included and were randomly allocated with a split ratio of 2:1 to the training and validation sets, respectively. The MRI images for metastatic LNs were evaluated by Faster R-CNN. Multivariate regression analyses were used to develop the predictive models. Faster R-CNN nomograms were constructed based on the multivariate analyses in the training sets and were validated in the validation sets., Results: The Faster R-CNN nomogram for predicting metastatic LN status contained predictors of age, metastatic LNs by Faster R-CNN and differentiation degrees of tumors, with areas under the curves (AUCs) of 0.862 (95% CI: 0.816-0.909) and 0.920 (95% CI: 0.876-0.964) in the training and validation sets, respectively. The Faster R-CNN nomogram for predicting LN metastasis degree contained predictors of metastatic LNs by Faster R-CNN and differentiation degrees of tumors, with AUCs of 0.859 (95% CI: 0.804-0.913) and 0.886 (95% CI: 0.822-0.950) in the training and validation sets, respectively. Calibration plots and decision curve analyses demonstrated good calibrations and clinical utilities. The two nomograms were used jointly as a kit for predicting metastatic LNs., Conclusion: The Faster R-CNN nomogram kit exhibits excellent performance in discrimination, calibration, and clinical utility and is convenient and reliable for predicting metastatic LNs preoperatively., Clinical Trial Registration: ChiCTR-DDD-17013842., (© 2020 The Authors. Cancer Medicine published by John Wiley & Sons Ltd.)

Background: Dental plaque causes many common oral diseases (e.g., caries, gingivitis, and periodontitis). Therefore, plaque detection and control are extremely important for children's oral health. The objectives of this study were to design a deep learning-based artificial intelligence (AI) model to detect plaque on primary teeth and to evaluate the diagnostic accuracy of the model., Methods: A conventional neural network (CNN) framework was adopted, and 886 intraoral photos of primary teeth were used for training. To validate clinical feasibility, 98 intraoral photos of primary teeth were assessed by the AI model. Additionally, tooth photos were acquired using a digital camera. One experienced pediatric dentist examined the photos and marked the regions containing plaque. Then, a plaque-disclosing agent was applied, and the areas with plaque were identified. After 1 week, the dentist drew the plaque area on the 98 photos taken by the digital camera again to evaluate the consistency of manual diagnosis. Additionally, 102 intraoral photos of primary teeth were marked to denote the plaque areas obtained by the AI model and the dentist to evaluate the diagnostic capacity of each approach based on lower-resolution photos. The mean intersection-over-union (MIoU) metric was employed to indicate detection accuracy., Results: The MIoU for detecting plaque on the tested tooth photos was 0.726 ± 0.165. The dentist's MIoU was 0.695 ± 0.269 when first diagnosing the 98 photos taken by the digital camera and 0.689 ± 0.253 after 1 week. Compared to the dentist, the AI model demonstrated a higher MIoU (0.736 ± 0.174), and the results did not change after 1 week. When the dentist and the AI model assessed the 102 intraoral photos, the MIoU was 0.652 ± 0.195 for the dentist and 0.724 ± 0.159 for the model. The results of a paired t-test found no significant difference between the AI model and human specialist (P > .05) in diagnosing dental plaque on primary teeth., Conclusions: The AI model showed clinically acceptable performance in detecting dental plaque on primary teeth compared with an experienced pediatric dentist. This finding illustrates the potential of such AI technology to help improve pediatric oral health.

Background: With recent advances in high-throughput technologies, matrix factorization techniques are increasingly being utilized for mapping quantitative omics profiling matrix data into low-dimensional embedding space, in the hope of uncovering insights in the underlying biological processes. Nevertheless, current matrix factorization tools fall short in handling noisy data and missing entries, both deficiencies that are often found in real-life data., Results: Here, we propose DeepMF, a deep neural network-based factorization model. DeepMF disentangles the association between molecular feature-associated and sample-associated latent matrices, and is tolerant to noisy and missing values. It exhibited feasible cancer subtype discovery efficacy on mRNA, miRNA, and protein profiles of medulloblastoma cancer, leukemia cancer, breast cancer, and small-blue-round-cell cancer, achieving the highest clustering accuracy of 76%, 100%, 92%, and 100% respectively. When analyzing data sets with 70% missing entries, DeepMF gave the best recovery capacity with silhouette values of 0.47, 0.6, 0.28, and 0.44, outperforming other state-of-the-art MF tools on the cancer data sets Medulloblastoma, Leukemia, TCGA BRCA, and SRBCT. Its embedding strength as measured by clustering accuracy is 88%, 100%, 84%, and 96% on these data sets, which improves on the current best methods 76%, 100%, 78%, and 87%., Conclusion: DeepMF demonstrated robust denoising, imputation, and embedding ability. It offers insights to uncover the underlying biological processes such as cancer subtype discovery. Our implementation of DeepMF can be found at https://github.com/paprikachan/DeepMF.

Background: Understanding the phenotypic drug response on cancer cell lines plays a vital role in anti-cancer drug discovery and re-purposing. The Genomics of Drug Sensitivity in Cancer (GDSC) database provides open data for researchers in phenotypic screening to build and test their models. Previously, most research in these areas starts from the molecular fingerprints or physiochemical features of drugs, instead of their structures., Results: In this paper, a model called twin Convolutional Neural Network for drugs in SMILES format (tCNNS) is introduced for phenotypic screening. tCNNS uses a convolutional network to extract features for drugs from their simplified molecular input line entry specification (SMILES) format and uses another convolutional network to extract features for cancer cell lines from the genetic feature vectors respectively. After that, a fully connected network is used to predict the interaction between the drugs and the cancer cell lines. When the training set and the testing set are divided based on the interaction pairs between drugs and cell lines, tCNNS achieves 0.826, 0.831 for the mean and top quartile of the coefficient of determinant (R 2 ) respectively and 0.909, 0.912 for the mean and top quartile of the Pearson correlation (R p ) respectively, which are significantly better than those of the previous works (Ammad-Ud-Din et al., J Chem Inf Model 54:2347-9, 2014), (Haider et al., PLoS ONE 10:0144490, 2015), (Menden et al., PLoS ONE 8:61318, 2013). However, when the training set and the testing set are divided exclusively based on drugs or cell lines, the performance of tCNNS decreases significantly and R p and R 2 drop to barely above 0., Conclusions: Our approach is able to predict the drug effects on cancer cell lines with high accuracy, and its performance remains stable with less but high-quality data, and with fewer features for the cancer cell lines. tCNNS can also solve the problem of outliers in other feature space. Besides achieving high scores in these statistical metrics, tCNNS also provides some insights into the phenotypic screening. However, the performance of tCNNS drops in the blind test.

Transcranial sonography (TCS) has been introduced to assess hyper-echogenicity in the substantia nigra of the midbrain for Parkinson's disease (PD); however, its subjective and resource-demanding nature has impeded its widespread application. An AI-empowered TCS-based PD classification tool is greatly demanding, yet relevant research is severely scarce. Therefore, we proposed a novel dual-channel CNXV2-DANet for TCS-based PD classification using a large cohort. A total of 1176 TCS images from 588 subjects were retrospectively enrolled from Beijing Tiantan Hospital, encompassing both the left and right side of the midbrain for each subject. The entire dataset was divided into a training/validation/testing set at a ratio of 70%/15%/15%. Development of the proposed CNXV2-DANet was performed on the training set with comparisons between the single-channel and dual-channel input settings; model evaluation was conducted on the independent testing set. The proposed dual-channel CNXV2-DANet was compared against three state-of-the-art networks (ConvNeXtV2, ConvNeXt, Swin Transformer). The results demonstrated that both CNXV2-DANet and ConvNeXt V2 performed more superiorly under dual-channel inputs than the single-channel input. The dual-channel CNXV2-DANet outperformed the single-channel, achieving superior average metrics for accuracy (0.839 ± 0.028), precision (0.849 ± 0.014), recall (0.845 ± 0.043), F1-score (0.820 ± 0.038), and AUC (0.906 ± 0.013) compared with the single channel metrics for accuracy (0.784 ± 0.037), precision (0.817 ± 0.090), recall (0.748 ± 0.093), F1-score (0.773 ± 0.037), and AUC (0.861 ± 0.047). Furthermore, the dual-channel CNXV2-DANet outperformed all other networks (all p-values < 0.001). These findings suggest that the proposed dual-channel CNXV2-DANet may provide the community with an AI-empowered TCS-based tool for PD assessment. [ABSTRACT FROM AUTHOR]

This paper proposes a deep learning-based intelligent detection device for insulation pull rod defects, addressing the issues of low detection accuracy, poor timeliness of intelligent analysis, and the difficulty in preserving detection results. Firstly, by constructing the pull rod defects dataset and training the YOLOv5s network, along with commonly used object detection algorithms in industrial defect detection, the feasibility of deep learning networks for insulation pull rod defects detection is explored. Secondly, the trained model is combined to build an intelligent detection device for pull rod defects, integrating insulation pull rod image acquisition and defect detection into a unified system. The research results demonstrate that the YOLOv5s network can quickly and accurately detect pull rod defects. On the test set constructed in this paper, the detection performance metric mAP@0.5:0.95 of the trained model reached 54.7%. Specifically, the mAP@0.5 score was 86.9% at a threshold of 0.5. The detection speed FPS reached 169.5, significantly improving the detection efficiency and accuracy compared to traditional object detection algorithms. By establishing an organic connection between the image hardware acquisition device and the deep learning network, the existing problems of inefficient detection and difficult storage of detection results in pull rod defects detection methods are effectively addressed. This research provides new insights for detecting insulation pull rod defects. [ABSTRACT FROM AUTHOR]

Surface defect inspection based on deep learning has demonstrated outstanding performance in improving detection accuracy and model generalization. However, the small scale of defect datasets always limits the application of deep models in industry. Generative models can obtain realistic samples in a very cheap way, which can effectively solve this problem and thus has received widespread attention in recent years. This paper provides a comprehensive analysis and summary of the current studies of surface defect inspection methods proposed between 2022 and 2024. First, according to the use of generative models, these methods are classified into four categories: Variational Auto-Encoders (VAEs), Generative Adversarial Networks (GANs), Diffusion Models (DMs), and multi-models. Second, the research status of surface defect inspection based on generative models in recent years is discussed from four aspects: sample generation, detection objective, inspection task, and learning model. Then, the public datasets and evaluation metrics that are commonly used for surface defect inspection are discussed, and a comparative evaluation of defect inspection methods based on generative models is provided. Finally, this study discusses the existing challenges for the defect inspection methods based on generative models, providing insights for future research. [ABSTRACT FROM AUTHOR]

Feature extraction and matching play integral roles in autonomous vision navigation. As a typical morphological feature, craters can be used as navigation landmarks. However, the spin of small bodies introduces substantial variations in illumination and viewing angles within acquired images, thereby posing challenges to feature extraction and matching algorithms. Addressing these challenges, a deep learning-based approach emerges as pivotal in mitigating the limitations of existing navigation algorithms. This paper presents an algorithm designed to segment noteworthy craters and irregular regions on such small bodies, thereby furnishing crucial autonomous navigation data for missions involving uncharted terrains. By seamlessly integrating deep learning techniques with conventional computer vision-based feature description methods, the algorithm achieves matching segmentation results of noteworthy precision. The experimental results show that the algorithm has better results compared to the existing mainstream segmentation networks. And it can achieve the accurate matching of segmentation results. • Accurate segmentation of small-body craters via deep-learning Algorithms. • Gradient descriptor construction method based on segmentation results. • The proposed method has viewpoint, illumination, scale, and rotation invariance. [ABSTRACT FROM AUTHOR]

Deep learning is a powerful technique to boost application performance in various fields, including face recognition, image classification, natural language understanding, and recommendation system. With the rapid increase in the computing power of mobile devices, developers can embed deep learning models into their apps for building more competitive products with more accurate and faster responses. Although there are several works of adversarial attacks against deep learning models in apps, they all need information about the models' internals (i.e., structures and weights) or need to modify the models. In this paper, we propose an effective black‐box approach by training substitute models to spoof the deep learning systems inside the apps. We evaluate our approach on 10 real‐world deep‐learning apps from Google Play to perform black‐box adversarial attacks. Through the study, we find three factors that can affect the performance of attacks. Our approach can reach a relatively high attack success rate of 66.60% on average. Compared with other adversarial attacks on mobile deep learning models, in terms of the average attack success rates, our approach outperforms its counterparts by 27.63%. [ABSTRACT FROM AUTHOR]

Recent advances in high-throughput electron microscopy (EM) have revolutionized the examination of microstructures by enabling fast EM image generation. However, accurately segmenting EM images remains challenging due to inherent characteristics, including low contrast and subtle grayscale variations. Moreover, as manually annotated EM images are limited, it is usually impractical to utilize deep learning techniques for EM image segmentation. To address these challenges, the pyramid multiscale channel attention network (PmcaNet) is specifically designed. PmcaNet employs a convolutional neural network-based architecture and a multiscale feature pyramid to effectively capture global context information, enhancing its ability to comprehend the intricate structures within EM images. To enable the rapid extraction of channel-wise dependencies, a novel attention module is introduced to enhance the representation of intricate nonlinear features within the images. The performance of PmcaNet is evaluated on two general EM image segmentation datasets as well as a homemade dataset of superalloy materials, regarding pixel-wise accuracy and mean intersection over union (mIoU) as evaluation metrics. Extensive experiments demonstrate that PmcaNet outperforms other models on the ISBI 2012 dataset, achieving 87.85% pixel-wise accuracy and 73.11% mean intersection over union (mIoU), while also advancing results on the Kathuri and SEM-material datasets. [ABSTRACT FROM AUTHOR]

A variety of neural networks have been presented to deal with issues in deep learning in the last decades. Despite the prominent success achieved by the neural network, it still lacks theoretical guidance to design an efficient neural network model, and verifying the performance of a model needs excessive resources. Previous research studies have demonstrated that many existing models can be regarded as different numerical discretizations of differential equations. This connection sheds light on designing an effective recurrent neural network (RNN) by resorting to numerical analysis. Simple RNN is regarded as a discretisation of the forward Euler scheme. Considering the limited solution accuracy of the forward Euler methods, a Taylor‐type discrete scheme is presented with lower truncation error and a Taylor‐type RNN (T‐RNN) is designed with its guidance. Extensive experiments are conducted to evaluate its performance on statistical language models and emotion analysis tasks. The noticeable gains obtained by T‐RNN present its superiority and the feasibility of designing the neural network model using numerical methods. [ABSTRACT FROM AUTHOR]

Although novel 3D animation techniques could be boosted by a large variety of deep learning methods, flexible automatic 3D applications (involving animated figures such as humans and low-life animals) are still rarely studied in 3D computer vision. This is due to lacking of arbitrary 3D data acquisition environment, especially those involving human populated scenes. Given a single image, the 3D animation aided by contextual inference is still plagued by limited reconstruction clues without prior knowledge pertinent to the identified figures/objects and/or their possible relationship w.r.t. the environment. To alleviate such difficulty in time-varying 3D animation, we devise a dynamic scene creation framework via a dynamic knowledge graph (DKG). The DKG encodes both temporal and spatial contextual clues to enable and facilitate human interactions with the affordance environment. Furthermore, we construct the DKG-driven variational auto-encoder (DVAE) upon animation kinematics knowledge conveyed by meta-motion sequences, which are disentangled from videos of prior scenes. It is then possible to utilize the DKG to induce the animations in certain scenes, thus, we could automatically and physically generate plausible 3D animations that afford vivid interactions among humans, low-and life animals in the environment. The extensive experimental results and comprehensive evaluations confirm our DKGs' representation and modeling power towards new animation production in 3D graphics and vision applications. [ABSTRACT FROM AUTHOR]

Semantic segmentation of large-scale indoor 3D point cloud scenes is crucial for scene understanding but faces challenges in effectively modeling long-range dependencies and multi-scale features. In this paper, we present RegionPVT, a novel Regional-to-Local Point-Voxel Transformer that synergistically integrates voxel-based regional self-attention and window-based point-voxel self-attention for concurrent coarse-grained and fine-grained feature learning. The voxel-based regional branch focuses on capturing regional context and facilitating inter-window communication. The window-based point-voxel branch concentrates on local feature learning while integrating voxel-level information within each window. This unique design enables the model to jointly extract local details and regional structures efficiently and provides an effective and efficient solution for multi-scale feature fusion and a comprehensive understanding of 3D point clouds. Extensive experiments on S3DIS and ScanNet v2 datasets demonstrate that our RegionPVT achieves competitive or superior performance compared with state-of-the-art approaches, attaining mIoUs of 71.0% and 73.9% respectively, with significantly lower memory footprint. [ABSTRACT FROM AUTHOR]

Magnetotelluric (MT) inversion and forward modeling are closely linked. The optimization and iteration processes of the inverse algorithm require frequent calls to forward modeling. However, traditional numerical simulations for forward modeling are computationally expensive; here, deep learning (DL) networks can simulate forward modeling and significantly improve forward speed. Applying DL for forward modeling in inversion problems requires a high-precision network capable of responding to fine changes in the model to achieve high accuracy in inversion optimization. Most existing MT studies have used a convolutional neural network, but this method is limited by the receptive field and cannot extract global feature information. In contrast, the Mix Transformer has the ability to globally model and extract features. In this study, we used a Mix Transformer to hierarchically extract feature information, adopted a multiscale approach to restore feature information to the decoder, and eliminated the skip connection between the encoder and decoder. We designed a forward modeling network model (MT-MitNet) oriented toward inversion. A sample dataset required for DL forward was established using the forward data generated from the traditional inverse calculation iteration process. The trained network quickly and accurately calculates the forward response. The experimental results indicate a high agreement between the forward results of MT-MitNet and those obtained with traditional methods. When MT-MitNet replaces the forward computation in traditional inversion, the inversion results obtained with it are also highly in agreement with the traditional inversion results. Importantly, under the premise of ensuring high accuracy, the forward speed of MT-MitNet is hundreds of times faster than that of traditional inversion methods in the same process. [ABSTRACT FROM AUTHOR]

Rapid reconnaissance of building damage is critical for disaster response and recovery. Drones have been utilized to collect aerial images of affected areas in order to assess building damage. However, there are two challenges. First, processing many aerial images to detect and classify building damage based on a consistent standard remains laborious and complex, necessitating a new automated solution to achieve accurate building damage detection and classification. Second, drone operations during disaster response rely primarily on human operators' experience and seldom use the obtained building damage information to optimize drone mission planning. Therefore, this study proposes a new method, which automates building damage reconnaissance with drone mission planning for disaster response operations. Specifically, a deep learning method is developed to detect and classify building damages using a newly labeled dataset consisting of 24,496 distinct instances of building damage. This deep learning method is validated, achieving 71.9% mean average precision. In addition, building damage information is modeled and integrated into mission planning, in order to optimize drones' task assignments and route calculations. A tornado disaster in Tennessee is used as a case study to quantitatively evaluate this methodology. The present study concludes that optimal drone mission planning during disaster response can be augmented using accurate building damage information acquired from deep learning methods. [ABSTRACT FROM AUTHOR]

Despite the successful application of multimodal deep learning (MDL) methods for land use/land cover (LULC) classification tasks, their fusion capacity has not yet been substantially examined for hyperspectral and synthetic aperture radar (SAR) data. Hyperspectral and SAR data have recently been widely used in land cover classification. However, the speckle noise of SAR and the heterogeneity with the imaging mechanism of hyperspectral data have hindered the application of MDL methods for integrating hyperspectral and SAR data. Accordingly, we proposed a deep feature fusion method called Refine-EndNet that combines a dynamic filter network (DFN), an attention mechanism (AM), and an encoder–decoder framework (EndNet). The proposed method is specifically designed for hyperspectral and SAR data and adopts an intra-group and inter-group feature fusion strategy. In intra-group feature fusion, the spectral information of hyperspectral data is integrated by fully connected neural networks in the feature dimension. The fusion filter generation network (FFGN) suppresses the presence of speckle noise and the influence of heterogeneity between multimodal data. In inter-group feature fusion, the fusion weight generation network (FWGN) further optimizes complementary information and improves fusion capacity. Experimental results from ZY1-02D satellite hyperspectral data and Sentinel-1A dual-polarimetric SAR data illustrate that the proposed method outperforms the conventional feature-level image fusion (FLIF) and MDL methods, such as S2ENet, FusAtNet, and EndNets, both visually and numerically. We first attempt to investigate the potentials of ZY1-02D satellite hyperspectral data affected by thick clouds, combined with SAR data for complex ground object classification in the land cover ecosystem. [ABSTRACT FROM AUTHOR]

Featured Application: P-wave first-motion detection; focal mechanism inversion. P-wave first-motion polarity is important for the inversion of earthquake focal mechanism solutions. The focal mechanism solution can further contribute to our understanding of the source rupture process, the fault structure, and the regional stress field characteristics. By using the abundant focal mechanism solutions of small and moderate earthquakes, we can deepen our understanding of fault geometry and the seismogenic environment. In this paper, we propose an automatic workflow, FocMech-Flow (Focal Mechanism-Flow), for identifying P-wave first-motion polarity and focal mechanism inversion with deep learning and applied it to the 2021 Yangbi earthquake sequence. We use a deep learning model named DiTingMotion to detect the P-wave first-motion polarity of 2389 waveforms, resulting in 98.49% accuracy of polarity discrimination compared with human experts. The focal mechanisms of 112 earthquakes are obtained by using the CHNYTX program, which is 3.7 times more than that of the waveform inversion method, and the results are highly consistent. The analysis shows that the focal mechanisms of the foreshock sequence of the Yangbi earthquake are highly consistent and are all of the strike-slip type; the focal mechanisms of the aftershock sequence are complex, mainly the strike-slip type, but there are also reverse and normal fault types. This study shows that the deep learning method has high reliability in determining the P-wave first-motion polarity, and FocMech-Flow can obtain a large number of focal mechanism solutions from small and moderate earthquakes, having promising application in fine-scale stress inversion. [ABSTRACT FROM AUTHOR]

Robotics has attracted broad attention as an emerging technology in construction to help workers with repetitive, physically demanding, and dangerous tasks, thus improving productivity and safety. Under the new era of human–robot coexistence and collaboration in dynamic and complex workspaces, it is critical for robots to navigate to the targets efficiently without colliding with moving workers. This study proposes a new deep reinforcement learning (DRL)–based robot path planning method that integrates the predicted movements of construction workers to achieve safe and efficient human–robot collaboration in construction. First, an uncertainty-aware long short-term memory network is developed to predict the movements of construction workers and associated uncertainties. Second, a DRL framework is formulated, where predicted movements of construction workers are innovatively integrated into the state space and the computation of the reward function. By incorporating predicted trajectories in addition to current locations, the proposed method enables proactive planning such that the robot could better adapt to human movements, thus ensuring both safety and efficiency. The proposed method was demonstrated and evaluated using simulations generated based on real construction scenarios. The results show that prediction-based DRL path planning achieved a 100% success rate (with a total of 10,000 episodes) for robots to achieve the destination along the near-shortest path. Furthermore, it reduced the collision rate with moving workers by 23% compared with the conventional DRL method, which does not consider predicted information. [ABSTRACT FROM AUTHOR]

Deep Learning has been increasingly utilized in structural damage detection. Existing relevant studies often highlight the benefits of supervised deep learning in the intelligent identification of bridge damage. Notably, however, supervised deep learning methods encounter specific challenges in processing real-world monitoring data to reflect damage. Typical challenges include: (i) the need for a large dataset derived from a vast number of labeled damaged cases, which are extremely difficult to obtain from real-world monitoring data, experiments, and numerical simulations; (ii) the inability of available damage sample spectra to fully capture the damage information underlying the dynamic responses of bridges; (iii) the likelihood of requiring manual intervention to discriminate damage from the outputs of deep learning models, which is inefficient when dealing with massive amounts of monitoring data. To address these challenges, this study proposes an intelligent unsupervised deep learning framework for damage identification in bridges. The framework is characterized by three innovative technical elements: (1) a Deep Convolutional Autoencoder (CAE) model with a hybrid loss function is developed to provide an intelligent system that can identify bridge damage without requiring a large dataset of damaged cases; (2) Wavelet transmissibility pattern spectra are established to characterize damage information embedded in dynamic responses in a more efficient manner; and (3) an Ordering Points To Identify the Clustering Structure (OPTICS)-based damage picker is proposed to achieve automatic discrimination of damage cases. The feasibility of this framework is numerically demonstrated through the detection of damage in a curved bridge, and its effectiveness is experimentally validated on a laboratory-scale suspension bridge with induced damage. The results indicate that the proposed framework can automatically and accurately identify damage in bridges. This framework provides a solution for intelligent and data-driven bridge damage detection. [ABSTRACT FROM AUTHOR]

The rapid growth of Internet of Things (IoT) and Internet of Vehicles (IoV) are rapidly moving to the 6G networks, which leads to dramatically raised security issues. Using machine learning, including deep learning, to find out malicious network traffic is one of practical ways. Though much work has been done in this direction, we found little investigating the effect of using fused network conversation datasets to train and test models. Thus, this work proposes to check conversation dataset characteristics and find suitable ones to fuse into one dataset in order to improve the capability of malicious traffic and malware detection performance. The experiments using real data show that conditioned combination of datasets can be used to enhance algorithm performance and improve detection results. For this reason, it is recommended to profile datasets and conduct conditional fusion of network conversation datasets before using machine learning or deep learning. As the characterization is done using general statistical calculation, it is promising to be used for other domains too. [ABSTRACT FROM AUTHOR]

In the field of educational data mining, the classification of students' grades is a subject that receives widespread attention. However, solving this problem based on machine learning algorithms and deep learning algorithms is usually limited by large datasets. The privacy problem of educational data platforms also limits the possibility of building an extensive dataset of students' information and behavior by gathering small datasets and then carrying out the federated training model. Therefore, the balance of educational data and the inconsistency of feature distribution are the critical problems that need to be solved urgently in educational data mining. Federated learning technology enables multiple participants to continue machine learning and deep learning in protecting data privacy and meeting legal compliance requirements to solve the data island problem. However, these methods are only applicable to the data environment with common characteristics or common samples under the alliance. This results in domain transfer between nodes. Therefore, in this paper, we propose a framework based on federated transfer learning for student classification with privacy protection. This framework introduces the domain adaptation method and extends the domain adaptation to the constraint of federated learning. Through the feature extractor, this method matches the feature distribution of each party in the feature space. Then, labels and domains are classified on each side, the model is trained, and the target model is updated by gradient aggregation. The federated learning framework based on this method can effectively solve the federated transfer learning on heterogeneous datasets. We evaluated the performance of the proposed framework for student classification on the datasets of two courses. We simulated four scenarios according to different situations in reality. Then, the results of only source domain training, only target domain training, and federated migration training are compared. The experimental results show that the heterogeneous federated transfer framework based on domain adaptation can solve federated learning and knowledge transfer problems when there are little data at the data source and can be used for students' grades classification in small datasets. [ABSTRACT FROM AUTHOR]

Existing disinfection robots are not intelligent enough to adapt their actions to object surface materials for precise and effective disinfection. To address this problem, a new framework is developed to enable the robot to recognize various object surface materials and to adapt its disinfection methods to be compatible with recognized object surface materials. Specifically, a new deep learning network is proposed that integrates multi‐level and multi‐scale features to classify the materials on contaminated surfaces requiring disinfection. The infection risk of contaminated surfaces is computed to choose the appropriate disinfection modes and parameters. The developed material recognition method demonstrates state‐of‐the‐art performance, achieving an accuracy of 92.24% and 91.84% on the Materials in Context Database validation and test datasets, respectively. The proposed method was also tested and evaluated in the context of healthcare facilities, where the material classification achieved an accuracy of 89.09%, and the adaptive robotic disinfection was successfully implemented. [ABSTRACT FROM AUTHOR]

Recent research advances in salient object detection (SOD) could largely be attributed to ever-stronger multi-scale feature representation empowered by the deep learning technologies. The existing SOD deep models extract multi-scale features via the off-the-shelf encoders and combine them smartly via various delicate decoders. However, the kernel sizes in this commonly-used thread are usually “fixed”. In our new experiments, we have observed that kernels of small size are preferable in scenarios containing tiny salient objects. In contrast, large kernel sizes could perform better for images with large salient objects. Inspired by this observation, we advocate the “dynamic” scale routing (as a brand-new idea) in this paper. It will result in a generic plug-in that could directly fit the existing feature backbone. This paper’s key technical innovations are two-fold. First, instead of using the vanilla convolution with fixed kernel sizes for the encoder design, we propose the dynamic pyramid convolution (DPConv), which dynamically selects the best-suited kernel sizes w.r.t. the given input. Second, we provide a self-adaptive bidirectional decoder design to accommodate the DPConv-based encoder best. The most significant highlight is its capability of routing between feature scales and their dynamic collection, making the inference process scale-aware. As a result, this paper continues to enhance the current SOTA performance. Both the code and dataset are publicly available at https://github.com/wuzhenyubuaa/DPNet. [ABSTRACT FROM AUTHOR]

The accurate detection of dental plaque at an early stage will definitely prevent periodontal diseases and dental caries. However, it remains difficult for the current dental examination to accurately recognize dental plaque without using medical dyeing reagent due to the low contrast between dental plaque and healthy teeth. To combat this problem, this paper proposes a novel network enhanced by a self-attention module for intelligent dental plaque segmentation. The key motivation is to directly utilize oral endoscope images (bypassing the need for dyeing reagent) and get accurate pixel-level dental plaque segmentation results. The algorithm needs to conduct self-attention at the super-pixel level and fuse the super-pixels’ local-to-global features. Our newly-designed network architecture will afford the simultaneous fusion of multiple-scale complementary information guided by the powerful deep learning paradigm. The critical fused information includes the statistical distribution of the plaques color, the heat kernel signature (HKS) based local-to-global structure relationship, and the circle-LBP based local texture pattern in the nearby regions centering around the plaque area. To further refine the fuzed multiple-scale features, we devise an attention module based on CNN, which could focalize the regions of interest in plaque more easily, especially for many challenging cases. Extensive experiments and comprehensive evaluations confirm that, for a small-scale training dataset, our method could outperform the state-of-the-art methods. Meanwhile, the user studies verify the claim that our method is more accurate than conventional dental practice conducted by experienced dentists. [ABSTRACT FROM AUTHOR]

Recurrent neural networks (RNNs) have been widely used to solve sequence problems due to their capability of modeling temporal dependency. Despite the rich varieties of RNN models proposed in the literature, the problem of different sampling rates or performing speeds in sequence tasks has not been explicitly considered in the network and the corresponding training and testing processes. This paper addresses the problem of different sampling rates or performing speeds in the skeleton-based action recognition with RNNs. Specifically, the recently proposed independently recurrent neural network (IndRNN) is used as the RNN network due to its well-behaved and easily regulated gradient backpropagation through time. Samples are extracted with variable sampling rates and thus of different lengths, then processed by IndRNN with different time steps. In order to accommodate the differences in terms of gradients introduced by the backpropagation through time under variable time steps, a learning rate adjustment method is further proposed in the paper. Different learning rate adjustment factors are obtained for different layers by analyzing the gradient behavior under IndRNN. Experiments on skeleton-based action recognition are conducted to verify its effectiveness, and the results show that the proposed variable rate IndRNN network can significantly improve the performance over the RNN models under the conventional training strategies. [ABSTRACT FROM AUTHOR]

The rice seed setting rate (RSSR) is an important component in calculating rice yields and a key phenotype for its genetic analysis. Automatic calculations of RSSR through computer vision technology have great significance for rice yield predictions. The basic premise for calculating RSSR is having an accurate and high throughput identification of rice grains. In this study, we propose a method based on image segmentation and deep learning to automatically identify rice grains and calculate RSSR. By collecting information on the rice panicle, our proposed image automatic segmentation method can detect the full grain and empty grain, after which the RSSR can be calculated by our proposed rice seed setting rate optimization algorithm (RSSROA). Finally, the proposed method was used to predict the RSSR during which process, the average identification accuracy reached 99.43%. This method has therefore been proven as an effective, non-invasive method for high throughput identification and calculation of RSSR. It is also applicable to soybean yields, as well as wheat and other crops with similar characteristics. [ABSTRACT FROM AUTHOR]

Directly benefiting from the deep learning methods, object detection has witnessed a great performance boost in recent years. However, drone-view object detection remains challenging for two main reasons: (1) Objects of tiny-scale with more blurs w.r.t. ground-view objects offer less valuable information towards accurate and robust detection; (2) The unevenly distributed objects make the detection inefficient, especially for regions occupied by crowded objects. Confronting such challenges, we propose an end-to-end global-local self-adaptive network (GLSAN) in this paper. The key components in our GLSAN include a global-local detection network (GLDN), a simple yet efficient self-adaptive region selecting algorithm (SARSA), and a local super-resolution network (LSRN). We integrate a global-local fusion strategy into a progressive scale-varying network to perform more precise detection, where the local fine detector can adaptively refine the target’s bounding boxes detected by the global coarse detector via cropping the original images for higher-resolution detection. The SARSA can dynamically crop the crowded regions in the input images, which is unsupervised and can be easily plugged into the networks. Additionally, we train the LSRN to enlarge the cropped images, providing more detailed information for finer-scale feature extraction, helping the detector distinguish foreground and background more easily. The SARSA and LSRN also contribute to data augmentation towards network training, which makes the detector more robust. Extensive experiments and comprehensive evaluations on the VisDrone2019-DET benchmark dataset and UAVDT dataset demonstrate the effectiveness and adaptivity of our method. Towards an industrial application, our network is also applied to a DroneBolts dataset with proven advantages. Our source codes have been available at https://github.com/dengsutao/glsan. [ABSTRACT FROM AUTHOR]

Existing RGB-D salient object detection methods treat depth information as an independent component to complement RGB and widely follow the bistream parallel network architecture. To selectively fuse the CNN features extracted from both RGB and depth as a final result, the state-of-the-art (SOTA) bistream networks usually consist of two independent subbranches: one subbranch is used for RGB saliency, and the other aims for depth saliency. However, depth saliency is persistently inferior to the RGB saliency because the RGB component is intrinsically more informative than the depth component. The bistream architecture easily biases its subsequent fusion procedure to the RGB subbranch, leading to a performance bottleneck. In this paper, we propose a novel data-level recombination strategy to fuse RGB with D (depth) before deep feature extraction, where we cyclically convert the original 4-dimensional RGB-D into DGB, RDB and RGD. Then, a newly lightweight designed triple-stream network is applied over these novel formulated data to achieve an optimal channel-wise complementary fusion status between the RGB and D, achieving a new SOTA performance. [ABSTRACT FROM AUTHOR]

Structural monitoring systems installed on bridges are capable of capturing large-scale dynamic responses online and in real-time. The response data of the bridge under different loading conditions is used for condition assessment of the bridge. There is a chasm between monitoring data and damage assessment due to the difficulty in revealing the relationship between the massive monitoring data and damage. this study proposes an intelligent framework for identifying (localizing and quantifying) damage in a bridge from massive acceleration responses. The framework features (i) physical big-data quantity and (ii) an intelligent identification model. The former specifies a mechanical quantity termed damage pattern spectrum of power-spectrum-density transmissibility (PSDT), the PSDT-damage pattern spectrum for short, constructed to protrude damage information from massive acceleration responses; the latter specifies an upgraded Capsules Networks (CapsNets), Up-CapsNets for short, which intelligently locate and quantify damage via deep learning with inputted PSDT-damage pattern spectrum. The proposed intelligent framework is verified through a numerically truss bridge model and an experimental suspension bridge model. The results demonstrate that this intelligent framework can identify locations and severity of damage with greater accuracy, stronger immunity to noise, and higher generalization than existing convolutional neural networks (CNNs) and CapsNets. Such a peculiarity can be attributed to two points: (i) the sophisticated function of the Up-CapsNets in recognizing damage patterns over conventional deep learning networks; and (ii) the unique qualification of PSDT-damage pattern spectrum in refining damage information over raw acceleration responses. The proposed intelligent framework provides a viable approach for monitoring the condition of a bridge by combining deep-learning deduction and big-data dynamic responses. • Propose the PSDT-damage pattern spectrum to refine damage information contained in the response signals. • Upgrade the standard CapsNets for locating and quantifying damage in operating bridges. • Create an intelligent framework for real time and high-accurate damage identification. • The intelligent framework provides a viable approach for monitoring the condition of bridges. [ABSTRACT FROM AUTHOR]

In this paper, we introduce the algorithms of Orthogonal Deep Neural Networks (OrthDNNs) to connect with recent interest of spectrally regularized deep learning methods. OrthDNNs are theoretically motivated by generalization analysis of modern DNNs, with the aim to find solution properties of network weights that guarantee better generalization. To this end, we first prove that DNNs are of local isometry on data distributions of practical interest; by using a new covering of the sample space and introducing the local isometry property of DNNs into generalization analysis, we establish a new generalization error bound that is both scale- and range-sensitive to singular value spectrum of each of networks’ weight matrices. We prove that the optimal bound w.r.t. the degree of isometry is attained when each weight matrix has a spectrum of equal singular values, among which orthogonal weight matrix or a non-square one with orthonormal rows or columns is the most straightforward choice, suggesting the algorithms of OrthDNNs. We present both algorithms of strict and approximate OrthDNNs, and for the later ones we propose a simple yet effective algorithm called Singular Value Bounding (SVB), which performs as well as strict OrthDNNs, but at a much lower computational cost. We also propose Bounded Batch Normalization (BBN) to make compatible use of batch normalization with OrthDNNs. We conduct extensive comparative studies by using modern architectures on benchmark image classification. Experiments show the efficacy of OrthDNNs. [ABSTRACT FROM AUTHOR]

Directly benefiting from the rapid advancement of deep learning methods, person re-identification (Re-ID) applications have been widespread with remarkable successes in recent years. Nevertheless, cross-scene Re-ID is still hindered by large view variation, since it is challenging to effectively exploit and leverage the temporal clues due to heavy computational burden and the difficulty in flexibly incorporating discriminative features. To alleviate, we articulate a long-short temporal–spatial clues excited network (LSTS-NET) for robust person Re-ID across different scenes. In essence, our LSTS-NET comprises a motion appearance model and a motion-refinement aggregating scheme. Of which, the former abstracts temporal clues based on multi-range low-rank analysis both in consecutive frames and in cross-camera videos, which can augment the person-related features with details while suppressing the clutter background across different scenes. In addition, to aggregate the temporal clues with spatial features, the latter is proposed to automatically activate the person-specific features by incorporating personalized motion-refinement layers and several motion-excitation CNN blocks into deep networks, which expedites the extraction and learning of discriminative features from different temporal clues. As a result, our LSTS-NET can robustly distinguish persons across different scenes. To verify the improvement of our LSTS-NET, we conduct extensive experiments and make comprehensive evaluations on 8 widely-recognized public benchmarks. All the experiments confirm that, our LSTS-NET can significantly boost the Re-ID performance of existing deep learning methods, and outperforms the state-of-the-art methods in terms of robustness and accuracy. [ABSTRACT FROM AUTHOR]

Despite growing progresses in recent years, cross-scenario person re-identification remains challenging, mainly due to the pedestrians commonly surrounded by highly-complex environment contexts. In reality, the human perception mechanism could adaptively find proper contextualized spatial-temporal clues towards pedestrian recognition. However, conventional methods fall short in adaptively leveraging the long-term spatial-temporal information due to ever-increasing computational cost. Moreover, CNN-based deep learning methods are hard to conduct optimization due to the non-differentiable property of the built-in context search operation. To ameliorate, this paper proposes a novel Context-Interactive CNN (CI-CNN) to dynamically find both spatial and temporal contexts by embedding multi-task Reinforcement Learning (MTRL). The CI-CNN streamlines the multi-task reinforcement learning by using an actor-critic agent to capture the temporal-spatial context simultaneously, which comprises a context-policy network and a context-critic network. The former network learns policies to determine the optimal spatial context region and temporal sequence range. Based on the inferred temporal-spatial cues, the latter one focuses on the identification task and provides feedback for the policy network. Thus, CI-CNN can simultaneously zoom in/out the perception field in spatial and temporal domain for the context interaction with the environment. By fostering the collaborative interaction between the person and context, our method could achieve outstanding performance on various public benchmarks, which confirms the rationality of our hypothesis, and verifies the effectiveness of our CI-CNN framework. [ABSTRACT FROM AUTHOR]

In recent years deep neural networks have been widely applied to visual saliency detection tasks with remarkable detection performance improvements. As for the salient object detection in single image, the automatically computed convolutional features frequently demonstrate high discriminative power to distinguish salient foregrounds from its non-salient surroundings in most cases. Yet, the obstinate feature conflicts still persist, which naturally gives rise to the learning ambiguity, arriving at massive failure detections. To solve such problem, we propose to jointly re-learn common consistency of inter-image saliency and then use it to boost the detection performance. Its core rationale is to utilize the easy-to-detect cases to re-boost much harder ones. Compared with the conventional methods, which focus on their problem domain within the single image scope, our method attempts to utilize those beyond-scope information to facilitate the current salient object detection. To validate our new approach, we have conducted a comprehensive quantitative comparisons between our approach and 13 state-of-the-art methods over 5 publicly available benchmarks, and all the results suggest the advantage of our approach in terms of accuracy, reliability, and versatility. [ABSTRACT FROM AUTHOR]