Search

Purpose: Prognostic prediction is crucial to guide individual treatment for locoregionally advanced nasopharyngeal carcinoma (LA-NPC) patients. Recently, multi-task deep learning was explored for joint prognostic prediction and tumor segmentation in various cancers, resulting in promising performance. This study aims to evaluate the clinical value of multi-task deep learning for prognostic prediction in LA-NPC patients., Methods: A total of 886 LA-NPC patients acquired from two medical centers were enrolled including clinical data, [ 18 F]FDG PET/CT images, and follow-up of progression-free survival (PFS). We adopted a deep multi-task survival model (DeepMTS) to jointly perform prognostic prediction (DeepMTS-Score) and tumor segmentation from FDG-PET/CT images. The DeepMTS-derived segmentation masks were leveraged to extract handcrafted radiomics features, which were also used for prognostic prediction (AutoRadio-Score). Finally, we developed a multi-task deep learning-based radiomic (MTDLR) nomogram by integrating DeepMTS-Score, AutoRadio-Score, and clinical data. Harrell's concordance indices (C-index) and time-independent receiver operating characteristic (ROC) analysis were used to evaluate the discriminative ability of the proposed MTDLR nomogram. For patient stratification, the PFS rates of high- and low-risk patients were calculated using Kaplan-Meier method and compared with the observed PFS probability., Results: Our MTDLR nomogram achieved C-index of 0.818 (95% confidence interval (CI): 0.785-0.851), 0.752 (95% CI: 0.638-0.865), and 0.717 (95% CI: 0.641-0.793) and area under curve (AUC) of 0.859 (95% CI: 0.822-0.895), 0.769 (95% CI: 0.642-0.896), and 0.730 (95% CI: 0.634-0.826) in the training, internal validation, and external validation cohorts, which showed a statistically significant improvement over conventional radiomic nomograms. Our nomogram also divided patients into significantly different high- and low-risk groups., Conclusion: Our study demonstrated that MTDLR nomogram can perform reliable and accurate prognostic prediction in LA-NPC patients, and also enabled better patient stratification, which could facilitate personalized treatment planning., (© 2023. The Author(s).)

Nasopharyngeal Carcinoma (NPC) is a malignant epithelial cancer arising from the nasopharynx. Survival prediction is a major concern for NPC patients, as it provides early prognostic information to plan treatments. Recently, deep survival models based on deep learning have demonstrated the potential to outperform traditional radiomics-based survival prediction models. Deep survival models usually use image patches covering the whole target regions (e.g., nasopharynx for NPC) or containing only segmented tumor regions as the input. However, the models using the whole target regions will also include non-relevant background information, while the models using segmented tumor regions will disregard potentially prognostic information existing out of primary tumors (e.g., local lymph node metastasis and adjacent tissue invasion). In this study, we propose a 3D end-to-end Deep Multi-Task Survival model (DeepMTS) for joint survival prediction and tumor segmentation in advanced NPC from pretreatment PET/CT. Our novelty is the introduction of a hard-sharing segmentation backbone to guide the extraction of local features related to the primary tumors, which reduces the interference from non-relevant background information. In addition, we also introduce a cascaded survival network to capture the prognostic information existing out of primary tumors and further leverage the global tumor information (e.g., tumor size, shape, and locations) derived from the segmentation backbone. Our experiments with two clinical datasets demonstrate that our DeepMTS can consistently outperform traditional radiomics-based survival prediction models and existing deep survival models.

Objective. Positron emission tomography-computed tomography (PET-CT) is regarded as the imaging modality of choice for the management of soft-tissue sarcomas (STSs). Distant metastases (DM) are the leading cause of death in STS patients and early detection is important to effectively manage tumors with surgery, radiotherapy and chemotherapy. In this study, we aim to early detect DM in patients with STS using their PET-CT data. Approach. We derive a new convolutional neural network method for early DM detection. The novelty of our method is the introduction of a constrained hierarchical multi-modality feature learning approach to integrate functional imaging (PET) features with anatomical imaging (CT) features. In addition, we removed the reliance on manual input, e.g. tumor delineation, for extracting imaging features. Main results. Our experimental results on a well-established benchmark PET-CT dataset show that our method achieved the highest accuracy (0.896) and AUC (0.903) scores when compared to the state-of-the-art methods (unpaired student's t-test p -value < 0.05). Significance. Our method could be an effective and supportive tool to aid physicians in tumor quantification and in identifying image biomarkers for cancer treatment., (© 2021 Institute of Physics and Engineering in Medicine.)

Soft-tissue Sarcomas (STS) are a heterogeneous group of malignant neoplasms with a relatively high mortality rate from distant metastases. Early prediction or quantitative evaluation of distant metastases risk for patients with STS is an important step which can provide better-personalized treatments and thereby improve survival rates. Positron emission tomography-computed tomography (PET-CT) image is regarded as the imaging modality of choice for the evaluation, staging and assessment of STS. Radiomics, which refers to the extraction and analysis of the quantitative of high-dimensional mineable data from medical images, is foreseen as an important prognostic tool for cancer risk assessment. However, conventional radiomics methods that depend heavily on hand-crafted features (e.g. shape and texture) and prior knowledge (e.g. tuning of many parameters) therefore cannot fully represent the semantic information of the image. In addition, convolutional neural networks (CNN) based radiomics methods present capabilities to improve, but currently, they are mainly designed for single modality e.g., CT or a particular body region e.g., lung structure. In this work, we propose a deep multi-modality collaborative learning to iteratively derive optimal ensembled deep and conventional features from PET-CT images. In addition, we introduce an end-to-end volumetric deep learning architecture to learn complementary PET-CT features optimised for image radiomics. Our experimental results using public PET-CT dataset of STS patients demonstrate that our method has better performance when compared with the state-of-the-art methods.

In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in http://medicaldecathlon.com/. In addition, both data and online evaluation are accessible via https://competitions.codalab.org/competitions/17094.

Efficient and accurate computation of the single‐scattering properties of black carbon (BC) aerosols is fundamental in various fields, including remote sensing and climate simulations. In this study, we developed a composite model of fractal aggregates of BC encapsulated with hygroscopic aerosols to represent the ambient BC. We used the invariant imbedding T‐matrix method to compute the optical properties of fully and partially encapsulated BC aerosols. In this new model, the traditional assumption of unoverlapped surfaces in the super‐position T‐matrix method is unnecessary. After extensive simulations, we established a database of single‐scattering properties, including the extinction efficiency, the single‐scattering albedo, the asymmetry factor and six phase matrix elements. Moreover, we obtained deep neural networks (DNNs) from this database using a deep learning method. These DNN models provide a universal interface for predicting the optical properties of ambient BC aerosols. Specifically, through a modified architecture of the DNN, we trained two models based on the database to predict three integrated optical properties (extinction efficiency, single‐scattering albedo, and asymmetry factor) and six phase matrix elements. We performed statistical assessments based on the true values in the database and the predicted values from the DNNs, demonstrating that the DNNs accurately predicted all single‐scattering properties. Therefore, the developed DNN models can be conveniently implemented in aerosol optical parameterization for remote sensing studies and atmospheric models. Plain Language Summary: Black carbon (BC) aerosol plays a significant role in the atmosphere, influencing climate through its interactions with radiation, such as scattering and absorbing. The accurate determination of the single‐scattering properties of BC, including the extinction efficiency, the single‐scattering albedo, the asymmetry factor and six phase matrix elements, is crucial for understanding its radiation effects. In this study, we developed a comprehensive model for BC particles that incorporates complex morphological characteristics and the mixing of BC with other hygroscopic aerosols. We established a database of single‐scattering properties for these models using the invariant imbedding T‐matrix method. However, the large storage space required by this database (over 20 GB) makes it impractical for widespread applications. Therefore, we employed a deep learning method to compress the storage consumption. Two deep neural networks (DNNs) were trained using the database, enabling the prediction of all single‐scattering properties. Remarkably, the performance of the DNNs was excellent, with a coefficient of determination greater than 0.99 for all single‐scattering properties, as determined through statistical assessment. These DNN models hold great potential for various atmospheric radiation and remote sensing studies in the future. Key Points: A flexible model of partially and fully encapsulated fractal black carbon particles was developedA database of single‐scattering properties of encapsulated fractal black carbon particles was constructedTwo deep neural networks were obtained, offering a flexible approach to calculating the optical properties of black carbon aerosols [ABSTRACT FROM AUTHOR]

The aim of this study was to computationally model, in an unsupervised manner, a manifold of symmetry variations in normal brains, such that the learned manifold can be used to segment brain tumors from magnetic resonance (MR) images that fail to exhibit symmetry. An unsupervised brain tumor segmentation method, named as symmetric driven generative adversarial network (SD-GAN), was proposed. SD-GAN model was trained to learn a non-linear mapping between the left and right brain images, and thus being able to present the variability of the (symmetry) normal brains. The trained SD-GAN was then used to reconstruct normal brains and to segment brain tumors based on higher reconstruction errors arising from their being unsymmetrical. SD-GAN was evaluated on two public benchmark datasets (Multi-modal Brain Tumor Image Segmentation (BRATS) 2012 and 2018). SD-GAN provided best performance with tumor segmentation accuracy superior to the state-of-the-art unsupervised segmentation methods and performed comparably (<3% lower in Dice score) to the supervised U-Net (the most widely used supervised method for medical images). This study demonstrated that symmetric features presenting variations (i.e., inherent anatomical variations) can be modelled using unannotated normal MR images and thus be used in segmenting tumors. [ABSTRACT FROM AUTHOR]

Highlights • A new dermoscopic segmentation method based on learning class-wise deep features. • A class-specific model helps balance the contribution of melanoma and non-melanoma data. • A step-wise refinement approach iteratively maximises the segmentation map agreement. • A probability based integration approach combines the relevant complementary segmentation results. • We achieve higher accuracy compared to the state-of-the-art methods on 3 datasets. Abstract The segmentation of abnormal regions on dermoscopic images is an important step for automated computer aided diagnosis (CAD) of skin lesions. Recent methods based on fully convolutional networks (FCN) have been very successful for dermoscopic image segmentation. However, they tend to overfit to the visual characteristics that are present in the dominant non-melanoma studies and therefore, perform poorly on the complex visual characteristics exhibited by melanoma studies, which usually consists of fuzzy boundaries and heterogeneous textures. In this paper, we propose a new method for automated skin lesion segmentation that overcomes these limitations via a novel deep class-specific learning approach which learns the important visual characteristics of the skin lesions of each individual class (melanoma vs. non-melanoma) on an individual basis. We also introduce a new probability-based, step-wise integration to combine complementary segmentation results derived from individual class-specific learning models. We achieved an average Dice coefficient of 85.66% on the ISBI 2017 Skin Lesion Challenge (SLC), 91.77% on the ISBI 2016 SLC and 92.10% on the PH2 datasets with corresponding Jaccard indices of 77.73%, 85.92% and 85.90%, respectively, for the same datasets. Our experiments on three well-established public benchmark datasets demonstrate that our method is more effective than other state-of-the-art methods for skin lesion segmentation. [ABSTRACT FROM AUTHOR]

Deformable image registration is fundamental for many medical image analyses. A key obstacle for accurate image registration lies in image appearance variations such as the variations in texture, intensities, and noise. These variations are readily apparent in medical images, especially in brain images where registration is frequently used. Recently, deep learning-based registration methods (DLRs), using deep neural networks, have shown computational efficiency that is several orders of magnitude faster than traditional optimization-based registration methods (ORs). DLRs rely on a globally optimized network that is trained with a set of training samples to achieve faster registration. DLRs tend, however, to disregard the target-pair-specific optimization inherent in ORs and thus have degraded adaptability to variations in testing samples. This limitation is severe for registering medical images with large appearance variations, especially since few existing DLRs explicitly take into account appearance variations. In this study, we propose an Appearance Adjustment Network (AAN) to enhance the adaptability of DLRs to appearance variations. Our AAN, when integrated into a DLR, provides appearance transformations to reduce the appearance variations during registration. In addition, we propose an anatomy-constrained loss function through which our AAN generates anatomy-preserving transformations. Our AAN has been purposely designed to be readily inserted into a wide range of DLRs and can be trained cooperatively in an unsupervised and end-to-end manner. We evaluated our AAN with three state-of-the-art DLRs - Voxelmorph (VM), Diffeomorphic Voxelmorph (DifVM), and Laplacian Pyramid Image Registration Network (LapIRN) – on three well-established public datasets of 3D brain magnetic resonance imaging (MRI) - IBSR18, Mindboggle101, and LPBA40. The results show that our AAN consistently improved existing DLRs and outperformed state-of-the-art ORs on registration accuracy, while adding a fractional computational load to existing DLRs. [ABSTRACT FROM AUTHOR]

Accurate characterization of visual attributes such as spiculation, lobulation, and calcification of lung nodules in computed tomography (CT) images is critical in cancer management. The characterization of these attributes is often subjective, which may lead to high inter- and intra-observer variability. Furthermore, lung nodules are often heterogeneous in the cross-sectional image slices of a 3D volume. Current state-of-the-art methods that score multiple attributes rely on deep learning-based multi-task learning (MTL) schemes. These methods, however, extract shared visual features across attributes and then examine each attribute without explicitly leveraging their inherent intercorrelations. Furthermore, current methods treat each slice with equal importance without considering their relevance or heterogeneity, which limits performance. In this study, we address these challenges with a new convolutional neural network (CNN)-based MTL model that incorporates multiple attention-based learning modules to simultaneously score 9 visual attributes of lung nodules in CT image volumes. Our model processes entire nodule volumes of arbitrary depth and uses a slice attention module to filter out irrelevant slices. We also introduce cross-attribute and attribute specialization attention modules that learn an optimal amalgamation of meaningful representations to leverage relationships between attributes. We demonstrate that our model outperforms previous state-of-the-art methods at scoring attributes using the well-known public LIDC-IDRI dataset of pulmonary nodules from over 1,000 patients. Our model also performs competitively when repurposed for benign-malignant classification. Our attention modules provide easy-to-interpret weights that offer insights into the predictions of the model. [Display omitted] [ABSTRACT FROM AUTHOR]

• We propose to use deep learning with few user-clicks to achieve accurate skin lesion segmentation results. • We propose to leverage user-inputs in optimizing the learning of skin lesion characteristics. • We propose hyper-integration modules (HIMs) to iteratively propagate user-input features and skin lesion image features to ensure the appearance of the segmented skin lesions is spatially consistent. • Our method is capable to segment skin lesions that are known to be challenging, such as those with fuzzy boundaries, inhomogeneous textures and low-contrast to the background. • We had consistently better segmentation results on three well-established public datasets (ISBI 2017, ISBI 2016 skin lesion challenge datasets and PH2 dataset). Segmentation of skin lesions is an important step for imaging-based clinical decision support systems. Automatic skin lesion segmentation methods based on fully convolutional networks (FCNs) are regarded as the state-of-the-art for accuracy. When there are, however, insufficient training data to cover all the variations in skin lesions, where lesions from different patients may have major differences in size/shape/texture, these methods failed to segment the lesions that have image characteristics, which are less common in the training datasets. FCN-based semi-automatic segmentation methods, which fuse user-inputs with high-level semantic image features derived from FCNs offer an ideal complement to overcome limitations of automatic segmentation methods. These semi-automatic methods rely on the automated state-of-the-art FCNs coupled with user-inputs for refinements, and therefore being able to tackle challenging skin lesions. However, there are a limited number of FCN-based semi-automatic segmentation methods and all these methods focused on 'early-fusion', where the first few convolutional layers are used to fuse image features and user-inputs and then derive fused image features for segmentation. For early-fusion based methods, because the user-input information can be lost after the first few convolutional layers, consequently, the user-input information will have limited guidance and constraint in segmenting the challenging skin lesions with inhomogeneous textures and fuzzy boundaries. Hence, in this work, we introduce a hyper-fusion network (HFN) to fuse the extracted user-inputs and image features over multiple stages. We separately extract complementary features which then allows for an iterative use of user-inputs along all the fusion stages to refine the segmentation. We evaluated our HFN on three well-established public benchmark datasets – ISBI Skin Lesion Challenge 2017, 2016 and PH2 – and our results show that the HFN is more accurate and generalizable than the state-of-the-art methods, in particular with challenging skin lesions. [Display omitted] [ABSTRACT FROM AUTHOR]

• We propose a spatial-sequential CNN network for temporal left ventricle segmentation on 4D cardiac sequence. • We utilize the sequential consistency (cardiac motion) to improve the LV segmentation during cardiac systole. • We present an unsupervised sequential CNN network to capture cardiac motion in heart cycle. • We introduce a bi-directional learning strategy to further refine the results. • We achieve higher accuracy compared to the state-of-the-art methods on 4D cardiac CT dataset. Automated segmentation of left ventricular cavity (LVC) in temporal cardiac image sequences (consisting of multiple time-points) is a fundamental requirement for quantitative analysis of cardiac structural and functional changes. Deep learning methods for segmentation are the state-of-the-art in performance; however, these methods are generally formulated to work on a single time-point, and thus disregard the complementary information available from the temporal image sequences that can aid in segmentation accuracy and consistency across the time-points. In particular, single time-point segmentation methods perform poorly in segmenting the end-systole (ES) phase image in the cardiac sequence, where the left ventricle deforms to the smallest irregular shape, and the boundary between the blood chamber and the myocardium becomes inconspicuous and ambiguous. To overcome these limitations in automatically segmenting temporal LVCs, we present a spatial sequential network (SS-Net) to learn the deformation and motion characteristics of the LVCs in an unsupervised manner; these characteristics are then integrated with sequential context information derived from bi-directional learning (BL) where both chronological and reverse-chronological directions of the image sequence are used. Our experimental results on a cardiac computed tomography (CT) dataset demonstrate that our spatial-sequential network with bi-directional learning (SS-BL-Net) outperforms existing methods for spatiotemporal LVC segmentation. [ABSTRACT FROM AUTHOR]