Search

Pathological primary tumor (pT) stage focuses on the infiltration degree of the primary tumor to surrounding tissues, which relates to the prognosis and treatment choices. The pT staging relies on the field-of-views from multiple magnifications in the gigapixel images, which makes pixel-level annotation difficult. Therefore, this task is usually formulated as a weakly supervised whole slide image (WSI) classification task with the slide-level label. Existing weakly-supervised classification methods mainly follow the multiple instance learning paradigm, which takes the patches from single magnification as the instances and extracts their morphological features independently. However, they cannot progressively represent the contextual information from multiple magnifications, which is critical for pT staging. Therefore, we propose a structure-aware hierarchical graph-based multi-instance learning framework (SGMF) inspired by the diagnostic process of pathologists. Specifically, a novel graph-based instance organization method is proposed, namely structure-aware hierarchical graph (SAHG), to represent the WSI. Based on that, we design a novel hierarchical attention-based graph representation (HAGR) network to capture the critical patterns for pT staging by learning cross-scale spatial features. Finally, the top nodes of SAHG are aggregated by a global attention layer for bag-level representation. Extensive studies on three large-scale multi-center pT staging datasets with two different cancer types demonstrate the effectiveness of SGMF, which outperforms state-of-the-art up to 5.6% in the F1 score.

Leukemia classification relies on a detailed cytomorphological examination of Bone Marrow (BM) smear. However, applying existing deep-learning methods to it is facing two significant limitations. Firstly, these methods require large-scale datasets with expert annotations at the cell level for good results and typically suffer from poor generalization. Secondly, they simply treat the BM cytomorphological examination as a multi-class cell classification task, thus failing to exploit the correlation among leukemia subtypes over different hierarchies. Therefore, BM cytomorphological estimation as a time-consuming and repetitive process still needs to be done manually by experienced cytologists. Recently, Multi-Instance Learning (MIL) has achieved much progress in data-efficient medical image processing, which only requires patient-level labels (which can be extracted from the clinical reports). In this paper, we propose a hierarchical MIL framework and equip it with Information Bottleneck (IB) to tackle the above limitations. First, to handle the patient-level label, our hierarchical MIL framework uses attention-based learning to identify cells with high diagnostic values for leukemia classification in different hierarchies. Then, following the information bottleneck principle, we propose a hierarchical IB to constrain and refine the representations of different hierarchies for better accuracy and generalization. By applying our framework to a large-scale childhood acute leukemia dataset with corresponding BM smear images and clinical reports, we show that it can identify diagnostic-related cells without the need for cell-level annotations and outperforms other comparison methods. Furthermore, the evaluation conducted on an independent test cohort demonstrates the high generalizability of our framework.

Objectives: To identify the feasibility of deep learning-based diagnostic models for detecting and assessing lower-extremity fatigue fracture severity on plain radiographs., Methods: This retrospective study enrolled 1151 X-ray images (tibiofibula/foot: 682/469) of fatigue fractures and 2842 X-ray images (tibiofibula/foot: 2000/842) without abnormal presentations from two clinical centers. After labeling the lesions, images in a center (tibiofibula/foot: 2539/1180) were allocated at 7:1:2 for model construction, and the remaining images from another center (tibiofibula/foot: 143/131) for external validation. A ResNet-50 and a triplet branch network were adopted to construct diagnostic models for detecting and grading. The performances of detection models were evaluated with sensitivity, specificity, and area under the receiver operating characteristic curve (AUC), while grading models were evaluated with accuracy by confusion matrix. Visual estimations by radiologists were performed for comparisons with models., Results: For the detection model on tibiofibula, a sensitivity of 95.4%/85.5%, a specificity of 80.1%/77.0%, and an AUC of 0.965/0.877 were achieved in the internal testing/external validation set. The detection model on foot reached a sensitivity of 96.4%/90.8%, a specificity of 76.0%/66.7%, and an AUC of 0.947/0.911. The detection models showed superior performance to the junior radiologist, comparable to the intermediate or senior radiologist. The overall accuracy of the diagnostic model was 78.5%/62.9% for tibiofibula and 74.7%/61.1% for foot in the internal testing/external validation set., Conclusions: The deep learning-based models could be applied to the radiological diagnosis of plain radiographs for assisting in the detection and grading of fatigue fractures on tibiofibula and foot., Key Points: • Fatigue fractures on radiographs are relatively difficult to detect, and apt to be misdiagnosed. • Detection and grading models based on deep learning were constructed on a large cohort of radiographs with lower-extremity fatigue fractures. • The detection model with high sensitivity would help to reduce the misdiagnosis of lower-extremity fatigue fractures., (© 2022. The Author(s), under exclusive licence to European Society of Radiology.)

Importance: Otitis media with effusion (OME) is one of the most common causes of acquired conductive hearing loss (CHL). Persistent hearing loss is associated with poor childhood speech and language development and other adverse consequence. However, to obtain accurate and reliable hearing thresholds largely requires a high degree of cooperation from the patients., Objective: To predict CHL from otoscopic images using deep learning (DL) techniques and a logistic regression model based on tympanic membrane features., Design, Setting, and Participants: A retrospective diagnostic/prognostic study was conducted using 2790 otoscopic images obtained from multiple centers between January 2015 and November 2020. Participants were aged between 4 and 89 years. Of 1239 participants, there were 209 ears from children and adolescents (aged 4-18 years [16.87%]), 804 ears from adults (aged 18-60 years [64.89%]), and 226 ears from older people (aged >60 years, [18.24%]). Overall, 679 ears (54.8%) were from men. The 2790 otoscopic images were randomly assigned into a training set (2232 [80%]), and validation set (558 [20%]). The DL model was developed to predict an average air-bone gap greater than 10 dB. A logistic regression model was also developed based on otoscopic features., Main Outcomes and Measures: The performance of the DL model in predicting CHL was measured using the area under the receiver operating curve (AUC), accuracy, and F1 score (a measure of the quality of a classifier, which is the harmonic mean of precision and recall; a higher F1 score means better performance). In addition, these evaluation parameters were compared to results obtained from the logistic regression model and predictions made by three otologists., Results: The performance of the DL model in predicting CHL showed the AUC of 0.74, accuracy of 81%, and F1 score of 0.89. This was better than the results from the logistic regression model (ie, AUC of 0.60, accuracy of 76%, and F1 score of 0.82), and much improved on the performance of the 3 otologists; accuracy of 16%, 30%, 39%, and F1 scores of 0.09, 0.18, and 0.25, respectively. Furthermore, the DL model took 2.5 seconds to predict from 205 otoscopic images, whereas the 3 otologists spent 633 seconds, 645 seconds, and 692 seconds, respectively., Conclusions and Relevance: The model in this diagnostic/prognostic study provided greater accuracy in prediction of CHL in ears with OME than those obtained from the logistic regression model and otologists. This indicates great potential for the use of artificial intelligence tools to facilitate CHL evaluation when CHL is unable to be measured.

Objectives: The molecular subtyping of diffuse gliomas is important. The aim of this study was to establish predictive models based on preoperative multiparametric MRI., Methods: A total of 1016 diffuse glioma patients were retrospectively collected from Beijing Tiantan Hospital. Patients were randomly divided into the training (n = 780) and validation (n = 236) sets. According to the 2016 WHO classification, diffuse gliomas can be classified into four binary classification tasks (tasks I-IV). Predictive models based on radiomics and deep convolutional neural network (DCNN) were developed respectively, and their performances were compared with receiver operating characteristic (ROC) curves. Additionally, the radiomics and DCNN features were visualized and compared with the t-distributed stochastic neighbor embedding technique and Spearman's correlation test., Results: In the training set, areas under the curves (AUCs) of the DCNN models (ranging from 0.99 to 1.00) outperformed the radiomics models in all tasks, and the accuracies of the DCNN models (ranging from 0.90 to 0.94) outperformed the radiomics models in tasks I, II, and III. In the independent validation set, the accuracies of the DCNN models outperformed the radiomics models in all tasks (0.74-0.83), and the AUCs of the DCNN models (0.85-0.89) outperformed the radiomics models in tasks I, II, and III. DCNN features demonstrated more superior discriminative capability than the radiomics features in feature visualization analysis, and their general correlations were weak., Conclusions: Both the radiomics and DCNN models could preoperatively predict the molecular subtypes of diffuse gliomas, and the latter performed better in most circumstances., Key Points: • The molecular subtypes of diffuse gliomas could be predicted with MRI. • Deep learning features tend to outperform radiomics features in large cohorts. • The correlation between the radiomics features and DCNN features was low., (© 2021. European Society of Radiology.)

Till March 31st, 2021, the coronavirus disease 2019 (COVID-19) had reportedly infected more than 127 million people and caused over 2.5 million deaths worldwide. Timely diagnosis of COVID-19 is crucial for management of individual patients as well as containment of the highly contagious disease. Having realized the clinical value of non-contrast chest computed tomography (CT) for diagnosis of COVID-19, deep learning (DL) based automated methods have been proposed to aid the radiologists in reading the huge quantities of CT exams as a result of the pandemic. In this work, we address an overlooked problem for training deep convolutional neural networks for COVID-19 classification using real-world multi-source data, namely, the data source bias problem. The data source bias problem refers to the situation in which certain sources of data comprise only a single class of data, and training with such source-biased data may make the DL models learn to distinguish data sources instead of COVID-19. To overcome this problem, we propose MIx-aNd-Interpolate (MINI), a conceptually simple, easy-to-implement, efficient yet effective training strategy. The proposed MINI approach generates volumes of the absent class by combining the samples collected from different hospitals, which enlarges the sample space of the original source-biased dataset. Experimental results on a large collection of real patient data (1,221 COVID-19 and 1,520 negative CT images, and the latter consisting of 786 community acquired pneumonia and 734 non-pneumonia) from eight hospitals and health institutions show that: 1) MINI can improve COVID-19 classification performance upon the baseline (which does not deal with the source bias), and 2) MINI is superior to competing methods in terms of the extent of improvement.

Purpose: Microvascular invasion (MVI) is a valuable predictor of survival in hepatocellular carcinoma (HCC) patients. This study developed predictive models using eXtreme Gradient Boosting (XGBoost) and deep learning based on CT images to predict MVI preoperatively., Methods: In total, 405 patients were included. A total of 7302 radiomic features and 17 radiological features were extracted by a radiomics feature extraction package and radiologists, respectively. We developed a XGBoost model based on radiomics features, radiological features and clinical variables and a three-dimensional convolutional neural network (3D-CNN) to predict MVI status. Next, we compared the efficacy of the two models., Results: Of the 405 patients, 220 (54.3%) were MVI positive, and 185 (45.7%) were MVI negative. The areas under the receiver operating characteristic curves (AUROCs) of the Radiomics-Radiological-Clinical (RRC) Model and 3D-CNN Model in the training set were 0.952 (95% confidence interval (CI) 0.923-0.973) and 0.980 (95% CI 0.959-0.993), respectively (p = 0.14). The AUROCs of the RRC Model and 3D-CNN Model in the validation set were 0.887 (95% CI 0.797-0.947) and 0.906 (95% CI 0.821-0.960), respectively (p = 0.83). Based on the MVI status predicted by the RRC and 3D-CNN Models, the mean recurrence-free survival (RFS) was significantly better in the predicted MVI-negative group than that in the predicted MVI-positive group (RRC Model: 69.95 vs. 24.80 months, p < 0.001; 3D-CNN Model: 64.06 vs. 31.05 months, p = 0.027)., Conclusion: The RRC Model and 3D-CNN models showed considerable efficacy in identifying MVI preoperatively. These machine learning models may facilitate decision-making in HCC treatment but requires further validation.

In the context of motor imagery, electroencephalography (EEG) data vary from subject to subject such that the performance of a classifier trained on data of multiple subjects from a specific domain typically degrades when applied to a different subject. While collecting enough samples from each subject would address this issue, it is often too time-consuming and impractical. To tackle this problem, we propose a novel end-to-end deep domain adaptation method to improve the classification performance on a single subject (target domain) by taking the useful information from multiple subjects (source domain) into consideration. Especially, the proposed method jointly optimizes three modules, including a feature extractor, a classifier, and a domain discriminator. The feature extractor learns the discriminative latent features by mapping the raw EEG signals into a deep representation space. A center loss is further employed to constrain an invariant feature space and reduce the intrasubject nonstationarity. Furthermore, the domain discriminator matches the feature distribution shift between source and target domains by an adversarial learning strategy. Finally, based on the consistent deep features from both domains, the classifier is able to leverage the information from the source domain and accurately predict the label in the target domain at the test time. To evaluate our method, we have conducted extensive experiments on two real public EEG data sets, data set IIa, and data set IIb of brain-computer interface (BCI) Competition IV. The experimental results validate the efficacy of our method. Therefore, our method is promising to reduce the calibration time for the use of BCI and promote the development of BCI.

Tissue segmentation is an essential task in computational pathology. However, relevant datasets for such a pixel-level classification task are hard to obtain due to the difficulty of annotation, bringing obstacles for training a deep learning-based segmentation model. Recently, contrastive learning has provided a feasible solution for mitigating the heavy reliance of deep learning models on annotation. Nevertheless, applying contrastive loss to the most abstract image representations, existing contrastive learning frameworks focus on global features, therefore, are less capable of encoding finer-grained features (e.g., pixel-level discrimination) for the tissue segmentation task. Enlightened by domain knowledge, we design three contrastive learning tasks with multi-granularity views (from global to local) for encoding necessary features into representations without accessing annotations. Specifically, we construct: (1) an image-level task to capture the difference between tissue components, i.e., encoding the component discrimination; (2) a superpixel-level task to learn discriminative representations of local regions with different tissue components, i.e., encoding the prototype discrimination; (3) a pixel-level task to encourage similar representations of different tissue components within a local region, i.e., encoding the spatial smoothness. Through our global-to-local pre-training strategy, the learned representations can reasonably capture the domain-specific and fine-grained patterns, making them easily transferable to various tissue segmentation tasks in histopathological images. We conduct extensive experiments on two tissue segmentation datasets, while considering two real-world scenarios with limited or sparse annotations. The experimental results demonstrate that our framework is superior to existing contrastive learning methods and can be easily combined with weakly supervised and semi-supervised segmentation methods. [ABSTRACT FROM AUTHOR]

Premature ventricular contractions (PVCs) are a common type of arrhythmia and can be life-threatening if deteriorated. There are several challenges remain to be resolved to the inter-patient paradigm of PVCs detection task by deep learning approaches. This research aims to overcome data imbalance, inaccurate ECG segments labeling and finding a generalized end-to-end model. In this paper, we present a deep learning network, PVCNet, to detect PVCs based on single-lead ECG. All ECG records are firstly split into fixed length segments based on the R-peak locations. To obtain more contextual information and improve the accuracy of segments labeling, we innovatively label the segment with sequence heartbeats label in the single heartbeat classification task, which was widely labeled with single heartbeat label in previous practices. Meanwhile, a loss function of redesigned weights is introduced to encourage the model to focus on detecting the specific heartbeat in the input segment. Additionally, a novel data augmentation method is conducted to alleviate the data imbalance problem. The proposed model is trained and validated on the MIT-BIH Arrhythmia Database (MITDB) with the inter-patient paradigm and tested on the St.Petersburg INCART Database (INCARTDB). It achieves a sensitivity of 89.91%, positive prediction value of 96.72%, F1-score of 93.19% and an overall accuracy of 98.50% on the testing database and outperforms existing approaches. The experimental results validate the effectiveness of the proposed approach and potentially can be applied to long-term ECG monitoring scenarios for the PVC detection task. [Display omitted] • We label the input ECG segment with a sequence of four heartbeats which decreases the possible label noise and provides more contextual information. • A loss function of redesigned weights is introduced to encourage the model to focus on detecting the specific heartbeat in the input segment. • Several comparisons and ablation studies results validate the effectiveness of the proposed PVC detection approach. [ABSTRACT FROM AUTHOR]

Due to the lack of properly annotated medical data, exploring the generalization capability of the deep model is becoming a public concern. Zero-shot learning (ZSL) has emerged in recent years to equip the deep model with the ability to recognize unseen classes. However, existing studies mainly focus on natural images, which utilize linguistic models to extract auxiliary information for ZSL. It is impractical to apply the natural image ZSL solutions directly to medical images, since the medical terminology is very domain-specific, and it is not easy to acquire linguistic models for the medical terminology. In this work, we propose a new paradigm of ZSL specifically for medical images utilizing cross-modality information. We make three main contributions with the proposed paradigm. First, we extract the prior knowledge about the segmentation targets, called relation prototypes, from the prior model and then propose a cross-modality adaptation module to inherit the prototypes to the zero-shot model. Second, we propose a relation prototype awareness module to make the zero-shot model aware of information contained in the prototypes. Last but not least, we develop an inheritance attention module to recalibrate the relation prototypes to enhance the inheritance process. The proposed framework is evaluated on two public cross-modality datasets including a cardiac dataset and an abdominal dataset. Extensive experiments show that the proposed framework significantly outperforms the state of the arts. [ABSTRACT FROM AUTHOR]

Medical images from multicentres often suffer from the domain shift problem, which makes the deep learning models trained on one domain usually fail to generalize well to another. One of the potential solutions for the problem is the generative adversarial network (GAN), which has the capacity to translate images between different domains. Nevertheless, the existing GAN-based approaches are prone to fail at preserving image-objects in image-to-image (I2I) translation, which reduces their practicality on domain adaptation tasks. In this regard, a novel GAN (namely IB-GAN) is proposed to preserve image-objects during cross-domain I2I adaptation. Specifically, we integrate the information bottleneck constraint into the typical cycle-consistency-based GAN to discard the superfluous information (e.g., domain information) and maintain the consistency of disentangled content features for image-object preservation. The proposed IB-GAN is evaluated on three tasks—polyp segmentation using colonoscopic images, the segmentation of optic disc and cup in fundus images and the whole heart segmentation using multi-modal volumes. We show that the proposed IB-GAN can generate realistic translated images and remarkably boost the generalization of widely used segmentation networks (e.g., U-Net). [ABSTRACT FROM AUTHOR]

As the labeled anomalous medical images are usually difficult to acquire, especially for rare diseases, the deep learning based methods, which heavily rely on the large amount of labeled data, cannot yield a satisfactory performance. Compared to the anomalous data, the normal images without the need of lesion annotation are much easier to collect. In this paper, we propose an anomaly detection framework, namely $\mathbb {SALAD}$ , extracting $\mathbb {S}$ elf-supervised and tr $\mathbb {A}$ ns $\mathbb {L}$ ation-consistent features for $\mathbb {A}$ nomaly $\mathbb {D}$ etection. The proposed SALAD is a reconstruction-based method, which learns the manifold of normal data through an encode-and-reconstruct translation between image and latent spaces. In particular, two constraints (i.e., structure similarity loss and center constraint loss) are proposed to regulate the cross-space (i.e., image and feature) translation, which enforce the model to learn translation-consistent and representative features from the normal data. Furthermore, a self-supervised learning module is engaged into our framework to further boost the anomaly detection accuracy by deeply exploiting useful information from the raw normal data. An anomaly score, as a measure to separate the anomalous data from the healthy ones, is constructed based on the learned self-supervised-and-translation-consistent features. Extensive experiments are conducted on optical coherence tomography (OCT) and chest X-ray datasets. The experimental results demonstrate the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

Fully convolutional neural networks have made promising progress in joint liver and liver tumor segmentation. Instead of following the debates over 2D versus 3D networks (for example, pursuing the balance between large-scale 2D pretraining and 3D context), in this paper, we novelly identify the wide variation in the ratio between intra- and inter-slice resolutions as a crucial obstacle to the performance. To tackle the mismatch between the intra- and inter-slice information, we propose a slice-aware 2.5D network that emphasizes extracting discriminative features utilizing not only in-plane semantics but also out-of-plane coherence for each separate slice. Specifically, we present a slice-wise multi-input multi-output architecture to instantiate such a design paradigm, which contains a Multi-Branch Decoder (MD) with a Slice-centric Attention Block (SAB) for learning slice-specific features and a Densely Connected Dice (DCD) loss to regularize the inter-slice predictions to be coherent and continuous. Based on the aforementioned innovations, we achieve state-of-the-art results on the MICCAI 2017 Liver Tumor Segmentation (LiTS) dataset. Besides, we also test our model on the ISBI 2019 Segmentation of THoracic Organs at Risk (SegTHOR) dataset, and the result proves the robustness and generalizability of the proposed method in other segmentation tasks. [ABSTRACT FROM AUTHOR]

Cervical cancer causes the fourth most cancer-related deaths of women worldwide. Early detection of cervical intraepithelial neoplasia (CIN) can significantly increase the survival rate of patients. In this paper, we propose a deep learning framework for the accurate identification of LSIL+ (including CIN and cervical cancer) using time-lapsed colposcopic images. The proposed framework involves two main components, i.e., key-frame feature encoding networks and feature fusion network. The features of the original (pre-acetic-acid) image and the colposcopic images captured at around 60s, 90s, 120s and 150s during the acetic acid test are encoded by the feature encoding networks. Several fusion approaches are compared, all of which outperform the existing automated cervical cancer diagnosis systems using a single time slot. A graph convolutional network with edge features (E-GCN) is found to be the most suitable fusion approach in our study, due to its excellent explainability consistent with the clinical practice. A large-scale dataset, containing time-lapsed colposcopic images from 7,668 patients, is collected from the collaborative hospital to train and validate our deep learning framework. Colposcopists are invited to compete with our computer-aided diagnosis system. The proposed deep learning framework achieves a classification accuracy of 78.33%—comparable to that of an in-service colposcopist—which demonstrates its potential to provide assistance in the realistic clinical scenario. [ABSTRACT FROM AUTHOR]

• A novel loss function for multi-organ segmentation. • Handling both input and output class imbalance. • Smoothing dice (or similar discrete) loss function(s). • Preventing potential gradient vanishing/exploding problem caused by Dice or similar loss functions. • First deep network for PET multi-organ segmentation. • Tested on various data-sets: MRI, PET, and CT. Simultaneous segmentation of multiple organs from different medical imaging modalities is a crucial task as it can be utilized for computer-aided diagnosis, computer-assisted surgery, and therapy planning. Thanks to the recent advances in deep learning, several deep neural networks for medical image segmentation have been introduced successfully for this purpose. In this paper, we focus on learning a deep multi-organ segmentation network that labels voxels. In particular, we examine the critical choice of a loss function in order to handle the notorious imbalance problem that plagues both the input and output of a learning model. The input imbalance refers to the class-imbalance in the input training samples (i.e., small foreground objects embedded in an abundance of background voxels, as well as organs of varying sizes). The output imbalance refers to the imbalance between the false positives and false negatives of the inference model. In order to tackle both types of imbalance during training and inference, we introduce a new curriculum learning based loss function. Specifically, we leverage Dice similarity coefficient to deter model parameters from being held at bad local minima and at the same time gradually learn better model parameters by penalizing for false positives/negatives using a cross entropy term. We evaluated the proposed loss function on three datasets: whole body positron emission tomography (PET) scans with 5 target organs, magnetic resonance imaging (MRI) prostate scans, and ultrasound echocardigraphy images with a single target organ i.e., left ventricular. We show that a simple network architecture with the proposed integrative loss function can outperform state-of-the-art methods and results of the competing methods can be improved when our proposed loss is used. [ABSTRACT FROM AUTHOR]

Robust and fast detection of anatomical structures is a prerequisite for both diagnostic and interventional medical image analysis. Current solutions for anatomy detection are typically based on machine learning techniques that exploit large annotated image databases in order to learn the appearance of the captured anatomy. These solutions are subject to several limitations, including the use of suboptimal feature engineering techniques and most importantly the use of computationally suboptimal search-schemes for anatomy detection. To address these issues, we propose a method that follows a new paradigm by reformulating the detection problem as a behavior learning task for an artificial agent. We couple the modeling of the anatomy appearance and the object search in a unified behavioral framework, using the capabilities of deep reinforcement learning and multi-scale image analysis. In other words, an artificial agent is trained not only to distinguish the target anatomical object from the rest of the body but also how to find the object by learning and following an optimal navigation path to the target object in the imaged volumetric space. We evaluated our approach on 1487 3D-CT volumes from 532 patients, totaling over 500,000 image slices and show that it significantly outperforms state-of-the-art solutions on detecting several anatomical structures with no failed cases from a clinical acceptance perspective, while also achieving a 20-30 percent higher detection accuracy. Most importantly, we improve the detection-speed of the reference methods by 2-3 orders of magnitude, achieving unmatched real-time performance on large 3D-CT scans. [ABSTRACT FROM AUTHOR]

Robust and fast solutions for anatomical object detection and segmentation support the entire clinical workflow from diagnosis, patient stratification, therapy planning, intervention and follow-up. Current state-of-the-art techniques for parsing volumetric medical image data are typically based on machine learning methods that exploit large annotated image databases. Two main challenges need to be addressed, these are the efficiency in scanning high-dimensional parametric spaces and the need for representative image features which require significant efforts of manual engineering. We propose a pipeline for object detection and segmentation in the context of volumetric image parsing, solving a two-step learning problem: anatomical pose estimation and boundary delineation. For this task we introduce Marginal Space Deep Learning (MSDL), a novel framework exploiting both the strengths of efficient object parametrization in hierarchical marginal spaces and the automated feature design of Deep Learning (DL) network architectures. In the 3D context, the application of deep learning systems is limited by the very high complexity of the parametrization. More specifically 9 parameters are necessary to describe a restricted affine transformation in 3D, resulting in a prohibitive amount of billions of scanning hypotheses. The mechanism of marginal space learning provides excellent run-time performance by learning classifiers in clustered, high-probability regions in spaces of gradually increasing dimensionality. To further increase computational efficiency and robustness, in our system we learn sparse adaptive data sampling patterns that automatically capture the structure of the input. Given the object localization, we propose a DL-based active shape model to estimate the non-rigid object boundary. Experimental results are presented on the aortic valve in ultrasound using an extensive dataset of 2891 volumes from 869 patients, showing significant improvements of up to 45.2% over the state-of-the-art. To our knowledge, this is the first successful demonstration of the DL potential to detection and segmentation in full 3D data with parametrized representations. [ABSTRACT FROM AUTHOR]