Search

This paper addresses digital staining and classification of the unstained white blood cell images obtained with a differential contrast microscope. We have data coming from multiple domains that are partially labeled and partially matching across the domains. Using unstained images removes time-consuming staining procedures and could facilitate and automatize comprehensive diagnostics. To this aim, we propose a method that translates unstained images to realistically looking stained images preserving the inter-cellular structures, crucial for the medical experts to perform classification. We achieve better structure preservation by adding auxiliary tasks of segmentation and direct reconstruction. Segmentation enforces that the network learns to generate correct nucleus and cytoplasm shape, while direct reconstruction enforces reliable translation between the matching images across domains. Besides, we build a robust domain agnostic latent space by injecting the target domain label directly to the generator, i.e., bypassing the encoder. It allows the encoder to extract features independently of the target domain and enables an automated domain invariant classification of the white blood cells. We validated our method on a large dataset composed of leukocytes of 24 patients, achieving state-of-the-art performance on both digital staining and classification tasks. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNN) have demonstrated their ability to segment 2D cardiac ultrasound images. However, despite recent successes according to which the intra-observer variability on end-diastole and end-systole images has been reached, CNNs still struggle to leverage temporal information to provide accurate and temporally consistent segmentation maps across the whole cycle. Such consistency is required to accurately describe the cardiac function, a necessary step in diagnosing many cardiovascular diseases. In this paper, we propose a framework to learn the 2D+time apical long-axis cardiac shape such that the segmented sequences can benefit from temporal and anatomical consistency constraints. Our method is a post-processing that takes as input segmented echocardiographic sequences produced by any state-of-the-art method and processes it in two steps to (i) identify spatio-temporal inconsistencies according to the overall dynamics of the cardiac sequence and (ii) correct the inconsistencies. The identification and correction of cardiac inconsistencies relies on a constrained autoencoder trained to learn a physiologically interpretable embedding of cardiac shapes, where we can both detect and fix anomalies. We tested our framework on 98 full-cycle sequences from the CAMUS dataset, which are available alongside this paper. Our temporal regularization method not only improves the accuracy of the segmentation across the whole sequences, but also enforces temporal and anatomical consistency. [ABSTRACT FROM AUTHOR]

Macrovascular invasion (MaVI) is a major threat to survival in hepatocellular carcinoma (HCC), which should be treated as early as possible to ensure safety and efficacy. In this aspect, MaVI prediction can be helpful. However, MaVI prediction is difficult because of the inter-class similarity and intra-class variation of HCC in computed tomography (CT) images. Moreover, existing methods fail to include clinical priori knowledge associated with HCC, leading to incomprehensive information extraction. In this paper, we proposed a prior knowledge-aware fusion network (PKAFnet) to accurately achieve MaVI prediction in CT images. First, a perception module was presented to extract features related to tumor marginal heterogeneity in the graph domain, which contributed to rotation invariance and captured intensity variations of tumor margin. Second, a tumor segmentation network was built to obtain global information of a 3D tumor image and information associated with tumor internal heterogeneity in the image domain. Finally, multi-domain features associated with the tumor margin and tumor region were combined by using a multi-domain attentional feature fusion module. Thus, by incorporating MaVI-related prior knowledge, our PKAFnet can alleviate overfitting, which can improve the discriminative ability. The proposed PKAFnet was validated on a multi-center dataset, and remarkable performance was achieved in an independent testing set. Moreover, the interpretability of perception module and segmentation network were presented in our paper, which illustrated the effectiveness and credibility of PKAFnet. Therefore, the proposed method showed great application potential for MaVI prediction. [ABSTRACT FROM AUTHOR]

This paper proposes a new method for joint design of radiofrequency (RF) and gradient waveforms in Magnetic Resonance Imaging (MRI), and applies it to the design of 3D spatially tailored saturation and inversion pulses. The joint design of both waveforms is characterized by the ODE Bloch equations, to which there is no known direct solution. Existing approaches therefore typically rely on simplified problem formulations based on, e.g., the small-tip approximation or constraining the gradient waveforms to particular shapes, and often apply only to specific objective functions for a narrow set of design goals (e.g., ignoring hardware constraints). This paper develops and exploits an auto-differentiable Bloch simulator to directly compute Jacobians of the (Bloch-simulated) excitation pattern with respect to RF and gradient waveforms. This approach is compatible with arbitrary sub-differentiable loss functions, and optimizes the RF and gradients directly without restricting the waveform shapes. For computational efficiency, we derive and implement explicit Bloch simulator Jacobians (approximately halving computation time and memory usage). To enforce hardware limits (peak RF, gradient, and slew rate), we use a change of variables that makes the 3D pulse design problem effectively unconstrained; we then optimize the resulting problem directly using the proposed auto-differentiation framework. We demonstrate our approach with two kinds of 3D excitation pulses that cannot be easily designed with conventional approaches: Outer-volume saturation (90° flip angle), and inner-volume inversion. [ABSTRACT FROM AUTHOR]

A flat-field correction method is proposed for multiple measured hyperspectral microscopy imaging in this paper. As the most crucial preprocessing process in quantitative microscopic analysis, flat-field correction solves the uneven illumination caused by vignetting in microscopic images, and guarantees the precision of spatial and spectral information in hyperspectral microscopic imaging. In order to carry out flat-field correction and extract uneven illumination among groups of hyperspectral microscopic data containing hundreds of bands simultaneously, two properties of vignetting have been exploited: i) low-rank property is reflected by little information contained in vignetting; ii) local smoothness can be observed as a gradual change in brightness of vignetting, which is typically equivalent to the sparseness in spatial frequency domain. Combining the two properties above, a novel Low Rank Smooth Flat-field Correction (LRSFC) model modified from common orthogonal basis extraction is proposed, while an optimization is solved based on alternating direction multiplier method (ADMM), obtaining a unique flat-field term with low-rank and smooth properties. Qualitative and quantitative experimental assessments indicate that LRSFC does not add extra cell texture to the extracted flat-field term, whose performance appears prior to other state-of-the-art flat-field correction methods. [ABSTRACT FROM AUTHOR]

This is the first article in a series of two dealing with a matrix approach for aberration quantification and correction in ultrasound imaging. Advanced synthetic beamforming relies on a double focusing operation at transmission and reception on each point of the medium. Ultrasound matrix imaging (UMI) consists in decoupling the location of these transmitted and received focal spots. The response between those virtual transducers form the so-called focused reflection matrix that actually contains much more information than a confocal ultrasound image. In this paper, a time-frequency analysis of this matrix is performed, which highlights the single and multiple scattering contributions as well as the impact of aberrations in the monochromatic and broadband regimes. Interestingly, this analysis enables the measurement of the incoherent input-output point spread function at any pixel of this image. A fitting process enables the quantification of the single scattering, multiple scattering and noise components in the image. From the single scattering contribution, a focusing criterion is defined, and its evolution used to quantify the amount of aberration throughout the ultrasound image. In contrast to the state-of-the-art coherence factor, this new indicator is robust to multiple scattering and electronic noise, thereby providing a contrasted map of the focusing quality at a much better transverse resolution. After a validation of the proof-of-concept based on time-domain simulations, UMI is applied to the in-vivo study of a human calf. Beyond this specific example, UMI opens a new route for speed-of-sound and scattering quantification in ultrasound imaging. [ABSTRACT FROM AUTHOR]

Automated detection of retinal structures, such as retinal vessels (RV), the foveal avascular zone (FAZ), and retinal vascular junctions (RVJ), are of great importance for understanding diseases of the eye and clinical decision-making. In this paper, we propose a novel Voting-based Adaptive Feature Fusion multi-task network (VAFF-Net) for joint segmentation, detection, and classification of RV, FAZ, and RVJ in optical coherence tomography angiography (OCTA). A task-specific voting gate module is proposed to adaptively extract and fuse different features for specific tasks at two levels: features at different spatial positions from a single encoder, and features from multiple encoders. In particular, since the complexity of the microvasculature in OCTA images makes simultaneous precise localization and classification of retinal vascular junctions into bifurcation/crossing a challenging task, we specifically design a task head by combining the heatmap regression and grid classification. We take advantage of three different en face angiograms from various retinal layers, rather than following existing methods that use only a single en face. We carry out extensive experiments on three OCTA datasets acquired using different imaging devices, and the results demonstrate that the proposed method performs on the whole better than either the state-of-the-art single-purpose methods or existing multi-task learning solutions. We also demonstrate that our multi-task learning method generalizes across other imaging modalities, such as color fundus photography, and may potentially be used as a general multi-task learning tool. We also construct three datasets for multiple structure detection, and part of these datasets with the source code and evaluation benchmark have been released for public access. [ABSTRACT FROM AUTHOR]

Annotating cancerous regions in whole-slide images (WSIs) of pathology samples plays a critical role in clinical diagnosis, biomedical research, and machine learning algorithms development. However, generating exhaustive and accurate annotations is labor-intensive, challenging, and costly. Drawing only coarse and approximate annotations is a much easier task, less costly, and it alleviates pathologists’ workload. In this paper, we study the problem of refining these approximate annotations in digital pathology to obtain more accurate ones. Some previous works have explored obtaining machine learning models from these inaccurate annotations, but few of them tackle the refinement problem where the mislabeled regions should be explicitly identified and corrected, and all of them require a – often very large – number of training samples. We present a method, named Label Cleaning Multiple Instance Learning (LC-MIL), to refine coarse annotations on a single WSI without the need for external training data. Patches cropped from a WSI with inaccurate labels are processed jointly within a multiple instance learning framework, mitigating their impact on the predictive model and refining the segmentation. Our experiments on a heterogeneous WSI set with breast cancer lymph node metastasis, liver cancer, and colorectal cancer samples show that LC-MIL significantly refines the coarse annotations, outperforming state-of-the-art alternatives, even while learning from a single slide. Moreover, we demonstrate how real annotations drawn by pathologists can be efficiently refined and improved by the proposed approach. All these results demonstrate that LC-MIL is a promising, lightweight tool to provide fine-grained annotations from coarsely annotated pathology sets. [ABSTRACT FROM AUTHOR]

Multi-modal imaging data fusion has attracted much attention in medical data analysis because it can provide complementary information for more accurate analysis. Integrating functional and structural multi-modal imaging data has been increasingly used in the diagnosis of brain diseases, such as epilepsy. Most of the existing methods focus on the feature space fusion of different modalities but ignore the valuable high-order relationships among samples and the discriminative fused features for classification. In this paper, we propose a novel framework by fusing data from two modalities of functional MRI (fMRI) and diffusion tensor imaging (DTI) for epilepsy diagnosis, which effectively captures the complementary information and discriminative features from different modalities by high-order feature extraction with the attention mechanism. Specifically, we propose a triple network to explore the discriminative information from the high-order representation feature space learned from multi-modal data. Meanwhile, self-attention is introduced to adaptively estimate the degree of importance between brain regions, and the cross-attention mechanism is utilized to extract complementary information from fMRI and DTI. Finally, we use the triple loss function to adjust the distance between samples in the common representation space. We evaluate the proposed method on the epilepsy dataset collected from Jinling Hospital, and the experiment results demonstrate that our method is significantly superior to several state-of-the-art diagnosis approaches. [ABSTRACT FROM AUTHOR]

Pathology analysis is crucial to precise cancer diagnoses and the succeeding treatment plan as well. To detect abnormality in histopathology images with prevailing patch-based convolutional neural networks (CNNs), contextual information often serves as a powerful cue. However, as whole-slide images (WSIs) are characterized by intense morphological heterogeneity and extensive tissue scale, a straightforward visual span to a larger context may not well capture the information closely associated with the focal patch. In this paper, we propose a novel pixel-offset based patch-location method to identify high-representative tissues, with a CNN backbone. Pathology Deformable Conditional Random Field (PDCRF) is proposed to learn the offsets and weights of neighboring contexts in a spatial-adaptive manner, to search for high-representative patches. A CNN structure with the localized patches as training input is then capable of consistently reaching superior classification outcomes for histology images. Overall, the proposed method has achieved state-of-the-art performance, in terms of the test classification accuracy improvement to the baseline by 1.15-2.60%, 0.78-1.78%, and 1.47-2.18% on TCGA public datasets of TCGA-STAD, TCGA-COAD, and TCGA-READ respectively. It also achieves 88.95% test accuracy and 0.920 test AUC on Camelyon 16. To show the effectiveness of the proposed framework on downstream tasks, we take a further step by incorporating an active learning model, which noticeably reduces the number of manual annotations by PDCRF to reach a parallel patch-based histology classifier. [ABSTRACT FROM AUTHOR]

Serial section transmission electron micro-scopy (ssTEM) reveals biological information at a scale of nanometer and plays an important role in the ultrastructural analysis. However, due to the imperfect preparation of biological samples, ssTEM images are usually degraded with various artifacts that greatly challenge the subsequent analysis and visualization. In this paper, we introduce a unified deep learning framework for ssTEM image restoration which addresses three main types of artifacts, i.e., Support Film Folds (SFF), Staining Precipitates (SP), and Missing Sections (MS). To achieve this goal, we first model the appearance of SFF and SP artifacts by conducting comprehensive analyses on the statistics of real degraded images, relying on which we can then simulate a large number of paired images (degraded/artifacts-free) for training a deep restoration network. Then, we design a coarse-to-fine restoration network consisting of three modules, i.e., interpolation, correction, and fusion. The interpolation module exploits the adjacent artifacts-free images for an initial restoration, while the correction module resorts to the degraded image itself to rectify the artifacts. Finally, the fusion module jointly utilizes the above two results to further improve the restoration fidelity. Experimental results on both synthetic and real test data validate the significantly improved performance of our proposed framework over existing solutions, in terms of both image restoration fidelity and neuron segmentation accuracy. To the best of our knowledge, this is the first unified deep learning framework for ssTEM image restoration from different types of artifacts. Code is available at https://github.com/sydeng99/ssTEM-restoration. [ABSTRACT FROM AUTHOR]

Tissue segmentation is the mainstay of pathological examination, whereas the manual delineation is unduly burdensome. To assist this time-consuming and subjective manual step, researchers have devised methods to automatically segment structures in pathological images. Recently, automated machine and deep learning based methods dominate tissue segmentation research studies. However, most machine and deep learning based approaches are supervised and developed using a large number of training samples, in which the pixel-wise annotations are expensive and sometimes can be impossible to obtain. This paper introduces a novel unsupervised learning paradigm by integrating an end-to-end deep mixture model with a constrained indicator to acquire accurate semantic tissue segmentation. This constraint aims to centralise the components of deep mixture models during the calculation of the optimisation function. In so doing, the redundant or empty class issues, which are common in current unsupervised learning methods, can be greatly reduced. By validation on both public and in-house datasets, the proposed deep constrained Gaussian network achieves significantly (Wilcoxon signed-rank test) better performance (with the average Dice scores of 0.737 and 0.735, respectively) on tissue segmentation with improved stability and robustness, compared to other existing unsupervised segmentation approaches. Furthermore, the proposed method presents a similar performance (p-value >0.05) compared to the fully supervised U-Net. [ABSTRACT FROM AUTHOR]

Deep learning-based approaches to delineating 3D structure depend on accurate annotations to train the networks. Yet in practice, people, no matter how conscientious, have trouble precisely delineating in 3D and on a large scale, in part because the data is often hard to interpret visually and in part because the 3D interfaces are awkward to use. In this paper, we introduce a method that explicitly accounts for annotation inaccuracies. To this end, we treat the annotations as active contour models that can deform themselves while preserving their topology. This enables us to jointly train the network and correct potential errors in the original annotations. The result is an approach that boosts performance of deep networks trained with potentially inaccurate annotations. [ABSTRACT FROM AUTHOR]

Clinically oriented deep learning algorithms, combined with large-scale medical datasets, have significantly promoted computer-aided diagnosis. To address increasing ethical and privacy issues, Federated Learning (FL) adopts a distributed paradigm to collaboratively train models, rather than collecting samples from multiple institutions for centralized training. Despite intensive research on FL, two major challenges are still existing when applying FL in the real-world medical scenarios, including the performance degradation (i.e., retrogress) after each communication and the intractable class imbalance. Thus, in this paper, we propose a novel personalized FL framework to tackle these two problems. For the retrogress problem, we first devise a Progressive Fourier Aggregation (PFA) at the server side to gradually integrate parameters of client models in the frequency domain. Then, at the client side, we design a Deputy-Enhanced Transfer (DET) to smoothly transfer global knowledge to the personalized local model. For the class imbalance problem, we propose the Conjoint Prototype-Aligned (CPA) loss to facilitate the balanced optimization of the FL framework. Considering the inaccessibility of private local data to other participants in FL, the CPA loss calculates the global conjoint objective based on global imbalance, and then adjusts the client-side local training through the prototype-aligned refinement to eliminate the imbalance gap with such a balanced goal. Extensive experiments are performed on real-world dermoscopic and prostate MRI FL datasets. The experimental results demonstrate the advantages of our FL framework in real-world medical scenarios, by outperforming state-of-the-art FL methods with a large margin. The source code is available at https://github.com/CityU-AIM-Group/PRR-Imbalance https://github.com/CityU-AIM-Group/PRR-Imbalance. [ABSTRACT FROM AUTHOR]

Domain adaptation techniques have been demonstrated to be effective in addressing label deficiency challenges in medical image segmentation. However, conventional domain adaptation based approaches often concentrate on matching global marginal distributions between different domains in a class-agnostic fashion. In this paper, we present a dual-attention domain-adaptative segmentation network (DADASeg-Net) for cross-modality medical image segmentation. The key contribution of DADASeg-Net is a novel dual adversarial attention mechanism, which regularizes the domain adaptation module with two attention maps respectively from the space and class perspectives. Specifically, the spatial attention map guides the domain adaptation module to focus on regions that are challenging to align in adaptation. The class attention map encourages the domain adaptation module to capture class-specific instead of class-agnostic knowledge for distribution alignment. DADASeg-Net shows superior performance in two challenging medical image segmentation tasks. [ABSTRACT FROM AUTHOR]

Complete tumor coverage by the thermal ablation zone and with a safety margin (5 or 10 mm) is required to achieve the entire tumor eradication in liver tumor ablation procedures. However, 2D ultrasound (US) imaging has limitations in evaluating the tumor coverage by imaging only one or multiple planes, particularly for cases with multiple inserted applicators or irregular tumor shapes. In this paper, we evaluate the intra-procedural tumor coverage using 3D US imaging and investigate whether it can provide clinically needed information. Using data from 14 cases, we employed surface- and volume-based evaluation metrics to provide information on any uncovered tumor region. For cases with incomplete tumor coverage or uneven ablation margin distribution, we also proposed a novel margin uniformity -based approach to provide quantitative applicator adjustment information for optimization of tumor coverage. Both the surface- and volume-based metrics showed that 5 of 14 cases had incomplete tumor coverage according to the estimated ablation zone. After applying our proposed applicator adjustment approach, the simulated results showed that 92.9% (13 of 14) cases achieved 100% tumor coverage and the remaining case can benefit by increasing the ablation time or power. Our proposed method can evaluate the intra-procedural tumor coverage and intuitively provide applicator adjustment information for the physician. Our 3D US-based method is compatible with the constraints of conventional US-guided ablation procedures and can be easily integrated into the clinical workflow. [ABSTRACT FROM AUTHOR]

Tensor fields are useful for modeling the structure of biological tissues. The challenge to measure tensor fields involves acquiring sufficient data of scalar measurements that are physically achievable and reconstructing tensors from as few projections as possible for efficient applications in medical imaging. In this paper, we present a filtered back-projection algorithm for the reconstruction of a symmetric second-rank tensor field from directional X-ray projections about three axes. The tensor field is decomposed into a solenoidal and irrotational component, each of three unknowns. Using the Fourier projection theorem, a filtered back-projection algorithm is derived to reconstruct the solenoidal and irrotational components from projections acquired around three axes. A simple illustrative phantom consisting of two spherical shells and a 3D digital cardiac diffusion image obtained from diffusion tensor MRI of an excised human heart are used to simulate directional X-ray projections. The simulations validate the mathematical derivations and demonstrate reasonable noise properties of the algorithm. The decomposition of the tensor field into solenoidal and irrotational components provides insight into the development of algorithms for reconstructing tensor fields with sufficient samples in terms of the type of directional projections and the necessary orbits for the acquisition of the projections of the tensor field. [ABSTRACT FROM AUTHOR]

Medical images contain various abnormal regions, most of which are closely related to the lesions or diseases. The abnormality or lesion is one of the major concerns during clinical practice and therefore becomes the key in answering questions about medical images. However, the recent efforts still focus on constructing a generic Visual Question Answering framework for medical-domain tasks, which is not adequate for practical medical requirements and applications. In this paper, we present two novel medical-specific modules named multiplication anomaly sensitive module and residual anomaly sensitive module to utilize weakly supervised anomaly localization information in medical Visual Question Answering. Firstly, the proposed multiplication anomaly sensitive module designed for anomaly-related questions can mask the feature of the whole image according to the anomaly location map. Secondly, the residual anomaly sensitive module could learn a flexible anomaly feature while preserving the information of the original questioned image, which is more helpful in answering anomaly-unrelated questions. Thirdly, the transformer decoder and multi-task learning strategy are combined to further enhance the question-reasoning ability and the model generalization performance. Finally, qualitative and quantitative experiments on a variety of medical datasets exhibit the superiority of the proposed approaches compared to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Medical image segmentation is fundamental and essential for the analysis of medical images. Although prevalent success has been achieved by convolutional neural networks (CNN), challenges are encountered in the domain of medical image analysis by two aspects: 1) lack of discriminative features to handle similar textures of distinct structures and 2) lack of selective features for potential blurred boundaries in medical images. In this paper, we extend the concept of contrastive learning (CL) to the segmentation task to learn more discriminative representation. Specifically, we propose a novel patch-dragsaw contrastive regularization (PDCR) to perform patch-level tugging and repulsing. In addition, a new structure, namely uncertainty-aware feature re- weighting block (UAFR), is designed to address the potential high uncertainty regions in the feature maps and serves as a better feature re- weighting. Our proposed method achieves state-of-the-art results across 8 public datasets from 6 domains. Besides, the method also demonstrates robustness in the limited-data scenario. The code is publicly available at https://github.com/lzh19961031/PDCR_UAFR-MIShttps://github.com/lzh19961031/PDCR_UAFR-MIS. [ABSTRACT FROM AUTHOR]

Accurately segmenting teeth and identifying the corresponding anatomical landmarks on dental mesh models are essential in computer-aided orthodontic treatment. Manually performing these two tasks is time-consuming, tedious, and, more importantly, highly dependent on orthodontists’ experiences due to the abnormality and large-scale variance of patients’ teeth. Some machine learning-based methods have been designed and applied in the orthodontic field to automatically segment dental meshes (e.g., intraoral scans). In contrast, the number of studies on tooth landmark localization is still limited. This paper proposes a two-stage framework based on mesh deep learning (called TS-MDL) for joint tooth labeling and landmark identification on raw intraoral scans. Our TS-MDL first adopts an end-to-end iMeshSegNet method (i.e., a variant of the existing MeshSegNet with both improved accuracy and efficiency) to label each tooth on the downsampled scan. Guided by the segmentation outputs, our TS-MDL further selects each tooth’s region of interest (ROI) on the original mesh to construct a light-weight variant of the pioneering PointNet (i.e., PointNet-Reg) for regressing the corresponding landmark heatmaps. Our TS-MDL was evaluated on a real-clinical dataset, showing promising segmentation and localization performance. Specifically, iMeshSegNet in the first stage of TS-MDL reached an averaged Dice similarity coefficient (DSC) at ${0.964}\pm {0.054}$ , significantly outperforming the original MeshSegNet. In the second stage, PointNet-Reg achieved a mean absolute error (MAE) of ${0}.{597}\pm {0}.{761} \, mm$ in distances between the prediction and ground truth for 66 landmarks, which is superior compared with other networks for landmark detection. All these results suggest the potential usage of our TS-MDL in orthodontics. [ABSTRACT FROM AUTHOR]

Medical visual question answering (VQA) aims to correctly answer a clinical question related to a given medical image. Nevertheless, owing to the expensive manual annotations of medical data, the lack of labeled data limits the development of medical VQA. In this paper, we propose a simple yet effective data augmentation method, VQAMix, to mitigate the data limitation problem. Specifically, VQAMix generates more labeled training samples by linearly combining a pair of VQA samples, which can be easily embedded into any visual-language model to boost performance. However, mixing two VQA samples would construct new connections between images and questions from different samples, which will cause the answers for those new fabricated image-question pairs to be missing or meaningless. To solve the missing answer problem, we first develop the Learning with Missing Labels (LML) strategy, which roughly excludes the missing answers. To alleviate the meaningless answer issue, we design the Learning with Conditional-mixed Labels (LCL) strategy, which further utilizes language-type prior to forcing the mixed pairs to have reasonable answers that belong to the same category. Experimental results on the VQA-RAD and PathVQA benchmarks show that our proposed method significantly improves the performance of the baseline by about 7% and 5% on the averaging result of two backbones, respectively. More importantly, VQAMix could improve confidence calibration and model interpretability, which is significant for medical VQA models in practical applications. All code and models are available at https://github.com/haifangong/VQAMix. [ABSTRACT FROM AUTHOR]

Knee osteoarthritis (KOA) as a disabling joint disease has doubled in prevalence since the mid-20th century. Early diagnosis for the longitudinal KOA grades has been increasingly important for effective monitoring and intervention. Although recent studies have achieved promising performance for baseline KOA grading, longitudinal KOA grading has been seldom studied and the KOA domain knowledge has not been well explored yet. In this paper, a novel deep learning architecture, namely adversarial evolving neural network (A-ENN), is proposed for longitudinal grading of KOA severity. As the disease progresses from mild to severe level, ENN involves the progression patterns for accurately characterizing the disease by comparing an input image it to the template images of different KL grades using convolution and deconvolution computations. In addition, an adversarial training scheme with a discriminator is developed to obtain the evolution traces. Thus, the evolution traces as fine-grained domain knowledge are further fused with the general convolutional image representations for longitudinal grading. Note that ENN can be applied to other learning tasks together with existing deep architectures, in which the responses characterize progressive representations. Comprehensive experiments on the Osteoarthritis Initiative (OAI) dataset were conducted to evaluate the proposed method. An overall accuracy was achieved as 62.7%, with the baseline, 12-month, 24-month, 36-month, and 48-month accuracy as 64.6%, 63.9%, 63.2%, 61.8% and 60.2%, respectively. [ABSTRACT FROM AUTHOR]

Reconstructing the 3D geometry of the surgical site and detecting instruments within it are important tasks for surgical navigation systems and robotic surgery automation. Traditional approaches treat each problem in isolation and do not account for the intrinsic relationship between segmentation and stereo matching. In this paper, we present a learning-based framework that jointly estimates disparity and binary tool segmentation masks. The core component of our architecture is a shared feature encoder which allows strong interaction between the aforementioned tasks. Experimentally, we train two variants of our network with different capacities and explore different training schemes including both multi-task and single-task learning. Our results show that supervising the segmentation task improves our network’s disparity estimation accuracy. We demonstrate a domain adaptation scheme where we supervise the segmentation task with monocular data and achieve domain adaptation of the adjacent disparity task, reducing disparity End-Point-Error and depth mean absolute error by 77.73% and 61.73% respectively compared to the pre-trained baseline model. Our best overall multi-task model, trained with both disparity and segmentation data in subsequent phases, achieves 89.15% mean Intersection-over-Union in RIS and 3.18 millimetre depth mean absolute error in SCARED test sets. Our proposed multi-task architecture is real-time, able to process ($1280\times 1024$) stereo input and simultaneously estimate disparity maps and segmentation masks at 22 frames per second. The model code and pre-trained models are made available: https://github.com/dimitrisPs/msdesis [ABSTRACT FROM AUTHOR]

Automatic surgical phase recognition plays a vital role in robot-assisted surgeries. Existing methods ignored a pivotal problem that surgical phases should be classified by learning segment-level semantics instead of solely relying on frame-wise information. This paper presents a segment-attentive hierarchical consistency network (SAHC) for surgical phase recognition from videos. The key idea is to extract hierarchical high-level semantic-consistent segments and use them to refine the erroneous predictions caused by ambiguous frames. To achieve it, we design a temporal hierarchical network to generate hierarchical high-level segments. Then, we introduce a hierarchical segment-frame attention module to capture relations between the low-level frames and high-level segments. By regularizing the predictions of frames and their corresponding segments via a consistency loss, the network can generate semantic-consistent segments and then rectify the misclassified predictions caused by ambiguous low-level frames. We validate SAHC on two public surgical video datasets, i.e., the M2CAI16 challenge dataset and the Cholec80 dataset. Experimental results show that our method outperforms previous state-of-the-arts and ablation studies prove the effectiveness of our proposed modules. Our code has been released at: https://github.com/xmed-lab/SAHC. [ABSTRACT FROM AUTHOR]

Representation learning is the critical task for medical image analysis in computer-aided diagnosis. However, it is challenging to learn discriminative features due to the limited size of the dataset and the lack of labels. In this paper, we propose a stochastic routing normalization and neighborhood embedding framework with application to breast tissue classification by learning discriminative features of probe-based confocal laser endomicroscopy. In order to align the low-level and mid-level of pCLE and histology domain, we firstly build the domain-specific normalization module with stochastic activation strategy considering both depth-wise and feature-wise criterion. For high-level features, the latent centers are learned from the histology domain as the template for feature matching. The proposed method is evaluated on a clinical database with 700 pCLE mosaics. The accuracy of image classification with limited training samples demonstrates that the proposed method can outperform previous works on domain alignment. [ABSTRACT FROM AUTHOR]

Surgical instrument segmentation can be used in a range of computer assisted interventions and automation in surgical robotics. While deep learning architectures have rapidly advanced the robustness and performance of segmentation models, most are still reliant on supervision and large quantities of labelled data. In this paper, we present a novel method for surgical image generation that can fuse robotic instrument simulation and recent domain adaptation techniques to synthesize artificial surgical images to train surgical instrument segmentation models. We integrate attention modules into well established image generation pipelines and propose a novel cost function to support supervision from simulation frames in model training. We provide an extensive evaluation of our method in terms of segmentation performance along with a validation study on image quality using evaluation metrics. Additionally, we release a novel segmentation dataset from real surgeries that will be shared for research purposes. Both binary and semantic segmentation have been considered, and we show the capability of our synthetic images to train segmentation models compared with the latest methods from the literature. [ABSTRACT FROM AUTHOR]

Effectively integrating multi-scale information is of considerable significance for the challenging multi-class segmentation of fundus lesions because different lesions vary significantly in scales and shapes. Several methods have been proposed to successfully handle the multi-scale object segmentation. However, two issues are not considered in previous studies. The first is the lack of interaction between adjacent feature levels, and this will lead to the deviation of high-level features from low-level features and the loss of detailed cues. The second is the conflict between the low-level and high-level features, this occurs because they learn different scales of features, thereby confusing the model and decreasing the accuracy of the final prediction. In this paper, we propose a progressive multi-scale consistent network (PMCNet) that integrates the proposed progressive feature fusion (PFF) block and dynamic attention block (DAB) to address the aforementioned issues. Specifically, PFF block progressively integrates multi-scale features from adjacent encoding layers, facilitating feature learning of each layer by aggregating fine-grained details and high-level semantics. As features at different scales should be consistent, DAB is designed to dynamically learn the attentive cues from the fused features at different scales, thus aiming to smooth the essential conflicts existing in multi-scale features. The two proposed PFF and DAB blocks can be integrated with the off-the-shelf backbone networks to address the two issues of multi-scale and feature inconsistency in the multi-class segmentation of fundus lesions, which will produce better feature representation in the feature space. Experimental results on three public datasets indicate that the proposed method is more effective than recent state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Automatic surgical scene segmentation is fundamental for facilitating cognitive intelligence in the modern operating theatre. Previous works rely on conventional aggregation modules (e.g., dilated convolution, convolutional LSTM), which only make use of the local context. In this paper, we propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance, by progressively capturing the global context. We firstly develop a hierarchy Transformer to capture intra-video relation that includes richer spatial and temporal cues from neighbor pixels and previous frames. A joint space-time window shift scheme is proposed to efficiently aggregate these two cues into each pixel embedding. Then, we explore inter-video relation via pixel-to-pixel contrastive learning, which well structures the global embedding space. A multi-source contrast training objective is developed to group the pixel embeddings across videos with the ground-truth guidance, which is crucial for learning the global property of the whole data. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches. Code is available at https://github.com/YuemingJin/STswinCL. [ABSTRACT FROM AUTHOR]

Image reconstruction for positron emission tomography (PET) is challenging because of the ill-conditioned tomographic problem and low counting statistics. Kernel methods address this challenge by using kernel representation to incorporate image prior information in the forward model of iterative PET image reconstruction. Existing kernel methods construct the kernels commonly using an empirical process, which may lead to unsatisfactory performance. In this paper, we describe the equivalence between the kernel representation and a trainable neural network model. A deep kernel method is then proposed by exploiting a deep neural network to enable automated learning of an improved kernel model and is directly applicable to single subjects in dynamic PET. The training process utilizes available image prior data to form a set of robust kernels in an optimized way rather than empirically. The results from computer simulations and a real patient dataset demonstrate that the proposed deep kernel method can outperform the existing kernel method and neural network method for dynamic PET image reconstruction. [ABSTRACT FROM AUTHOR]

Instance segmentation is of great importance for many biological applications, such as study of neural cell interactions, plant phenotyping, and quantitatively measuring how cells react to drug treatment. In this paper, we propose a novel box-based instance segmentation method. Box-based instance segmentation methods capture objects via bounding boxes and then perform individual segmentation within each bounding box region. However, existing methods can hardly differentiate the target from its neighboring objects within the same bounding box region due to their similar textures and low-contrast boundaries. To deal with this problem, in this paper, we propose an object-guided instance segmentation method. Our method first detects the center points of the objects, from which the bounding box parameters are then predicted. To perform segmentation, an object-guided coarse-to-fine segmentation branch is built along with the detection branch. The segmentation branch reuses the object features as guidance to separate target object from the neighboring ones within the same bounding box region. To further improve the segmentation quality, we design an auxiliary feature refinement module that densely samples and refines point-wise features in the boundary regions. Experimental results on three biological image datasets demonstrate the advantages of our method. The code will be available at https://github.com/yijingru/ObjGuided-Instance-Segmentation. [ABSTRACT FROM AUTHOR]

The aim of this paper is to provide a comprehensive overview of the MICCAI 2020 AutoImplant Challenge. The approaches and publications submitted and accepted within the challenge will be summarized and reported, highlighting common algorithmic trends and algorithmic diversity. Furthermore, the evaluation results will be presented, compared and discussed in regard to the challenge aim: seeking for low cost, fast and fully automated solutions for cranial implant design. Based on feedback from collaborating neurosurgeons, this paper concludes by stating open issues and post-challenge requirements for intra-operative use. The codes can be found at https://github.com/Jianningli/tmi. [ABSTRACT FROM AUTHOR]

Deep learning models are sensitive to domain shift phenomena. A model trained on images from one domain cannot generalise well when tested on images from a different domain, despite capturing similar anatomical structures. It is mainly because the data distribution between the two domains is different. Moreover, creating annotation for every new modality is a tedious and time-consuming task, which also suffers from high inter- and intra- observer variability. Unsupervised domain adaptation (UDA) methods intend to reduce the gap between source and target domains by leveraging source domain labelled data to generate labels for the target domain. However, current state-of-the-art (SOTA) UDA methods demonstrate degraded performance when there is insufficient data in source and target domains. In this paper, we present a novel UDA method for multi-modal cardiac image segmentation. The proposed method is based on adversarial learning and adapts network features between source and target domain in different spaces. The paper introduces an end-to-end framework that integrates: a) entropy minimization, b) output feature space alignment and c) a novel point-cloud shape adaptation based on the latent features learned by the segmentation model. We validated our method on two cardiac datasets by adapting from the annotated source domain, bSSFP-MRI (balanced Steady-State Free Procession-MRI), to the unannotated target domain, LGE-MRI (Late-gadolinium enhance-MRI), for the multi-sequence dataset; and from MRI (source) to CT (target) for the cross-modality dataset. The results highlighted that by enforcing adversarial learning in different parts of the network, the proposed method delivered promising performance, compared to other SOTA methods. [ABSTRACT FROM AUTHOR]

Fetal development relies on a complex circulatory network. Accurate assessment of flow distribution is important for understanding pathologies and potential therapies. In this paper, we demonstrate a method for volumetric imaging of fetal flow with magnetic resonance imaging (MRI). Fetal MRI faces challenges: small vascular structures, unpredictable motion, and inadequate traditional cardiac gating methods. Here, orthogonal multislice stacks are acquired with accelerated multidimensional radial phase contrast (PC) MRI. Slices are reconstructed into flow sensitive time-series images with motion correction and image-based cardiac gating. They are then combined into a dynamic volume using slice-to-volume reconstruction (SVR) while resolving interslice spatiotemporal coregistration. Compared to prior methods, this approach achieves higher spatiotemporal resolution ($1\times 1\times 1$ mm3, ~30 ms) with reduced scan time – important features for the quantification of flow through small fetal structures. Validation is demonstrated in adults by comparing SVR with 4D radial PCMRI (flow bias and limits of agreement: −1.1 ml/s and [−11.8 9.6] ml/s). Feasibility is demonstrated in late gestation fetuses by comparing SVR with 2D Cartesian PCMRI (flow bias and limits of agreement: −0.9 ml/min/kg and [−39.7 37.8] ml/min/kg). With SVR, we demonstrate complex flow pathways (such as parallel flow streams in the proximal inferior vena cava, preferential shunting of blood from the ductus venosus into the left atrium, and blood from the brain leaving the heart through the main pulmonary artery) for the first time in human fetal circulation. This method allows for comprehensive evaluation of the fetal circulation and enables future studies of fetal physiology. [ABSTRACT FROM AUTHOR]

Positron emission tomography is widely used in clinical and preclinical applications. Positronium lifetime carries information about the tissue microenvironment where positrons are emitted, but such information has not been captured because of two technical challenges. One challenge is the low sensitivity in detecting triple coincidence events. This problem has been mitigated by the recent developments of PET scanners with long (1-2 m) axial field of view. The other challenge is the low spatial resolution of the positronium lifetime images formed by existing methods that is determined by the time-of-flight (TOF) resolution (200-500 ps) of existing PET scanners. This paper solves the second challenge by developing a new image reconstruction method to generate high-resolution positronium lifetime images using existing TOF PET. Simulation studies demonstrate that the proposed method can reconstruct positronium lifetime images at much better spatial resolution than the limit set by the TOF resolution of the PET scanner. The proposed method opens up the possibility of performing positronium lifetime imaging using existing TOF PET scanners. The lifetime information can be used to understand the tissue microenvironment in vivo which could facilitate the study of disease mechanism and selection of proper treatments. [ABSTRACT FROM AUTHOR]

Age-related macular degeneration (AMD) is the leading cause of visual impairment among elderly in the world. Early detection of AMD is of great importance, as the vision loss caused by this disease is irreversible and permanent. Color fundus photography is the most cost-effective imaging modality to screen for retinal disorders. Cutting edge deep learning based algorithms have been recently developed for automatically detecting AMD from fundus images. However, there are still lack of a comprehensive annotated dataset and standard evaluation benchmarks. To deal with this issue, we set up the Automatic Detection challenge on Age-related Macular degeneration (ADAM), which was held as a satellite event of the ISBI 2020 conference. The ADAM challenge consisted of four tasks which cover the main aspects of detecting and characterizing AMD from fundus images, including detection of AMD, detection and segmentation of optic disc, localization of fovea, and detection and segmentation of lesions. As part of the ADAM challenge, we have released a comprehensive dataset of 1200 fundus images with AMD diagnostic labels, pixel-wise segmentation masks for both optic disc and AMD-related lesions (drusen, exudates, hemorrhages and scars, among others), as well as the coordinates corresponding to the location of the macular fovea. A uniform evaluation framework has been built to make a fair comparison of different models using this dataset. During the ADAM challenge, 610 results were submitted for online evaluation, with 11 teams finally participating in the onsite challenge. This paper introduces the challenge, the dataset and the evaluation methods, as well as summarizes the participating methods and analyzes their results for each task. In particular, we observed that the ensembling strategy and the incorporation of clinical domain knowledge were the key to improve the performance of the deep learning models. [ABSTRACT FROM AUTHOR]

Limited angle reconstruction is a typical ill-posed problem in computed tomography (CT). Given incomplete projection data, images reconstructed by conventional analytical algorithms and iterative methods suffer from severe structural distortions and artifacts. In this paper, we proposed a self-augmented multi-stage deep-learning network (Sam’s Net) for end-to-end reconstruction of limited angle CT. With the merit of the alternating minimization technique, Sam’s Net integrates multi-stage self-constraints into cross-domain optimization to provide additional constraints on the manifold of neural networks. In practice, a sinogram completion network (SCNet) and artifact suppression network (ASNet), together with domain transformation layers constitute the backbone for cross-domain optimization. An online self-augmentation module was designed following the manner defined by alternating minimization, which enables a self-augmented learning procedure and multi-stage inference manner. Besides, a substitution operation was applied as a hard constraint for the solution space based on the data fidelity and a learnable weighting layer was constructed for data consistency refinement. Sam’s Net forms a new framework for ill-posed reconstruction problems. In the training phase, the self-augmented procedure guides the optimization into a tightened solution space with enriched diverse data distribution and enhanced data consistency. In the inference phase, multi-stage prediction can improve performance progressively. Extensive experiments with both simulated and practical projections under 90-degree and 120-degree fan-beam configurations validate that Sam’s Net can significantly improve the reconstruction quality with high stability and robustness. [ABSTRACT FROM AUTHOR]

Cephalometric analysis relies on accurate detection of craniomaxillofacial (CMF) landmarks from cone-beam computed tomography (CBCT) images. However, due to the complexity of CMF bony structures, it is difficult to localize landmarks efficiently and accurately. In this paper, we propose a deep learning framework to tackle this challenge by jointly digitalizing 105 CMF landmarks on CBCT images. By explicitly learning the local geometrical relationships between the landmarks, our approach extends Mask R-CNN for end-to-end prediction of landmark locations. Specifically, we first apply a detection network on a down-sampled 3D image to leverage global contextual information to predict the approximate locations of the landmarks. We subsequently leverage local information provided by higher-resolution image patches to refine the landmark locations. On patients with varying non-syndromic jaw deformities, our method achieves an average detection accuracy of 1.38± 0.95mm, outperforming a related state-of-the-art method. [ABSTRACT FROM AUTHOR]

Automated radiographic report generation is challenging in at least two aspects. First, medical images are very similar to each other and the visual differences of clinic importance are often fine-grained. Second, the disease-related words may be submerged by many similar sentences describing the common content of the images, causing the abnormal to be misinterpreted as the normal in the worst case. To tackle these challenges, this paper proposes a pure transformer-based framework to jointly enforce better visual-textual alignment, multi-label diagnostic classification, and word importance weighting, to facilitate report generation. To the best of our knowledge, this is the first pure transformer-based framework for medical report generation, which enjoys the capacity of transformer in learning long range dependencies for both image regions and sentence words. Specifically, for the first challenge, we design a novel mechanism to embed an auxiliary image-text matching objective into the transformer’s encoder-decoder structure, so that better correlated image and text features could be learned to help a report to discriminate similar images. For the second challenge, we integrate an additional multi-label classification task into our framework to guide the model in making correct diagnostic predictions. Also, a term-weighting scheme is proposed to reflect the importance of words for training so that our model would not miss key discriminative information. Our work achieves promising performance over the state-of-the-arts on two benchmark datasets, including the largest dataset MIMIC-CXR. [ABSTRACT FROM AUTHOR]

The gold standard for diagnosing lymph node metastasis of papillary thyroid carcinoma is to analyze the whole slide histopathological images (WSIs). Due to the large size of WSIs, recent computer-aided diagnosis approaches adopt the multi-instance learning (MIL) strategy and the key part is how to effectively aggregate the information of different instances (patches). In this paper, a novel transformer-guided framework is proposed to predict lymph node metastasis from WSIs, where we incorporate the transformer mechanism to improve the accuracy from three different aspects. First, we propose an effective transformer-based module for discriminative patch feature extraction, including a lightweight feature extractor with a pruned transformer (Tiny-ViT) and a clustering-based instance selection scheme. Next, we propose a new Transformer-MIL module to capture the relationship of different discriminative patches with sparse distribution on WSIs and better nonlinearly aggregate patch-level features into the slide-level prediction. Considering that the slide-level annotation is relatively limited to training a robust Transformer-MIL, we utilize the pathological relationship between the primary tumor and its lymph node metastasis and develop an effective attention-based mutual knowledge distillation (AMKD) paradigm. Experimental results on our collected WSI dataset demonstrate the efficiency of the proposed Transformer-MIL and attention-based knowledge distillation. Our method outperforms the state-of-the-art methods by over 2.72% in AUC (area under the curve). [ABSTRACT FROM AUTHOR]

Registration of brain MRI images requires to solve a deformation field, which is extremely difficult in aligning intricate brain tissues, e.g., subcortical nuclei, etc. Existing efforts resort to decomposing the target deformation field into intermediate sub-fields with either tiny motions, i.e., progressive registration stage by stage, or lower resolutions, i.e., coarse-to-fine estimation of the full-size deformation field. In this paper, we argue that those efforts are not mutually exclusive, and propose a unified framework for robust brain MRI registration in both progressive and coarse-to-fine manners simultaneously. Specifically, building on a dual-encoder U-Net, the fixed-moving MRI pair is encoded and decoded into multi-scale sub-fields from coarse to fine. Each decoding block contains two proposed novel modules: i) in Deformation Field Integration (DFI), a single integrated deformation sub-field is calculated, warping by which is equivalent to warping progressively by sub-fields from all previous decoding blocks, and ii) in Non-rigid Feature Fusion (NFF), features of the fixed-moving pair are aligned by DFI-integrated deformation field, and then fused to predict a finer sub-field. Leveraging both DFI and NFF, the target deformation field is factorized into multi-scale sub-fields, where the coarser fields alleviate the estimate of a finer one and the finer field learns to make up those misalignments insolvable by previous coarser ones. The extensive and comprehensive experimental results on both private and two public datasets demonstrate a superior registration performance of brain MRI images over progressive registration only and coarse-to-fine estimation only, with an increase by at most 8% in the average Dice. [ABSTRACT FROM AUTHOR]

Constructing and analyzing functional brain networks (FBN) has become a promising approach to brain disorder classification. However, the conventional successive construct-and-analyze process would limit the performance due to the lack of interactions and adaptivity among the subtasks in the process. Recently, Transformer has demonstrated remarkable performance in various tasks, attributing to its effective attention mechanism in modeling complex feature relationships. In this paper, for the first time, we develop Transformer for integrated FBN modeling, analysis and brain disorder classification with rs-fMRI data by proposing a Diffusion Kernel Attention Network to address the specific challenges. Specifically, directly applying Transformer does not necessarily admit optimal performance in this task due to its extensive parameters in the attention module against the limited training samples usually available. Looking into this issue, we propose to use kernel attention to replace the original dot-product attention module in Transformer. This significantly reduces the number of parameters to train and thus alleviates the issue of small sample while introducing a non-linear attention mechanism to model complex functional connections. Another limit of Transformer for FBN applications is that it only considers pair-wise interactions between directly connected brain regions but ignores the important indirect connections. Therefore, we further explore diffusion process over the kernel attention to incorporate wider interactions among indirectly connected brain regions. Extensive experimental study is conducted on ADHD-200 data set for ADHD classification and on ADNI data set for Alzheimer’s disease classification, and the results demonstrate the superior performance of the proposed method over the competing methods. [ABSTRACT FROM AUTHOR]

We present a spherical harmonics-based convolutional neural network (CNN) for cortical parcellation, which we call SPHARM-Net. Recent advances in CNNs offer cortical parcellation on a fine-grained triangle mesh of the cortex. Yet, most CNNs designed for cortical parcellation employ spatial convolution that depends on extensive data augmentation and allows only predefined neighborhoods of specific spherical tessellation. On the other hand, a rotation-equivariant convolutional filter avoids data augmentation, and rotational equivariance can be achieved in spectral convolution independent of a neighborhood definition. Nevertheless, the limited resources of a modern machine enable only a finite set of spectral components that might lose geometric details. In this paper, we propose (1) a constrained spherical convolutional filter that supports an infinite set of spectral components and (2) an end-to-end framework without data augmentation. The proposed filter encodes all the spectral components without the full expansion of spherical harmonics. We show that rotational equivariance drastically reduces the training time while achieving accurate cortical parcellation. Furthermore, the proposed convolution is fully composed of matrix transformations, which offers efficient and fast spectral processing. In the experiments, we validate SPHARM-Net on two public datasets with manual labels: Mindboggle-101 (N=101) and NAMIC (N=39). The experimental results show that the proposed method outperforms the state-of-the-art methods on both datasets even with fewer learnable parameters without rigid alignment and data augmentation. Our code is publicly available at https://github.com/Shape-Lab/SPHARM-Net. [ABSTRACT FROM AUTHOR]

Bioluminescence tomography (BLT) is a promising pre-clinical imaging technique for a wide variety of biomedical applications, which can non-invasively reveal functional activities inside living animal bodies through the detection of visible or near-infrared light produced by bioluminescent reactions. Recently, reconstruction approaches based on deep learning have shown great potential in optical tomography modalities. However, these reports only generate data with stationary patterns of constant target number, shape, and size. The neural networks trained by these data sets are difficult to reconstruct the patterns outside the data sets. This will tremendously restrict the applications of deep learning in optical tomography reconstruction. To address this problem, a self-training strategy is proposed for BLT reconstruction in this paper. The proposed strategy can fast generate large-scale BLT data sets with random target numbers, shapes, and sizes through an algorithm named random seed growth algorithm and the neural network is automatically self-trained. In addition, the proposed strategy uses the neural network to build a map between photon densities on surface and inside the imaged object rather than an end-to-end neural network that directly infers the distribution of sources from the photon density on surface. The map of photon density is further converted into the distribution of sources through the multiplication with stiffness matrix. Simulation, phantom, and mouse studies are carried out. Results show the availability of the proposed self-training strategy. [ABSTRACT FROM AUTHOR]

Automatic liver tumor segmentation could offer assistance to radiologists in liver tumor diagnosis, and its performance has been significantly improved by recent deep learning based methods. These methods rely on large-scale well-annotated training datasets, but collecting such datasets is time-consuming and labor-intensive, which could hinder their performance in practical situations. Learning from synthetic data is an encouraging solution to address this problem. In our task, synthetic tumors can be injected to healthy images to form training pairs. However, directly applying the model trained using the synthetic tumor images on real test images performs poorly due to the domain shift problem. In this paper, we propose a novel approach, namely Synthetic-to-Real Test-Time Training (SR-TTT), to reduce the domain gap between synthetic training images and real test images. Specifically, we add a self-supervised auxiliary task, i.e., two-step reconstruction, which takes the output of the main segmentation task as its input to build an explicit connection between these two tasks. Moreover, we design a scheduled mixture strategy to avoid error accumulation and bias explosion in the training process. During test time, we adapt the segmentation model to each test image with self-supervision from the auxiliary task so as to improve the inference performance. The proposed method is extensively evaluated on two public datasets for liver tumor segmentation. The experimental results demonstrate that our proposed SR-TTT can effectively mitigate the synthetic-to-real domain shift problem in the liver tumor segmentation task, and is superior to existing state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

In clinical practice, multi-modal magnetic resonance imaging (MRI) with different contrasts is usually acquired in a single study to assess different properties of the same region of interest in the human body. The whole acquisition process can be accelerated by having one or more modalities under-sampled in the ${k}$ -space. Recent research has shown that, considering the redundancy between different modalities, a target MRI modality under-sampled in the ${k}$ -space can be more efficiently reconstructed with a fully-sampled reference MRI modality. However, we find that the performance of the aforementioned multi-modal reconstruction can be negatively affected by subtle spatial misalignment between different modalities, which is actually common in clinical practice. In this paper, we improve the quality of multi-modal reconstruction by compensating for such spatial misalignment with a spatial alignment network. First, our spatial alignment network estimates the displacement between the fully-sampled reference and the under-sampled target images, and warps the reference image accordingly. Then, the aligned fully-sampled reference image joins the multi-modal reconstruction of the under-sampled target image. Also, considering the contrast difference between the target and reference images, we have designed a cross-modality-synthesis-based registration loss in combination with the reconstruction loss, to jointly train the spatial alignment network and the reconstruction network. The experiments on both clinical MRI and multi-coil ${k}$ -space raw data demonstrate the superiority and robustness of the multi-modal MRI reconstruction empowered with our spatial alignment network. Our code is publicly available at https://github.com/woxuankai/SpatialAlignmentNetwork. [ABSTRACT FROM AUTHOR]

Learning how to capture long-range dependencies and restore spatial information of down-sampled feature maps are the basis of the encoder-decoder structure networks in medical image segmentation. U-Net based methods use feature fusion to alleviate these two problems, but the global feature extraction ability and spatial information recovery ability of U-Net are still insufficient. In this paper, we propose a Global Feature Reconstruction (GFR) module to efficiently capture global context features and a Local Feature Reconstruction (LFR) module to dynamically up-sample features, respectively. For the GFR module, we first extract the global features with category representation from the feature map, then use the different level global features to reconstruct features at each location. The GFR module establishes a connection for each pair of feature elements in the entire space from a global perspective and transfers semantic information from the deep layers to the shallow layers. For the LFR module, we use low-level feature maps to guide the up-sampling process of high-level feature maps. Specifically, we use local neighborhoods to reconstruct features to achieve the transfer of spatial information. Based on the encoder-decoder architecture, we propose a Global and Local Feature Reconstruction Network (GLFRNet), in which the GFR modules are applied as skip connections and the LFR modules constitute the decoder path. The proposed GLFRNet is applied to four different medical image segmentation tasks and achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Many machine learning tasks in neuroimaging aim at modeling complex relationships between a brain’s morphology as seen in structural MR images and clinical scores and variables of interest. A frequently modeled process is healthy brain aging for which many image-based brain age estimation or age-conditioned brain morphology template generation approaches exist. While age estimation is a regression task, template generation is related to generative modeling. Both tasks can be seen as inverse directions of the same relationship between brain morphology and age. However, this view is rarely exploited and most existing approaches train separate models for each direction. In this paper, we propose a novel bidirectional approach that unifies score regression and generative morphology modeling and we use it to build a bidirectional brain aging model. We achieve this by defining an invertible normalizing flow architecture that learns a probability distribution of 3D brain morphology conditioned on age. The use of full 3D brain data is achieved by deriving a manifold-constrained formulation that models morphology variations within a low-dimensional subspace of diffeomorphic transformations. This modeling idea is evaluated on a database of MR scans of more than 5000 subjects. The evaluation results show that our bidirectional brain aging model (1) accurately estimates brain age, (2) is able to visually explain its decisions through attribution maps and counterfactuals, (3) generates realistic age-specific brain morphology templates, (4) supports the analysis of morphological variations, and (5) can be utilized for subject-specific brain aging simulation. [ABSTRACT FROM AUTHOR]