Search

Objectives: Most countries have adopted public activity intervention policies to control the coronavirus disease 2019 (COVID-19) pandemic. Nevertheless, empirical evidence of the effectiveness of different interventions on the containment of the epidemic was inconsistent.Methods: We retrieved time-series intervention policy data for 145 countries from the Oxford COVID-19 Government Response Tracker from December 31, 2019, to July 1, 2020, which included 8 containment and closure policies. We investigated the association of timeliness, stringency, and duration of intervention with cumulative infections per million population on July 1, 2020. We introduced a novel counterfactual estimator to estimate the effects of these interventions on COVID-19 time-varying reproduction number (Rt).Results: There is some evidence that earlier implementation, longer durations, and more strictness of intervention policies at the early but not middle stage were associated with reduced infections of COVID-19. The counterfactual model proved to have controlled for unobserved time-varying confounders and established a valid causal relationship between policy intervention and Rt reduction. The average intervention effect revealed that all interventions significantly decrease Rt after their implementation. Rt decreased by 30% (22%-41%) in 25 to 32 days after policy intervention. Among the 8 interventions, school closing, workplace closing, and public events cancellation demonstrated the strongest and most consistent evidence of associations.Conclusions: Our study provides more reliable evidence of the quantitative effects of policy interventions on the COVID-19 epidemic and suggested that stricter public activity interventions should be implemented at the early stage of the epidemic for improved containment. [ABSTRACT FROM AUTHOR]

• A novel loss function for multi-organ segmentation. • Handling both input and output class imbalance. • Smoothing dice (or similar discrete) loss function(s). • Preventing potential gradient vanishing/exploding problem caused by Dice or similar loss functions. • First deep network for PET multi-organ segmentation. • Tested on various data-sets: MRI, PET, and CT. Simultaneous segmentation of multiple organs from different medical imaging modalities is a crucial task as it can be utilized for computer-aided diagnosis, computer-assisted surgery, and therapy planning. Thanks to the recent advances in deep learning, several deep neural networks for medical image segmentation have been introduced successfully for this purpose. In this paper, we focus on learning a deep multi-organ segmentation network that labels voxels. In particular, we examine the critical choice of a loss function in order to handle the notorious imbalance problem that plagues both the input and output of a learning model. The input imbalance refers to the class-imbalance in the input training samples (i.e., small foreground objects embedded in an abundance of background voxels, as well as organs of varying sizes). The output imbalance refers to the imbalance between the false positives and false negatives of the inference model. In order to tackle both types of imbalance during training and inference, we introduce a new curriculum learning based loss function. Specifically, we leverage Dice similarity coefficient to deter model parameters from being held at bad local minima and at the same time gradually learn better model parameters by penalizing for false positives/negatives using a cross entropy term. We evaluated the proposed loss function on three datasets: whole body positron emission tomography (PET) scans with 5 target organs, magnetic resonance imaging (MRI) prostate scans, and ultrasound echocardigraphy images with a single target organ i.e., left ventricular. We show that a simple network architecture with the proposed integrative loss function can outperform state-of-the-art methods and results of the competing methods can be improved when our proposed loss is used. [ABSTRACT FROM AUTHOR]

Self-supervised learning has emerged as a powerful tool for pretraining deep networks on unlabeled data, prior to transfer learning of target tasks with limited annotation. The relevance between the pretraining pretext and target tasks is crucial to the success of transfer learning. Various pretext tasks have been proposed to utilize properties of medical image data (e.g., three dimensionality), which are more relevant to medical image analysis than generic ones for natural images. However, previous work rarely paid attention to data with anatomy-oriented imaging planes, e.g., standard cardiac magnetic resonance imaging views. As these imaging planes are defined according to the anatomy of the imaged organ, pretext tasks effectively exploiting this information can pretrain the networks to gain knowledge on the organ of interest. In this work, we propose two complementary pretext tasks for this group of medical image data based on the spatial relationship of the imaging planes. The first is to learn the relative orientation between the imaging planes and implemented as regressing their intersecting lines. The second exploits parallel imaging planes to regress their relative slice locations within a stack. Both pretext tasks are conceptually straightforward and easy to implement, and can be combined in multitask learning for better representation learning. Thorough experiments on two anatomical structures (heart and knee) and representative target tasks (semantic segmentation and classification) demonstrate that the proposed pretext tasks are effective in pretraining deep networks for remarkably boosted performance on the target tasks, and superior to other recent approaches. [Display omitted] • Complementary self-supervised tasks for medical images with anatomy-oriented planes. • Conceptually straightforward and easy to implement. • Superior transfer learning performance on targeted groups of data to related works. [ABSTRACT FROM AUTHOR]

The paradigm of "pretraining-then-finetuning" (PT-FT) has been extensively explored to enhance the performance of clinical applications with limited annotations. A major impediment to applying such workflow to various medical imaging modalities is the lack of parameter-free approaches boosting the transferability against notable domain shifts. The success of normalization techniques in domain adaptation provides a promising solution, while inaccessible source data in finetuning (FT) poses new challenges. In this paper, we revisit the Batch Normalization module in PT-FT and propose a unified framework for both fast transferability estimation and transferability-aware finetuning. We discovered that the vanilla FT suffers from the issue that feature patterns of corresponding channels could be misaligned across domains. Hence, we present an Affine Collaborative Normalization (AC-Norm) to dynamically recalibrate the channels in the target model according to the cross-domain channel-wise correlations without any source data and extra parameters. AC-Norm provides very competitive results compared to state-of-the-art FT methods in six classification/segmentation tasks. We also demonstrate the capability of AC-Norm in estimating the transferability of pretrained models in only single-step backpropagation. Our code is available at https://github.com/EndoluminalSurgicalVision-IMR/ACNorm. • We explore the issue of channel misalignment within the transfer learning paradigm and propose an Affine Collaborative Normalization (AC-Norm) to improve the finetuning process and enable the fast selection of pretrained models. • AC-Norm permits transferability-aware finetuning by recalibrating target BN channels based on the cross-channel collaboration. • AC-Norm derives an AC-Correlation (AC-Corr) metric for fast transferability estimation on various pretrained models, which for the first time considers the dynamic representations during finetuning. [ABSTRACT FROM AUTHOR]

Since the labels for medical images are challenging to collect in real scenarios, especially for rare diseases, fully supervised methods cannot achieve robust performance for clinical anomaly detection. Recent research tried to tackle this problem by training the anomaly detection framework using only normal data. Reconstruction-based methods, e.g., auto-encoder, achieved impressive performances in the anomaly detection task. However, most existing methods adopted the straightforward backbone architecture (i.e., encoder-and-decoder) for image reconstruction. The design of a skip connection, which can directly transfer information between the encoder and decoder, is rarely used. Since the existing U-Net has demonstrated the effectiveness of skip connections for image reconstruction tasks, in this paper, we first use the dynamic gating strategy to achieve the usage of skip connections in existing reconstruction-based anomaly detection methods and then propose a novel gating highway connection module to adaptively integrate skip connections into the framework and boost its anomaly detection performance, namely GatingAno. Furthermore, we formulate an auxiliary task, namely histograms of oriented gradients (HOG) prediction, to encourage the framework to exploit contextual information from fundus images in a self-driven manner, which increases the robustness of feature representation extracted from the healthy samples. Last but not least, to improve the model generalization for anomalous data, we introduce an adversarial strategy for the training of our multi-task framework. Experimental results on the publicly available datasets, i.e., IDRiD and ADAM, validate the superiority of our method for detecting abnormalities in retinal fundus images. The source code is available at https://github.com/WentianZhang-ML/GatingAno. • A gating highway connection module is proposed, which can be applied in a reconstruction-based anomaly detection method to exploit and constrain the intrinsic features during training for better anomaly detection performance. • We integrate histograms of oriented gradients (HOG) prediction as an auxiliary task in the reconstruction network to improve its sensitivity to abnormal images. • Adversarial learning is combined in our multi-task encoder–decoder network in the training phase to improve generalization ability for anomaly detection. • We evaluate our model on two publicly available datasets. The experimental results on image-level and pixel-level tasks prove the effectiveness of gating highway connections, HOG prediction, and adversarial learning for anomaly detection. [ABSTRACT FROM AUTHOR]

Rare diseases are characterized by low prevalence and are often chronically debilitating or life-threatening. Imaging phenotype classification of rare diseases is challenging due to the severe shortage of training examples. Few-shot learning (FSL) methods tackle this challenge by extracting generalizable prior knowledge from a large base dataset of common diseases and normal controls and transferring the knowledge to rare diseases. Yet, most existing methods require the base dataset to be labeled and do not make full use of the precious examples of rare diseases. In addition, the extremely small size of the training samples may result in inter-class performance imbalance due to insufficient sampling of the true distributions. To this end, we propose in this work a novel hybrid approach to rare disease imaging phenotype classification, featuring three key novelties targeted at the above drawbacks. First, we adopt the unsupervised representation learning (URL) based on self-supervising contrastive loss, whereby to eliminate the overhead in labeling the base dataset. Second, we integrate the URL with pseudo-label supervised classification for effective self-distillation of the knowledge about the rare diseases, composing a hybrid approach taking advantage of both unsupervised and (pseudo-) supervised learning on the base dataset. Third, we use the feature dispersion to assess the intra-class diversity of training samples, to alleviate the inter-class performance imbalance via dispersion-aware correction. Experimental results of imaging phenotype classification of both simulated (skin lesions and cervical smears) and real clinical rare diseases (retinal diseases) show that our hybrid approach substantially outperforms existing FSL methods (including those using a fully supervised base dataset) via effective integration of the URL, pseudo-label driven self-distillation, and dispersion-aware imbalance correction, thus establishing a new state of the art. • A simple yet effective approach to rare disease classification. • Eliminate the heavy burden of labeling the large base dataset. • Surprisingly superior performance to comparable supervised approaches. • Hybrid unsupervised and pseudo-label supervised self-distillation. • Dispersion-aware correction of inter-rare-disease performance imbalance. [ABSTRACT FROM AUTHOR]

Segmenting prostate from magnetic resonance imaging (MRI) is a critical procedure in prostate cancer staging and treatment planning. Considering the nature of labeled data scarcity for medical images, semi-supervised learning (SSL) becomes an appealing solution since it can simultaneously exploit limited labeled data and a large amount of unlabeled data. However, SSL relies on the assumption that the unlabeled images are abundant, which may not be satisfied when the local institute has limited image collection capabilities. An intuitive solution is to seek support from other centers to enrich the unlabeled image pool. However, this further introduces data heterogeneity, which can impede SSL that works under identical data distribution with certain model assumptions. Aiming at this under-explored yet valuable scenario, in this work, we propose a separated collaborative learning (SCL) framework for semi-supervised prostate segmentation with multi-site unlabeled MRI data. Specifically, on top of the teacher-student framework, SCL exploits multi-site unlabeled data by: (i) Local learning, which advocates local distribution fitting, including the pseudo label learning that reinforces confirmation of low-entropy easy regions and the cyclic propagated real label learning that leverages class prototypes to regularize the distribution of intra-class features; (ii) External multi-site learning, which aims to robustly mine informative clues from external data, mainly including the local-support category mutual dependence learning, which takes the spirit that mutual information can effectively measure the amount of information shared by two variables even from different domains, and the stability learning under strong adversarial perturbations to enhance robustness to heterogeneity. Extensive experiments on prostate MRI data from six different clinical centers show that our method can effectively generalize SSL on multi-site unlabeled data and significantly outperform other semi-supervised segmentation methods. Besides, we validate the extensibility of our method on the multi-class cardiac MRI segmentation task with data from four different clinical centers. • We present a new and practical scenario of multi-site semi-supervised learning (MS-SSL), which allows the enrichment of the unlabeled pool with heterogeneous unlabeled data from multiple arbitrary sites and support the semi-supervised learning in local centers. • We propose a new separated collaborative learning (SCL) framework, including local learning and external multi-site learning, for this under-explored scenario. • We propose a novel local-support category mutual dependence learning scheme, which advocates mutual information-based distribution-insensitive relationship modeling on region-of-interests, for effective collaboration between local labeled data and heterogeneous external unlabeled data to support local learning. • Our method is extensively evaluated on public prostate MRI datasets from six different institutes with varying scanning protocols and patient demographics. The experimental results demonstrate the superiority of our approach. • We also validated the extensibility of our method on the multi-class cardiac MRI segmentation task with data from four different clinical centers. [ABSTRACT FROM AUTHOR]

Existing Universal Domain Adaptation (UniDA) approaches can handle various domain adaptation (DA) tasks, which need no prior information about the category overlap across target and source domains. However, traditional UniDA scenario cannot fully cover every DA scenario, e.g. , Multi-Source DA is absent. Therefore, aiming to simultaneously handle more DA scenarios in nature, we propose the More Universal Domain Adaptation (MUniDA) task. There are three challenges in MUniDA: (i) Category shift between source and target domains; (ii) Domain shift, especially the domain shift among multiple modalities in the source, which is ignored by the current UniDA approaches; (iii) How to recognize common categories across domains? We propose a more universally applicable DA approach that can tackle above challenges without any modification called A ttention Cycle -consistent U niversal N etwork (A-CycleUN). We show through extensive experiments on several benchmarks that A-CycleUN works stably and outperforms baselines across different MUniDA settings. • We introduce a more practical setting called More Universal Domain Adaptation. • A novel end-to-end framework Attention Cycle-consistent Universal Network is proposed. • The proposed method can achieve state-of-the-art performance under the new setting. [ABSTRACT FROM AUTHOR]

Since different disease grades require different treatments from physicians, i.e., the low-grade patients may recover with follow-up observations whereas the high-grade may need immediate surgery, the accuracy of disease grading is pivotal in clinical practice. In this paper, we propose a Triplet-Branch Network with ContRastive priOr-knoWledge embeddiNg (TBN-CROWN) for the accurate disease grading, which enables physicians to accordingly take appropriate treatments. Specifically, our TBN-CROWN has three branches, which are implemented for representation learning, classifier learning and grade-related prior-knowledge learning, respectively. The former two branches deal with the issue of class-imbalanced training samples, while the latter one embeds the grade-related prior-knowledge via a novel auxiliary module, termed contrastive embedding module. The proposed auxiliary module takes the features embedded by different branches as input, and accordingly constructs positive and negative embeddings for the model to deploy grade-related prior-knowledge via contrastive learning. Extensive experiments on our private and two publicly available disease grading datasets show that our TBN-CROWN can effectively tackle the class-imbalance problem and yield a satisfactory grading accuracy for various diseases, such as fatigue fracture, ulcerative colitis, and diabetic retinopathy. [Display omitted] • The auxiliary ranking task in previous TBN is improved to a novel contrastive embedding module. • The proposed TBN-CROWN is validated on the publicly available ulcerative colitis grading dataset. • This is the first work trying to develop an automated system for fatigue fracture diagnosis. [ABSTRACT FROM AUTHOR]

Abstract: Trans-catheter aortic valve implantation (TAVI) is a new breakthrough in the field of minimally invasive surgery applied on high-risk patients with aortic valve defects. 2-D X-ray angiographic and fluoroscopic images are typically used to guide TAVI procedures, for which contrast agent needs to be injected from time to time in order to make the anatomy of the aortic root visible under X-ray. Advanced visualization and guidance technology involving patient-specific 3-D models of the aorta can greatly facilitate the relatively complex TAVI procedures by providing a more realistic anatomy of the aortic root and more accurate C-Arm angulation. In this paper, a fully automatic and efficient system for contrast-based 2-D/3-D fusion for TAVI is presented. Contrast agent injection into the aortic root is automatically detected based on histogram analysis and a likelihood ratio test on the X-ray images. A hybrid method is then applied for contrast-based 2-D/3-D registration between the 3-D model and the detected angiographic frame. By integrating the information of aorta segmentation and aortic landmark detection into intensity-based registration, the proposed method combines the merits of intensity-based registration and feature/landmark-based registration. Experiments on 34 clinical data sets from TAVI patients achieve 100% correct detection on the contrast-enhanced frame, and a mean registration error of 0.66±0.47mm for 2-D/3-D registration. The proposed method is furthermore highly efficient with an average processing time of 2.5s after the most contrast-enhanced frame is available, demonstrating the efficacy of the proposed method to be adopted in a clinical setup. [Copyright &y& Elsevier]

• Crossover-Net is designed for non-elongated tissues segmentation in medical images. • Crossover-patch provides crossover relation information to Crossover-Net. • Crossover-Net learns features of crossover-patch from two directions simultaneously. • Target-guided information could be effectively highlighted by new loss function. Accurate boundary segmentation in medical images is significant yet challenging due to large variation of shape, size and appearance within intra- and inter- samples. In this paper, we present a novel deep model termed as Crossover-Net for robust segmentation in medical images. The proposed model is inspired by an interesting observation – the features learned from horizontal and vertical directions can provide informative and complement contextual information to enhance discriminative ability between different tissues. Specifically, we first originally propose a cross-shaped patch, namely crossover-patch which consists of a pair of (orthogonal and overlapping) vertical and horizontal patches. Then, we develop our Crossover-Net to learn the vertical and horizontal crossover relation according to the proposed crossover-patches. To train our model end-to-end, we design a novel loss function to (1) impose the consistency on overlapping region of vertical and horizontal patches and (2) preserve the diversity on their non-overlapping regions. We have extensively evaluated our method on CT kidney tumor, MR cardiac, and X-ray breast mass segmentation tasks, showing promising results compared with the current state-of-the-art methods. The code is available at https://github.com/Qianyu1226/Crossover-Net. [ABSTRACT FROM AUTHOR]

Layer segmentation is important to quantitative analysis of retinal optical coherence tomography (OCT). Recently, deep learning based methods have been developed to automate this task and yield remarkable performance. However, due to the large spatial gap and potential mismatch between the B-scans of an OCT volume, all of them were based on 2D segmentation of individual B-scans, which may lose the continuity and diagnostic information of the retinal layers in 3D space. Besides, most of these methods required dense annotation of the OCT volumes, which is labor-intensive and expertise-demanding. This work presents a novel framework based on hybrid 2D–3D convolutional neural networks (CNNs) to obtain continuous 3D retinal layer surfaces from OCT volumes, which works well with both full and sparse annotations. The 2D features of individual B-scans are extracted by an encoder consisting of 2D convolutions. These 2D features are then used to produce the alignment displacement vectors and layer segmentation by two 3D decoders coupled via a spatial transformer module. Two losses are proposed to utilize the retinal layers' natural property of being smooth for B-scan alignment and layer segmentation, respectively, and are the key to the semi-supervised learning with sparse annotation. The entire framework is trained end-to-end. To the best of our knowledge, this is the first work that attempts 3D retinal layer segmentation in volumetric OCT images based on CNNs. Experiments on a synthetic dataset and three public clinical datasets show that our framework can effectively align the B-scans for potential motion correction, and achieves superior performance to state-of-the-art 2D deep learning methods in terms of both layer segmentation accuracy and cross-B-scan 3D continuity in both fully and semi-supervised settings, thus offering more clinical values than previous works. • Simultaneous B-scan alignment and 3D layer segmentation of retinal OCT images. • Hybrid 2D–3D architecture accommodating highly anisotropic OCT data. • Spatial transformer module coupling B-scan motion correction with layer segmentation. • Global coherence loss encouraging coherent layer surfaces in 3D. • Effective fully and semi-supervised learning with a single unified framework. [Display omitted] [ABSTRACT FROM AUTHOR]

In the field of medical image analysis, accurate lesion segmentation is beneficial for the subsequent clinical diagnosis and treatment planning. Currently, various deep learning-based methods have been proposed to deal with the segmentation task. Albeit achieving some promising performances, the fully-supervised learning approaches require pixel-level annotations for model training, which is tedious and time-consuming for experienced radiologists to collect. In this paper, we propose a weakly semi-supervised segmentation framework, called Point Segmentation Transformer (Point SEGTR). Particularly, the framework utilizes a small amount of fully-supervised data with pixel-level segmentation masks and a large amount of weakly-supervised data with point-level annotations (i.e. , annotating a point inside each object) for network training, which largely reduces the demand of pixel-level annotations significantly. To fully exploit the pixel-level and point-level annotations, we propose two regularization terms, i.e. , multi-point consistency and symmetric consistency, to boost the quality of pseudo labels, which are then adopted to train a student model for inference. Extensive experiments are conducted on three endoscopy datasets with different lesion structures and several body sites (e.g. , colorectal and nasopharynx). Comprehensive experimental results finely substantiate the effectiveness and the generality of our proposed method, as well as its potential to loosen the requirements of pixel-level annotations, which is valuable for clinical applications. [Display omitted] • We propose a new weakly semi-supervised lesion segmentation framework. • We propose two consistency strategies to regularize the learning of our framework. • Comprehensive experiments demonstrate the superior clinical potential of our framework. [ABSTRACT FROM AUTHOR]

Nuclei segmentation is a crucial task for whole slide image analysis in digital pathology. Generally, the segmentation performance of fully-supervised learning heavily depends on the amount and quality of the annotated data. However, it is time-consuming and expensive for professional pathologists to provide accurate pixel-level ground truth, while it is much easier to get coarse labels such as point annotations. In this paper, we propose a weakly-supervised learning method for nuclei segmentation that only requires point annotations for training. First, coarse pixel-level labels are derived from the point annotations based on the Voronoi diagram and the k -means clustering method to avoid overfitting. Second, a co-training strategy with an exponential moving average method is designed to refine the incomplete supervision of the coarse labels. Third, a self-supervised visual representation learning method is tailored for nuclei segmentation of pathology images that transforms the hematoxylin component images into the H&E stained images to gain better understanding of the relationship between the nuclei and cytoplasm. We comprehensively evaluate the proposed method using two public datasets. Both visual and quantitative results demonstrate the superiority of our method to the state-of-the-art methods, and its competitive performance compared to the fully-supervised methods. Codes are available at https://github.com/hust-linyi/SC-Net. • An annotation-efficient nuclei segmentation network using point label. • A co-training strategy to automatically adjust incomplete and inaccurate coarse label. • A self-supervised learning method tailored for nuclei segmentation in pathology images. • Comprehensive analysis and evaluation on two public datasets. [ABSTRACT FROM AUTHOR]

• FedMed-GAN, to the best of our knowledge, is the first work to establish a new benchmark for federally cross-modality brain image synthesis, which greatly facilitates the development of medical GAN with differential privacy guarantees. • We provide comprehensive explanations for treating mode collapse and performance drop compared to centralized training. • The proposed work simulates as much as possible proportions of unpaired and paired data for each client with various data distributions for all clients. The performance of FedMed-GAN remains stable when facing long-tail data distributions. Utilizing multi-modal neuroimaging data is proven to be effective in investigating human cognitive activities and certain pathologies. However, it is not practical to obtain the full set of paired neuroimaging data centrally since the collection faces several constraints, e.g. , high examination cost, long acquisition time, and image corruption. In addition, these data are dispersed into different medical institutions and thus cannot be aggregated for centralized training considering the privacy issues. There is a clear need to launch federated learning and facilitate the integration of dispersed data from different institutions. In this paper, we propose a new benchmark for federated domain translation on unsupervised brain image synthesis (FedMed-GAN) to bridge the gap between federated learning and medical GAN. FedMed-GAN mitigates the mode collapse without sacrificing the performance of generators, and is widely applied to different proportions of unpaired and paired data with variation adaptation properties. We treat the gradient penalties using the federated averaging algorithm and then leverage the differential privacy gradient descent to regularize the training dynamics. A comprehensive evaluation is provided for comparing FedMed-GAN and other centralized methods, demonstrating that the proposed algorithm outperforms the state-of-the-art. Our code is available at: https://github.com/M-3LAB/FedMed-GAN. [ABSTRACT FROM AUTHOR]

Semi-supervised learning has greatly advanced medical image segmentation since it effectively alleviates the need of acquiring abundant annotations from experts, wherein the mean-teacher model, known as a milestone of perturbed consistency learning, commonly serves as a standard and simple baseline. Inherently, learning from consistency can be regarded as learning from stability under perturbations. Recent improvement leans toward more complex consistency learning frameworks, yet, little attention is paid to the consistency target selection. Considering that the ambiguous regions from unlabeled data contain more informative complementary clues, in this paper, we improve the mean-teacher model to a novel ambiguity-consensus mean-teacher (AC-MT) model. Particularly, we comprehensively introduce and benchmark a family of plug-and-play strategies for ambiguous target selection from the perspectives of entropy, model uncertainty and label noise self-identification, respectively. Then, the estimated ambiguity map is incorporated into the consistency loss to encourage consensus between the two models' predictions in these informative regions. In essence, our AC-MT aims to find out the most worthwhile voxel-wise targets from the unlabeled data, and the model especially learns from the perturbed stability of these informative regions. The proposed methods are extensively evaluated on left atrium segmentation and brain tumor segmentation. Encouragingly, our strategies bring substantial improvement over recent state-of-the-art methods. The ablation study further demonstrates our hypothesis and shows impressive results under various extreme annotation conditions. • Experimentally identify a neglected-yet-critical problem, i.e., consistency target selection, that limits the performance of the classical mean-teacher semi-supervised segmentation model. • Propose a new perspective that encouraging perturbed stability in ambiguous-yet-informative regions can drive the model to learn more useful representations from unlabeled data. • Improve the classical mean-teacher model to a novel ambiguity-consensus mean-teacher (AC-MT) model, equipped with four plug-and-play strategies for ambiguous target selection. [ABSTRACT FROM AUTHOR]

Highlights • This paper shows extensive experiments to validate our method, including CT and MRI translation for pancreas segmentation and domain adaptation for mammogram X-rays for breast lesion segmentation. • This paper validates our method using a variety of advanced segmentation networks and show that our method performs generally well and consistently boosts medical 2D/3D image segmentation performance. • This paper systemically analyzes the effect of synthetic data on segmentation, with the goal to investigate the limitation of synthetic data, and inspire new scopes. Graphical abstract Abstract Synthesized medical images have several important applications. For instance, they can be used as an intermedium in cross-modality image registration or used as augmented training samples to boost the generalization capability of a classifier. In this work, we propose a generic cross-modality synthesis approach with the following targets: 1) synthesizing realistic looking 2D/3D images without needing paired training data, 2) ensuring consistent anatomical structures, which could be changed by geometric distortion in cross-modality synthesis and 3) more importantly, improving volume segmentation by using synthetic data for modalities with limited training samples. We show that these goals can be achieved with an end-to-end 2D/3D convolutional neural network (CNN) composed of mutually-beneficial generators and segmentors for image synthesis and segmentation tasks. The generators are trained with an adversarial loss, a cycle-consistency loss, and also a shape-consistency loss (supervised by segmentors) to reduce the geometric distortion. From the segmentation view, the segmentors are boosted by synthetic data from generators in an online manner. Generators and segmentors prompt each other alternatively in an end-to-end training fashion. We validate our proposed method on three datasets, including cardiovascular CT and magnetic resonance imaging (MRI), abdominal CT and MRI, and mammography X-rays from different data domains, showing both tasks are beneficial to each other and coupling these two tasks results in better performance than solving them exclusively. [ABSTRACT FROM AUTHOR]

• A novel deep CNN architecture is proposed for jointly learning the expert consensus label and the annotator's label. The proposed architecture (Fig. 1) consists of two coupled CNNs where one estimates the expert consensus label probabilities and the other models the characteristics of individual annotators (e.g., tendency to over-segmentation, mix-up between different classes, etc) by estimating the pixel-wise confusion matrices (CMs) on a per image basis. Unlike STAPLE [25] and its variants, our method models, and disentangles with deep neural networks, the complex mappings from the input images to the annotator behaviours and to the expert consensus label. • The parameters of our CNNs are "global variables" that are optimised across different image samples; this enables the model to disentangle robustly the annotators' mistakes and the expert consensus label based on correlations between similar image samples, even when the number of available annotations is small per image (e.g., a single annotation per image). In contrast, this would not be possible with STAPLE [25] and its variants [5,8] where the annotators' parameters are estimated on every target image separately. • This paper extends the preliminary version of our method presented at the NeurIPS Thirty-fourth Annual Conference on Neural Information Processing Systems [30] , by extensively evaluating our model on a new created real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK). This dataset is generated with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). Additionally, we presented a comprehensive discussion about our model's potential applications (e.g., estimate annotator's quality and annotation's quality), the future works we are going to explore, and the potential limitations of our model. [Display omitted] Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the "actual" segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator's "unreliable behavior" (we call it "maximally unreliable") while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method's efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators' mistakes. [ABSTRACT FROM AUTHOR]

During the computed tomography (CT) imaging process, metallic implants within patients often cause harmful artifacts, which adversely degrade the visual quality of reconstructed CT images and negatively affect the subsequent clinical diagnosis. For the metal artifact reduction (MAR) task, current deep learning based methods have achieved promising performance. However, most of them share two main common limitations: (1) the CT physical imaging geometry constraint is not comprehensively incorporated into deep network structures; (2) the entire framework has weak interpretability for the specific MAR task; hence, the role of each network module is difficult to be evaluated. To alleviate these issues, in the paper, we construct a novel deep unfolding dual domain network, termed InDuDoNet+, into which CT imaging process is finely embedded. Concretely, we derive a joint spatial and Radon domain reconstruction model and propose an optimization algorithm with only simple operators for solving it. By unfolding the iterative steps involved in the proposed algorithm into the corresponding network modules, we easily build the InDuDoNet+ with clear interpretability. Furthermore, we analyze the CT values among different tissues, and merge the prior observations into a prior network for our InDuDoNet+, which significantly improve its generalization performance. Comprehensive experiments on synthesized data and clinical data substantiate the superiority of the proposed methods as well as the superior generalization performance beyond the current state-of-the-art (SOTA) MAR methods. Code is available at https://github.com/hongwang01/InDuDoNet_plus. [Display omitted] • For the MAR task, we propose a concise dual domain reconstruction model. • We build a dual domain network InDuDoNet+ with fine physical interpretability. • We design a knowledge-driven Prior-net for better generalization performance. • Comprehensive experiments fully substantiate the effectiveness of our methods. [ABSTRACT FROM AUTHOR]

Cancer region detection (CRD) and subtyping are two fundamental tasks in digital pathology image analysis. The development of data-driven models for CRD and subtyping on whole-slide images (WSIs) would mitigate the burden of pathologists and improve their accuracy in diagnosis. However, the existing models are facing two major limitations. Firstly, they typically require large-scale datasets with precise annotations, which contradicts with the original intention of reducing labor effort. Secondly, for the subtyping task, the non-cancerous regions are treated as the same as cancerous regions within a WSI, which confuses a subtyping model in its training process. To tackle the latter limitation, the previous research proposed to perform CRD first for ruling out the non-cancerous region, then train a subtyping model based on the remaining cancerous patches. However, separately training ignores the interaction of these two tasks, also leads to propagating the error of the CRD task to the subtyping task. To address these issues and concurrently improve the performance on both CRD and subtyping tasks, we propose a semi-supervised multi-task learning (MTL) framework for cancer classification. Our framework consists of a backbone feature extractor, two task-specific classifiers, and a weight control mechanism. The backbone feature extractor is shared by two task-specific classifiers, such that the interaction of CRD and subtyping tasks can be captured. The weight control mechanism preserves the sequential relationship of these two tasks and guarantees the error back-propagation from the subtyping task to the CRD task under the MTL framework. We train the overall framework in a semi-supervised setting, where datasets only involve small quantities of annotations produced by our minimal point-based (min-point) annotation strategy. Extensive experiments on four large datasets with different cancer types demonstrate the effectiveness of the proposed framework in both accuracy and generalization. [Display omitted] • A novel annotation method (Min-point annotation) for whole slide images. • A network targeted at perform cancer region detection and subtyping simultaneously. • A semi-supervised multi-task learning framework that learns two sequential tasks. • Four cancer classification datasets with complete region and min-point annotation. [ABSTRACT FROM AUTHOR]

• We propose an automatic end-to-end topology and width aware network for artery/vein classification, which, for the first time, integrates topology and vessel width information into the deep learning framework to boost the A/V classification performance. • A topology-aware module is proposed to increase the topological connectivity of the segmented artery and vein maps, which contains a topology-ranking discriminator and a topology preserving regularization module. • A width-aware module is designed to extract features related to the vessel width via predicting the width maps for the dilated/non-dilated ground truth A/V masks to enhance the model's perception of the vessel width, which is regularized by a width-aware loss. • The proposed framework is validated quantitatively and qualitatively on three publicly available datasets, including AV-DRIVE, INSPIRE-AVR and high-resolution fundus (HRF). In addition, we have manually annotated the pixel-wise artery and vein classification labels for the HRF dataset, which will be released for public access. • Source code of the proposed framework will be made publicly available. [Display omitted] Automatic artery/vein (A/V) classification, as the basic prerequisite for the quantitative analysis of retinal vascular network, has been actively investigated in recent years using both conventional and deep learning based methods. The topological connection relationship and vessel width information, which have been proved effective in improving the A/V classification performance for the conventional methods, however, have not yet been exploited by the deep learning based methods. In this paper, we propose a novel Topology and Width Aware Generative Adversarial Network (named as TW-GAN), which, for the first time, integrates the topology connectivity and vessel width information into the deep learning framework for A/V classification. To improve the topology connectivity, a topology-aware module is proposed, which contains a topology ranking discriminator based on ordinal classification to rank the topological connectivity level of the ground-truth mask, the generated A/V mask and the intentionally shuffled mask. In addition, a topology preserving triplet loss is also proposed to extract the high-level topological features and further to narrow the feature distance between the predicted A/V mask and the ground-truth mask. Moreover, to enhance the model's perception of vessel width, a width-aware module is proposed to predict the width maps for the dilated/non-dilated ground-truth masks. Extensive empirical experiments demonstrate that the proposed framework effectively increases the topological connectivity of the segmented A/V masks and achieves state-of-the-art A/V classification performance on the publicly available AV-DRIVE and HRF datasets. Source code and data annotations are available at https://github.com/o0t1ng0o/TW-GAN. [ABSTRACT FROM AUTHOR]

Examinations of the spinal column with both, Magnetic Resonance (MR) imaging and Computed Tomography (CT), often require a precise three-dimensional positioning, angulation and labeling of the spinal disks and the vertebrae. A fully automatic and robust approach is a prerequisite for an automated scan alignment as well as for the segmentation and analysis of spinal disks and vertebral bodies in Computer Aided Diagnosis (CAD) applications. In this article, we present a novel method that combines Marginal Space Learning (MSL), a recently introduced concept for efficient discriminative object detection, with a generative anatomical network that incorporates relative pose information for the detection of multiple objects. It is used to simultaneously detect and label the spinal disks. While a novel iterative version of MSL is used to quickly generate candidate detections comprising position, orientation, and scale of the disks with high sensitivity, the anatomical network selects the most likely candidates using a learned prior on the individual nine dimensional transformation spaces. Finally, we propose an optional case-adaptive segmentation approach that allows to segment the spinal disks and vertebrae in MR and CT respectively. Since the proposed approaches are learning-based, they can be trained for MR or CT alike. Experimental results based on 42 MR and 30 CT volumes show that our system not only achieves superior accuracy but also is among the fastest systems of its kind in the literature. On the MR data set the spinal disks of a whole spine are detected in 11.5s on average with 98.6% sensitivity and 0.073 false positive detections per volume. On the CT data a comparable sensitivity of 98.0% with 0.267 false positives is achieved. Detected disks are localized with an average position error of 2.4mm/3.2mm and angular error of 3.9°/4.5° in MR/CT, which is close to the employed hypothesis resolution of 2.1mm and 3.3°. [ABSTRACT FROM AUTHOR]

• We introduce a new task named Noisy Supervised Domain Adaptation (NSDA), which is both challenging and meaningful for real-world applications in medical image segmentation. • We propose a novel self-cleansing unsupervised domain adaptation (S-CUDA) method to address the NSDA problem for medical image segmentation, which trains two peer adversarial networks simultaneously in a co-training and co-cleansing style. For each peer network, we divide the training samples into the high-confidence clean labeled set (i.e., clean data) and noisy labeled set (i.e., noisy data) based on its per-sample loss. The identified clean data are then used to train the peer networks, which keep diverging to another one, to filter different type of errors and avoid confirmation bias in the self-training. • We propose to clean and reuse the identified high-confidence noisy labeled samples to provide more diversity and useful information to learn a robust model. For the identified high-confidence noisy data, we clean and refine their labels using the network's predictions as pseudo labels, which will be further reused in the following learning iterations. • We design a novel loss function to use both the original noisy labels and the updated cleaned labels from the peer network to break up the chain of "error propagation". The new loss function can help the cross-reviewing framework alleviate the "learning trap" problem exiting in self-training process (i.e., the quality of pseudo labels does not get improved and the new supervision from them becomes weaker and weaker). • We conduct extensive experiments on public datasets, i.e., two retinal fundus image datasets and a public spinal cord gray matter segmentation challenge dataset, to demonstrate the effectiveness of our proposed S-CUDA framework. The comprehensive results demonstrate the superior performance of our method compared with existing state-of-the-art methods. [Display omitted] Medical image segmentation tasks hitherto have achieved excellent progresses with large-scale datasets, which empowers us to train potent deep convolutional neural networks (DCNNs). However, labeling such large-scale datasets is laborious and error-prone, which leads the noisy (or incorrect) labels to be an ubiquitous problem in the real-world scenarios. In addition, data collected from different sites usually exhibit significant data distribution shift (or domain shift). As a result, noisy label and domain shift become two common problems in medical imaging application scenarios, especially in medical image segmentation, which degrade the performance of deep learning models significantly. In this paper, we identify a novel problem hidden in medical image segmentation, which is unsupervised domain adaptation on noisy labeled data, and propose a novel algorithm named "Self-Cleansing Unsupervised Domain Adaptation" (S-CDUA) to address such issue. S-CUDA sets up a realistic scenario to solve the above problems simultaneously where training data (i.e., source domain) not only shows domain shift w.r.t. unsupervised test data (i.e., target domain) but also contains noisy labels. The key idea of S-CUDA is to learn noise-excluding and domain invariant knowledge from noisy supervised data, which will be applied on the highly corrupted data for label cleansing and further data-recycling, as well as on the test data with domain shift for supervised propagation. To this end, we propose a novel framework leveraging noisy-label learning and domain adaptation techniques to cleanse the noisy labels and learn from trustable clean samples, thus enabling robust adaptation and prediction on the target domain. Specifically, we train two peer adversarial networks to identify high-confidence clean data and exchange them in companions to eliminate the error accumulation problem and narrow the domain gap simultaneously. In the meantime, the high-confidence noisy data are detected and cleansed in order to reuse the contaminated training data. Therefore, our proposed method can not only cleanse the noisy labels in the training set but also take full advantage of the existing noisy data to update the parameters of the network. For evaluation, we conduct experiments on two popular datasets (REFUGE and Drishti-GS) for optic disc (OD) and optic cup (OC) segmentation, and on another public multi-vendor dataset for spinal cord gray matter (SCGM) segmentation. Experimental results show that our proposed method can cleanse noisy labels efficiently and obtain a model with better generalization performance at the same time, which outperforms previous state-of-the-art methods by large margin. Our code can be found at https://github.com/zzdxjtu/S-cuda. [ABSTRACT FROM AUTHOR]

• A dataset is constructed for GRAding cervical intraepithelial Neoplasia with fine-grained lesion Description (GRAND). • We benchmark a multi-rater model for our dataset. • The proposed benchmark framework achieves a comparable CIN grading accuracy to that of a professional colposcopist. Exemplar colposcopic images in our GRAND dataset. The CIN1 patient (a) may have a similar appearance to CIN2/3 (b) under colposcopy. There are some difficulties for accurate CIN grading, such as the false lesion area caused by the light reflection on normal cervix surface (c) and the occlusion caused by artifacts (d). [Display omitted] Cervical cancer causes the fourth most cancer-related deaths of women worldwide. Early detection of cervical intraepithelial neoplasia (CIN) can significantly increase the survival rate of patients. World Health Organization (WHO) divided the CIN into three grades (CIN1, CIN2 and CIN3). In clinical practice, different CIN grades require different treatments. Although existing studies proposed computer aided diagnosis (CAD) systems for cervical cancer diagnosis, most of them are fail to perform accurate separation between CIN1 and CIN2/3, due to the similar appearances under colposcopy. To boost the accuracy of CAD systems, we construct a colposcopic image dataset for GRA ding cervical intraepithelial N eoplasia with fine-grained lesion D escription (GRAND). The dataset consists of colposcopic images collected from 8,604 patients along with the pathological reports. Additionally, we invite the experienced colposcopist to annotate two main clues, which are usually adopted for clinical diagnosis of CIN grade, i.e., texture of acetowhite epithelium (TAE) and appearance of blood vessel (ABV). A multi-rater model using the annotated clues is benchmarked for our dataset. The proposed framework contains several sub-networks (raters) to exploit the fine-grained lesion features TAE and ABV, respectively, by contrastive learning and a backbone network to extract the global information from colposcopic images. A comprehensive experiment is conducted on our GRAND dataset. The experimental results demonstrate the benefit of using additional lesion descriptions (TAE and ABV), which increases the CIN grading accuracy by over 10%. Furthermore, we conduct a human-machine confrontation to evaluate the potential of the proposed benchmark framework for clinical applications. Particularly, three colposcopists on different professional levels (intern, in-service and professional) are invited to compete with our benchmark framework by investigating a same extra test set—our framework achieves a comparable CIN grading accuracy to that of a professional colposcopist. [ABSTRACT FROM AUTHOR]

• Pairwise learning for medical image segmentation. • A proxy supervision for addressing intra-class heterogeneity and boundary ambiguity. • Inherent prior information exploiting from the label space. • Segmentation probability inferred by the learned logical relation of input pairs in the test phase. • Evaluation on two public medical image segmentation datasets. Fully convolutional networks (FCNs) trained with abundant labeled data have been proven to be a powerful and efficient solution for medical image segmentation. However, FCNs often fail to achieve satisfactory results due to the lack of labelled data and significant variability of appearance in medical imaging. To address this challenging issue, this paper proposes a conjugate fully convolutional network (CFCN) where pairwise samples are input for capturing a rich context representation and guide each other with a fusion module. To avoid the overfitting problem introduced by intra-class heterogeneity and boundary ambiguity with a small number of training samples, we propose to explicitly exploit the prior information from the label space, termed as proxy supervision. We further extend the CFCN to a compact conjugate fully convolutional network (C 2 FCN), which just has one head for fitting the proxy supervision without incurring two additional branches of decoders fitting ground truth of the input pairs compared to CFCN. In the test phase, the segmentation probability is inferred by the learned logical relation implied in the proxy supervision. Quantitative evaluation on the Liver Tumor Segmentation (LiTS) and Combined (CT-MR) Healthy Abdominal Organ Segmentation (CHAOS) datasets shows that the proposed framework achieves a significant performance improvement on both binary segmentation and multi-category segmentation, especially with a limited amount of training data. The source code is available at https://github.com/renzhenwang/pairwise_segmentation. [ABSTRACT FROM AUTHOR]

• A benchmark study of a global segmentation challenge conducted on the largest atrial LGE-MRI dataset. • Performed rigorous subgroup analysis and hyper-parameter tuning experiments. • U-Net achieved better performance compared to others. • 2D and 3D CNN methods had comparable accuracies. • The double, sequentially used CNNs achieved superior results. Segmentation of medical images, particularly late gadolinium-enhanced magnetic resonance imaging (LGE-MRI) used for visualizing diseased atrial structures, is a crucial first step for ablation treatment of atrial fibrillation. However, direct segmentation of LGE-MRIs is challenging due to the varying intensities caused by contrast agents. Since most clinical studies have relied on manual, labor-intensive approaches, automatic methods are of high interest, particularly optimized machine learning approaches. To address this, we organized the 2018 Left Atrium Segmentation Challenge using 154 3D LGE-MRIs, currently the world's largest atrial LGE-MRI dataset, and associated labels of the left atrium segmented by three medical experts, ultimately attracting the participation of 27 international teams. In this paper, extensive analysis of the submitted algorithms using technical and biological metrics was performed by undergoing subgroup analysis and conducting hyper-parameter analysis, offering an overall picture of the major design choices of convolutional neural networks (CNNs) and practical considerations for achieving state-of-the-art left atrium segmentation. Results show that the top method achieved a Dice score of 93.2% and a mean surface to surface distance of 0.7 mm, significantly outperforming prior state-of-the-art. Particularly, our analysis demonstrated that double sequentially used CNNs, in which a first CNN is used for automatic region-of-interest localization and a subsequent CNN is used for refined regional segmentation, achieved superior results than traditional methods and machine learning approaches containing single CNNs. This large-scale benchmarking study makes a significant step towards much-improved segmentation methods for atrial LGE-MRIs, and will serve as an important benchmark for evaluating and comparing the future works in the field. Furthermore, the findings from this study can potentially be extended to other imaging datasets and modalities, having an impact on the wider medical imaging community. Image, graphical abstract [ABSTRACT FROM AUTHOR]