Search

With the recent advancement in the medical diagnostic tools, multi-modality medical images are extensively utilized as a lifesaving tool. An efficient fusion of medical images can improve the performance of various medical diagnostic tools. But, gathering of all modalities for a given patient is defined as an ill-posed problem as medical images suffer from poor visibility and frequent patient dropout. Therefore, in this paper, a novel image fusion model is designed to fuse multi-modality medical images. The proposed model model suffers from hyper-parameters tuning issue, therefore, a multi-objective differential evolution (MDE) is used to optimize the initial parameters of the proposed model. The fusion factor and edge strength metrics are utilized to form a multi-objective fitness function. The performance of the proposed model is validated by comparing the proposed model models with nine competitive models over fifteen benchmark images. Performance analysis reveal that the proposed model outperforms the competitive fusion models.

In this article, a three-dimensional detection framework for detecting non-small cell lung cancer (NSCLC) in 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography/computed tomography (PET/CT) images and guided by a multi-modality attention fusion is proposed. A total of 250 18F-FDG PET/CT scans between January 1, 2015 and December 31, 2019 of patients who had histopathologically proven NSCLC were acquired. A customized dual-path 3D CenterNet is used for NSCLC detection. Moreover, we propose a multi-modality attention module that adaptively refines the multi-modality feature map fusion. Since the 3D convolutional neural network (CNN) requires many graphic memories and sliding windows, a 384×384×32 patch size is designed and used for training and testing. Five-fold cross-validation is applied for this study. The sensitivity and false positive per scan (FPPS) obtained by our proposed method are 0.96 and 1.04, respectively. Our method significantly outperforms 3D CenterNet in terms of sensitivity (P =.031). This module demonstrates the potential to be implemented in other multi-modality applications. Our result performs competitively against other lung cancer detections. Furthermore, case studies show that the proposed method can detect difficult-to-diagnose NSCLCs. Our result shows the proposed method can help radiologists and medical physicists diagnose NSCLC.

Nowadays, with the development of various kinds of sensors in smartphones or wearable devices, human activity recognition (HAR) has been widely researched and has numerous applications in healthcare, smart city, etc. Many techniques based on hand-crafted feature engineering or deep neural network have been proposed for sensor based HAR. However, these existing methods usually recognize activities offline, which means the whole data should be collected before training, occupying large-capacity storage space. Moreover, once the offline model training finished, the trained model can't recognize new activities unless retraining from the start, thus with a high cost of time and space. In this paper, we propose a multi-modality incremental learning model, called HarMI, with continuous learning ability. The proposed HarMI model can start training quickly with little storage space and easily learn new activities without storing previous training data. In detail, we first adopt attention mechanism to align heterogeneous sensor data with different frequencies. In addition, to overcome catastrophic forgetting in incremental learning, HarMI utilizes the elastic weight consolidation and canonical correlation analysis from a multi-modality perspective. Extensive experiments based on two public datasets demonstrate that HarMI can achieve a superior performance compared with several state-of-the-arts.

Magnetic resonance imaging (MRI) can provide multi-modality MR images by setting task-specific scan parameters, and has been widely used in various disease diagnosis and planned treatments. However, in practical clinical applications, it is often difficult to obtain multi-modality MR images simultaneously due to patient discomfort, and scanning costs, etc. Therefore, how to effectively utilize the existing modality images to synthesize missing modality image has become a hot research topic. In this paper, we propose a novel confidence-guided aggregation and cross-modality refinement network (CACR-Net) for multi-modality MR image synthesis, which effectively utilizes complementary and correlative information of multiple modalities to synthesize high-quality target-modality images. Specifically, to effectively utilize the complementary modality-specific characteristics, a confidence-guided aggregation module is proposed to adaptively aggregate the multiple target-modality images generated from multiple source-modality images by using the corresponding confidence maps. Based on the aggregated target-modality image, a cross-modality refinement module is presented to further refine the target-modality image by mining correlative information among the multiple source-modality images and aggregated target-modality image. By training the proposed CACR-Net in an end-to-end manner, high-quality and sharp target-modality MR images are effectively synthesized. Experimental results on the widely used benchmark demonstrate that the proposed method outperforms state-of-the-art methods.

Big challenges are usually occurring in non-rigid 3D shape retrieval, for the shapes undergoing arbitrarily non-affine transformations. In this work, a novel design of feature learning approach is proposed for non-rigid 3D shape retrieval, dubbed Structured Sparsity Regularized Multi-Modality Method (SSR-MM). The shape signatures which capture the deformation-invariant characteristics are averaged and stacked for a multi-modality machine learning approach, and a transform matrix based on the structure sparsity regularization is utilized to map those signatures obtaining the discriminative features for retrieval. The proposed framework is evaluated on the publicly available non-rigid 3D human benchmarks, and the experimental results show the efficacy of our contributions and the advantages of our method over existing ones.

Craniofacial fibrous dysplasia [CF-FD] with orbital complications is a known but rare entity. This is a multi-modality imaging Case Report of extensive CF-FD in a 31-year-old male with right eye pain, swelling, and redness, along with histopathology correlation. In this patient, the CT scan demonstrates the classic ground glass bony appearance in great detail and helps with the diagnosis of FD, while excluding other bony pathology. The patient’s corroborative MRI brain with and without contrast and MRI brain perfusion images are presented to further characterize this pathology along with its orbital and ocular complications. Given the significant mass-effect on the ocular structures, the patient underwent orbital surgery with removal of as much of the lesion as possible. On macroscopic pathology evaluation, the affected bone was rubbery and gritty when sectioned. Microscopically, remnant fragments of woven bone of various size and shapes were seen with lack of an osteoblast rim. The bony fragments had a characteristic curvilinear, trabecular, and/or branching pattern. Post-surgical imaging demonstrated improvement in the mass-effect on orbital structures and proptosis, along with residual bony lesion.

The tricuspid valve has gained interest recently because of the poor outcomes with current treatments and advances in percutaneous valve interventions. A sound understanding of the anatomy and pathologies of the tricuspid valve is critical in its evaluation and management of tricuspid regurgitation (TR). A multi-modality imaging approach with transthoracic echocardiography, transesophageal echocardiography, computed tomography, magnetic resonance imaging all have their individual and collective roles in the evaluation of TR and guidance of surgical and percutaneous procedures. This combined with clinical factors will contribute to defining timing, indications, modality selection and risk stratification for tricuspid valve interventions, which currently remains controversial.

The left atrium (LA) plays a vital role in maintaining normal cardiac function. LA volume and function have been utilised as important imaging biomarkers, with their prognostic value demonstrated in multiple cardiac conditions. More recently, there has been a sharp increase in the number of publications utilising LA strain by echocardiography and cardiac magnetic resonance (CMR) imaging. However, little is known about its prognostic value or reproducibility as a technique. In this review, we aim to highlight the conventional and novel imaging techniques available for LA assessment, using echocardiography and CMR, their role as an imaging biomarker in cardiovascular disease, the reproducibility of the techniques and the current limitations to their clinical application. We identify a need for further standardisation of techniques, with establishment of ‘normal’ cut-offs before routine clinical application can be made. Supplementary Information The online version contains supplementary material available at 10.1007/s10554-021-02316-x.

Hypertrophic cardiomyopathy (HCM) represents one of the primary cardiomyopathies and may lead to heart failure and sudden cardiac death. Among various histologic features of the disease examined, assessment of myocardial fibrosis may offer valuable information, since it may be considered the common nominator for all HCM connected complications. Late gadolinium-enhanced cardiac magnetic resonance (LGE-CMR) has emerged as the reference noninvasive method for visualizing and quantifying myocardial fibrosis in patients with HCM. T1 mapping, a promising new CMR technique, may provide an advantage over conventional LGE-CMR, by permitting a more valid quantification of diffuse fibrosis. On the other hand, echocardiography offers a significantly more portable, affordable, and easily accessible solution for the study of fibrosis. Various echocardiographic techniques ranging from integrated backscatter and contrast-enhanced ultrasound to two- (2D) or three-dimensional (3D) deformation and shear wave imaging may offer new insights into substrate characterization in HCM. The aim of this review is to describe thoroughly all different modalities that may be used in everyday clinical practice for HCM fibrosis evaluation (with special focus on echocardiographic techniques), to concisely present available evidence and to argue in favor of multi-modality imaging application. It is essential to understand that the role of various imaging modalities is not competitive but complementary, since the information provided by each one is necessary to illuminate the complex pathophysiologic pathways of HCM, offering a personalized approach and treatment in every patient.

A dural arteriovenous fistula (AVF) is a rare condition in a child and is not evident clinically. It is a type of an acquired cerebral vascular malformation that usually occurs after a thrombotic event of the cerebral venous sinuses. Dural AVF is not suspected clinically and is revealed through imaging done for evaluation of cranial symptoms. Therefore, it is essential to revisit the pathophysiology and the clinical situations leading to intracranial dural AVF. Equally crucial is identifying the imaging findings on computed tomography, or magnetic resonance imaging brain scans done as a preliminary work-up in these patients. However, for optimal management decision and prognostication of dural AVF, a digital subtraction angiography is essential. As the entire burden of establishing the diagnosis rests on the radiologists, we would like to present this rare case report highlighting both the clinical and imaging aspects and the management options available for dural AVFs.

In this study, we focus on improving the efficiency and accuracy of nonrigid multi-modality registration of medical images. In this regard, we analyze the potentials of using the point similarity measurement approach as an alternative to global computation of mutual information (MI), which is still the most renown multi-modality similarity measure. The improvement capabilities are illustrated using the popular B-spline transformation model. The proposed solution is a combination of three related improvements of the most straightforward implementation, i.e., efficient computation of the voxel displacement field, local estimation of similarity and usage of a static image intensity dependence estimate. Five image registration prototypes were implemented to show contribution and dependence of the proposed improvements. When all the proposed improvements are applied, a significant reduction of computational cost and increased accuracy are obtained. The concept offers additional improvement opportunities by incorporating prior knowledge and machine learning techniques into the static intensity dependence estimation.

Optic neuropathy is kind of common eye diseases, which usually causes irreversible vision loss. Early diagnosis is key to saving patients’ vision. Due to the similar early clinical manifestations of common optic neuropathy, it may cause misdiagnosis and delays in treatment. Worse, most diagnoses rely on experienced doctors. In this paper, we proposed a novel deep learning architecture GroupFusionNet (GFN) to diagnose five normal optic neuropathy diseases, including Anterior Ischemic Optic Neuropathy (AION), papilledema, papillitis, Optic Disc Vasculitis (ODV), and optic atrophy (OA). Specifically, we combined multi-modalities in clinic examination such as fundus image, visual field tests and age of each patient. GFN utilized two ResNet pathways to extract and fuse both features of fundus image and visual field tests, and the information of structured data was embedded in the end. Experimental results demonstrate that multi-modality feature aggregation is effective for optic neuropathy diseases diagnosis, and GFN achieved a five-classes classification accuracy of 87.82% on the test dataset.

This paper proposes an automated registration method for multi‐modality retinal fundus photographs based on the directional vessel skeleton. The main purpose is to register two retinal fundus photographs with different modalities of the same scanning region, which can provide multi‐modality information for clinicians to diagnose retinal diseases or to make a treatment decision. The directional vessel skeleton of each fundus image is first detected by bias field correction and Gabor filter. The final registered fundus photographs are then obtained by the iterative affine registration between the detected directional vessel skeletons of two photographs. In this work, four kinds of fundus photographs in the macular regions of the patient with diseases, consisting of 20 optical coherence tomography fundus images, 20 colour fundus photographs, 20 fluorescein fundus angiography images and 20 indocyanine green angiography images, are utilised to quantitatively evaluate the proposed method. The root‐mean‐square errors show an advantageous performance in both registration success rate and accuracy.

In this paper, we propose a triple intersecting U-Nets (TIU-Nets) for brain glioma segmentation. First, the proposed TIU-Nets is composed of binary-class segmentation U-Net (BU-Net) and multi-class segmentation U-Net (MU-Net), in which MU-Net reuses multi-resolution features from BU-Net. Second, we introduce a segmentation soft-mask predicted by BU-Net, that is, candidate glioma region is generated by removing most of non-glioma backgrounds, which guides multi-category segmentation of MU-Net in a weighted manner. Third, an edge branch in MU-Net is leveraged to enhance boundary information of glioma substructure, which facilitates to locate glioma true boundaries and improve segmentation accuracy. Finally, we propose a sigmoid-evolution based polarized cross-entropy loss (S-CE) to resolve class unbalance problem, and apply S-CE loss to soft-mask prediction loss in BU-Net, multi-class segmentation loss in MU-Net and edge prediction loss in edge branch. Experimental results have demonstrated that the proposed 2D/3D TIU-Nets achieves a higher segmentation accuracy than corresponding 2D/3D state-of-the-art segmentation methods including FCN, U-Net, SegNet, CRDN, IVD-Net, FCDenseNet, DeepMedic, DMFNet, etc, evaluating on publicly available brain tumor segmentation challenge 2015 (BRATS2015) datasets. To show the universality of the proposed method, we also give a comparison of segmentation performance on BrainWeb dataset.

We present the case of a 19-year-old woman who presented with cyanosis and on investigation (Multi Modality imaging) was found to have biventricular non-compaction associated with pulmonary arteriovenous malformation - an association not hitherto reported.

Background: Imaging of salivary gland tumors is a challenge for the radiologist due to varied histological features and overlapping clinical presentation.The salivary glands have a glandular structure with tubules, secretory acini and myoepithelial cells to produce and secrete saliva. They are divided into two types: major and minor. Parotid, submandibular, and sublingual glands are the major paired salivary glands of the body. Multiple small minor salivary glands are distributed in the upper aerodigestive tract including PNS & parapharyngeal spaces. Objective: This study evaluates the role of different modalities used to assess salivary gland tumors. Results: 50 patients presenting with suspected salivary gland masses were evaluated. 22 males and 28 females were assessed with age ranging from 15 to 66 years (mean age of 52 years). 27 cases were of pleomorphic adenoma, 10 warthin tumors, 2 intraductal papilloma,1 lipoma, 1 oncocytoma,1 intra parotid paraganglioma, 2 hemangioma, 3 muco-epidermoid carcinoma, 2 adeno cystic cancer and 1 acinic cell cancer of salivary gland. 90% of patients (45 out of 50) presented with swelling / lump. 26% of patients had pain /discomfort and 4% of patients (2 out of 50) had facial nerve palsy. Conclusion: The conventional radiological evaluation was carried out using Xray and ultrasound with USG as rst line of investigation.. Cross sectional imaging with CT and MRI is useful in assessing for local extension , invasion of neighbouring structures as well as lymph nodal and systemic metastasis.MRI is superior to CT in assesing for extent and perineural spread.

Purpose Dorsal arachnoid web (DAW) is a rare intradural abnormality which is associated with progressive myelopathy. Our objective was to review multi-modality imaging techniques demonstrating the scalpel sign appearance in evaluation of DAW. Methods We retrospectively reviewed various imaging modalities of patients found to have DAW at our institution during January 2015 to February 2020. Five patients underwent surgical decompression with pathological correlation. The remaining patients were presumptively diagnosed based on the characteristic finding of scalpel sign. Clinical data were evaluated and correlated to imaging findings. All imaging modalities demonstrated the characteristic scalpel sign. Results Sixteen patients (10 females, and six males) with multi-imaging modalities were evaluated. Their mean age was 52 year (range 23–74 years). Fifteen patients underwent conventional spine MRI. Further high-resolution MR imaging techniques, e.g. 3D T2 myelographic sequence, were utilized with two patients. MRI spine CSF flow study was performed to evaluate the flow dynamic across the arachnoid web in one patient. Eight patients were evaluated with CT myelogram. Syrinx formation was discovered in seven (44%) patients; five (71%) of them underwent surgical resection and decompression. Two patients underwent successful catheter-directed fenestration of the web with clinical improvement. We found a statically significant positive correlation between the degree of cord displacement and compression with syrinx formation (r = 0.55 and 0.65 with p-value of 0.03 and 0.009, respectively). Conclusion DAW has characteristic scalpel sign independent of imaging modality. Multi-modality imaging evaluation of DAW is helpful for evaluation and surgical planning.

The automatic segmentation of brain tumors in magnetic resonance (MR) images is very important in the diagnosis, radiotherapy planning, surgical navigation and several other clinical processes. As the location, size, shape, boundary of gliomas are heterogeneous, segmenting gliomas and intratumoral structures is very difficult. Besides, the multi-center issue makes it more challenging that multimodal brain gliomas images (such as T1, T2, fluid-attenuated inversion recovery (FLAIR), and T1c images) are from different radiation centers. This paper presents a multimodal, multi-scale, double-pathway, 3D residual convolution neural network (CNN) for automatic gliomas segmentation. In the pre-processing step, a robust gray-level normalization method is proposed to solve the multi-center problem, that the intensity range from deferent centers varies a lot. Then, a doublepathway 3D architecture based on DeepMedic toolkit is trained using multi-modality information to fuse the local and context features. In the post-processing step, a fully connected conditional random field (CRF) is built to improve the performance, filling and connecting the isolated segmentations and holes. Experiments on the Multimodal Brain Tumor Segmentation (BRATS) 2017 and 2019 dataset showed that this methods can delineate the whole tumor with a Dice coefficient, a sensitivity and a positive predictive value (PPV) of 0.88, 0.89 and 0.88, respectively. As for the segmentation of the tumor core and the enhancing area, the sensitivity reached 0.80. The results indicated that this method can segment gliomas and intratumoral structures from multimodal MR images accurately, and it possesses a clinical practice value.

Background Heart failure with preserved ejection fraction (HFpEF) is thought to be highly prevalent yet remains underdiagnosed. Evidence-based treatments are available that increase quality of life and decrease hospitalization. We sought to develop a data-driven diagnostic model to predict from electronic health records (EHR) the likelihood of HFpEF among patients with unexplained dyspnea and preserved left ventricular EF. Methods and results The derivation cohort comprised patients with dyspnea and echocardiography results. Structured and unstructured data were extracted using an automated informatics pipeline. Patients were retrospectively diagnosed as HFpEF (cases), non-HF (control cohort I), or HF with reduced EF (HFrEF; control cohort II). The ability of clinical parameters and investigations to discriminate cases from controls was evaluated by extreme gradient boosting. A likelihood scoring system was developed and validated in a separate test cohort. The derivation cohort included 1585 consecutive patients: 133 cases of HFpEF (9%), 194 non-HF cases (Control cohort I) and 1258 HFrEF cases (Control cohort II). Two HFpEF diagnostic signatures were derived, comprising symptoms, diagnoses and investigation results. A final prediction model was generated based on the averaged likelihood scores from these two models. In a validation cohort consisting of 269 consecutive patients [with 66 HFpEF cases (24.5%)], the diagnostic power of detecting HFpEF had an AUROC of 90% (P Conclusion This diagnostic signature enables discrimination of HFpEF from non-cardiac dyspnea or HFrEF from EHR and can assist in the diagnostic evaluation in patients with unexplained dyspnea. This approach will enable identification of HFpEF patients who may then benefit from new evidence-based therapies.

Introduction: The purpose of this study was to determine the incidence of radiation induced hypothyroidism after treatment with radiotherapy alone or in combination with surgery/chemotherapy in head & neck cancer patients. Methods: This study was a retrospective non-randomized done on 100 patients of head & neck cancer in whom definitive radiotherapy, postoperative radiotherapy or radiotherapy in combination with chemotherapy was given. Values of TSH, T3 & T4 were analyzed at baseline and at 6 monthly follow up. Subclinical hypothyroidism was defined as TSH value of >4mU/L and Clinical hypothyroidism was taken as TSH >10mU/L with decreased T3 & T4. Results: Out of 100 patients 73 patients were euthyroid at the end of 2 year follow up, 21% had subclinical hypothyroidism and 6% had clinical hypothyroidism. Incidence of subclinical hypothyroidism in surgery plus radiotherapy group and radiotherapy group was 22.3% & 50% respectively. Incidence of clinical hypothyroidism in surgery plus radiotherapy group and radiotherapy group was 6.5% & nil respectively. Conclusion: The incidence of hypothyroidism is high in head & neck cancer patients receiving radiotherapy. The risk is higher in patients who undergo surgery in combination with radiotherapy. Regular thyroid function test is, therefore, recommended.

Neurophysiological signals are crucial intermediaries, through which brain activity can be quantitatively measured and brain mechanisms are able to be revealed. In particular, non‐invasive neurophysiological signals, such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), are welcomed and frequently utilised in various studies since these signals can be non‐invasively recorded without harming the human brain while they convey abundant information pertaining to brain activity. The recorded neurophysiological signals are analysed to mine meaningful information for the understanding of brain mechanisms or are classified to distinguish different patterns (e.g., different cognitive states, brain diseases versus healthy controls). To date, remarkable progress has been made in both the analysis and classification of neurophysiological signals, but scholars are not feeling complacent. Consistent effort ought to be paid to advance the research of analysis and classification based on neurophysiological signals. In this paper, I express my thoughts regarding promising future directions in neurophysiological signal analysis and classification based on current developments and accomplishments. I will elucidate the thoughts after brief summaries of relevant backgrounds, accomplishments, and tendencies. According to my personal selection and preference, I mainly focus on brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment. Hopefully, my thoughts could give a little help to inspire new ideas and contribute to the research of the analysis and classification of neurophysiological signals in some way.

This essay departs from a stand-up performance by comic and critic Hannah Gadsby to think through divestments of whiteness alongside investments in indigeneity, multi-modality, and a “politics of possibility” for the future of media studies.

Cardiac amyloidosis is a heterogeneous and challenging diagnostic disease with poor prognosis that is now being altered by introduction of new therapies. Echocardiography remains the first-line imaging tool, and when disease is suspected on echocardiography, cardiac magnetic resonance imaging and nuclear imaging play critical roles in the non-invasive diagnosis and evaluation of cardiac amyloidosis. Advances in multi-modality cardiac imaging allowing earlier diagnosis and initiation of novel therapies have significantly improved the outcomes in these patients. Cardiac imaging also plays important roles in the risk stratification of patients presenting with cardiac amyloidosis. In the current review, we provide a clinical and imaging focused update, and importantly outline the imaging protocols, diagnostic and prognostic utility of multimodality cardiac imaging in the assessment of cardiac amyloidosis.

Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manual segmenting of NPC is time-consuming and labor-intensive. In addition, single-modality MRI generally cannot provide enough information for accurate delineation. Therefore, a novel multi-modality MRI fusion network (MMFNet) is proposed to complete accurate segmentation of NPC via utilizing T1, T2 and contrast-enhanced T1 of MRI. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders and one decoder, where the encoders aim to capture modality-specific features and the decoder is to obtain fused features for NPC segmentation. A fusion block consisting of a 3D Convolutional Block Attention Module (3D-CBAM) and a residual fusion block (RFBlock) is presented. The 3D-CBAM recalibrates low-level features captured from modality-specific encoders to highlight both informative features and regions of interest (ROIs) whereas the RFBlock fuses re-weighted features to keep balance between fused ones and high-level features from decoder. Moreover, a training strategy named self-transfer is also proposed which utilizes pre-trained modality-specific encoders to initialize multi-encoder-based network in order to make full mining of individual information from multi-modality MRI. The proposed method based on multi-modality MRI can effectively segment NPC and its advantages are validated by extensive experiments.

Radiomics has great prospects in terms of tumour grading, diagnosis and prediction of prognosis by analysing multifaceted data from sources such as clinical treatments, medical images, and pathology. However, exploring an effective way to manage miscellaneous clinical information, as well as to select an appropriate classifier for prediction modelling, is still demanding in a practical clinical context. In this study, we propose a multi-criterion decision-making (MCDM) based classifier fusion (MCF) strategy to combine different classifiers within an MCDM framework. A hierarchical predictive scheme (H-MCF) based on the proposed MCF is also investigated to reliably link the multi-modality features and multi-classifiers. Ten public UCI datasets and two clinical datasets were used to validate the proposed MCF and H-MCF. The experimental results showed that H-MCF has superior predictive performance when compared with the traditional fusion strategies and other fusion architectures, thus demonstrating the feasibility of the proposed H-MCF in integrating information from features of diversified modalities and different classifiers.

Recent deep learning methods rely on a large amount of labeled data to achieve high performance. These methods may be impractical in some scenarios, where manual data annotation is costly or the samples of certain categories are scarce (e.g., tumor lesions, endangered animals and rare individual activities). When only limited annotated samples are available, these methods usually suffer from the overfitting problem severely, which degrades the performance significantly. In contrast, humans can recognize the objects in the images rapidly and correctly with their prior knowledge after exposed to only a few annotated samples. To simulate the learning schema of humans and relieve the reliance on the large-scale annotation benchmarks, researchers start shifting towards the few-shot learning problem: they try to learn a model to correctly recognize novel categories with only a few annotated samples.

In this talk, we present our experiences and applications of large-scale multi-modality pretrained models, developed at Alibaba and Ant Group. We first present a cross-modal pretraining method called M6 (Multi-Modality to Multi-Modality Multitask Mega-transformer) [1], for unified pretraining on the data of multiple modalities. We scale the model size up to 1 trillion parameters [2], and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation [3], and show that the finetuned M6 can create high-quality images with high resolution and fidelity. We also present research and applications of image editing with pretrained Generative Adversarial Networks (GANs). A general principle between the underlying manifold and the generator is discovered. Based on our discovery, we propose an algorithm for GANs with low-rank factorization [4], which can be harnessed for image editing with pretrained GAN models.

Background/Introduction Multi-modality imaging is crucial for confirming diagnosis and assessing prognosis in patients with hypertrophic cardiomyopathy (HCM). However, inter-modality discrepancies in key parameters are commonly reported. Purpose To assess real-world inter-modal reporting discrepancies between transthoracic echocardiography (TTE) and cardiac magnetic resonance (CMR) imaging in the measurement of four key parameters in HCM patients. Methods Consecutive HCM patients with TTE and CMR performed within 6 months of each other at a tertiary centre were retrospectively assessed for reported maximum wall thickness (MWT), left atrial diameter (LAd), left ventricular ejection fraction (LVEF) and presence of left ventricular apical aneurysm (LVAA). The CMR report was considered gold standard. Data are reported as mean ± standard deviation (SD) or median and interquartile range (IQR) as appropriate. Results 353 consecutive HCM patients (72% male, median age 60.9 years, IQR 49.8–71.6 years) with TTE and CMR within 6 months (median difference 1.7 months, IQR 1.1–3.4 months) were assessed between 4th January 2018 and 9th April 2019. Of 284 patients with paired MWT data, median difference was 0.0 mm (IQR −1.0 to 3.0 mm, p=0.02), likely representing a difference in distributions of MWT. TTE both over and underestimated MWT (in 36% and 46% cases respectively). Of the 94 patients with paired LAd data, mean difference was 0.4±5.7 mm (95% CI −0.8010 to 1.546, p=0.5). N=320 patients with paired LVEF data (after excluding patients with atrial fibrillation (n=20)). Median difference in LVEF was 12% (IQR 5–19% p LVAA was accurately identified by TTE in only 9/30 (30%) of those patients with demonstrable LVAA by CMR (p=0.0008). TTE evidence of a discreet apical chamber (paradoxical jet on spectral or colour Doppler) was present in 16/21 (76%) cases where TTE failed to overtly identify LVAA. However, apical or mid-cavity obliteration was reported in 15/21 (71%) cases where TTE failed to identify LVAA. Conclusion(s) Echocardiography and CMR measurements are often used interchangeably in clinical practice but inter-modality discrepancies can affect diagnosis and sudden cardiac death (SCD) risk assessment. This is particularly important for binary risk factors such as LVEF Funding Acknowledgement Type of funding sources: None.

Purpose: This study was performed to explore the value of multi-modality technology, with a combination of narrow acquisition window, isocentric scanning, low tube voltage, low tube current and iterative reconstruction (IR), for reducing the radiation dose in multi-slice spiral computed tomography coronary angiography (MSCTCA). Materials and methods: In this prospective randomised controlled study, 154 patients with coronary heart disease (CHD) were classified according to body mass index (BMI) as normal weight (BMI 18–27kg/m2) or overweight (BMI ≥ 27 kg/m2), and divided into four groups: multi-modality–normal BMI group (A1, n = 82); multi-modality–overweight group (B1, n = 17); conventional–normal BMI group (A2, n = 39); and conventional–overweight group (B2, n = 16). The parameters in the multi-modality groups were as follows: isocentric scan, tube voltage = 80 kV, tube current control using 80% “smart milliampere”, and maximum current during 60–80% of the RR interval. The parameters in the conventional groups were as follows: normal position, tube voltage = 100 kV, tube current control using smart milliampere, and maximum current during 30–80% of the RR interval. The effective radiation dose (ED), objective image quality (IQ), noise, signal-to-noise ratio (SNR), contrast signal-to-noise ratio (CNR) and subjective 5-point Likert scale IQ scores of MSCTCA images were compared among the four groups. Results: The average EDs of groups A1, A2, B1and B2 were( 1.13±0.35 ) mSv, ( 3.36±1.30 ) mSv, ( 1.54±0.53 ) mSv and ( 5.90±0.93 ) mSv, respectively. There were statistically significant differences in ED between groups A1 and A2, and between groups B1 and B2 (all P 2 than group A1 (all P 1 and B2 (P = 0.14–0.51). The average IQ scores of groups A1, A2, B1and B2 were 4.46±0.59(Fig.3), 4.45±0.62(Fig.4), 4.39±0.68(Fig.5) and 4.42±0.66(Fig.6),respectively. There were no significant differences in subjective IQ scores among the four groups (P = 0.12). Consistency among observers in the subjective IQ scores of the four groups was very good, with intraclass correlation coefficients (ICCs) of 0.71–0.90. The subjective IQ scores of the coronary artery were excellent in all four groups, with a total good-to-excellent rate of ≥ 92.64%, and the total number of evaluable segments in the images of all four groups was ≥ 98.26%. Conclusions: Under conditions appropriate for clinical diagnosis, multi-modality technology can reduce the radiation dose of MSCTCA scans in both normal weight and overweight patients.

Simple Summary The lymphatic system is an essential component of the human circulatory system that plays a critical role in antigen presentation, mounting immune reactions, gastrointestinal tract lipid absorption, and maintenance of interstitial homeostasis. This complex network of specially adapted vessels and lymphoid organs also represents major pathway for cancer spread. Knowledge of lymphatic anatomy, physiology, and expected imaging appearances is crucial in understanding the pattern of cancer spread, with great implications for treatment and management. In this review article, we discuss lymphatic anatomy, physiology, imaging techniques, and radiographic appearances of cancer spread with relevant illustrative cases. Abstract The lymphatic system is an anatomically complex vascular network that is responsible for interstitial fluid homeostasis, transport of large interstitial particles and cells, immunity, and lipid absorption in the gastrointestinal tract. This network of specially adapted vessels and lymphoid tissue provides a major pathway for metastatic spread. Many malignancies produce vascular endothelial factors that induce tumoral and peritumoral lymphangiogenesis, increasing the likelihood for lymphatic spread. Radiologic evaluation for disease staging is the cornerstone of oncologic patient treatment and management. Multiple imaging modalities are available to access both local and distant metastasis. In this manuscript, we review the anatomy, physiology, and imaging of the lymphatic system.

Multimodal pretraining has demonstrated success in the downstream tasks of cross-modal representation learning. However, it is limited to the English data, and there is still a lack of large-scale dataset for multimodal pretraining in Chinese. In this work, we propose the largest dataset for pretraining in Chinese, which consists of over 1.9TB images and 292GB texts. The dataset has large coverage over domains, including encyclopedia, question answering, forum discussion, etc. Besides, we propose a method called M6, referring to Multi-Modality-to-Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. The model is pretrained with our proposed tasks, including text-to-text transfer, image-to-text transfer, as well as multi-modality-to-text transfer. The tasks endow the model with strong capability of understanding and generation. We scale the model to 10 billion parameters, and build the largest pretrained model in Chinese. Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities, and the 10B-parameter pretrained model demonstrates strong potential in the setting of zero-shot learning.

Locoregional recurrence (LRR) remains one of leading causes in head and neck (H&N) cancer treatment failure despite the advancement of multidisciplinary management. Accurately predicting LRR in early stage can help physicians make an optimal personalized treatment strategy. In this study, we propose an end-to-end multi-modality and multi-view convolutional neural network model (mMmV-CNN) for LRR prediction in H&N cancer. In mMmV, a dimension reduction operator is designed, projecting the 3D volume onto 2D images in different directions, and a multi-view strategy is used to replace the original 3D method, which reduces the complexity of the algorithm while preserving important 3D information. Meanwhile, multi-modal data is used for the classification by making full use of the complementary information from cross modality data. Furthermore, we design a multi-modality deep neural network which is trained in an end-to-end manner and jointly optimize the deep features of CT, PET and clinical features. A H&N dataset which consists of 206 patients was used to evaluate the performance. Experimental results demonstrated that mMm V-CNN can obtain an AUC value of 0.81 and outperform a state of the art CNN-based method.

State-of-the-art multi-stream methods for image manipulation detection suffer from gaps between features. Moreover, the rapid development of GANs makes it an emerging method of image tampering. However, existing natural scene image tampered datasets are limited to manual tampering. In this paper, we address these two issues. Firstly, we propose a novel two-stream multi-modality image manipulation detection model (MM-net) that abandons the way of fusion. The main idea of the proposed model is to exploit one stream to guide the learning of the other stream through attention mechanism. Therefore, our approach enjoys the benefits of the multi-stream methods while avoiding the semantic gaps caused by bridging gaps between different streams. Secondly, we build the first tampered dataset of natural images based on GANs, pushing manipulation detection toward more realistic and challenging scenarios. Extensive experimental results demonstrate that our model outperforms state-of-the-art approaches on both manually and GANs tampered images.

Background Widespread abnormalities of the myocardium have been reported in patients with COVID-19. However, these patients often have substantial co-morbidities and it is essential to understand whether cardiac abnormalities represent pre-existing disease or are the consequence of COVID-19. Objective To determine the contribution and cardiac impact of co-morbidities in patients who have recovered from COVID-19. Methods In a prospective observational study, adult patients hospitalized with confirmed COVID-19 were recruited from the Edinburgh Heart Centre between May and November 2020 and compared to healthy and co-morbidity-matched volunteers. Patients underwent gadolinium and manganese-enhanced magnetic resonance imaging and coronary computed tomography angiography. Results Twenty-three patients (54±11 years, 20 male) who recovered from COVID-19 were recruited. Half (n=11, 48%) required admission to the intensive care unit and a third (n=7, 31%) received non-invasive or invasive ventilation. Patients had a high prevalence of known cardiovascular disease (n=18, 78%), associated risk factors (n=11, 45%) and coronary artery disease (n=8, 35%). Compared with younger healthy volunteers (n=10), myocardial native T1 values (1202±25 versus 1162±27 ms, P=0.008, figure 1) and extracellular volume fraction (31.9±1.7 versus 29.8±0.5 %, P=0.001, figure 1) were higher with no differences in manganese uptake. Compared to co-morbidity-matched volunteers (n=20), there were no differences in native T1 values (1202±25 versus 1196±39 ms, P=0.61, figure 1), extracellular volume fraction (31.9±1.7 versus 31.0±0.5 %, P=0.11), presence of late gadolinium enhancement or manganese uptake. These findings remained irrespective of COVID-19 disease severity, presence of concomitant myocardial injury or coronary artery disease. Conclusions Patients who have recovered following hospitalization with COVID-19 have no evidence of a major excess in myocardial injury or dysfunction compared to co-morbidity-matched volunteers. The presence of co-morbidities likely explains many of the previously reported myocardial abnormalities. Conflict of Interest None

Background Several Arab countries as well as many parts of the world are currently involved in armed conflicts. Characterization and documentation of combat-related injury patterns and their impact on healthcare are a difficult challenge. However, it is crucial in planning and developing of strategies capable of addressing the demands of ill-equipped medical facilities. The aim of this study was to record the different patterns of combat-related injuries sustained by civilians transferred to Egyptian tertiary hospitals for further investigation and definite treatment after primary stabilization by an emergency trauma team in their countries. Results Metallic foreign bodies were detected in different body locations in 49.1% of patients, while bullets were identified in 12.3%. Injuries involving the musculoskeletal system of the extremities were the most frequent (77.2%), followed in descending order by maxillofacial injuries (41%), thoracic injuries (32.1%), cranial injuries (31%), abdominal injuries (21.1%), spinal injuries (8.8%), and vascular injuries (4.7%). Among the extremities injuries, bone fractures were the most common (68.5%). Conclusion Radiological assessment aids in the evaluation of patients coming from conflict zones pre- and postoperatively, to recognize the precise sites and types of their injuries, the number and location of metallic shrapnel or bullets, and detection of possible associated complications, which aid in accurate demonstration of the extent and patterns of combat-related injuries and guide the management plan.

Takotsubo cardiomyopathy (TC) is an acute but reversible non-ischemic heart failure syndrome. It is characterized by a transient form of ventricular dysfunction typically manifesting as basal hyperkinesis with hypokinesia and ballooning of left ventricle mid-cavity and apex. Imaging helps in both diagnosis and follow up. Echocardiogram is the first-line modality to assess the typical contractile dysfunction in suspected patients with catheter angiography showing normal coronary arteries. Cardiac MRI is currently the modality of choice for the non-invasive initial assessment of TC and for follow up imaging. The current review focusses on historical background of TC, its pathophysiology, diagnostic work up and differential diagnosis and provides multimodality imaging work up of TC including role of echocardiogram, invasive catheterization, nuclear imaging, cardiac computed tomography and cardiac MRI including basic and advanced MRI sequences.

Hereditary transthyretin amyloidosis (ATTRv) gives rise to varying patterns of multi-organ amyloid infiltration, characteristically with cardiomyopathic and/or neuropathic phenotypes. Splenic amylo...

IntroductionFaecal transplantation is an evidence based treatment forClostridiodes difficile. Patients infected with SARS-CoV-2 have been shown to shed the virus in stool for up to 33 days, well beyond the average clearance time for upper respiratory tract shedding. We carried out an analytical and clinical validation of reverse-transcriptase quantitative (RT-qPCR) as well as LAMP, LamPORE and droplet digital PCR in the detection of SARS-CoV-2 RNA in stool from donated samples for FMT, spiked samples and asymptomatic inpatients in an acute surgical unit.MethodsKilled SARS-CoV-2 viral lysate and extracted RNA was spiked into donor stool & FMT and a linear dilution series from 10−1to 10−5and tested via RT-qPCR, LAMP, LamPORE and ddPCR against SARS-CoV-2. Patients admitted to the critical care unit with symptomatic SARS-CoV-2 and sequential asymptomatic patients from acute presentation to an acute surgical unit were also tested.ResultsIn a linear dilution series, detection of the lowest dilution series was found to be 8 copies per microlitre of sample. Spiked lysate samples down to 10−2dilution were detected in FMT samples using RTQPCR, LamPORE and ddPCR and down to 10−1with LAMP. In symptomatic patients 5/12 had detectable SARS-CoV-2 in stool via RT-qPCR and 6/12 via LamPORE, and in 1/97 asymptomatic patients via RT-qPCR.ConclusionsRT-qPCR can be detected in FMT donor samples using RT-qPCR, LamPORE and ddPCR to low levels using validated pathways. As previously demonstrated, nearly half of symptomatic and less than one percent of asymptomatic patients had detectable SARS-CoV-2 in stool.