Search

ResumoA evolução na avaliação por imagens dos sarcomas musculoesqueléticos contribuiu para melhora significativa no prognóstico e na sobrevida dos portadores destas neoplasias. A caracterização precisa destas lesões, mediante utilização das modalidades de imagem mais adequadas a cada condição clínica apresentada, é de suma importância no delineamento da abordagem terapêutica a ser instituída, com impacto direto sobre os desfechos clínicos. O presente artigo busca atualizar o leitor a propósito das metodologias de imagem no contexto da avaliação local e sistêmica dos sarcomas ósseos e das partes moles.

В огляді літератури розглянуто історичні аспекти та ключові питання клінічного застосування позитронно-емісійної томографії (ПЕТ)/комп’ютерної томографії як нового методу гібридної візуалізації. Ця оглядова робота аналізує проблеми, пов’язані з процесом злиття ПET з магнітно-резонансною томографією (МРТ) і розвитком даних систем протягом багатьох років. У роботі досліджуються діагностичні аспекти мультимодального зображення з корекцією ослаблення та методи реконструкції з метою отримання кількісного зображення. Розвиток ПЕТ/МРТ-візуалізації відіграє важливу роль у галузі нейрохірургії, онкології, опор-но-рухового апарату тощо, таким чином, може розглядатись як сучасний мультимодальний діагностичний метод додаткової гібридної візуалізації. Однак питання діагностичного застосування ПЕТ/МРТ-візуалізації і надалі потребує досліджень щодо інших можливих способів впровадження ПЕТ/МРТ для нових доклінічних і клінічних випробувань.

Multimodal medical imaging is an indispensable requirement in the treatment of various pathologies to accelerate care. Rather than discrete images, a composite image combining complementary features from multimodal images is highly informative for clinical examinations, surgical planning, and progress monitoring. In this paper, a deep learning fusion model is proposed for the fusion of medical multimodal images. Based on pyramidal and residual learning units, the proposed model, strengthened with adaptive fusion rules, is tested on image pairs from a standard dataset. The potential of the proposed model for enhanced image exams is shown by fusion studies with deep network images and quantitative output metrics of magnetic resonance imaging and positron emission tomography (MRI/PET) and magnetic resonance imaging and single-photon emission computed tomography (MRI/SPECT). The proposed fusion model achieves the Structural Similarity Index Measure (SSIM) values of 0.9502 and 0.8103 for the MRI/SPECT and MRI/PET MRI/SPECT image sets, signifying the perceptual visual consistency of the fused images. Testing is performed on 20 pairs of MRI/SPECT and MRI/PET images. Similarly, the Mutual Information (MI) values of 2.7455 and 2.7776 obtained for the MRI/SPECT and MRI/PET image sets, indicating the model’s ability to capture the information content from the source images to the composite image. Further, the proposed model allows deploying its variants, introducing refinements on the basic model suitable for the fusion of low and high-resolution medical images of diverse modalities.

Multimodal image registration is an important technique for many clinical applications. However, it is particularly challenging to obtain good spatial alignment. This paper introduces a novel archi...

Introduction: Retained subretinal perfluoro-n-octane droplets occur in nearly 1% of the procedures. When subretinal cystoid cavities are registered, differential diagnosis including this condition is needed. Clinical case: A 77-year-old male presented three small bubbles of perfluoro-n-octane under the papillomacular bundle after vitrectomy due to a retina redetachment, evidenced by multimodal image. Conclusions: Multimodal image can help to diagnose this condition. There are several signs that can support this diagnosis. If visual acuity is threatened, removing the bubbles is mandatory, accepting risks.

We present a case of a woman with idiopathic macular telangiectasia type 1, characterized by the presence of a capillary macroanaeurysm associated with macular edema with a macular star pattern. Several differential diagnoses were proposed, in which the study with a multimodal image was key to clarify the diagnosis. In addition, the various treatments used and their effectiveness are discussed.

It is well known that there exist great differences in human brain anatomy and functions between males and females. With the development of noninvasive neuroimaging techniques, the sex differences in adult human brain have been well studied in some researches. However, the sexual dimorphism of human brain anatomy and functions has not been sufficiently described during the developmental period of early childhood brain when the sex differences emerge in behavior. This study was to identify specific patterns of the sexual dimorphism in developing brain structure of early childhood using multimodal brain image analysis. We have performed a multivariate and data-driven analysis by combining multiple neuroimaging technologies including the 3D T1-weighted structural MR images (sMRI) and diffusion tensor imaging (DTI) in a prospective cohort of 188 children (128 males and 60 females) between the ages of 0 and 15. First, the brain images were segmented into 90 regions of interest (ROIs) based on the AAL template to extract the ROI volume and connectivity features. Then, the individual multimodal imaging biomarkers were identified associated with the sex differences. Finally, the selected features from multi-modality neuroimages were combined using multi-kernel support vector machine for classifications of male or female. The method achieved sex classification accuracy 72% for the children between the ages of 0 and 15. And the volumes of right precuneus and right postcentral were more related to sexual dimorphism of developing brain structure than those of other regions in sMRI (p

The purpose of this paper is to serve as a template for greater understanding for the practicing radiologist about key steps to perform multimodality computer analysis of MRI images, specifically in multiple sclerosis patients. With this understanding, radiologists will be better equipped about how best to process and analyze MRI imaging data and obtain accurate quantitative information for MS patient evaluation. A secondary intent of this article is to improve radiologist understanding of how artificial intelligence will be employed in the future for better patient stratification, and for evaluation of response to therapy in both clinical care and drug trials.

The case is reported of a 73-year-old male presenting with the recently described peripapillary pachychoroid syndrome. This syndrome is included in the pachychoroid diseases, which is a relatively new condition characterised by increased choroidal thickness with attenuation of the choriocapillary layer located above dilated choroidal vessels. It is associated with progressive malfunction of the retinal pigment epithelium and neovascularisation. The case is presented with a multimodal image, in which some tests show signs not previously described. In addition, a description is presented of the asymmetric outcome of both eyes after multiple treatments, since the left eye has a very severe affectation, and is refractory to the treatment.

Aim To classify polypoidal choroidal vasculopathy (PCV) into 2 subtypes based on the subfoveal choroidal thickness (SFCT) and to further evaluate their multimodal image features. Methods A retrospective observational case series study. Sixty-four eyes of 64 patients with PCV were enrolled and classified into 2 groups based on SFCT (thick-choroid group/thin-choroid group). Then further analyze the spectrum domain optical coherence tomography (SD-OCT) and indocyanine green angiography (ICGA) differences of the two subtypes. Imaging analysis included measurement of SFCT, maximum vascular diameter ratio (MVDR), choroidal vascularity index (CVI), central macular thickness (CMT), and the presence of pigment epithelial detachment (PED) on SD-OCT. Polypoidal lesions (polyps) number, branching vascular network (BVN) area, greatest linear dimension (GLD), and the choroidal vascular hyperpermeability (CVH) were analyzed by ICGA. Results The distribution of SFCT was bimodal with two peaks at 195 and 285 µm, and a trough at 225 µm. The 225 µm was taken as the cutoff point for the following classification of thick/thin choroid groups. The PCV eyes in the thick-choroid group presented with greater MVDR, CVI within 3 and 6 mm of the fovea, but lower CMT, less PED, small PED diameters on SD-OCT scans, and fewer polyps, smaller BVN and GLD, but more frequency of CVH on ICGA. Conclusion The SFCT at 225 µm can be used as a readily available indicator for the classification of PCV subtypes. The thick-choroid group presents much apparent enlargement of the choroidal layer and vasculature expansion, which indicates different pathogenesis of the two subtypes.

Medical robots have demonstrated the ability to manipulate percutaneous instruments into soft tissue anatomy while working beyond the limits of human perception and dexterity. Robotic technologies further offer the promise of autonomy in carrying out critical tasks with minimal supervision when resources are limited. Here, we present a portable robotic device capable of introducing needles and catheters into deformable tissues such as blood vessels to draw blood or deliver fluids autonomously. Robotic cannulation is driven by predictions from a series of deep convolutional neural networks that encode spatiotemporal information from multimodal image sequences to guide real-time servoing. We demonstrate, through imaging and robotic tracking studies in volunteers, the ability of the device to segment, classify, localize and track peripheral vessels in the presence of anatomical variability and motion. We then evaluate robotic performance in phantom and animal models of difficult vascular access and show that the device can improve success rates and procedure times compared to manual cannulations by trained operators, particularly in challenging physiological conditions. These results suggest the potential for autonomous systems to outperform humans on complex visuomotor tasks, and demonstrate a step in the translation of such capabilities into clinical use. Getting safe and fast access to blood vessels is vital to many methods of treatment and diagnosis in medicine. Robot-assisted or even fully autonomous methods can potentially do the task more reliably than humans, especially when veins are hard to detect. In this work, a method is tested that uses deep learning to find blood vessels and track the movement of a patient’s arm.

Objective To address the feasibility and predictive value of multimodal image-based 3D reconstruction in assessing features of neurovascular relationship particularly regarding the detection of offending vessels, degree of compression exerted on the nerve root, in patients who underwent microvascular decompression for primary trigeminal neuralgia. Methods This study includes 58 patients who underwent microvascular decompression for primary trigeminal neuralgia. All patients underwent preoperative 3.0-T magnetic resonance imaging (MRI) scan of 3D-TOF MRA and 3D-FIESTA sequences in combination. Multimodal MRIs were then registered with open-source software 3D Slicer, followed by 3D image reconstruction to get 3D models for detection of possible neurovascular relationship in the cerebellopontine angle (CPA). Evaluations were performed by 2 doctors and compared with the intraoperative findings and surgery video. Results For detection of offending vessels, multimodal image-based 3D reconstruction sensitivity was 93.1% (54/58). Compared with the intraoperative findings, the k coefficients for evaluating the offending vessel and the degree of compression were >0.70 (P Conclusions Multimodal image-based 3D reconstruction proved to be reliable and practical in detecting offending vessel and in predicting the degree of nerve compression. The 3D anatomical model is consistent with the structural position found during surgery, which means that the method is valuable in the formulation of preoperative plans.

3D histology, slice-based connectivity atlases, and diffusion MRI are common techniques to map brain wiring. While there are many modality-specific tools to process these data, there is a lack of integration across modalities. We develop an automated resource that combines histologically cleared volumes with connectivity atlases and MRI, enabling the analysis of histological features across multiple fiber tracts and networks, and their correlation with in-vivo biomarkers. We apply our pipeline in a murine stroke model, demonstrating not only strong correspondence between MRI abnormalities and CLARITY-tissue staining, but also uncovering acute cellular effects in areas connected to the ischemic core. We provide improved maps of connectivity by quantifying projection terminals from CLARITY viral injections, and integrate diffusion MRI with CLARITY viral tracing to compare connectivity maps across scales. Finally, we demonstrate tract-level histological changes of stroke through this multimodal integration. This resource can propel investigations of network alterations underlying neurological disorders., Many approaches exist to process data from individual imaging modalities, but integrating them is challenging. The authors develop an automated resource that enables co-registered network- and tract-level analysis of macroscopic in-vivo imaging and microscopic imaging of cleared tissue.

International audience; Hierarchical data representations are powerful tools to analyze images and have found numerous applications in image processing. When it comes to multimodal images however, the fusion of multiple hierarchies remains an open question. Recently, the concept of braids of partitions has been proposed as a theoretical tool and possible solution to this issue. In this paper, we demonstrate the relevance of the braid structure for the hierarchical representation of multimodal images. We first propose a fully operable procedure to build a braid of partitions from two hierarchical representations. We then derive a framework for multimodal image segmentation, relying on an energetic minimization scheme conducted on the braid structure. The proposed approach is investigated on different multimodal images scenarios, and the obtained results confirm its ability to efficiently handle the multimodal information to produce more accurate segmentation outputs.

Background The automatic detection of common subregions and registration in multimodal functional and structural images is challenging. This article gives an overview of multimodal image registration and the developments and technical issues with automatic detection and registration of subregions of interest in multimodal images. Discussion The available knowledge about subregion detection and registration in multimodal images are described in detail. Besides the provision of compact knowledge on subregion detection and registration, the challenges and proposed solutions are also discussed. Conclusion This article provides research guidelines for the development of automatic detection and registration of subregions of interest in functional and structural images with high accuracy and efficiency.

We propose a new two-stage joint image reconstruction method by recovering edges directly from observed data and then assembling an image using the recovered edges. More specifically, we reformulat...

Insights from multiple imaging modalities have recently been applied in solving many computer vision related applications. In this paper, we model the cross-modal dependencies between different modalities for Multimodal Image Super-Resolution (MISR), i.e., enhance the Low Resolution (LR) image of target modality with the guidance of a High Resolution (HR) image from another modality. We introduce a joint optimization based transform learning frame-work referred to as Joint Coupled Transform Learning (JCTL) to combine the information from multiple modalities to generate the HR image of the target modality. All the necessary intermediate steps and the corresponding closed form solution updates are pro-vided. The performance of the proposed JCTL is benchmarked against the state-of-the-art MISR approaches on different multi-modal datasets with different upscaling factors. The results show better performance with the proposed JCTL approach compared to other state-of-the-art techniques both in terms of PSNR and SSIM.

International audience

Clinical detection of premalignant lesions often requires labeling with molecular probes. In the gastrointestinal tract, most cancers originate in epithelial layers which are interrogated using topically-applied probes with conjugated fluorescence dyes. As part of clinical trials using fluorescence peptide probes, we have developed two different instruments that rapidly image the biopsy at the bedside before fixation. These multimodal images (visible light reflectance and near infrared fluorescence) provide verification that the targeted lesion was sampled, and provide feedback to the clinician at the bedside for any follow-up procedure. Performance of these two prototypes are compared for fluorescence sensitivity and multimodal image quality.

An efficient and automated image analysis pipeline is essential for extracting quantitative information from multimodal image datasets. In this study, a multimodal optical imaging platform was used to capture CARS, 2PF, and FLIM images from control and drug-treated cells. Images were collected using both fluorescent label-based and label-free approaches. Here we present a single-cell analysis pipeline for the multimodal cellular image analysis. The results demonstrate the capability of our single-cell analysis pipeline for quantitatively measuring the intracellular drug distribution and its longitudinal uptake using a multimodal optical imaging platform, which can provide novel insights into the uptake pathways and target-sites.

The purpose of this study was to present an innovative approach for the preoperative assessment and intraoperative targeted excision of masses occupying the intraorbital space based on multimodal image fusion, segmentation, virtual models, digital planning, and navigation. Nineteen patients were studied and underwent surgery using the presented workflow, in both open and endoscopic procedures. Three main scenarios were standardized for the application of computer-guided surgery: single masses of the superior-lateral compartment, single masses of the inferior-medial compartment, and multifocal masses. An operative protocol was devised, and the accuracy of the osteotomies was analysed. All patients were managed successfully by applying the same protocol. No intraoperative complications were reported. The accuracy of the osteotomies was evaluated as a surrogate endpoint for the overall precision of surgery, showing average discrepancies of1 mm for lateral marginotomies and0.5 mm for endoscopic osteotomies. This study outlines an operative workflow for the implementation of virtual models to excise orbital masses, enhancing in-depth preoperative understanding of the anatomical relationships within the orbital space and increasing precision in both open and endoscopic approaches.

Salp Swarm Algorithm (SSA) is a relatively new metaheuristic optimization technique inspired by the swarming behavior of salps in oceans. Though used successfully in various engineering optimization problems, the use of SSA in image registration is yet to be explored. This work tests the effectiveness of SSA algorithm for multimodal image registration applications. The proposed algorithm is tested with different sets of infrared and visible images and its convergence rate is compared with the Particle Swarm Optimization (PSO) algorithm, frequently used in image registration. Results show that the Salp Swarm algorithm can effectively optimize the objective function and can provide accurate image registration results.

Across all stages of drug discovery and development, experimental assays are performed to understand the effect of a drug or drug candidate—at the molecular, cellular, organ, or organism level. Imaging is a key technology in this process, and an imaging assay consisting of sample preparation, image acquisition, and image analysis provides a quantitative readout of a system. Historically, chemical assays have been the workhorse of early discovery, screening millions of compounds for a simple endpoint, whereas imaging was primarily used in lower throughput mechanistic studies. However, with development of high-throughput high-content microscopy platforms, the throughput of imaging assays now rivals chemical screens. Similarly, innovations in image analysis mean that robust quantitative conclusions can be derived from complex and multimodal image data, driving informed decision making later in the drug development process. For these reasons, imaging is widely used throughout the pharmaceutical industry and throughout multiple stages of the drug discovery process.

Multimodal image fusion combines the complementary information of multimodality images into a single image that preserves the information of all the source images. This paper proposes a multimodal image fusion method situated on image enhancement, edge detection, multiscale sliding window, and image matting to obtain the detailed region information of the input images. In the proposed system, firstly the multimodality input images are rectified via a contrast enhancement method through which the intensity distribution is refined for clear vision. The spatial gradient edge detection method is utilized for separating the edge information from the contrast-enhanced images. These edges are then used by a multiscale sliding window method to provide global and local activity level maps. These activity maps further generate trimap and decision maps. Finally, by employing the improved decision maps and fusion rule the fused image is acquired.

Recognition of key morphological features in histological slides is crucial for pathological diagnosis and monitoring therapeutic progress. However, the typical routine microscopic workflow is conducted by hand which is time-consuming and has unavoidable intra- and inter-observer variability like all human work. Therefore, we propose a bifocal classification and fusion network for the automated recognition and cross-modality analysis of diagnostic features in whole-slide multimodal images (WSIs). In brief, paired image tiles cropped from digitized tissue sections were fed into a modified dual-path CNN which accepts asymmetric inputs for classification, and then the inference results were converted to feature distribution heatmaps, which permit qualitative as well as quantitative morphological analyses of entire histological sections, even in combination with adjacent sections that have been stained differently. The multimodal heatmaps were aligned using image registration and fused for cross-modality analysis. Our experiments showed that the network achieved high recognition performance (AUCs of 0.985 and 0.988, and accuracies of 94.7% and 96.1% on two WSI modalities, respectively, against expert markings) and outperformed state-of-the-art methods without training on a large cohort or utilizing domain transfer. In addition, the new method involves a self-contained inference and fusion process and thus harbors significant potential for speeding up microscopic analysis workflows.

The detection and classification of tumors like Glioblastoma is largely dependent on the imaging techniques like Computed Tomography and Magnetic Resonance Imaging and its variants. Although these techniques are widely accepted in representing different characteristics of the brain, they face challenges starting from acquisition to analysis, leading to delay. These challenges make it difficult to detect or classify the different grades of tumors at an early stage. Also, a single imaging technique is never enough and fails to detect the tumor in some cases. In order to overcome these challenges, a framework is proposed to combine these images and generate a single image that is visually enhanced and assist in early detection of Glioblastoma. The framework commences with preprocessing the CT and MR images, registering the images, fusing the multi-modal images into a single image, followed by enhancement and segmentation.

Deformable image registration between Computed Tomography (CT) images and Magnetic Resonance (MR) imaging is essential for many image-guided therapies. In this paper, we propose a novel translation-based unsupervised deformable image registration method. Distinct from other translation-based methods that attempt to convert the multimodal problem (e.g., CT-to-MR) into a unimodal problem (e.g., MR-to-MR) via image-to-image translation, our method leverages the deformation fields estimated from both: (i) the translated MR image and (ii) the original CT image in a dual-stream fashion, and automatically learns how to fuse them to achieve better registration performance. The multimodal registration network can be effectively trained by computationally efficient similarity metrics without any ground-truth deformation. Our method has been evaluated on two clinical datasets and demonstrates promising results compared to state-of-the-art traditional and learning-based methods.

We present a new multimodal image-to-image translation model for the generation of gastritis images using X-ray and blood inspection results. In clinical situations, clinicians estimate the prognosis of the target disease by considering multiple inspection results. Similarly, we take a multimodal approach in the task of gastric cancer risk prediction. Visual characteristics of the gastric X-ray image and blood index values are highly related in the evaluation of gastric cancer risk. If we can generate a prediction image from blood index values, it contributes to the clinicians’ sophisticated and integrated diagnosis. Hence, we learn a model that can map non-gastritis images to gastritis images based on the blood index values. Although this is a challenging multimodal task in medical image analysis, experimental results showed the effectiveness of our model.

Images captured by different sensors with different spectral bands cause non-linear intensity changes between image pairs. Classic feature descriptors cannot handle this problem and are prone to yielding unsatisfactory results. Inspired by the illumination and contrast invariant properties of phase congruency, here, we propose a new descriptor to tackle this problem. The proposed descriptor generation mainly involves three steps. (1) Images are convolved with a bank of log-Gabor filters with different scales and orientations. (2) A window of fixed size is selected and divided into several blocks for each keypoint, and an oriented magnitude histogram and the orientation of the minimum moment of a phase congruency-based histogram are calculated in each block. (3) These two histograms are normalized respectively and concatenated to form the proposed descriptor. Performance evaluation experiments on three datasets were carried out to validate the superiority of the proposed method. Experimental results indicated that the proposed descriptor outperformed most of the classic and state-of-art descriptors in terms of precision and recall within an acceptable computational time.

Background Analytical methods of brain research involving across-voxel correlation between multimodal images are currently tedious and slow due to the amount of manual interaction required. We have developed a new software package to automate and simplify many of these tasks. New method and results Our software performs four primary functions to aid in research. First, it helps with consistent renaming of files preprocessed with other software, enabling more accurate analysis. Second, it automates ROI extraction using data from existing and custom brain atlases. Third, it performs coupling analysis to obtain across-voxel Pearson correlation coefficients between images of different modalities based on these brain atlases or custom ROIs. Fourth, it automatically performs multiple comparison correction to correct the P-value using two false discovery rate (FDR) methods and a Bonferroni method to reduce the false-positive rate. Comparison with existing methods Previous researchers have investigated the couplings between blood supply and brain functional topology in healthy brains and those from patients with type 2 diabetes, chronic migraine, and schizophrenia. These studies conducted analyses of both the whole and parts of the brain in terms of neuronal activity and blood perfusion, but the procedures were laborious and time-consuming. Conclusion We have developed a convenient and time-saving software package using MATLAB 2014a to automate the data preparation and analysis of across-voxel coupling between multimodal images.

Many tasks in computer vision and graphics fall within the framework of conditional image synthesis. In recent years, generative adversarial nets (GANs) have delivered impressive advances in quality of synthesized images. However, it remains a challenge to generate both diverse and plausible images for the same input, due to the problem of mode collapse. In this paper, we develop a new generic multimodal conditional image synthesis method based on Implicit Maximum Likelihood Estimation (IMLE) and demonstrate improved multimodal image synthesis performance on two tasks, single image super-resolution and image synthesis from scene layouts. We make our implementation publicly available., To appear in International Journal of Computer Vision (IJCV). arXiv admin note: text overlap with arXiv:1811.12373

Objective To introduce the application of robot-assisted frameless stereoelectroencephalography (SEEG) based on multimodal image fusion technology in pediatric refractory epilepsy in a pediatric center from a developing country. Methods We retrospectively evaluated pediatric patients with drug-resistant epilepsy who underwent SEEG monitoring at the Children's Hospital of Fudan University from July 2014 to August 2017. Application of multimodal image fusion technology in SEEG was described in detail. Seizure outcomes were assessed according to the International League Against Epilepsy classification. Results A total of 208 patients were initially eligible and underwent a rigorous phase I evaluation. SEEG explorations were performed in 20 patients who entered phase II assessment (11 male and 9 female patients) with a median age of 7.99 ± 4.07 years. In total, 181 electrodes were implanted (9 per implantation), among which 16 implantations were unilateral (6 left and 10 right) and 4 were bilateral. The mean operating time was 3 hours and no obvious hemorrhage occurred. Electrode displacement and pneumocephalus were observed in 1 and 2 patients, respectively. Thirteen and 7 patients underwent tailored resection and radiofrequency thermocoagulation, respectively. Among resection cases, focal cortical dysplasia was the predominant pathologic type. The overall seizure outcome after a mean follow-up of 2.65 years was International League Against Epilepsy class 1 in 13, class 2 in 2, class 3 in 3, class 4 in 1, and class 5 in 1 patient, respectively. Conclusions The combination of multimodal image fusion and frameless robot-assisted SEEG is demonstrated to be safe and effective on children with refractory epilepsy in developing countries.

Objective To develop a revised evaluation method for accuracy of multimodal image fusion for oral and maxillofacial tumors and explore its application for comparing the accuracy of three commonly used fusion algorithms, automatic fusion, manual fusion, and registration point–based fusion. Materials and methods Image sets of patients with oral and maxillofacial tumor were fused using the iPlan 3.0 navigation system. Fusion accuracy included two aspects: (1) overall fusion accuracy: represented by the mean value of the coordinate differences along the x-, y-, and z- axes (Δx, Δy, and Δz), mean deviation (MD), and root mean square (RMS) of six pairs of landmarks on the two image sets; (2) tumor volume fusion accuracy: represented by Fusion Index (FI), which was calculated based on the volume of tumor delineated on the two image sets. Results Eighteen pairs of image sets of 17 patients were enrolled in this study. The Δx and Δy values for the three algorithms were less than 1.5 mm. The Δz values for automatic fusion, manual fusion and registration point–based fusion was 1.049 mm, 1.864 mm and 1.254 mm. The MD for automatic fusion, manual fusion and registration point–based fusion was 1.978 mm, 2.788 mm and 1.926 mm. Significant differences existed in Δz for manual fusion and that for automatic fusion (P = 0.058), in MD for manual fusion and that for automatic fusion (P = 0.087), and in MD for manual fusion and that for registration point–based fusion (P = 0.069). The FI for automatic fusion, manual fusion, and registration point–based fusion was 0.594, 0.520, and 0.549; the inter-algorithm differences were not significant (P = 0.290). Conclusion The automatic fusion and the registration point–based fusion were more accurate than manual fusion, and therefore were recommended to be used in multimodal image fusion for oral and maxillofacial tumors.

Background Cerebrovascular reactivity (CVR) is the change in cerebral blood flow in response to a vaso-active stimulus, and may assist the treatment strategy of ischemic stroke. However, previous studies reported that a therapeutic strategy for stroke mainly depends on the degree of vascular stenosis with steady-state vascular parameters (e.g., cerebral blood flow and CVR). Hence, measurement of CVR by multimodal imaging techniques may improve the treatment of ischemic stroke. Methods/design This is a prospective, randomized, controlled clinical trial that aimed to examine the capability of multimodal imaging techniques for the evaluation of CVR to improve treatment of patients with ischemic stroke. A total of 66 eligible patients will be recruited from Renji Hospital, Shanghai Jiaotong University School of Medicine. The patients will be categorized based on CVR into two subgroups as follows: CVR > 10% group and CVR Discussion Measurement of cerebrovascular reserve by multimodal image is recommended by most recent studies to guide the treatment of ischemic stroke, and thus its efficacy and evaluation accuracy need to be established in randomized controlled settings. This prospective, parallel, randomized, controlled registry study, together with other ongoing studies, should present more evidence for optimal individualized accurate treatment of ischemic stroke. Trial registration Chinese Clinical Trial Registry, ID: ChiCTR-IOR-16009635; Registered on 16 October 2016. All items are from the World Health Organization Trial Registration Data Set and registration in the Chinese Clinical Trial Registry: ChiCTR-IOR-16009635.

Multimodal image registration (MIR) is a fundamental procedure in many image-guided therapies. Recently, unsupervised learning-based methods have demonstrated promising performance over accuracy and efficiency in deformable image registration. However, the estimated deformation fields of the existing methods fully rely on the to-be-registered image pair. It is difficult for the networks to be aware of the mismatched boundaries, resulting in unsatisfactory organ boundary alignment. In this paper, we propose a novel multimodal registration framework, which leverages the deformation fields estimated from both: (i) the original to-be-registered image pair, (ii) their corresponding gradient intensity maps, and adaptively fuses them with the proposed gated fusion module. With the help of auxiliary gradient-space guidance, the network can concentrate more on the spatial relationship of the organ boundary. Experimental results on two clinically acquired CT-MRI datasets demonstrate the effectiveness of our proposed approach., Comment: 5 pages, under review

Learning Classifiers are essential analysis tools in todays era due to the rapid expansion in data-intensive applications. Every type of the classifier has own distinct architectural properties which allow to apply them in various application-specific requirements. Here, an experimental work is performed to analyse the performance and accuracy of most commonly used learning classifiers applied within the scope of multimodal image classification problem space. The purpose of this study is to investigate the usefulness of the learning-classifiers with the multimodal datasets which are in many cases have the curse of dimensionality and may contain a very high volume of noise. Validation is conducted over multimodal datasets with the proposed classification algorithms operating in a parallel manner. Finally, results are discussed to explore possible ways to apply every learning classifier in multi-modal image classification applications.

Multimodal registration is a challenging task due to the significant variations exhibited from images of different modalities. CT and MRI are two of the most commonly used medical images in clinical diagnosis, since MRI with multicontrast images, together with CT, can provide complementary auxiliary information. The deformable image registration between MRI and CT is essential to analyze the relationships among different modality images. Here, we proposed an indirect multimodal image registration method, i.e., sCT-guided multimodal image registration and problematic image completion method. In addition, we also designed a deep learning-based generative network, Conditional Auto-Encoder Generative Adversarial Network, called CAE-GAN, combining the idea of VAE and GAN under a conditional process to tackle the problem of synthetic CT (sCT) synthesis. Our main contributions in this work can be summarized into three aspects: (1) We designed a new generative network called CAE-GAN, which incorporates the advantages of two popular image synthesis methods, i.e., VAE and GAN, and produced high-quality synthetic images with limited training data. (2) We utilized the sCT generated from multicontrast MRI as an intermediary to transform multimodal MRI-CT registration into monomodal sCT-CT registration, which greatly reduces the registration difficulty. (3) Using normal CT as guidance and reference, we repaired the abnormal MRI while registering the MRI to the normal CT.

In medical imaging it is common practice to acquire a wide range of modalities (MRI, CT, PET, etc.), to highlight different structures or pathologies. As patient movement between scans or scanning session is unavoidable, registration is often an essential step before any subsequent image analysis. In this paper, we introduce a cost function based on joint total variation for such multimodal image registration. This cost function has the advantage of enabling principled, groupwise alignment of multiple images, whilst being insensitive to strong intensity non-uniformities. We evaluate our algorithm on rigidly aligning both simulated and real 3D brain scans. This validation shows robustness to strong intensity non-uniformities and low registration errors for CT/PET to MRI alignment. Our implementation is publicly available at https://github.com/brudfors/coregistration-njtv.

Multimodal medical image fusion, which aims at integrating different multimodal information into a single output, plays an important role in the clinical applicability of medical images such as noninvasive diagnosis and image-guided surgery. The main motivation of this study is to model the coefficient selection step of medical image fusion as a pattern recognition task. The proposed method first decomposes the source images by the tetrolet transform. Subsequently, different activity measures are used to extract salient features from patches of the tetrolet subbands. The features are then fed to the sparse un-mixing by variable splitting and augmented Lagrangian (SUnSAL) classifier. Coefficients to be incorporated into the fused image are chosen by the classifier. Finally, the cycle-spinning technique is exploited to avoid artifacts. Experimental results on three pairs of medical images validate the reliability and credibility of the proposed method in clinical applications when compared with four state-of-the-art fusion methods. More specifically, the proposed framework does not suffer from contrast reduction, color distortion and loss of fine details.

Prostate cancer (PCa) is one of the most common cancer diseases in men in the western countries [1]. Besides the palpation, and the amount of prostate-specific-antigen’s (PSA) inside the blood, the current diagnostic imaging technologies are not appropriate. Early diagnosis defining the exact tumor location, spread and margins could make efficient targeted biopsies and image-guided surgery. A multimodal imaging technique containing a transmit-receive surface coil for anatomical MR imaging, a (S)PET detector module, consisting of silicon photomultipliers (SiPM), for functional imaging and an ultrasound (US) probe are placed as close as possible to the prostate designed as an endorectal tube to increase sensitivity and spatial resolution. All materials that are used are non-magnetic. Advantages of the SiPM are diversified, like non-sensitive to magnetic fields, higher gain (105–106) than standard avalanche photodiodes (APD), good timing properties and compactness. The PET detector should reach approximately 1mm3 spatial resolution together with 60ps FWHM Time-of-Flight resolution and a high efficiency to reduce scanning time and injected dose. A home-made transmit-receive coil surrounding the PET module improves signal-to-noise-ratio (SNR) with respect to standard coils will be present. The system will be used as a MRI-insert and be able to visualize anatomic and metabolic information together. The US-probe is guiding examination for correct overlapping of the multimodal images. This procedure will save time, costs and the need of co-registration. By combining all advantages of each system, it will necessarily update the non-invasive treatment of PCa. The system is adapted and tested to a 3 Tesla MR scanner called Trio A Tim system and Allegra system from the company Siemens healthcare with a larmor frequency of 123.2 MHz and an input of 50 Ω free from artifacts. First results on homogeneity of the transmit-receive coil will be presented. Preliminary measurements showing the proposed device is challenging but feasible.

Background Interstitial brachytherapy (BT) is becoming an accepted treatment option for head and neck cancer patients for whom surgery poses high risks. Multimodal, image-guided, robotic surgery has the potential to allow precise seed implantation into tumors. Our aim was to introduce a new multimodal, image-guided surgical robot during the performance of interstitial BT for the treatment of tumors in the head and neck regions. Methods Clinical data for three patients were analyzed, retrospectively; patients had received 125 I seed implantations from July 2019 to October 2019. Multimodal, image-guided, robotic surgery was performed in all patients. Postoperative computed tomography data were imported to software to evaluate the accuracy of the seed position and the operation times. Results The mean placement error of the 125 I seed was 1.9 ± 0.74 mm. The mean operation time is 47 minutes. Conclusion The experimental results showed that the Remebot has promise for use during BT for the head and neck.