Search

Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 The Author(s). Published by Elsevier B.V. All rights reserved.)

Purpose: CT is routinely used to detect cranial abnormalities in pediatric patients with head trauma or craniosynostosis. This study aimed to develop a deep learning method to synthesize pseudo-CT (pCT) images for MR high-resolution pediatric cranial bone imaging to eliminating ionizing radiation from CT., Methods: 3D golden-angle stack-of-stars MRI were obtained from 44 pediatric participants. Two patch-based residual UNets were trained using paired MR and CT patches randomly selected from the whole head (NetWH) or in the vicinity of bone, fractures/sutures, or air (NetBA) to synthesize pCT. A third residual UNet was trained to generate a binary brain mask using only MRI. The pCT images from NetWH (pCT NetWH ) in the brain area and NetBA (pCT NetBA ) in the nonbrain area were combined to generate pCT Com . A manual processing method using inverted MR images was also employed for comparison., Results: pCT Com (68.01 ± 14.83 HU) had significantly smaller mean absolute errors (MAEs) than pCT NetWH (82.58 ± 16.98 HU, P < 0.0001) and pCT NetBA (91.32 ± 17.2 HU, P < 0.0001) in the whole head. Within cranial bone, the MAE of pCT Com (227.92 ± 46.88 HU) was significantly lower than pCT NetWH (287.85 ± 59.46 HU, P < 0.0001) but similar to pCT NetBA (230.20 ± 46.17 HU). Dice similarity coefficient of the segmented bone was significantly higher in pCT Com (0.90 ± 0.02) than in pCT NetWH (0.86 ± 0.04, P < 0.0001), pCT NetBA (0.88 ± 0.03, P < 0.0001), and inverted MR (0.71 ± 0.09, P < 0.0001). Dice similarity coefficient from pCT Com demonstrated significantly reduced age dependence than inverted MRI. Furthermore, pCT Com provided excellent suture and fracture visibility comparable to CT., Conclusion: MR high-resolution pediatric cranial bone imaging may facilitate the clinical translation of a radiation-free MR cranial bone imaging method for pediatric patients., (© 2022 International Society for Magnetic Resonance in Medicine.)

Attenuation correction remains a challenge in pelvic PET/MRI. In addition to the segmentation/model-based approaches, deep learning methods have shown promise in synthesizing accurate pelvic attenuation maps (μ-maps). However, these methods often misclassify air pockets in the digestive tract, potentially introducing bias in the reconstructed PET images. The aims of this work were to develop deep learning-based methods to automatically segment air pockets and generate pseudo-CT images from CAIPIRINHA-accelerated MR Dixon images. Methods: A convolutional neural network (CNN) was trained to segment air pockets using 3-dimensional CAIPIRINHA-accelerated MR Dixon datasets from 35 subjects and was evaluated against semiautomated segmentations. A separate CNN was trained to synthesize pseudo-CT μ-maps from the Dixon images. Its accuracy was evaluated by comparing the deep learning-, model-, and CT-based μ-maps using data from 30 of the subjects. Finally, the impact of different μ-maps and air pocket segmentation methods on the PET quantification was investigated. Results: Air pockets segmented using the CNN agreed well with semiautomated segmentations, with a mean Dice similarity coefficient of 0.75. The volumetric similarity score between 2 segmentations was 0.85 ± 0.14. The mean absolute relative changes with respect to the CT-based μ-maps were 2.6% and 5.1% in the whole pelvis for the deep learning-based and model-based μ-maps, respectively. The average relative change between PET images reconstructed with deep learning-based and CT-based μ-maps was 2.6%. Conclusion: We developed a deep learning-based method to automatically segment air pockets from CAIPIRINHA-accelerated Dixon images, with accuracy comparable to that of semiautomatic segmentations. The μ-maps synthesized using a deep learning-based method from CAIPIRINHA-accelerated Dixon images were more accurate than those generated with the model-based approach available on integrated PET/MRI scanners., (© 2022 by the Society of Nuclear Medicine and Molecular Imaging.)

Recently,deep learning (DL)-based methods for the generation of synthetic computed tomography (sCT) have received significant research attention as an alternative to classical ones. We present here a systematic review of these methods by grouping them into three categories, according to their clinical applications: (i) to replace computed tomography in magnetic resonance (MR) based treatment planning, (ii) facilitate cone-beam computed tomography based image-guided adaptive radiotherapy, and (iii) derive attenuation maps for the correction of positron emission tomography. Appropriate database searching was performed on journal articles published between January 2014 and December 2020. The DL methods' key characteristics were extracted from each eligible study, and a comprehensive comparison among network architectures and metrics was reported. A detailed review of each category was given, highlighting essential contributions, identifying specific challenges, and summarizing the achievements. Lastly, the statistics of all the cited works from various aspects were analyzed, revealing the popularity and future trends and the potential of DL-based sCT generation. The current status of DL-based sCT generation was evaluated, assessing the clinical readiness of the presented methods., (© 2021 The Authors. Medical Physics published by Wiley Periodicals LLC on behalf of American Association of Physicists in Medicine.)

Whole-body attenuation correction (AC) is still challenging in combined PET/MR scanners. We describe Dixon-VIBE Deep Learning (DIVIDE), a deep-learning network that allows synthesizing pelvis pseudo-CT maps based only on the standard Dixon volumetric interpolated breath-hold examination (Dixon-VIBE) images currently acquired for AC in some commercial scanners. Methods: We propose a network that maps between the four 2-dimensional (2D) Dixon MR images (water, fat, in-phase, and out-of-phase) and their corresponding 2D CT image. In contrast to previous methods, we used transposed convolutions to learn the up-sampling parameters, we used whole 2D slices to provide context information, and we pretrained the network with brain images. Twenty-eight datasets obtained from 19 patients who underwent PET/CT and PET/MR examinations were used to evaluate the proposed method. We assessed the accuracy of the μ-maps and reconstructed PET images by performing voxel- and region-based analysis comparing the SUVs (in g/mL) obtained after AC using the Dixon-VIBE (PET Dixon ), DIVIDE (PET DIVIDE ), and CT-based (PET CT ) methods. Additionally, the bias in quantification was estimated in synthetic lesions defined in the prostate, rectum, pelvis, and spine. Results: Absolute mean relative change values relative to CT AC were lower than 2% on average for the DIVIDE method in every region of interest except for bone tissue, where it was lower than 4% and 6.75 times smaller than the relative change of the Dixon method. There was an excellent voxel-by-voxel correlation between PET CT and PET DIVIDE ( R 2 = 0.9998, P < 0.01). The Bland-Altman plot between PET CT and PET DIVIDE showed that the average of the differences and the variability were lower (mean PET CT -PET DIVIDE SUV, 0.0003; PET CT -PET DIVIDE SD, 0.0094; 95% confidence interval, [-0.0180,0.0188]) than the average of differences between PET CT and PET Dixon (mean PET CT -PET Dixon SUV, 0.0006; PET CT -PET Dixon The DIVIDE method can accurately synthesize a pelvis pseudo-CT scan from standard Dixon-VIBE images, allowing for accurate AC in combined PET/MR scanners. Additionally, our implementation allows rapid pseudo-CT synthesis, making it suitable for routine applications and even allowing retrospective processing of Dixon-VIBE data.Conclusion: The DIVIDE method can accurately synthesize a pelvis pseudo-CT scan from standard Dixon-VIBE images, allowing for accurate AC in combined PET/MR scanners. Additionally, our implementation allows rapid pseudo-CT synthesis, making it suitable for routine applications and even allowing retrospective processing of Dixon-VIBE data., (© 2019 by the Society of Nuclear Medicine and Molecular Imaging.)

In the field of knee imaging, the incorporation of MR-based pseudo-CT synthesis holds the potential to mitigate the need for separate CT scans, simplifying workflows, enhancing patient comfort, and reducing radiation exposure. In this work, we present a novel DL framework, grounded in the development of the Cycle-Consistent Generative Adversarial Network (CycleGAN) method, tailored specifically for the synthesis of pseudo-CT images in knee imaging to surmount the limitations of current methods. Upon visually examining the outcomes, it is evident that the synthesized pseudo-CTs show an excellent quality and high robustness. Despite the limited dataset employed, the method is able to capture the particularities of the bone contours in the resulting image. The experimental Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Zero-Normalized Cross Correlation (ZNCC), Mutual Information (MI), Relative Change (RC), and absolute Relative Change (|RC|) report values of 30.4638 ± 7.4770, 28.1168 ± 1.5245, 0.9230 ± 0.0217, 0.9807 ± 0.0071, 0.8548 ± 0.1019, 0.0055 ± 0.0265, and 0.0302 ± 0.0218 (median ± median absolute deviation), respectively. The voxel-by-voxel correlation plot shows an excellent correlation between pseudo-CT and ground-truth CT Hounsfield units (m = 0.9785; adjusted R 2 = 0.9988; ρ = 0.9849; p < 0.001). The Bland–Altman plot shows that the average of the differences is low (( H U C T − H U p s e u d o − C T = 0.7199 ± 35.2490; 95% confidence interval [−68.3681, 69.8079]). This study represents the first reported effort in the field of MR-based knee pseudo-CT synthesis, shedding light to significantly advance the field of knee imaging. [ABSTRACT FROM AUTHOR]

In the field of knee imaging, the incorporation of MR-based pseudo-CT synthesis holds the potential to mitigate the need for separate CT scans, simplifying workflows, enhancing patient comfort, and reducing radiation exposure. In this work, we present a novel DL framework, grounded in the development of the Cycle-Consistent Generative Adversarial Network (CycleGAN) method, tailored specifically for the synthesis of pseudo-CT images in knee imaging to surmount the limitations of current methods. Upon visually examining the outcomes, it is evident that the synthesized pseudo-CTs show an excellent quality and high robustness. Despite the limited dataset employed, the method is able to capture the particularities of the bone contours in the resulting image. The experimental Mean Absolute Error (MAE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Zero-Normalized Cross Correlation (ZNCC), Mutual Information (MI), Relative Change (RC), and absolute Relative Change (|RC|) report values of 30.4638 ± 7.4770, 28.1168 ± 1.5245, 0.9230 ± 0.0217, 0.9807 ± 0.0071, 0.8548 ± 0.1019, 0.0055 ± 0.0265, and 0.0302 ± 0.0218 (median ± median absolute deviation), respectively. The voxel-by-voxel correlation plot shows an excellent correlation between pseudo-CT and ground-truth CT Hounsfield units (m = 0.9785; adjusted R2 = 0.9988; ρ = 0.9849; p < 0.001). The Bland–Altman plot shows that the average of the differences is low ((HUCT−HUpseudo−CT = 0.7199 ± 35.2490; 95% confidence interval [−68.3681, 69.8079]). This study represents the first reported effort in the field of MR-based knee pseudo-CT synthesis, shedding light to significantly advance the field of knee imaging.

Despite being thirteen years since the installation of the first PET-MR system, the scanners constitute a very small proportion of the total hybrid PET systems installed. This is in stark contrast to the rapid expansion of the PET-CT scanner, which quickly established its importance in patient diagnosis within a similar timeframe. One of the main hurdles is the development of an accurate, reproducible and easy-to-use method for attenuation correction. Quantitative discrepancies in PET images between the manufacturer-provided MR methods and the more established CT- or transmission-based attenuation correction methods have led the scientific community in a continuous effort to develop a robust and accurate alternative. These can be divided into four broad categories: (i) MR-based, (ii) emission-based, (iii) atlas-based and the (iv) machine learning-based attenuation correction, which is rapidly gaining momentum. The first is based on segmenting the MR images in various tissues and allocating a predefined attenuation coefficient for each tissue. Emission-based attenuation correction methods aim in utilising the PET emission data by simultaneously reconstructing the radioactivity distribution and the attenuation image. Atlas-based attenuation correction methods aim to predict a CT or transmission image given an MR image of a new patient, by using databases containing CT or transmission images from the general population. Finally, in machine learning methods, a model that could predict the required image given the acquired MR or non-attenuation-corrected PET image is developed by exploiting the underlying features of the images. Deep learning methods are the dominant approach in this category. Compared to the more traditional machine learning, which uses structured data for building a model, deep learning makes direct use of the acquired images to identify underlying features. This up-to-date review goes through the literature of attenuation correction approaches in PET-MR after categorising them. The various approaches in each category are described and discussed. After exploring each category separately, a general overview is given of the current status and potential future approaches along with a comparison of the four outlined categories. [ABSTRACT FROM AUTHOR]

Introduction: The five-class Dixon-based PET/MR attenuation correction (AC) model, which adds bone information to the four-class model by registering major bones from a bone atlas, has been shown to be error-prone. In this study, we introduce a novel method of accounting for bone in pelvic PET/MR AC by directly predicting the errors in the PET image space caused by the lack of bone in four-class Dixon-based attenuation correction. Methods: A convolutional neural network was trained to predict the four-class AC error map relative to CT-based attenuation correction. Dixon MR images and the four-class attenuation correction µ-map were used as input to the models. CT and PET/MR examinations for 22 patients ([18F]FDG) were used for training and validation, and 17 patients were used for testing (6 [18F]PSMA-1007 and 11 [68Ga]Ga-PSMA-11). A quantitative analysis of PSMA uptake using voxel- and lesion-based error metrics was used to assess performance. Results: In the voxel-based analysis, the proposed model reduced the median root mean squared percentage error from 12.1% and 8.6% for the four- and fiveclass Dixon-based AC methods, respectively, to 6.2%. The median absolute percentage error in the maximum standardized uptake value (SUVmax) in bone lesions improved from 20.0% and 7.0% for four- and five-class Dixon-based AC methods to 3.8%. Conclusion: The proposed method reduces the voxel-based error and SUVmax errors in bone lesions when compared to the four- and five-class Dixon-based AC models. [ABSTRACT FROM AUTHOR]

This paper provides an overview of the different deep convolutional neural network (DCNNs) architectures that have been investigated in the past years for the generation of synthetic computed tomography (CT) or pseudo-CT from magnetic resonance (MR). The U-net, the Atrous-net and the Residual-net architectures were analyzed, implemented and compared. Each network was implemented using 2D filters and 3D filters with 2D slices and 3D patches respectively as inputs. Two datasets were used for training and evaluation. The first one is composed by pairs of 3D T1-weighted MR and Low-dose CT images from the head of 19 healthy women. The second database contains dual echo Dixon-VIBE MR images and CT images from the pelvis of 13 colorectal and 6 prostate cancer patients. Bone structures in the target anatomy were key in choosing the right deep learning approach. This work provides a deep explanation of the architectures in order to know which DCNN fits better each medical application. According to this study, the 3D U-net architecture would be the best option to generate head pseudo-CTs while the 2D Residual-net provides the most accurate results for the pelvis anatomy. [ABSTRACT FROM AUTHOR]

This paper provides an overview of the different deep convolutional neural network (DCNNs) architectures that have been investigated in the past years for the generation of synthetic computed tomography (CT) or pseudo-CT from magnetic resonance (MR). The U-net, the Atrous-net and the Residual-net architectures were analyzed, implemented and compared. Each network was implemented using 2D filters and 3D filters with 2D slices and 3D patches respectively as inputs. Two datasets were used for training and evaluation. The first one is composed by pairs of 3D T1-weighted MR and Low-dose CT images from the head of 19 healthy women. The second database contains dual echo Dixon-VIBE MR images and CT images from the pelvis of 13 colorectal and 6 prostate cancer patients. Bone structures in the target anatomy were key in choosing the right deep learning approach. This work provides a deep explanation of the architectures in order to know which DCNN fits better each medical application. According to this study, the 3D U-net architecture would be the best option to generate head pseudo-CTs while the 2D Residual-net provides the most accurate results for the pelvis anatomy.

Featured Application: The Franken-CT approach allows synthesizing pseudo-CT images from using diverse anatomical overlapping MR-CT datasets as a potential application in PET/MR attenuation correction. Typically, pseudo-Computerized Tomography (CT) synthesis schemes proposed in the literature rely on complete atlases acquired with the same field of view (FOV) as the input volume. However, clinical CTs are usually acquired in a reduced FOV to decrease patient ionization. In this work, we present the Franken-CT approach, showing how the use of a non-parametric atlas composed of diverse anatomical overlapping Magnetic Resonance (MR)-CT scans and deep learning methods based on the U-net architecture enable synthesizing extended head and neck pseudo-CTs. Visual inspection of the results shows the high quality of the pseudo-CT and the robustness of the method, which is able to capture the details of the bone contours despite synthesizing the resulting image from knowledge obtained from images acquired with a completely different FOV. The experimental Zero-Normalized Cross-Correlation (ZNCC) reports 0.9367 ± 0.0138 (mean ± SD) and 95% confidence interval (0.9221, 0.9512); the experimental Mean Absolute Error (MAE) reports 73.9149 ± 9.2101 HU and 95% confidence interval (66.3383, 81.4915); the Structural Similarity Index Measure (SSIM) reports 0.9943 ± 0.0009 and 95% confidence interval (0.9935, 0.9951); and the experimental Dice coefficient for bone tissue reports 0.7051 ± 0.1126 and 95% confidence interval (0.6125, 0.7977). The voxel-by-voxel correlation plot shows an excellent correlation between pseudo-CT and ground-truth CT Hounsfield Units (m = 0.87; adjusted R2 = 0.91; p < 0.001). The Bland–Altman plot shows that the average of the differences is low (−38.6471 ± 199.6100; 95% CI (−429.8827, 352.5884)). This work serves as a proof of concept to demonstrate the great potential of deep learning methods for pseudo-CT synthesis and their great potential using real clinical datasets. [ABSTRACT FROM AUTHOR]

Magnetic resonance (MR) images have distinctive advantages in radiation treatment (RT) planning due to their superior, anatomic and functional information compared with computed tomography (CT). For the RT dose calculation, MR images cannot be directly used because of the lack of electron density information. To address this issue, we propose to generate pseudo-CT (pCT) in terms of multiple matching Dixon MR images to support MR-only RT, particularly in the challenging body section of the abdomen. To this end, we design the dedicated multichannel residual conditional generative adversarial network (MCRCGAN). The significance of our efforts is three-fold: 1) The MCRCGAN organically incorporates multiple theories and techniques, such as multichannel residual network (ResNet) and conditional generative adversarial network (cGAN), which facilitate its more authentic pCT generation than many existing methods. 2) The usage of residual modules effectively deepens the network without performance degradation, and the multichannel ResNet helps to simultaneously capture the substance of images, as extensively as possible, which is implicitly contained in the multiple different MR images of the same subject. 3) Due to the designed dedicated network structure, the MCRCGAN is capable of generating satisfactory pCTs under the condition of limited training data as well as prompt prediction response. Experimental studies on ten patients' paired MR-CT images demonstrate the effectiveness of our proposed MCRCGAN model on both the pCT generation quality and the performance stability.

Purpose: To develop and evaluate a novel method for pseudo‐CT generation from multi‐parametric MR images using multi‐channel multi‐path generative adversarial network (MCMP‐GAN). Methods: Pre‐ and post‐contrast T1‐weighted (T1‐w), T2‐weighted (T2‐w) MRI, and treatment planning CT images of 32 nasopharyngeal carcinoma (NPC) patients were employed to train a pixel‐to‐pixel MCMP‐GAN. The network was developed based on a 5‐level Residual U‐Net (ResU‐Net) with the channel‐based independent feature extraction network to generate pseudo‐CT images from multi‐parametric MR images. The discriminator with five convolutional layers was added to distinguish between the real CT and pseudo‐CT images, improving the nonlinearity and prediction accuracy of the model. Eightfold cross validation was implemented to validate the proposed MCMP‐GAN. The pseudo‐CT images were evaluated against the corresponding planning CT images based on mean absolute error (MAE), peak signal‐to‐noise ratio (PSNR), Dice similarity coefficient (DSC), and Structural similarity index (SSIM). Similar comparisons were also performed against the multi‐channel single‐path GAN (MCSP‐GAN), the single‐channel single‐path GAN (SCSP‐GAN). Results: It took approximately 20 h to train the MCMP‐GAN model on a Quadro P6000, and less than 10 s to generate all pseudo‐CT images for the subjects in the test set. The average head MAE between pseudo‐CT and planning CT was 75.7 ± 14.6 Hounsfield Units (HU) for MCMP‐GAN, significantly (P‐values < 0.05) lower than that for MCSP‐GAN (79.2 ± 13.0 HU) and SCSP‐GAN (85.8 ± 14.3 HU). For bone only, the MCMP‐GAN yielded a smaller mean MAE (194.6 ± 38.9 HU) than MCSP‐GAN (203.7 ± 33.1 HU), SCSP‐GAN (227.0 ± 36.7 HU). The average PSNR of MCMP‐GAN (29.1 ± 1.6) was found to be higher than that of MCSP‐GAN (28.8 ± 1.2) and SCSP‐GAN (28.2 ± 1.3). In terms of metrics for image similarity, MCMP‐GAN achieved the highest SSIM (0.92 ± 0.02) but did not show significantly improved bone DSC results in comparison with MCSP‐GAN. Conclusions: We developed a novel multi‐channel GAN approach for generating pseudo‐CT from multi‐parametric MR images. Our preliminary results in NPC patients showed that the MCMP‐GAN method performed apparently superior to the U‐Net‐GAN and SCSP‐GAN, and slightly better than MCSP‐GAN. [ABSTRACT FROM AUTHOR]

Typically, pseudo-Computerized Tomography (CT) synthesis schemes proposed in the literature rely on complete atlases acquired with the same field of view (FOV) as the input volume. However, clinical CTs are usually acquired in a reduced FOV to decrease patient ionization. In this work, we present the Franken-CT approach, showing how the use of a non-parametric atlas composed of diverse anatomical overlapping Magnetic Resonance (MR)-CT scans and deep learning methods based on the U-net architecture enable synthesizing extended head and neck pseudo-CTs. Visual inspection of the results shows the high quality of the pseudo-CT and the robustness of the method, which is able to capture the details of the bone contours despite synthesizing the resulting image from knowledge obtained from images acquired with a completely different FOV. The experimental Zero-Normalized Cross-Correlation (ZNCC) reports 0.9367 ± 0.0138 (mean ± SD) and 95% confidence interval (0.9221, 0.9512); the experimental Mean Absolute Error (MAE) reports 73.9149 ± 9.2101 HU and 95% confidence interval (66.3383, 81.4915); the Structural Similarity Index Measure (SSIM) reports 0.9943 ± 0.0009 and 95% confidence interval (0.9935, 0.9951); and the experimental Dice coefficient for bone tissue reports 0.7051 ± 0.1126 and 95% confidence interval (0.6125, 0.7977). The voxel-by-voxel correlation plot shows an excellent correlation between pseudo-CT and ground-truth CT Hounsfield Units (m = 0.87; adjusted R2 = 0.91; p < 0.001). The Bland–Altman plot shows that the average of the differences is low (−38.6471 ± 199.6100; 95% CI (−429.8827, 352.5884)). This work serves as a proof of concept to demonstrate the great potential of deep learning methods for pseudo-CT synthesis and their great potential using real clinical datasets.