Search

Artificial intelligence (AI) has great potential to transform the clinical workflow of radiotherapy. Since the introduction of deep neural networks (DNNs), many AI-based methods have been proposed to address challenges in different aspects of radiotherapy. Commercial vendors have started to release AI-based tools that can be readily integrated to the established clinical workflow. To show the recent progress in AI-aided radiotherapy, we have reviewed AI-based studies in five major aspects of radiotherapy, including image reconstruction, image registration, image segmentation, image synthesis, and automatic treatment planning. In each section, we summarized and categorized the recently published methods, followed by a discussion of the challenges, concerns, and future development. Given the rapid development of AI-aided radiotherapy, the efficiency and effectiveness of radiotherapy in the future could be substantially improved through intelligent automation of various aspects of radiotherapy.

Open set domain adaptation (OSDA) methods have been proposed to leverage the difference between the source and target domains, as well as to recognize the known and unknown classes in the target domain. Such methods typically require the entire source and target data simultaneously to train the target model. However, in real scenarios, data are distributed and stored in various clients. They cannot be exchanged among clients because of privacy protection. Federated learning (FL) is a decentralized approach for training an effective global model with the training data distributed among the clients. Despite its potential in addressing the privacy concerns of data sharing, FL methods for OSDA that can handle unknown classes is not yet available. To tackle this problem, we have developed a novel federated OSDA (FOSDA) algorithm. More specifically, FOSDA adopts an uncertainty-aware mechanism to generate a global model from all client models. It reduces the uncertainty of the federated aggregation by focusing on the contribution of source clients with high uncertainty while retaining those with high consistency. Moreover, a federated class-based weighted strategy is also implemented in FOSDA to maintain the category information of the source clients. We have conducted comprehensive experiments on three benchmark datasets to evaluate the performance of the proposed method, and the results demonstrate the effectiveness of FOSDA.

Purpose Cone-beam computed tomography (CBCT) scanning is used daily or weekly (i.e., on-treatment CBCT) for accurate patient setup in image-guided radiotherapy. However, inaccuracy of CT numbers prevents CBCT from performing advanced tasks such as dose calculation and treatment planning. Motivated by the promising performance of deep learning in medical imaging, we propose a deep U-net-based approach that synthesizes CT-like images with accurate numbers from planning CT, while keeping the same anatomical structure as on-treatment CBCT. Methods We formulated the CT synthesis problem under a deep learning framework, where a deep U-net architecture was used to take advantage of the anatomical structure of on-treatment CBCT and image intensity information of planning CT. U-net was chosen because it exploits both global and local features in the image spatial domain, matching our task to suppress global scattering artifacts and local artifacts such as noise in CBCT. To train the synthetic CT generation U-net (sCTU-net), we include on-treatment CBCT and initial planning CT of 37 patients (30 for training, seven for validation) as the input. Additional replanning CT images acquired on the same day as CBCT after deformable registration are utilized as the corresponding reference. To demonstrate the effectiveness of the proposed sCTU-net, we use another seven independent patient cases (560 slices) for testing. Results We quantitatively compared the resulting synthetic CT (sCT) with the original CBCT image using deformed same-day pCT images as reference. The averaged accuracy measured by mean absolute error (MAE) between sCT and reference CT (rCT) on testing data is 18.98 HU, while MAE between CBCT and rCT is 44.38 HU. Conclusions The proposed sCTU-net can synthesize CT-quality images with accurate CT numbers from on-treatment CBCT and planning CT. This potentially enables advanced CBCT applications for adaptive treatment planning.

Diagnosis of lung cancer requires radiologists to review every lung nodule in CT images. Such a process can be very time-consuming, and the accuracy is affected by many factors, such as experience of radiologists and available diagnosis time. To address this problem, we proposed to develop a deep learning-based system to automatically classify benign and malignant lung nodules. The proposed method automatically determines benignity or malignancy given the 3D CT image patch of a lung nodule to assist diagnosis process. Motivated by the fact that real structure among data is often embedded on a low-dimensional manifold, we developed a novel manifold regularized classification deep neural network (MRC-DNN) to perform classification directly based on the manifold representation of lung nodule images. The concise manifold representation revealing important data structure is expected to benefit the classification, while the manifold regularization enforces strong, but natural constraints on network training, preventing over-fitting. The proposed method achieves accurate manifold learning with reconstruction error of ~ 30 HU on real lung nodule CT image data. In addition, the classification accuracy on testing data is 0.90 with sensitivity of 0.81 and specificity of 0.95, which outperforms state-of-the-art deep learning methods. The proposed MRC-DNN facilitates an accurate manifold learning approach for lung nodule classification based on 3D CT images. More importantly, MRC-DNN suggests a new and effective idea of enforcing regularization for network training, possessing the potential impact to a board range of applications.

Purpose Applicator digitization is one of the most critical steps in 3D high-dose-rate brachytherapy (HDRBT) treatment planning. Motivated by recent advances in deep-learning, we propose a deep-learning–assisted applicator digitization method for 3D CT image-based HDRBT. This study demonstrates its feasibility and potential in gynecological cancer HDRBT. Methods and Materials Our method consisted of two steps. The first step used a U-net to segment applicator regions. We trained the U-net using two-dimensional CT images with a tandem-and-ovoid (T&O) applicator and corresponding applicator mask images. The second step applied a spectral clustering method and a polynomial curve fitting method to extract applicator central paths. We evaluated the accuracy, efficiency, and robustness of our method in different scenarios including other T&O cases that were not used in training, a T&O case scanned with cone-beam CT, and Y-tandem and cylinder-applicator cases. Results In test cases with a T&O applicator, average 3D Dice similarity coefficient between automatic and manual segmented applicator regions was 0.93. Average distance between tip positions and average Hausdorff distance between applicator channels determined by our method and manually were 0.64 mm and 0.68 mm, respectively. Although trained only using CT images of T&O cases, our tool can also digitize Y-tandem, cylinder applicator, and T&O applicator scanned in cone-beam CT with error of tip position and Hausdorff distance Conclusions We have developed a deep-learning–assisted applicator digitization tool for 3D CT image–based HDRBT of gynecological cancer. The achieved accuracy, efficiency, and robustness made our tool clinically attractive.

Purpose External beam radiotherapy combined with interstitial brachytherapy is commonly used to treat patients with bulky, advanced gynecologic cancer. However, the high radiation dose needed to control the tumor may result in fistula development. There is a clinical need to identify patients at high risk for fistula formation such that treatment may be managed to prevent this toxic side effect. This work aims to develop a fistula prediction model framework using machine learning based on patient, tumor, and treatment features. Methods and Materials This retrospective study included 35 patients treated at our institution using interstitial brachytherapy for various gynecological malignancies. Five patients developed rectovaginal fistula and two developed both rectovaginal and vesicovaginal fistula. For each patient, 31 clinical features of multiple data types were collected to develop a fistula prediction framework. A nonlinear support vector machine was used to build the prediction model. Sequential backward feature selection and sequential floating backward feature selection methods were used to determine optimal feature sets. To overcome data imbalance issues, the synthetic minority oversampling technique was used to generate synthetic fistula cases for model training. Results Seven mixed data features were selected by both sequential backward selection and sequential floating backward selection methods. Our prediction model using these features achieved a high prediction accuracy, that is, 0.904 area under the curve, 97.1% sensitivity, and 88.5% specificity. Conclusions A machine-learning–based prediction model of fistula formation has been developed for patients with advanced gynecological malignancies treated using interstitial brachytherapy. This model may be clinically impactful pending refinement and validation in a larger series.

PURPOSE: We have previously proposed an intelligent automatic treatment planning (IATP) framework that builds a virtual treatment planner network (VTPN) to operate a treatment planning system (TPS) to generate high-quality radiation therapy (RT) treatment plans. While the potential of IATP in automating RT treatment planning has been demonstrated, its poor scalability caused by an almost linear growth of network size with the number of treatment planning parameters (TPPs) is a bottleneck, preventing its application in complicate, but clinically relevant treatment planning problems. The decision-making behavior of the trained network is hard to understand. Motivated by the decision-making process of a human planner, this study proposes a hierarchical IATP framework. METHODS AND MATERIALS: The hierarchical VTPN (HieVTPN) consists of three networks, i.e. Structure-Net, Parameter-Net, and Action-Net. When interacting with a TPS, the networks are employed in a sequential order in each step to decide the structure to adjust, the TPP to adjust for the selected structure, and the specific adjustment manner for the parameter, respectively. We developed an end-to-end hierarchical deep reinforcement learning (DRL) scheme to simultaneously train the three networks. We then evaluated the effectiveness of the proposed framework in the treatment planning problems for prostate cancer intensity modulated RT (IMRT) and stereotactic body RT (SBRT). We benchmarked the performance of our approach by comparing plans made by VTPN of a parallel architecture, and the human plans submitted for competition in the 2016 American Association of Medical Dosimetrist (AAMD)/Radiosurgery Society (RSS) Plan Study. We analyzed scalability of the network size with respect to the number of TPPs. Numerical experiments were also performed to understand the rationale of the decision-making behaviors of the trained HieVTPN. RESULTS: Both HieVTPNs for prostate IMRT and SBRT were trained successfully using 10 training patient cases and 5 validation cases. For IMRT, HieVTPN was able to generate high-quality plans for 59 testing patient cases that were not included in training process, achieving an average plan score of 8.62 (±0.83), with 9 being the maximal score. The score was comparable to that of the VTPN, 8.45 (±0.48). For SBRT planning, HieVTPN achieved an average plan score of 139.07 on five testing patient cases compared to the score of 132.21 averaged over the human plans summited for competition in AAMD/RSS plan study. Different from VTPN with network size linearly scaling with the number of TPPs, the network size of HieVTPN is almost independent of the number of TPPs. It was also observed that the decision-making behaviors of HieVTPN were understandable and generally agreed with the human experience. CONCLUSIONS: With the scalability and explainability, the hierarchical IATP framework is more favorable than the previous framework in terms of handling treatment planning problems involving a large number of TPPs.

Robustness is an important aspect when evaluating a method of medical image analysis. In this study, we investigated the robustness of a deep learning (DL)-based lung-nodule classification model for CT images with respect to noise perturbations. A deep neural network (DNN) was established to classify 3D CT images of lung nodules into malignant or benign groups. The established DNN was able to predict malignancy rate of lung nodules based on CT images, achieving the area under the curve of 0.91 for the testing dataset in a tenfold cross validation as compared to radiologists’ prediction. We then evaluated its robustness against noise perturbations. We added to the input CT images noise signals generated randomly or via an optimization scheme using a realistic noise model based on a noise power spectrum for a given mAs level, and monitored the DNN’s output. The results showed that the CT noise was able to affect the prediction results of the established DNN model. With random noise perturbations at 100 mAs, DNN’s predictions for 11.2% of training data and 17.4% of testing data were successfully altered by at least once. The percentage increased to 23.4% and 34.3%, respectively, for optimization-based perturbations. We further evaluated robustness of models with different architectures, parameters, number of output labels, etc, and robustness concern was found in these models to different degrees. To improve model robustness, we empirically proposed an adaptive training scheme. It fine-tuned the DNN model by including perturbations in the training dataset that successfully altered the DNN’s perturbations. The adaptive scheme was repeatedly performed to gradually improve DNN’s robustness. The numbers of perturbations at 100 mAs affecting DNN’s predictions were reduced to 10.8% for training and 21.1% for testing by the adaptive training scheme after two iterations. Our study illustrated that robustness may potentially be a concern for an exemplary DL-based lung-nodule classification model for CT images, indicating the needs for evaluating and ensuring model robustness when developing similar models. The proposed adaptive training scheme may be able to improve model robustness.

Driving pattern recognition based on features, such as GPS, gear, and speed information, is essential to develop intelligent transportation systems. However, it is usually expensive and labor intensive to collect a large amount of labeled driving data from real-world driving scenes. The lack of a labeled data problem in a driving scene substantially hinders the driving pattern recognition accuracy. To handle the scarcity of labeled data, we have developed a novel discriminative transfer learning method for driving pattern recognition to leverage knowledge from related scenes with labeled data to improve recognition performance in unlabeled scenes. Note that data from different scenes may have different distributions, which is a major bottleneck limiting the performance of transfer learning. To address this issue, the proposed method adopts a discriminative distribution matching scheme with the aid of pseudolabels in unlabeled scenes. It is able to reduce the intraclass distribution disagreement for the same driving pattern among labeled and unlabeled scenes while increasing the interclass distance among different patterns. Pseudolabels in unlabeled scenes are updated iteratively via an ensemble strategy that preserves the data structure while enhancing the model robustness. To evaluate the performance of the proposed method, we conducted comprehensive experiments on real-world parking lot datasets. The results show that the proposed method can substantially outperform state-of-the-art methods in driving pattern recognition.

Automatic sigmoid colon segmentation in CT for radiotherapy treatment planning is challenging due to complex organ shape, close distances to other organs, and large variations in size, shape, and filling status. The patient bowel is often not evacuated, and CT contrast enhancement is not used, which further increase problem difficulty. Deep learning (DL) has demonstrated its power in many segmentation problems. However, standard 2-D approaches cannot handle the sigmoid segmentation problem due to incomplete geometry information and 3-D approaches often encounters the challenge of a limited training data size. Motivated by human's behavior that segments the sigmoid slice by slice while considering connectivity between adjacent slices, we proposed an iterative 2.5-D DL approach to solve this problem. We constructed a network that took an axial CT slice, the sigmoid mask in this slice, and an adjacent CT slice to segment as input and output the predicted mask on the adjacent slice. We also considered other organ masks as prior information. We trained the iterative network with 50 patient cases using five-fold cross validation. The trained network was repeatedly applied to generate masks slice by slice. The method achieved average Dice similarity coefficients of 0.82 0.06 and 0.88 0.02 in 10 test cases without and with using prior information.

PURPOSE In the treatment planning process of intensity-modulated radiation therapy (IMRT), a human planner operates the treatment planning system (TPS) to adjust treatment planning parameters, for example, dose volume histogram (DVH) constraints' locations and weights, to achieve a satisfactory plan for each patient. This process is usually time-consuming, and the plan quality depends on planer's experience and available planning time. In this study, we proposed to model the behaviors of human planners in treatment planning by a deep reinforcement learning (DRL)-based virtual treatment planner network (VTPN), such that it can operate the TPS in a human-like manner for treatment planning. METHODS AND MATERIALS Using prostate cancer IMRT as an example, we established the VTPN using a deep neural network developed. We considered an in-house optimization engine with a weighted quadratic objective function. Virtual treatment planner network was designed to observe an intermediate plan DVHs and decide the action to improve the plan by changing weights and threshold dose in the objective function. We trained the VTPN in an end-to-end DRL process in 10 patient cases. A plan score was used to measure plan quality. We demonstrated the feasibility and effectiveness of the trained VTPN in another 64 patient cases. RESULTS Virtual treatment planner network was trained to spontaneously learn how to adjust treatment planning parameters to generate high-quality treatment plans. In the 64 testing cases, with initialized parameters, quality score was 4.97 (±2.02), with 9.0 being the highest possible score. Using VTPN to perform treatment planning improved quality score to 8.44 (±0.48). CONCLUSIONS To our knowledge, this was the first time that intelligent treatment planning behaviors of human planner in external beam IMRT are autonomously encoded in an artificial intelligence system. The trained VTPN is capable of behaving in a human-like way to produce high-quality plans.

As one of the most popular approaches in artificial intelligence, deep learning (DL) has attracted a lot of attention in the medical physics field over the past few years. The goals of this topical review article are twofold. First, we will provide an overview of the method to medical physics researchers interested in DL to help them start the endeavor. Second, we will give in-depth discussions on the DL technology to make researchers aware of its potential challenges and possible solutions. As such, we divide the article into two major parts. The first part introduces general concepts and principles of DL and summarizes major research resources, such as computational tools and databases. The second part discusses challenges faced by DL, present available methods to mitigate some of these challenges, as well as our recommendations.

PURPOSE: Cone beam CT (CBCT) has been widely used in radiation therapy. However, its main application is still to acquire anatomical information for patient positioning. This study proposes a multi-energy element-resolved (MEER) CBCT framework that employs energy-resolved data acquisition on a conventional CBCT platform and then simultaneously reconstructs images of x-ray attenuation coefficients, electron density relative to water (rED) and elemental composition (EC) to support advanced applications. METHODS: The MEER-CBCT framework is realized on a Varian TrueBeam CBCT platform using a kVp-switching scanning scheme. A simultaneous image reconstruction and elemental decomposition model is formulated as an optimization problem. The objective function uses a least square term to enforce fidelity between x-ray attenuation coefficients and projection measurements. Spatial regularization is introduced via sparsity under a tight wavelet-frame transform. Consistency is imposed among rED, EC and attenuation coefficients and inherently serves as a regularization term along the energy direction. The EC is further constrained by a sparse combination of ECs in a dictionary containing tissues commonly existing in humans. The optimization problem is solved by a novel alternating-direction minimization scheme. The MEER-CBCT framework was tested in a simulation study using an NCAT phantom and an experimental study using a Gammex phantom. RESULTS: MEER-CBCT framework was successfully realized on a clinical Varian TrueBeam on-board CBCT platform with three energy channels of 80, 100 and 120 kVp. In the simulation study, the attenuation coefficient image achieved a structural similarity index of 0.98, compared to 0.61 for the image reconstructed by the conventional conjugate gradient least square (CGLS) algorithm, primarily because of reduction of artifacts. In the experimental study, the attenuation image obtained a contrast-to-noise ratio of [Formula: see text] 60, much higher than that of CGLS results (~16) because of noise reduction. The median errors in rED and EC were 0.5% and 1.4% in the simulation study, and 1.4% and 2.3% in the experimental study. CONCLUSION: We proposed a novel MEER-CBCT framework realized on a clinical CBCT platform. Simulation and experimental studies demonstrated its capability to simultaneously reconstruct x-ray attenuation coefficient, rED and EC images accurately.

Objective. Robustness is an important aspect to consider, when developing methods for medical image analysis. This study investigated robustness properties of deep neural networks (DNNs) for a lung nodule classification problem based on CT images and proposed a solution to improve robustness. Approach. We firstly constructed a class of four DNNs with different widths, each predicting an output label (benign or malignant) for an input CT image cube containing a lung nodule. These networks were trained to achieve Area Under the Curve of 0.891–0.914 on a testing dataset. We then added to the input CT image cubes noise signals generated randomly using a realistic CT image noise model based on a noise power spectrum at 100 mAs, and monitored the DNNs output change. We defined SAR 5(%) to quantify the robustness of the trained DNN model, indicating that for 5% of CT image cubes, the noise can change the prediction results with a chance of at least SAR 5(%). To understand robustness, we viewed the information processing pipeline by the DNN as a two-step process, with the first step using all but the last layers to extract representations of the input CT image cubes in a latent space, and the second step employing the last fully-connected layer as a linear classifier to determine the position of the sample representations relative to a decision plane. To improve robustness, we proposed to retrain the last layer of the DNN with a Supporting Vector Machine (SVM) hinge loss function to enforce the desired position of the decision plane. Main results. SAR 5 ranged in 47.0%–62.0% in different DNNs. The unrobustness behavior may be ascribed to the unfavorable placement of the decision plane in the latent representation space, which made some samples be perturbed to across the decision plane and hence susceptible to noise. The DNN-SVM model improved robustness over the DNN model and reduced SAR 5 by 8.8%–21.0%. Significance. This study provided insights about the potential reason for the unrobustness behavior of DNNs and the proposed DNN-SVM model improved model robustness.

Purpose We previously proposed an intelligent automatic treatment planning framework for radiotherapy, in which a virtual treatment planner network (VTPN) is built using deep reinforcement learning (DRL) to operate a treatment planning system (TPS) by adjusting treatment planning parameters in it to generate high-quality plans. We demonstrated the potential feasibility of this idea in prostate cancer intensity-modulated radiation therapy (IMRT). Despite the success, the process to train a VTPN via the standard DRL approach with an e-greedy algorithm was time consuming. The required training time was expected to grow with the complexity of the treatment planning problem, preventing the development of VTPN for more complicated but clinically relevant scenarios. In this study, we proposed a novel knowledge-guided DRL (KgDRL) approach that incorporated knowledge from human planners to guide the training process to improve the efficiency of training a VTPN. Method Using prostate cancer IMRT as a testbed, we first summarized a number of rules in the actions of adjusting treatment planning parameters of our in-house TPS. During the training process of VTPN, in addition to randomly navigating the large state-action space, as in the standard DRL approach using the e-greedy algorithm, we also sampled actions defined by the rules. The priority of sampling actions from rules decreased over the training process to encourage VTPN to explore new policy on parameter adjustment that were not covered by the rules. To test this idea, we trained a VTPN using KgDRL and compared its performance with another VTPN trained using the standard DRL approach. Both networks were trained using 10 training patient cases and 5 additional cases for validation, while another 59 cases were employed for the evaluation purpose. Results It was found that both VTPNs trained via KgDRL and standard DRL spontaneously learned how to operate the in-house TPS to generate high-quality plans, achieving plan quality scores of 8.82 (±0.29) and 8.43 (±0.48), respectively. Both VTPNs outperformed treatment planning purely based on the rules, which had a plan score of 7.81 (±1.59). VTPN trained with eight episodes using KgDRL was able to perform similarly to that trained using DRL with 100 epochs. The training time was reduced from more than a week to ~13 hours. Conclusion The proposed KgDRL framework was effective in accelerating the training process of a VTPN by incorporating human knowledge, which will facilitate the development of VTPN for more complicated treatment planning scenarios.

Time-resolved imaging becomes popular in radiotherapy in that it significantly reduces blurring artifacts in volumetric images reconstructed from a set of 2D X-ray projection data. We aim at developing a neural network (NN) based machine learning algorithm that allows for reconstructing an instantaneous image from a single projection. In our approach, each volumetric image is represented as a deformation of a chosen reference image, in which the deformation is modeled as a linear combination of a few basis functions through principal component analysis (PCA). Based on this PCA deformation model, we train an ensemble of neural networks to find a mapping from a projection image to PCA coefficients. For image reconstruction, we apply the learned mapping on an instantaneous projection image to obtain the PCA coefficients, thus getting a deformation. Then, a volumetric image can be reconstructed by applying the deformation on the reference image. Experimentally, we show promising results on a set of simulated data.

Digitization of interstitial needles is a complicated and tedious process for the treatment planning of 3D CT image based interstitial high dose-rate brachytherapy (HDRBT) of gynecological cancer. We developed a deep-learning assisted auto-digitization method for interstitial needles. The digitization method consisted of two steps. The first step used a deep neural network with a U-net structure to segment all needles from CT images. The second step simultaneously clustered the segmented voxels into different needle groups and generated the needle central trajectories by solving an optimization problem. We evaluated the effectiveness of the developed method in ten interstitial HDRBT patient cases that were not used in the training of the U-net. Average number of needles per case was 20.7. For the segmentation step, average Dice similarity coefficient between automatic and manual segmentation was 0.93. For the digitization step, Hausdorff distance between needle trajectories determined by our method and manually by qualified medical physicists was ~0.71 mm on average and mean difference of tip positions was ~0.63 mm, which were considered acceptable for HDRBT treatment planning. It took ~5 min to complete the digitization process of an interstitial HDRBT case. The achieved accuracy and efficiency made our method clinically attractive.

An indoor, real-time location system (RTLS) can benefit both hospitals and patients by improving clinical efficiency through data-driven optimization of procedures. Bluetooth-based RTLS systems are cost-effective but lack accuracy and robustness because Bluetooth signal strength is subject to fluctuation. We developed a machine learning-based solution using a Long Short-Term Memory (LSTM) network followed by a Multilayer Perceptron classifier and a posterior constraint algorithm to improve RTLS performance. Training and validation datasets showed that most machine learning models perform well in classifying individual location zones, although LSTM was most reliable. However, when faced with data indicating cross-zone trajectories, all models showed erratic zone switching. Thus, we implemented a history-based posterior constraint algorithm to reduce the variability in exchange for a slight decrease in responsiveness. This network increases robustness at the expense of latency. When latency is less of a concern, we computed the latency-corrected accuracy which is 100% for our testing data, significantly improved from LSTM without constraint which is 96.2%. The balance between robustness and responsiveness can be considered and adjusted on a case-by-case basis, according to the specific needs of downstream clinical applications. This system was deployed and validated in an academic medical center. Industry best practices enabled system scaling without substantial compromises to performance or cost., Comment: 20 pages, 6 figures, submitted to Physics in Medicine & Biology

Tuning parameters in a reconstruction model is of central importance to iterative CT reconstruction, since it critically affects the resulting image quality. Manual parameter tuning is not only tedious, but becomes impractical when there exits a number of parameters. In this paper, we develop a novel deep reinforcement learning (DRL) framework to train a parameter-tuning policy network (PTPN) to automatically adjust parameters in a human-like manner. A quality assessment network (QAN) is trained together with PTPN to learn how to judge CT image quality, serving as a reward function to guide the reinforcement learning. We demonstrate our idea in an iterative CT reconstruction problem with pixel-wise total-variation regularization. Experimental results demonstrates the effectiveness of both PTPN and QAN, in terms of tuning parameter and evaluating image quality, respectively.

Cervical tumor segmentation on 3D (18)FDG PET images is a challenging task because of the proximity between cervix and bladder, both of which can uptake (18)FDG tracers. This problem makes traditional segmentation based on intensity variation methods ineffective and reduces overall accuracy. Based on anatomy knowledge, including “roundness” of the cervical tumor and relative positioning between the bladder and cervix, we propose a supervised machine learning method that integrates convolutional neural network (CNN) with this prior information to segment cervical tumors. First, we constructed a spatial information embedded CNN model (S-CNN) that maps the PET image to its corresponding label map, in which bladder, other normal tissue, and cervical tumor pixels are labeled as −1, 0, and 1, respectively. Then, we obtained the final segmentation from the output of the network by a prior information constrained (PIC) thresholding method. We evaluated the performance of the PIC-S-CNN method on PET images from 50 cervical cancer patients. The PIC-S-CNN method achieved a mean Dice similarity coefficient (DSC) of 0.84 while region-growing, Chan-Vese, graph-cut, fully convolutional neural networks (FCN) based FCN-8 stride, and FCN-2 stride, and U-net achieved 0.55, 0.64, 0.67, 0.71, 0.77, and 0.80 mean DSC, respectively. The proposed PIC-S-CNN provides a more accurate way for segmenting cervical tumors on 3D PET images. Our results suggest that combining deep learning and anatomic prior information may improve segmentation accuracy for cervical tumors.

Adaptive-radiation-therapy (ART) is applied to account for anatomical variations observed over the treatment course. Daily or weekly cone-beam computed tomography (CBCT) is commonly used in clinic for patient positioning, but CBCT’s inaccuracy in Hounsfield units (HU) prevents its application to dose calculation and treatment planning. Adaptive re-planning can be performed by deformably registering planning CT (pCT) to CBCT. However, scattering artifacts and noise in CBCT decrease the accuracy of deformable registration and induce uncertainty in treatment plan. Hence, generating from CBCT a synthetic CT (sCT) that has the same anatomical structure as CBCT but accurate HU values is desirable for ART. We proposed an unsupervised style-transfer-based approach to generate sCT based on CBCT and pCT. Unsupervised learning was desired because exactly matched CBCT and CT are rarely available, even when they are taken a few minutes apart. In the proposed model, CBCT and pCT are two inputs that provide anatomical structure and accurate HU information, respectively. The training objective function is designed to simultaneously minimize (1) contextual loss between sCT and CBCT to maintain the content and structure of CBCT in sCT and (2) style loss between sCT and pCT to achieve pCT-like image quality in sCT. We used CBCT and pCT images of 114 patients to train and validate the designed model, and another 29 independent patient cases to test the model’s effectiveness. We quantitatively compared the resulting sCT with the original CBCT using the deformed same-day pCT as reference. Structure-similarity-index, peak-signal-to-noise-ratio, and mean-absolute-error in HU of sCT were 0.9723, 33.68, and 28.52, respectively, while those of CBCT were 0.9182, 29.67, and 49.90, respectively. We have demonstrated the effectiveness of the proposed model in using CBCT and pCT to synthesize CT-quality images. This model may permit using CBCT for advanced applications such as adaptive treatment planning.

Multi-instance multi-label (MIML) learning is one of challenging research problems in machine learning. In the literature, there are several methods for solving MIML problems. However, they may take a long computational time and have a huge storage cost for large MIML data sets. The main aim of this paper is to propose and develop an efficient Markov Chain learning algorithm for MIML problems, especially for data represented by non-negative features. Our idea is to perform labels classification iteratively through two Markov chains constructed by using objects and features respectively. The classification of objects can be obtained by using labels propagation via training data in the iterative method. Moreover, we demonstrate that the proposed method can be formulated by considering normalized linear kernel. Because linear kernel function is explicit and separable, it is not necessary to compute and store a huge affinity matrix among objects/instances compared with the use of other kernel functions. Therefore, both the storage and computational time of the proposed algorithm are very efficient. Experimental results are presented to show that the classification performance of the proposed method using normalized linear kernel function is about the same as those using the other kernel functions, while the required computational time is much less, which together suggest that the linear kernel can be good enough for MIML problem.Also experimental results on some benchmark data sets are reported to illustrate the effectiveness of the proposed method in one-error, ranking loss, coverage and average precision, and show that it is competitive with the other MIML methods.

Driving pattern recognition based on driving status features (GPS, gear, and speed etc.) is of central importance in the development of intelligent transportation. While it is expensive and labor intensive to obtain a large amount of labeled driving data in real applications. It makes the driving pattern recognition particularly difficult for those domains without labeled data. In this paper, to tackle this challenging recognition task, we propose a novel and robust Transfer Learning method for Driving Pattern Recognition (TLDPR) that can transfer knowledge from other related source domains with labeled data to the target domain. Compared to the traditional supervised learning, one of the major difficulties of transfer learning is that the data from different domains may have distinct distributions. The proposed TLDPR is able to reduce the distribution difference in RKHS between the samples in target and source domain with the same driving pattern, and it can preserve the local manifold structure simultaneously. In addition, an iterative ensemble strategy is implemented to make the model more robust using the pseudo-labels. To evaluate the performance of TLDPR, comprehensive experiments have been conducted on parking lots datasets. The results show TLDPR can substantially outperform the state-of-the-art methods.

Cervical tumor segmentation on 3D 18FDG PET images is a challenging task due to the proximity between cervix and bladder. Since bladder has high capacity of 18FDG tracers, bladder intensity is similar to cervical tumor intensity in the PET image. This inhibits traditional segmentation methods based on intensity variation of the image to achieve high accuracy. We propose a supervised machine learning method that integrates a convolutional neural network (CNN) with prior information of cervical tumor. In the proposed prior information constraint CNN (PIC-CNN) algorithm, we first construct a CNN to weaken the bladder intensity value in the image. Based on the roundness of cervical tumor and relative positioning information between bladder and cervix, we obtain the final segmentation result from the output of the network by an auto-thresholding method. We evaluate the performance of the proposed PIC-CNN method on PET images from 50 cervical cancer patients whose cervix and bladder are abutting. The PIC-CNN method achieves a mean DSC value of 0.84 while transfer learning method based on fully convolutional neural networks (FCN) achieves 0.77 DSC. In addition, traditional segmentation methods such as automatic threshold and region-growing method only achieve 0.59 and 0.52 DSC values, respectively. The proposed method provides a more accurate way for segmenting cervical tumor in 3D PET image.

Because in PET imaging cervical tumors are close to the bladder with high capacity for the secreted (18)FDG tracer, conventional intensity-based segmentation methods often misclassify the bladder as a tumor. Based on the observation that tumor position and area do not change dramatically from slice to slice, we propose a two-stage scheme that facilitates segmentation. In the first stage, we used a graph-cut based algorithm to obtain initial contouring of the tumor based on local similarity information between voxels; this was achieved through manual contouring of the cervical tumor on one slice. In the second stage, initial tumor contours were fine-tuned to more accurate segmentation by incorporating similarity information on tumor shape and position among adjacent slices, according to an intensity-spatial-distance map. Experimental results illustrate that the proposed two-stage algorithm provides a more effective approach to segmenting cervical tumors in 3D (18)FDG PET images than the benchmarks used for comparison.

Inverse treatment planning in radiation therapy is formulated as solving optimization problems. The objective function and constraints consist of multiple terms designed for different clinical and practical considerations. Weighting factors of these terms are needed to define the optimization problem. While a treatment planning optimization engine can solve the optimization problem with given weights, adjusting the weights to yield a high-quality plan is typically performed by a human planner. Yet the weight-tuning task is labor intensive, time consuming, and it critically affects the final plan quality. An automatic weight-tuning approach is strongly desired. The procedure of weight adjustment to improve the plan quality is essentially a decision-making problem. Motivated by the tremendous success in deep learning for decision making with human-level intelligence, we propose a novel framework to adjust the weights in a human-like manner. This study used inverse treatment planning in high-dose-rate brachytherapy (HDRBT) for cervical cancer as an example. We developed a weight-tuning policy network (WTPN) that observes dose volume histograms of a plan and outputs an action to adjust organ weighting factors, similar to the behaviors of a human planner. We trained the WTPN via end-to-end deep reinforcement learning. Experience replay was performed with the epsilon greedy algorithm. After training was completed, we applied the trained WTPN to guide treatment planning of five testing patient cases. It was found that the trained WTPN successfully learnt the treatment planning goals and was able to guide the weight tuning process. On average, the quality score of plans generated under the WTPN's guidance was improved by ~8.5% compared to the initial plan with arbitrarily set weights, and by 10.7% compared to the plans generated by human planners. To our knowledge, this was the first time that a tool was developed to adjust organ weights for the treatment planning optimization problem in a human-like fashion based on intelligence learnt from a training process, which was different from existing strategies based on pre-defined rules. The study demonstrated potential feasibility to develop intelligent treatment planning approaches via deep reinforcement learning.

A number of image-processing problems can be formulated as optimization problems. The objective function typically contains several terms specifically designed for different purposes. Parameters in front of these terms are used to control the relative weights among them. It is of critical importance to tune these parameters, as quality of the solution depends on their values. Tuning parameter is a relatively straightforward task for a human, as one can intelligently determine the direction of parameter adjustment based on the solution quality. Yet manual parameter tuning is not only tedious in many cases, but becomes impractical when a number of parameters exist in a problem. Aiming at solving this problem, this paper proposes an approach that employs deep reinforcement learning to train a system that can automatically adjust parameters in a human-like manner. We demonstrate our idea in an example problem of optimization-based iterative CT reconstruction with a pixel-wise total-variation regularization term. We set up a parameter tuning policy network (PTPN), which maps an CT image patch to an output that specifies the direction and amplitude by which the parameter at the patch center is adjusted. We train the PTPN via an end-to-end reinforcement learning procedure. We demonstrate that under the guidance of the trained PTPN for parameter tuning at each pixel, reconstructed CT images attain quality similar or better than in those reconstructed with manually tuned parameters., Comment: 8 pages, 8 figures, 2 tables

Purpose: To reduce cone beam CT (CBCT) imaging dose, we previously proposed a progressive dose control (PDC) scheme to employ temporal correlation between CBCT images at different fractions for image quality enhancement. A temporal non-local means (TNLM) method was developed to enhance quality of a new low-dose CBCT using existing high-quality CBCT. To enhance a voxel value, the TNLM method searches for similar voxels in a window. Due to patient deformation among the two CBCTs, a large searching window was required, reducing image quality and computational efficiency. This abstract proposes a deformation-assisted TNLM (DA-TNLM) method to solve this problem. Methods: For a low-dose CBCT to be enhanced using a high-quality CBCT, we first performed deformable image registration between the low-dose CBCT and the high-quality CBCT to approximately establish voxel correspondence between the two. A searching window for a voxel was then set based on the deformation vector field. Specifically, the search window for each voxel was shifted by the deformation vector. A TNLM step was then applied using only voxels within this determined window to correct image intensity at the low-dose CBCT. Results: We have tested the proposed scheme on simulated CIRS phantom data and real patient data. The CITS phantom was scanned on Varian onboard imaging CBCT system with coach shifting and dose reducing for each time. The real patient data was acquired in four fractions with dose reduced from standard CBCT dose to 12.5% of standard dose. It was found that the DA-TNLM method can reduce total dose by over 75% on average in the first four fractions. Conclusion: We have developed a PDC scheme which can enhance the quality of image scanned at low dose using a DA-TNLM method. Tests in phantom and patient studies demonstrated promising results.

The interactions among different genes, proteins and other small molecules are becoming more and more significant and have been studied intensively nowadays. One general way that helps people understand these interactions is to analyze networks constructed from genes/proteins. In particular, module structure as a common property of most biological networks has drawn much attention of researchers from different fields. In most cases, biological networks can be corrupted by noise in the data and the corruption may cause mis-identification of module structure. Besides, some structure may be destroyed when improper experimental settings are built up. Thus module structure may be unstable when one single network is employed. In this paper, we consider employing multiple networks for consistent module detection in order to reduce the effect of noise and experimental setting. Instead of considering different networks separately, our idea is to combine multiple networks together by building them into tensor structure data. Then given any node as prior label information, tensor-based Markov chains are constructed iteratively for identification of the modules shared by the multiple networks. In addition, the proposed tensor-based Markov chain algorithm is capable of simultaneously evaluating the contribution from each network. It would be useful to measure the consistency of modules in the multiple networks. In the experiments, we test our method on two groups of gene co-expression networks from human beings. We also validate the modules identified by the proposed method.

Multi-Instance Multi-Label MIML learning is one of challenging research problems in machine learning. The main aim of this paper is to propose and develop a novel sparsity-based MIML learning algorithm. Our idea is to formulate and construct a transductive objective function for labels indicator to be learned by using the method of random walk with restart that exploits the relationships among instances and labels of objects, and computes the affinities among the objects. Then sparsity can be introduced in the labels indicator of the objective function such that relevant and irrelevant objects with respect to a given class can be distinguished. The resulting sparsity-based MIML model can be given as a constrained convex optimization problem, and it can be solved very efficiently by using the augmented Lagrangian method. Experimental results on benchmark data have shown that the proposed sparse-MIML algorithm is computationally efficient, and effective in label prediction for MIML data. We demonstrate that the performance of the proposed method is better than the other testing MIML learning algorithms.

Purpose: Multi-energy computed tomography (MECT) is an emerging application in medical imaging due to its ability of material differentiation and potential for molecular imaging. In MECT, image correlations at different spatial and channels exist. It is desirable to incorporate these correlations in reconstruction to improve image quality. For this purpose, this study proposes a MECT reconstruction technique that employes spatial spectral non-local means (ssNLM) regularization. Methods: We consider a kVp-switching scanning method in which source energy is rapidly switched during data acquisition. For each energy channel, this yields projection data acquired at a number of angles, whereas projection angles among channels are different. We formulate the reconstruction task as an optimziation problem. A least square term enfores data fidelity. A ssNLM term is used as regularization to encourage similarities among image patches at different spatial locations and channels. When comparing image patches at different channels, intensity difference were corrected by a transformation estimated via histogram equalization during the reconstruction process. Results: We tested our method in a simulation study with a NCAT phantom and an experimental study with a Gammex phantom. For comparison purpose, we also performed reconstructions using conjugate-gradient least square (CGLS) method and conventional NLM method that only considers spatial correlation in an image. ssNLM is able to better suppress streak artifacts. The streaks are along different projection directions in images at different channels. ssNLM discourages this dissimilarity and hence removes them. True image structures are preserved in this process. Measurements in regions of interests yield 1.1 to 3.2 and 1.5 to 1.8 times higher contrast to noise ratio than the NLM approach. Improvements over CGLS is even more profound due to lack of regularization in the CGLS method and hence amplified noise. Conclusion: The proposed ssNLM method for kVp-switching MECT reconstruction can achieve high quality MECT images.