Search

BACKGROUND The Skeletal Oncology Research Group machine-learning algorithms (SORG-MLAs) estimate 90-day and 1-year survival in patients with long-bone metastases undergoing surgical treatment and have demonstrated good discriminatory ability on internal validation. However, the performance of a prediction model could potentially vary by race or region, and the SORG-MLA must be externally validated in an Asian cohort. Furthermore, the authors of the original developmental study did not consider the Eastern Cooperative Oncology Group (ECOG) performance status, a survival prognosticator repeatedly validated in other studies, in their algorithms because of missing data. QUESTIONS/PURPOSES (1) Is the SORG-MLA generalizable to Taiwanese patients for predicting 90-day and 1-year mortality? (2) Is the ECOG score an independent factor associated with 90-day and 1-year mortality while controlling for SORG-MLA predictions? METHODS All 356 patients who underwent surgery for long-bone metastases between 2014 and 2019 at one tertiary care center in Taiwan were included. Ninety-eight percent (349 of 356) of patients were of Han Chinese descent. The median (range) patient age was 61 years (25 to 95), 52% (184 of 356) were women, and the median BMI was 23 kg/m2 (13 to 39 kg/m2). The most common primary tumors were lung cancer (33% [116 of 356]) and breast cancer (16% [58 of 356]). Fifty-five percent (195 of 356) of patients presented with a complete pathologic fracture. Intramedullary nailing was the most commonly performed type of surgery (59% [210 of 356]), followed by plate screw fixation (23% [81 of 356]) and endoprosthetic reconstruction (18% [65 of 356]). Six patients were lost to follow-up within 90 days; 30 were lost to follow-up within 1 year. Eighty-five percent (301 of 356) of patients were followed until death or for at least 2 years. Survival was 82% (287 of 350) at 90 days and 49% (159 of 326) at 1 year. The model's performance metrics included discrimination (concordance index [c-index]), calibration (intercept and slope), and Brier score. In general, a c-index of 0.5 indicates random guess and a c-index of 0.8 denotes excellent discrimination. Calibration refers to the agreement between the predicted outcomes and the actual outcomes, with a perfect calibration having an intercept of 0 and a slope of 1. The Brier score of a prediction model must be compared with and ideally should be smaller than the score of the null model. A decision curve analysis was then performed for the 90-day and 1-year prediction models to evaluate their net benefit across a range of different threshold probabilities. A multivariate logistic regression analysis was used to evaluate whether the ECOG score was an independent prognosticator while controlling for the SORG-MLA's predictions. We did not perform retraining/recalibration because we were not trying to update the SORG-MLA algorithm in this study. RESULTS The SORG-MLA had good discriminatory ability at both timepoints, with a c-index of 0.80 (95% confidence interval 0.74 to 0.86) for 90-day survival prediction and a c-index of 0.84 (95% CI 0.80 to 0.89) for 1-year survival prediction. However, the calibration analysis showed that the SORG-MLAs tended to underestimate Taiwanese patients' survival (90-day survival prediction: calibration intercept 0.78 [95% CI 0.46 to 1.10], calibration slope 0.74 [95% CI 0.53 to 0.96]; 1-year survival prediction: calibration intercept 0.75 [95% CI 0.49 to 1.00], calibration slope 1.22 [95% CI 0.95 to 1.49]). The Brier score of the 90-day and 1-year SORG-MLA prediction models was lower than their respective null model (0.12 versus 0.16 for 90-day prediction; 0.16 versus 0.25 for 1-year prediction), indicating good overall performance of SORG-MLAs at these two timepoints. Decision curve analysis showed SORG-MLAs provided net benefits when threshold probabilities ranged from 0.40 to 0.95 for 90-day survival prediction and from 0.15 to 1.0 for 1-year prediction. The ECOG score was an independent factor associated with 90-day mortality (odds ratio 1.94 [95% CI 1.01 to 3.73]) but not 1-year mortality (OR 1.07 [95% CI 0.53 to 2.17]) after controlling for SORG-MLA predictions for 90-day and 1-year survival, respectively. CONCLUSION SORG-MLAs retained good discriminatory ability in Taiwanese patients with long-bone metastases, although their actual survival time was slightly underestimated. More international validation and incremental value studies that address factors such as the ECOG score are warranted to refine the algorithms, which can be freely accessed online at https://sorg-apps.shinyapps.io/extremitymetssurvival/. LEVEL OF EVIDENCE Level III, therapeutic study.

The purpose of this work is to develop a machine learning model to improve the GNSS position usage when applying the GNSS position as a Kalman filter measurement update in a loosely coupled GNSS/IMU system. The motivation comes from a perplexing issue that it is difficult to determine the GNSS position quality in challenging urban areas. In a loosely coupled GNSS/IMU system, it is critical to filter out the bad GNSS positions before applying GNSS position as the measurement update to the Kalman filter. Although the Kalman filter itself is a process which can automatically weight the measurement quality based on the reported confidence level from the GNSS receiver, it still cannot solve the issue well. The reason is that the reported confidence level from the GNSS receiver could easily be either underestimated or overestimated in weak signal conditions. As a result, a low quality GNSS position might be thought as a benign measurement and it is then used in the Kalman filter measurement update. Consequently, a large positioning error occurs in the fused solution; and furthermore, the inertial sensor bias states could be wrongly estimated. On the other hand, a good quality GNSS position could be excluded from being a measurement update to the Kalman filter. This would lead to an unnecessary dead reckoning and thus the positioning error is accumulated. In summary, purely rely on the reported confidence level from the GNSS receiver cannot provide reliable results in fusing the GNSS with the IMU together. We propose a new method by applying the machine learning model for the GNSS position quality classification, which is proved to be effective and suitable for the integration with the traditional rule based methods. The learning based and rule based methods are complementary. The rule based method is deterministic and predictable, but having plenty of exceptions. It is hard to find a general rule which applies to all cases. A simple rule based method is a heuristic threshold to classify the good and bad GNSS outputs, according to the values of confidence level, C/N0, and GDOP. There are other advance methods such as fault detection and integrity monitoring to guarantee the GNSS output quality and reliability. However, these methods rely on redundant strong GNSS signals. In challenging urban area, the condition with plenty of strong signals is not practical. On the contrary, a machine learning based method is suitable for classifying the GNSS outputs given the invisible clues under weak signal conditions. In this work, the focus is to determine whether a GNSS position solution is good or bad for the step of positioning fusion with the inertial sensors. Thus, this is a classification problem. We set the GNSS position quality in three classes, excellent, normal, and poor. The position error in terms of 2-D distance error for the three classes are: 10 m, respectively. Given the classification problem with the target classes, we then determine the features which are the input to the model. The features are GNSS signal conditions and positioning related parameters. For example, the features can be C/N0, the number of satellites used for position fix, the number of satellites tracked, the number of C/N0 higher than 35 dB-Hz, the horizontal position variance, the vertical position variance, the GDOP, and the correlation of derived velocity from consecutive position fixes with the direct velocity solution from the GNSS receiver. Furthermore, the above features are observed for a period of time. So we can calculate the mean, the standard deviation, the Quartile, the statistics in time domain, and the properties in frequency domain. As a result, we have hundreds of features as the inputs to the model. In the next step, we apply Linear Discriminant Analysis (LDA) and decreasing Gini index to do dimension reduction for the final model. In the model selection, we started from simple to complex models. For example, decision tree, random forest, support vector machine, bag of words were studied. To determine the best model, we used metrics such as sensitivity, precision, confusion matrix which is then used to derive the receiver operating characteristic (ROC) curve and the area under the curve of ROC (AUC). After the above process, we determined the best machine learning model for our system. With the aforementioned machine learning model built, we then combined the rule based method together with machine learning model to see the percentage of the bad GNSS position outputs can be classified. From the results of a database which composes of accumulated 800 km of driving, the true positive rate is 90%. This means that 90% of the bad GNSS position outputs were identified. We have implemented the rule based + machine learning based model in a loosely coupled GNSS/IMU system, together with vehicle’s speed (i.e., an automotive dead reckoning (ADR) system). The solution was implemented on an embedded system which was running in real time. In summary, we propose a machine learning model for GNSS position classification. We evaluated the important features for the model. We analyzed the best model to be used. We implemented the system in real time and concluded the final performance. We will present the final model selected, the features used, and the overall solution in the embedded system.