American Joint Replacement Registry Risk Calculator Does Not Predict 90-day Mortality in Veterans Undergoing Total Joint Replacement

Background The American Joint Replacement Registry (AJRR) Total Joint Risk Calculator uses demographic and clinical parameters to provide risk estimates for 90-day mortality and 2-year periprosthetic joint infection (PJI). The tool is intended to help surgeons counsel their Medicare-eligible patients about their risk of death and PJI after total joint arthroplasty (TJA). However, for a predictive risk model to be useful, it must be accurate when applied to new patients; this has yet to be established for this calculator. Questions/purposes To produce accuracy metrics (ie, discrimination, calibration) for the AJRR mortality calculator using data from Medicare-eligible patients undergoing TJA in the Veterans Health Administration (VHA), the largest integrated healthcare system in the United States, where more than 10,000 TJAs are performed annually. Methods We used the AJRR calculator to predict risk of death within 90 days of surgery among 31,214 VHA patients older than 64 years of age who underwent primary TJA; data was drawn from the Veterans Affairs Surgical Quality Improvement Project (VASQIP) and VA Corporate Data Warehouse (CDW). We then used VHA mortality data to evaluate the extent to which the AJRR calculator estimates distinguished individuals who died compared with those who did not (C-statistic), and graphically depicted the relationship between estimated risk and observed mortality (calibration). As a secondary evaluation of the calculator, a sample of 39,300 patients younger than 65 years old was assigned to the youngest age group available to the user (65-69 years) as might be done in real-world practice. Results C-statistics for 90-day mortality for the older samples were 0.62 (95% CI, 0.60-0.64) and for the younger samples they were 0.46 (95% CI, 0.43-0.49), suggesting poor discrimination. Calibration analysis revealed poor correspondence between deciles of predicted risk and observed mortality rates. Poor discrimination and calibration mean that patients who died will frequently have a lower estimated risk of death than surviving patients. Conclusions For Medicare-eligible patients receiving TJA in the VA, the AJRR risk calculator had a poor performance in the prediction of 90-day mortality. There are several possible reasons for the model's poor performance. Veterans Health Administration patients, 97% of whom were men, represent only a subset of the broader Medicare population. However, applying the calculator to a subset of the target population should not affect its accuracy. Other reasons for poor performance include a lack of an underlying statistical model in the calculator's implementation and simply the challenge of predicting rare events. External validation in a more representative sample of Medicare patients should be conducted to before assuming this tool is accurate for its intended use. Level of evidence Level I, diagnostic study.