Search

BackgroundTackling biases in medical artificial intelligence requires multi-centre collaboration, however, ethical, legal and entrustment considerations may restrict providers’ ability to participate. Federated learning (FL) may eliminate the need for data sharing by allowing algorithm development across multiple hospitals without data transfer.Previously, we have shown an AI-driven screening solution for COVID-19 in emergency departments using clinical data routinely available within 1h of arrival to hospital (vital signs & blood tests; CURIAL-Lab). Here, we aimed to extend and federate our COVID-19 screening test, demonstrating development and evaluation of a rapidly scalable and user-friendly FL solution across 4 UK hospital groups.MethodsWe supplied a Raspberry Pi 4 Model B device, preloaded with our end-to-end FL pipeline, to 4 NHS hospital groups or their locally-linked research university (Oxford University Hospitals/University of Oxford (OUH), University Hospitals Birmingham/University of Birmingham (UHB), Bedfordshire Hospitals (BH) and Portsmouth Hospitals University (PUH) NHS trusts). OUH, PUH and UHB participated in federated training and calibration, training a deep neural network (DNN) and logistic regressor to predict COVID-19 status using clinical data for pre-pandemic (COVID-19-negative) admissions and COVID-19-positive cases from the first wave. We performed federated prospective evaluation at PUH & OUH, and external evaluation at BH, evaluating the resultant global and site-tuned models for admissions to the respective sites during the second pandemic wave. Removable microSD storage was destroyed on study completion.FindingsRoutinely collected clinical data from a total 130,941 patients (1,772 COVID-19 positive) across three hospital groups were included in federated training. OUH, PUH and BH participated in prospective federated evaluation, with sets comprising 32,986 patient admissions (3,549 positive) during the second pandemic wave. Federated training improved DNN performance by a mean of 27.6% in terms of AUROC when compared to models trained locally, from AUROC of 0.574 & 0.622 at OUH & PUH to 0.872 & 0.876 for the federated global model. Performance improvement was more modest for a logistic regressor with a mean AUROC increase of 13.9%. During federated external evaluation at BH, the global DNN model achieved an AUROC of 0.917 (0.893-0.942), with 89.7% sensitivity (83.6-93.6) and 76.7% specificity (73.9-79.1). Site-personalisation of the global model did not give a significant improvement in overall performance (AUROC improvement InterpretationsWe present a rapidly scalable hardware and software FL solution, developing a COVID-19 screening test across four UK hospital groups using inexpensive micro-computing hardware. Federation improved model performance and generalisability, and shows promise as an enabling technology for deep learning in healthcare.Funding University of Oxford Medical & Life Sciences Translational Fund/Wellcome

As machine learning-based models continue to be developed for healthcare applications, greater effort is needed in ensuring that these technologies do not reflect or exacerbate any unwanted or discriminatory biases that may be present in the data. In this study, we introduce a reinforcement learning framework capable of mitigating biases that may have been acquired during data collection. In particular, we evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments, and aimed to mitigate any site-specific (hospital) and ethnicity-based biases present in the data. Using a specialized reward function and training procedure, we show that our method achieves clinically-effective screening performances, while significantly improving outcome fairness compared to current benchmarks and state-of-the-art machine learning methods. We performed external validation across three independent hospitals, and additionally tested our method on a patient ICU discharge status task, demonstrating model generalizability.

Background Uncertainty in patients' COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, the typical turnaround time for laboratory PCR remains 12–24 h and lateral flow devices (LFDs) have limited sensitivity. Previously, we have shown that artificial intelligence-driven triage (CURIAL-1.0) can provide rapid COVID-19 screening using clinical data routinely available within 1 h of arrival to hospital. Here, we aimed to improve the time from arrival to the emergency department to the availability of a result, do external and prospective validation, and deploy a novel laboratory-free screening tool in a UK emergency department. Methods We optimised our previous model, removing less informative predictors to improve generalisability and speed, developing the CURIAL-Lab model with vital signs and readily available blood tests (full blood count [FBC]; urea, creatinine, and electrolytes; liver function tests; and C-reactive protein) and the CURIAL-Rapide model with vital signs and FBC alone. Models were validated externally for emergency admissions to University Hospitals Birmingham, Bedfordshire Hospitals, and Portsmouth Hospitals University National Health Service (NHS) trusts, and prospectively at Oxford University Hospitals, by comparison with PCR testing. Next, we compared model performance directly against LFDs and evaluated a combined pathway that triaged patients who had either a positive CURIAL model result or a positive LFD to a COVID-19-suspected clinical area. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser to provide laboratory-free COVID-19 screening at the John Radcliffe Hospital (Oxford, UK). Our primary improvement outcome was time-to-result, and our performance measures were sensitivity, specificity, positive and negative predictive values, and area under receiver operating characteristic curve (AUROC). Findings 72 223 patients met eligibility criteria across the four validating hospital groups, in a total validation period spanning Dec 1, 2019, to March 31, 2021. CURIAL-Lab and CURIAL-Rapide performed consistently across trusts (AUROC range 0·858–0·881, 95% CI 0·838–0·912, for CURIAL-Lab and 0·836–0·854, 0·814–0·889, for CURIAL-Rapide), achieving highest sensitivity at Portsmouth Hospitals (84·1%, Wilson's 95% CI 82·5–85·7, for CURIAL-Lab and 83·5%, 81·8–85·1, for CURIAL-Rapide) at specificities of 71·3% (70·9–71·8) for CURIAL-Lab and 63·6% (63·1–64·1) for CURIAL-Rapide. When combined with LFDs, model predictions improved triage sensitivity from 56·9% (51·7–62·0) for LFDs alone to 85·6% with CURIAL-Lab (81·6–88·9; AUROC 0·925) and 88·2% with CURIAL-Rapide (84·4–91·1; AUROC 0·919), thereby reducing missed COVID-19 cases by 65% with CURIAL-Lab and 72% with CURIAL-Rapide. For the prospective deployment of CURIAL-Rapide, 520 patients were enrolled for point-of-care FBC analysis between Feb 18 and May 10, 2021, of whom 436 received confirmatory PCR testing and ten (2·3%) tested positive. Median time from arrival to a CURIAL-Rapide result was 45 min (IQR 32–64), 16 min (26·3%) sooner than with LFDs (61 min, 37–99; log-rank p Interpretation Our findings show the generalisability, performance, and real-world operational benefits of artificial intelligence-driven screening for COVID-19 over standard-of-care in emergency departments. CURIAL-Rapide provided rapid, laboratory-free screening when used with near-patient FBC analysis, and was able to reduce the number of patients who tested negative for COVID-19 but were triaged to COVID-19-suspected areas.

Machine learning is becoming increasingly prominent in healthcare. Although its benefits are clear, growing attention is being given to how machine learning may exacerbate existing biases and disparities. In this study, we introduce an adversarial training framework that is capable of mitigating biases that may have been acquired through data collection or magnified during model development. For example, if one class is over-presented or errors/inconsistencies in practice are reflected in the training data, then a model can be biased by these. To evaluate our adversarial training framework, we used the statistical definition of equalized odds. We evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments, and aimed to mitigate regional (hospital) and ethnic biases present. We trained our framework on a large, real-world COVID-19 dataset and demonstrated that adversarial training demonstrably improves outcome fairness (with respect to equalized odds), while still achieving clinically-effective screening performances (NPV>0.98). We compared our method to the benchmark set by related previous work, and performed prospective and external validation on four independent hospital cohorts. Our method can be generalized to any outcomes, models, and definitions of fairness.

As patient health information is highly regulated due to privacy concerns, the majority of machine learning (ML)-based healthcare studies are unable to test on external patient cohorts, resulting in a gap between locally reported model performance and cross-site generalizability. Different approaches have been introduced for developing models across multiple clinical sites, however no studies have compared methods for translating ready-made models for adoption in new settings. We introduce three methods to do this – (1) applying a ready-made model “as-is”; (2) readjusting the decision threshold on the output of a ready-made model using site-specific data; and (3) finetuning a ready-made model using site-specific data via transfer learning. Using a case study of COVID-19 diagnosis across four NHS Hospital Trusts, we show that all methods achieve clinically-effective performances (NPV >0.959), with transfer learning achieving the best results (mean AUROCs between 0.870-0.925). Our models demonstrate that site-specific customization improves predictive performance when compared to other ready-made approaches.