Implementing an artificial intelligence command centre in the NHS: a mixed-methods study

Background Hospital ‘command centres’ use digital technologies to collect, analyse and present real-time information that may improve patient flow and patient safety. Bradford Royal Infirmary has trialled this approach and presents an opportunity to evaluate effectiveness to inform future adoption in the United Kingdom. Objective To evaluate the impact of the Bradford Command Centre on patient care and organisational processes. Design A comparative mixed-methods study. Operational data from a study and control site were collected and analysed. The intervention was observed, and staff at both sites were interviewed. Analysis was grounded in a literature review and the results were synthesised to form conclusions about the intervention. Setting The study site was Bradford Royal Infirmary, a large teaching hospital in the city of Bradford, United Kingdom. The control site was Huddersfield Royal Infirmary in the nearby city of Huddersfield. Participants Thirty-six staff members were interviewed and/or observed. Intervention The implementation of a digitally enabled hospital command centre. Main outcome measures Qualitative perspectives on hospital management. Quantitative metrics on patient flow, patient safety, data quality. Data sources Anonymised electronic health record data. Ethnographic observations including interviews with hospital staff. Cross-industry review including relevant literature and expert panel interviews. Results The Command Centre was implemented successfully and has improved staff confidence of better operational control. Unintended consequences included tensions between localised and centralised decision-making and variable confidence in the quality of data available. The Command Centre supported the hospital through the COVID-19 pandemic, but the direct impact of the Command Centre was difficult to measure as the pandemic forced all hospitals, including the study and control sites, to innovate rapidly. Late in the study we learnt that the control site had visited the study site and replicated some aspects of the command centre themselves; we were unable to explore this in detail. There was no significant difference between pre- and post-intervention periods for the quantitative outcome measures and no conclusive impact on patient flow and data quality. Staff and patients supported the command-centre approaches but patients expressed concern that individual needs might get lost to ‘the system’. Conclusions Qualitative evidence suggests the Command Centre implementation was successful, but it proved challenging to link quantitative evidence to specific technology interventions. Staff were positive about the benefits and emphasised that these came from the way they adapted to and used the new technology rather than the technology per se. Limitations The COVID-19 pandemic disrupted care patterns and forced rapid innovation which reduced our ability to compare study and control sites and data before, during and after the intervention. Future work We plan to follow developments at Bradford and in command centres in the National Health Service in order to share learning. Our mixed-methods approach should be of interest to future studies attempting similar evaluation of complex digitally enabled whole-system changes. Study registration The study is registered as IRAS No.: 285933. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR129483) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 41. See the NIHR Funding and Awards website for further award information. Plain language summary Safety-critical industries like airports often use ‘command centres’ to manage operations safely and efficiently. In contrast, most National Health Service hospitals have operational management that is fragmented across many departments and poorly co-ordinated. This may pose risks to the safety and care of patients and may partially explain excessive waiting times. Bradford Royal Infirmary is one of the first National Health Service hospitals to try out a command-centre approach using new digital technologies. Hospital staff at Bradford now work together in a purpose-built Command Centre room and monitor a ‘wall of analytics’ that displays real-time data from the hospital’s information systems. This study examines the implementation at Bradford in order to learn lessons that may help the National Health Service improve the way hospitals provide safe and efficient patient care. We reviewed what is known about the approach, analysed hospital data, observed teams working in the Command Centre, interviewed staff and ran workshops with patients. We also compared Bradford with a similar hospital that did not initially have a command centre. Our results showed the Bradford Command Centre does work. Staff told us the Command Centre helped them manage the hospital through the COVID-19 pandemic. Staff described how they used the new technology to improve efficiency and safety. We observed some challenges, including tension between the central team and local departments and concerns to make sure data were good quality. We could not measure the direct impact of the Command Centre on patient flow and safety using the hospital’s data because the pandemic disrupted normal operations so much. Patients expressed strong support for what they saw as modern methods but were concerned to ensure that the National Health Service tradition of personal care was preserved. Other National Health Service hospitals are interested in following a command-centre approach and our results should help them to learn from Bradford. Scientific summary Background A hospital command centre (CC) is a new approach to the management of hospital operations based on the colocation of decision-making staff and supported by digital technology to provide these staff with close to real-time information. Recent adoption of hospital CCs in the USA predominantly has demonstrated that the approach can be applied to manage hospital operations, despite their complexity. The supporting technology often includes decision-support algorithms that trigger digital notifications and alerts that identify potential safety or flow issues. These algorithms may be based on simple rules or more complex rules generated by machine learning from historic data, and the software technology has therefore been described as AI, meaning either artificial intelligence or, more accurately, augmented intelligence. In the UK, the implementation of a CC and associated technology by Bradford Teaching Hospitals NHS Trust represents a first of type for the UK NHS. To date, there has been limited evidence of the effectiveness of the CC approach and this study aims to address that evidence gap. Bradford Teaching Hospitals NHS Trust manages Bradford Royal Infirmary, an 800-bed NHS hospital located in Bradford in northern England. The Bradford Hospital Command Centre was implemented at Bradford Royal Infirmary through a phased approach in late 2019, was operational through the COVID-19 pandemic, and remains operational to date. It is designed to integrate and centralise operational decision-making to improve patient flow and patient safety across the whole hospital. The Comand Centre (CC) is implemented in a dedicated room in which up to 30 trained non-clinical management and support staff from different operational functions can sit together in teams facing a wall on which 8 large digital display screens are mounted. Each staff member answers telephone calls and performs their operational role using Information Technology (IT) systems on their desktop computer, information on the wall of display screens and communicating with team members and other teams within the room. A senior clinician or manager supports decision-making. The CC was implemented in November 2019, several months before the COVID-19 pandemic began to have a major impact on hospitals in the UK and globally. It was operational throughout the pandemic and, to date, it is firmly established as the centre for operational management of the hospital. The implementation of the CC at Bradford Royal Infirmary presents an opportunity to evaluate the potential strengths and weaknesses of the approach and to generate learning that can inform other hospitals considering adopting the approach. Objectives We aimed to evaluate the impact of the Bradford Command Centre on patient care and organisational processes. We hypothesised that the CC would improve patient flow, reduce bottlenecks and delays, enhance situational awareness to support operational decision-making, and facilitate identification and timely mitigation of threats to patient safety. This study had four research objectives: to evaluate the impact of the CC on patient safety, hospital operational efficiency and related organisational processes to understand the process of implementation of the CC and its integration into hospital management to contextualise the findings using cross-sector and cross-industry perspectives to synthesise the research findings to inform future investment and practice. Methods We conducted a comparative mixed-method case study at two sites: Study site – the CC at Bradford Royal Infirmary. Control site – Huddersfield Royal Infirmary, a hospital in the city of Huddersfield. The control site was selected as being geographically close (15 miles) and part of a similar-sized NHS hospital trust with similar challenges serving areas of high deprivation. At the start of this study, the control site had no plans to implement a CC but, during the study, they learnt from the study site and replicated some aspects of the Bradford Command Centre as part of their own pandemic response. The study combined ethnographic observation and interviews with data analytics of time-series operational data. We accessed and analysed data in the form of anonymised electronic health records from the study and control sites between January 2018 and August 2021 to cover the period before and after implementation of the Bradford Command Centre in November 2019. We conducted 72 hours of ethnographic observations of the CC operation over a period of 9 months after implementation. We conducted 15 interviews with hospital staff at the study site and 4 interviews with staff at the control site. To evaluate the impact of the CC (Objective 1), we described (qualitatively) and evaluated (statistically) the effect on hospital operations and outcomes. We used Interrupted Time Series Analysis to analyse variation in key output indicators in patient safety, patient flow and data quality. We qualitatively investigated situational awareness, operational decision-making, risk and co-ordination/communication across organisational units, from multiple stakeholder perspectives. To understand the process of implementation (Objective 2) our interviews explored staff recall of the implementation, including critical implementation factors and exploring unintended consequences. We used the interviews with staff at the control site to understand how a similar organisation adapted and changed over the same study period. To contextualise the findings (Objective 3), we reviewed the literature on command and control processes in non-healthcare safety-critical operations to extract key principles and contextual factors that may influence transferability of these models into a hospital setting. We also searched for evidence of other hospital CC implementations in the UK and worldwide. To synthesise the research findings (Objective 4), we developed a logic model to map system preconditions, processes, technology and outcomes. Results We were unable to evaluate the impact of the CC as fully as we had planned because the study was impacted by the COVID-19 pandemic. Hospital staff were extremely busy and access on site was challenging. The additional work negotiating access and the direct impact of the pandemic on our own team meant that we had fewer resources and were unable to complete all our objectives. We were able to observe how the CC helped support the hospital manage its operations through the pandemic and received strong positive evidence of its success. Complex pandemic challenges and rapid innovation to meet these challenges made it difficult to attribute outcomes to the specific intervention of the CC. Evaluation of the impact of the Command Centre (Objective 1) Our ethnographic observations and interviews with 15 study-site staff provided documentary evidence of successful use in a complex environment. The CC made a significant impact on the management of the hospital through the pandemic including through the introduction of a COVID-19 ‘tile’ which was used to managed COVID-19-specific processes. The CC and its staff worked with the new technology to change the way that the hospital operated. We identified unintended consequences that included front-line staff developing a sense of being monitored and a fear of interventions from the CC team that were perceived as unwelcome. Linked to this were challenges keeping electronic records up to date and acting on evidence of operational issues that were seen as being under local autonomy or ownership. Data quality was a constant concern for staff working in and around the CC and there were limitations in how up to date and accurate (or complete) records were, often necessitating triangulation and verification from multiple sources and systems and discrepancy between the data in the systems and what front-line staff reported. Compromise, goodwill and a shared sense of purpose were necessary to ensure the CC was effective. We were able to extract time-series data on patient safety, patient flow and data quality from operational systems by selecting representative indicators and plotting these over time. We were able to measure changes in these indicators over time and evaluate statistically the long-term impact of the CC on these indicators. We were not able to isolate improvements in these indicators that could directly be attributed to the introduction of the CC. Similarly, we were unable to isolate noticeable improvements in these indicators between study and control site. We found improvements in mortality and reduced rates of re-admission at the study site but caution against drawing conclusions from this at a time when the pandemic was raging. Some indicators, notably data quality, worsened rather than improved. We reason that the pandemic had such a profound impact on all aspects of operation that it is not possible to separate out and measure the impact of the CC. Similarly, the later adoption of a CC approach by the control site means we cannot use it to draw strong comparisons. Understanding the process of implementation of the Command Centre (Objective 2) Our project started after the CC had been implemented so our results rely on staff recall of the implementation. We identified five phases in the implementation: (1) pre-intervention, (2) a patient flow change programme, (3) Command Centre tile roll-in, (4) Command Centre go-live and (5) post-intervention engagement. Phase 2 was an organisational change, Phase 3 represents a soft-implementation period of training and familiarisation and Phase 4 represents the hard implementation of the new technology and new ways of working. Staff interviews suggest that the overall implementation approach was effective but that they found the implementation challenging and identified some need for more training and software improvements. The intention had been for a period of post-intervention engagement to support staff in getting used to new ways of working and to adapt procedures and technologies to optimise the new approach. This was disrupted by the pandemic, which started to impact on hospital operations only a few months after Phase 4 was complete. Staff recollections are therefore mixed between the pandemic response and the new technology but there is strong evidence that staff worked well together to find ways of working that were consistent with the CC approach while solving immediate challenges. Analysis of time-series data on patient safety, patient flow and data quality at different stages of implantation revealed patterns of change in response to the implementation, but these were confounded by the impact of the pandemic on the same outcome measures. When only the technology part of the Command Centre was assumed as the intervention, there was no significant difference between the pre- and post-intervention periods in the patient safety and patient flow indicators. The data quality had largely worsened in the post-implementation phase and we attribute this to the impact of the pandemic. Qualitative results show that the Command Centre has had a long bedding-in process and that this is expected to be a long process as the hospital and its staff adapt to new ways of working. Our qualitative results suggest that major improvements in patient flow, patient safety and data quality have yet to be achieved. Contextualising the findings using cross-sector and cross-industry perspectives (Objective 3) Results from the literature review found a strong body of research to support the adoption of a CC approach as part of a successful and resilient organisation. CCs are described as supporting situational awareness, decision-making, team structure and workload with the main aim of successfully delivering safety-critical operations reliably over time and in the face of dynamic risks and variations in the operating environment and system conditions. Digital technologies need to be tailored to the work done in the respective domain and should contribute to system resilience. Most articles attribute performance improvements to the physical and functional features of the centres themselves; this often includes the use of technology to generate and display real-time and/or predictive data in the centres. The implementation process usually affects process and policy changes in the organisation, including introducing new ways of working and workload distribution, adding new roles and altering the existing hierarchy of decision-making and responsibility. The literature advises caution in attributing improvements to the physical and functional aspects of the CCs versus the process and policy changes within the organisation that often arise out of the implementation process. There is emerging evidence that a CC approach can be adopted in acute health care. Effective implementations are characterised by a strong sense of shared situational awareness within a team with a shared focus on specific focal points for intelligence and intensification of this focus as threat level increases. System resilience is maintained in these implementations through simultaneous responsive and anticipatory strategies with variable resource allocation for both proactive planning for expected deviations and events with varying timescales. One paper advised caution in using the term ‘command and control’, as it may overly restrict the exploring of new ideas and new approaches seen as important to meeting the specific needs of health care, hospitals and staff given the strong culture of autonomy on the clinical front line. Synthesis of research findings to inform future investment and practice (Objective 4) We found evidence that the approach to implementation was broadly successful but that benefits take time and significant additional innovation to realise. We consolidate our learning as an intervention logic model that can be used by other hospitals planning an implementation of a digitally enabled hospital CC. Conclusions This study has presented a case study of a successful implementation of a CC in the UK NHS. The Bradford Command Centre demonstrates that systems are available but it also reveals challenges in the reliability, timeliness and quality of these data that reduce transparency and limit confidence. Management and staff have managed to overcome many of these challenges through determination, negotiation and gradual improvements while also dealing with the pandemic. There was a strong sense that the CC had been invaluable during the pandemic but we were not able to empirically validate this. Our case study was affected by the pandemic and does not provide sufficient evidence to demonstrate major benefits on its own. We therefore recommend: Command centres are a viable approach that should be considered. Reliable, modern hospital-wide information systems are an essential foundation for command-centre technology and poor data quality will undermine implementation if not addressed. Further work should follow the evolving use of the Bradford Command Centre and disseminate learning to other hospitals considering adoption. Further studies that use our time-series approach for performance metrics would allow comparison across more hospitals and support the evaluation of other implementations. Studies should consider mixed methods rather than relying solely on qualitative or quantitative approaches to draw conclusions. Study registration The study is registered as IRAS No.: 285933. Funding This award was funded by the National Institute for Health and Care Research (NIHR) Health and Social Care Delivery Research programme (NIHR award ref: NIHR129483) and is published in full in Health and Social Care Delivery Research; Vol. 12, No. 41. See the NIHR Funding and Awards website for further award information.