Search

Data extraction is a crucial, yet labor‐intensive and error‐prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof‐of‐concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English‐language, open‐access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test–retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero‐shot learning). Based on findings of our proof‐of‐concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses. [ABSTRACT FROM AUTHOR]

Antibiotic exposure can lead to unintended outcomes, including drug-drug interactions, adverse drug events, and healthcare-associated infections like Clostridioides difficile infection (CDI). Improving antibiotic use is critical to reduce an individual's CDI risk. Antibiotic stewardship initiatives can reduce inappropriate antibiotic prescribing (e.g., unnecessary antibiotic prescribing, inappropriate antibiotic selection), impacting both hospital (healthcare)-onset (HO)-CDI and community-associated (CA)-CDI. Previous computational and mathematical modeling studies have demonstrated a reduction in CDI incidence associated with antibiotic stewardship initiatives in hospital settings. Although the impact of antibiotic stewardship initiatives in long-term care facilities (LTCFs), including nursing homes, and in outpatient settings have been documented, the effects of specific interventions on CDI incidence are not well understood. We examined the relative effectiveness of antibiotic stewardship interventions on CDI incidence using a geospatially explicit agent-based model of a regional healthcare network in North Carolina. We simulated reductions in unnecessary antibiotic prescribing and inappropriate antibiotic selection with intervention scenarios at individual and network healthcare facilities, including short-term acute care hospitals (STACHs), nursing homes, and outpatient locations. Modeled antibiotic prescription rates were calculated using patient-level data on antibiotic length of therapy for the 10 modeled network STACHs. By simulating a 30% reduction in antibiotics prescribed across all inpatient and outpatient locations, we found the greatest reductions on network CDI incidence among tested scenarios, namely a 17% decrease in HO-CDI incidence and 7% decrease in CA-CDI. Among intervention scenarios of reducing inappropriate antibiotic selection, we found a greater impact on network CDI incidence when modeling this reduction in nursing homes alone compared to the same intervention in STACHs alone. These results support the potential importance of LTCF and outpatient antibiotic stewardship efforts on network CDI burden and add to the evidence that a coordinated approach to antibiotic stewardship across multiple facilities, including inpatient and outpatient settings, within a regional healthcare network could be an effective strategy to reduce network CDI burden. [ABSTRACT FROM AUTHOR]

Agent-based models (ABMs) describe and simulate complex systems comprising unique agents, or individuals, while accounting for geospatial and temporal variability among dynamic processes. ABMs are increasingly used to study healthcare-associated infections (ie, infections acquired during admission to a healthcare facility), including Clostridioides difficile infection, currently the most common healthcare-associated infection in the United States. The overall burden and transmission dynamics of healthcare-associated infections, including C difficile infection, may be influenced by community sources and movement of people among healthcare facilities and communities. These complex dynamics warrant geospatially explicit ABMs that extend beyond single healthcare facilities to include entire systems (eg, hospitals, nursing homes and extended care facilities, the community). The agents in ABMs can be built on a synthetic population, a model-generated representation of the actual population with associated spatial (eg, home residence), temporal (eg, change in location over time), and nonspatial (eg, sociodemographic features) attributes. We describe our methods to create a geospatially explicit ABM of a major regional healthcare network using a synthetic population as microdata input. We illustrate agent movement in the healthcare network and the community, informed by patient-level medical records, aggregate hospital discharge data, healthcare facility licensing data, and published literature. We apply the ABM output to visualize agent movement in the healthcare network and the community served by the network. We provide an application example of the ABM to C difficile infection using a natural history submodel. We discuss the ABM's potential to detect network areas where disease risk is high; simulate and evaluate interventions to protect public health; adapt to other geographic locations and healthcare-associated infections, including emerging pathogens; and meaningfully translate results to public health practitioners, healthcare providers, and policymakers. Agent-based models describe and simulate complex systems comprising unique agents, or individuals, while accounting for geospatial and temporal variability among dynamic processes. They are increasingly used to study healthcare-associated infections, including Clostridioides difficile infection. The authors describe their methods of creating a geospatially explicit model of a major regional healthcare network using a synthetic population as microdata input. [ABSTRACT FROM AUTHOR]

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT‐4) for extracting pre‐specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT‐4. GPT‐4 required use of a third‐party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT‐4 with the plug‐in was lower (68.8%); however, most of the errors were due to the plug‐in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT‐4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open‐source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions. [ABSTRACT FROM AUTHOR]