Search

This article provides three major contributions to the literature: we provide granular information on the development of student argumentative writing across secondary school; we replicate the MacArthur et al. model of Natural Language Processing (NLP) writing features that predict quality with a younger group of students; and we are able to examine the differences for students across language status. In our study, we sought to find the average levels of text length, cohesion, connectives, syntactic complexity, and word-level complexity in this sample across Grades 7-12 by sex, by English learner status, and for essays scoring above and below the median holistic score. Mean levels of variables by grade suggest a developmental progression with respect to text length, with the text length increasing with grade level, but the other variables in the model were fairly stable. Sex did not seem to affect the model in meaningful ways beyond the increased fluency of women writers. We saw text length and word level differences between initially designated and redesignated bilingual students compared to their English-only peers. Finally, we see that the model works better with our higher scoring essays and is less effective explaining the lower scoring essays.

Assessment is viewed as an important means to understand learners' performance in the learning process. A good assessment method is based on high-quality examination questions. However, generating high-quality examination questions manually by teachers is a time-consuming task, and it is not easy for students to obtain question banks. To solve this issue, this study proposes an automatic high-quality question generation system based on natural language processing and Topic Model. A two-stage test-question generation method (sentence selection and neural question generation) is proposed in this study. We apply multisource teaching materials to select declarative sentences, and then a neural question generation model called topic-embedding question generation (TE-QG) is employed to generate high-quality examination questions. This model is based on attention and the pointer-generator mechanism. The experimental results show that the sentence selection method can select sentences that meet the key points of the course, and the performance of the TE-QG model outperforms those of existing NQG models.

Using data from approximately 342,000 course-taking records collected from 4406 college students enrolled at Taipei Tech during the 2009-2012 academic years, we examine the impact of multidisciplinarity on students' academic performance. Our study contributes to the literature in three ways. First, by applying natural language processing (NLP), we analyze course descriptions of 375 subject areas from the Classification of Instructional Programs and measure the pairwise distances among them. Second, based on the course-taking records and the subject area distribution, we measure each student's degree of multidisciplinary learning using a proposed weighted entropy formula. Third, using the proposed multidisciplinary index, we find that the impact of multidisciplinary course-taking experience on individual students' academic performance varies across academic fields. In the college of engineering, the college of electrical engineering and computer science, and the college of mechanical and electrical engineering, a higher level of multidisciplinarity is associated with a higher average weighted GPA in core courses. However, a positive association does not exist for students in the college of management.

Purpose: The purpose of this study is to explore teen-adult dialogic interactions during the co-design of data literacy activities in order to determine the nature of teen thinking, their emotions, level of engagement, and the power of relationships between teens and adults in the context of data literacy. This study conceives of co-design as a learning space for data literacy. It investigates the teen - adult dialogic interactions and what these interactions say about the nature of teen thinking, their emotions, level of engagement and the power relationships between teens and adults. Design/methodology/approach: The study conceives of co-design as a learning space for teens. Linguistic Inquiry and Word Count (LIWC-22), a natural language processing (NLP) software tool, was used to examine the linguistic measures of Analytic Thinking, Clout, Authenticity, and Emotional Tone using transcriptions of recorded Data Labs with teens and adults. Linguistic Inquiry and Word Count (LIWC-22), a natural language processing (NLP) software tool, was used to examine the linguistic measures of Analytic Thinking, Clout, Authenticity and Emotional Tone using transcriptions of recorded Data Labs with teens and adults. Findings: LIWC-22 scores on the linguistic measures Analytic Thinking, Clout, Authenticity and Emotional Tone indicate that teens had a high level of friendly engagement, a relatively low sense of power compared with the adult co-designers, medium levels of spontaneity and honesty and the prevalence of positive emotions during the co-design sessions. Practical implications: This study provides a concrete example of how to apply NLP in the context of data literacy in the public library, mapping the LIWC-22 findings to STEM-focused informal learning. It adds to the understanding of assessment/measurement tools and methods for designing data literacy education, stimulating further research and discussion on the ways to empower youth to engage more actively in informal learning about data. Originality/value: This study applies a novel approach for exploring teen engagement within a co-design project tasked with the creation of youth-oriented data literacy activities.

Rater comments tend to be qualitatively analyzed to indicate raters' application of rating scales. This study applied natural language processing (NLP) techniques to quantify meaningful, behavioral information from a corpus of rater comments and triangulated that information with a many-facet Rasch measurement (MFRM) analysis of rater scores. The data consisted of ratings on 987 essays by 36 raters (a total of 3948 analytic scores and 1974 rater comments) on a post-admission English Placement Test (EPT) at a large US university. We computed a set of comment-based features based on the analytic components and evaluative language the raters used to infer whether raters were aligned to the scale. For data triangulation, we performed correlation analyses between the MFRM measures of rater performance and the comment-based measures. Although the EPT raters showed overall satisfactory performance, we found meaningful associations between rater comments and performance features. In particular, raters with higher precision and fit to what the Rasch model predicts used more analytic components and used evaluative language more similar to the scale descriptors. These findings suggest that NLP techniques have the potential to help language testers analyze rater comments and understand rater behavior.

Background: In the past few years there has been a growing interest in the employment of verbal productions as digital biomarkers, namely objective, quantifiable behavioural data that can be collected and measured by means of digital devices, allowing for a low-cost pathology detection, classification and monitoring. Numerous research papers have been published on the automatic detection of subtle verbal alteration, starting from written texts, raw speech recordings and transcripts, and such linguistic analysis has been singled out as a cost-effective method for diagnosing dementia and other medical conditions common among elderly patients (e.g., cognitive dysfunctions associated with metabolic disorders, dysarthria). Aims: To provide a critical appraisal and synthesis of evidence concerning the application of natural language processing (NLP) techniques for clinical purposes in the geriatric population. In particular, we discuss the state of the art on studying language in healthy and pathological ageing, focusing on the latest research efforts to build non-intrusive language-based tools for the early identification of cognitive frailty due to dementia. We also discuss some challenges and open problems raised by this approach. Methods & Procedures: We performed a scoping review to examine emerging evidence about this novel domain. Potentially relevant studies published up to November 2021 were identified from the databases of MEDLINE, Cochrane and Web of Science. We also browsed the proceedings of leading international conferences (e.g., ACL, COLING, Interspeech, LREC) from 2017 to 2021, and checked the reference lists of relevant studies and reviews. Main Contribution: The paper provides an introductory, but complete, overview of the application of NLP techniques for studying language disruption due to dementia. We also suggest that this technique can be fruitfully applied to other medical conditions (e.g., cognitive dysfunctions associated with dysarthria, cerebrovascular disease and mood disorders). Conclusions & Implications: Despite several critical points need to be addressed by the scientific community, a growing body of empirical evidence shows that NLP techniques can represent a promising tool for studying language changes in pathological aging, with a high potential to lead a significant shift in clinical practice.

This article analyses the significance of ideological and political education for college students in Tibet and proposes a natural language model for an ideological education curriculum to improve the accuracy of students' document search. The segmentation results are optimized to enhance literature search accuracy, promoting the development of ideological education in Tibetan universities. By utilizing N-gram language models and enhanced technical modes, key information can be quickly obtained, developing students' interest in ideological education. The construction of a corpus is deemed crucial for expedited access to ideological education documents. The study suggests that combining information technology with ideological education can create new opportunities for innovation and reform in the field.

Recent advances in publicly available natural language processors (NLP) may enhance the efficiency of analyzing student short-answer responses in physics education research (PER). We train a state-of-the-art NLP, IBM's Watson, and test its agreement with human coders using two different studies that gathered text responses in which students explain their reasoning on physics-related questions. The first study analyzes 479 student responses to a lab data analysis question and categorizes them by main idea. The second study analyzes 732 student answers to identify the presence or absence of each of the two conceptual themes. When training Watson with approximately one-third to half of the samples, we find that samples labeled with high confidence scores have similar accuracy to human agreement; yet for lower confidence scores, humans outperform the NLP's labeling accuracy. In addition to studying Watson's overall accuracy, we use this analysis to better understand factors that impact how Watson categorizes. Using the data from the categorization study, we find that Watson's algorithm does not appear to be impacted by the disproportionate representation of categories in the training set, and we examine mislabeled statements to identify vocabulary and phrasing that may increase the rate of false positives. Based on this work, we find that, with careful consideration of the research study design and an awareness of the NLP's limitations, Watson may present a useful tool for large-scale PER studies or classroom analysis tools.

Natural language processing is an exciting and emerging field in health care that can transform the field of orthopaedics. It can aid in the process of automated clinical data analysis, changing the way we extract data for various purposes including research and registry formation, diagnosis, and medical billing. This scoping review will look at the various applications of NLP in orthopaedics. Specific examples of NLP applications include identification of essential data elements from surgical and imaging reports, patient feedback analysis, and use of AI conversational agents for patient engagement. We will demonstrate how NLP has proven itself to be a powerful and valuable tool. Despite these potential advantages, there are drawbacks we must consider. Concerns with data quality, bias, privacy, and accessibility may stand as barriers in the way of widespread implementation of NLP technology. As natural language processing technology continues to develop, it has the potential to revolutionize orthopaedic research and clinical practices and enhance patient outcomes., (Copyright © 2024 by the American Academy of Orthopaedic Surgeons.)

Purpose: Learner development and promotion rely heavily on narrative assessment comments, but narrative assessment quality is rarely evaluated in medical education. Educators have developed tools such as the Quality of Assessment for Learning (QuAL) tool to evaluate the quality of narrative assessment comments; however, scoring the comments generated in medical education assessment programs is time intensive. The authors developed a natural language processing (NLP) model for applying the QuAL score to narrative supervisor comments., Method: Samples of 2,500 Entrustable Professional Activities assessments were randomly extracted and deidentified from the McMaster (1,250 comments) and Saskatchewan (1,250 comments) emergency medicine (EM) residency training programs during the 2019-2020 academic year. Comments were rated using the QuAL score by 25 EM faculty members and 25 EM residents. The results were used to develop and test an NLP model to predict the overall QuAL score and QuAL subscores., Results: All 50 raters completed the rating exercise. Approximately 50% of the comments had perfect agreement on the QuAL score, with the remaining resolved by the study authors. Creating a meaningful suggestion for improvement was the key differentiator between high- and moderate-quality feedback. The overall QuAL model predicted the exact human-rated score or 1 point above or below it in 87% of instances. Overall model performance was excellent, especially regarding the subtasks on suggestions for improvement and the link between resident performance and improvement suggestions, which achieved 85% and 82% balanced accuracies, respectively., Conclusions: This model could save considerable time for programs that want to rate the quality of supervisor comments, with the potential to automatically score a large volume of comments. This model could be used to provide faculty with real-time feedback or as a tool to quantify and track the quality of assessment comments at faculty, rotation, program, or institution levels., (Copyright © 2024 the Association of American Medical Colleges.)

Background: Recording and coding of ageing syndromes in hospital records is known to be suboptimal. Natural Language Processing algorithms may be useful to identify diagnoses in electronic healthcare records to improve the recording and coding of these ageing syndromes, but the feasibility and diagnostic accuracy of such algorithms are unclear., Methods: We conducted a systematic review according to a predefined protocol and in line with Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guidelines. Searches were run from the inception of each database to the end of September 2023 in PubMed, Medline, Embase, CINAHL, ACM digital library, IEEE Xplore and Scopus. Eligible studies were identified via independent review of search results by two coauthors and data extracted from each study to identify the computational method, source of text, testing strategy and performance metrics. Data were synthesised narratively by ageing syndrome and computational method in line with the Studies Without Meta-analysis guidelines., Results: From 1030 titles screened, 22 studies were eligible for inclusion. One study focussed on identifying sarcopenia, one frailty, twelve falls, five delirium, five dementia and four incontinence. Sensitivity (57.1%-100%) of algorithms compared with a reference standard was reported in 20 studies, and specificity (84.0%-100%) was reported in only 12 studies. Study design quality was variable with results relevant to diagnostic accuracy not always reported, and few studies undertaking external validation of algorithms., Conclusions: Current evidence suggests that Natural Language Processing algorithms can identify ageing syndromes in electronic health records. However, algorithms require testing in rigorously designed diagnostic accuracy studies with appropriate metrics reported., (© The Author(s) 2024. Published by Oxford University Press on behalf of the British Geriatrics Society.)

Aims: To examine whether natural language processing (NLP) using a clinically based large language model (LLM) could be used to predict patient selection for total hip or total knee arthroplasty (THA/TKA) from routinely available free-text radiology reports., Methods: Data pre-processing and analyses were conducted according to the Artificial intelligence to Revolutionize the patient Care pathway in Hip and knEe aRthroplastY (ARCHERY) project protocol. This included use of de-identified Scottish regional clinical data of patients referred for consideration of THA/TKA, held in a secure data environment designed for artificial intelligence (AI) inference. Only preoperative radiology reports were included. NLP algorithms were based on the freely available GatorTron model, a LLM trained on over 82 billion words of de-identified clinical text. Two inference tasks were performed: assessment after model-fine tuning (50 Epochs and three cycles of k-fold cross validation), and external validation., Results: For THA, there were 5,558 patient radiology reports included, of which 4,137 were used for model training and testing, and 1,421 for external validation. Following training, model performance demonstrated average (mean across three folds) accuracy, F1 score, and area under the receiver operating curve (AUROC) values of 0.850 (95% confidence interval (CI) 0.833 to 0.867), 0.813 (95% CI 0.785 to 0.841), and 0.847 (95% CI 0.822 to 0.872), respectively. For TKA, 7,457 patient radiology reports were included, with 3,478 used for model training and testing, and 3,152 for external validation. Performance metrics included accuracy, F1 score, and AUROC values of 0.757 (95% CI 0.702 to 0.811), 0.543 (95% CI 0.479 to 0.607), and 0.717 (95% CI 0.657 to 0.778) respectively. There was a notable deterioration in performance on external validation in both cohorts., Conclusion: The use of routinely available preoperative radiology reports provides promising potential to help screen suitable candidates for THA, but not for TKA. The external validation results demonstrate the importance of further model testing and training when confronted with new clinical cohorts., Competing Interests: L. Farrow is currently in receipt of a Chief Scientist Office (CSO) Scotland Clinical Academic Fellowship of which the submitted work will form part of a PhD thesis. L. Farrow also reports travel and accommodation support from LINK to attend an educational event, unrelated to this study, and is lead for the Aberdeen AI in Healthcare Collaborative, and chair of the Scottish Hip Fracture Audit Quality Improvement and Research Sub-Group, both of which are unpaid positions. M. Zhong and L. Anderson are supervisors for the PhD project., (© 2024 The British Editorial Society of Bone & Joint Surgery.)

Introduction: Responses to COVID-19 within medical education prompted significant changes to the surgical clerkship. We analyzed the changes in medical student end of course feedback before and after the COVID-19 outbreak., Methods: Postclerkship surveys from 2017 to 2022 were analyzed including both Likert scale data and free text, excluding the COVID outbreak year 2019-2020. Likert scale questions were compared between pre-COVID (2017-2019) and COVID-era cohorts (2020-2022) with the Mann-Whitney U-test. Free-text comments were analyzed using both thematic analysis and natural language processing including sentiment, word and phrase frequency, and topic modeling., Results: Of the 483 medical students surveyed from 2017 to 2022, 297 responded (61% response rate) to the included end of clerkship surveys. Most medical students rated the clerkship above average or excellent with no significant difference between the pre-COVID and COVID-era cohorts (70.4% Versus 64.8%, P = 0.35). Perception of grading expectations did significantly differ, 51% of pre-COVID students reported clerkship grading standards were almost always clear compared to 27.5% of COVID-era students (P = 0.01). Pre-COVID cohorts more frequently mentioned learning and feedback while COVID-era cohorts more frequently mentioned case, attending, and expectation. Natural language processing topic modeling and formal thematic analysis identified similar themes: team, time, autonomy, and expectations., Conclusions: COVID-19 presented many challenges to undergraduate medical education. Despite many changes, there was no significant difference in clerkship satisfaction ratings. Unexpectedly, the greater freedom and autonomy of asynchronous lectures and choice of cases became a highlight of the new curriculum. Future research should investigate if there are similar associations nationally with a multi-institutional study., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: Natural language processing (NLP) represents one of the adjunct technologies within artificial intelligence and machine learning, creating structure out of unstructured data. This study aims to assess the performance of employing NLP to identify and categorize unstructured data within the emergency medicine (EM) setting., Methods: We systematically searched publications related to EM research and NLP across databases including MEDLINE, Embase, Scopus, CENTRAL, and ProQuest Dissertations & Theses Global. Independent reviewers screened, reviewed, and evaluated article quality and bias. NLP usage was categorized into syndromic surveillance, radiologic interpretation, and identification of specific diseases/events/syndromes, with respective sensitivity analysis reported. Performance metrics for NLP usage were calculated and the overall area under the summary of receiver operating characteristic curve (SROC) was determined., Results: A total of 27 studies underwent meta-analysis. Findings indicated an overall mean sensitivity (recall) of 82%-87%, specificity of 95%, with the area under the SROC at 0.96 (95% CI 0.94-0.98). Optimal performance using NLP was observed in radiologic interpretation, demonstrating an overall mean sensitivity of 93% and specificity of 96%., Conclusions: Our analysis revealed a generally favorable performance accuracy in using NLP within EM research, particularly in the realm of radiologic interpretation. Consequently, we advocate for the adoption of NLP-based research to augment EM health care management., (© 2024 The Authors. Academic Emergency Medicine published by Wiley Periodicals LLC on behalf of Society for Academic Emergency Medicine.)

Objective: To develop and validate TraumaICDBERT, a natural language processing algorithm to predict injury International Classification of Diseases, 10th edition (ICD-10) diagnosis codes from trauma tertiary survey notes., Background: The adoption of ICD-10 diagnosis codes in clinical settings for injury prediction is hindered by the lack of real-time availability. Existing natural language processing algorithms have limitations in accurately predicting injury ICD-10 diagnosis codes., Methods: Trauma tertiary survey notes from hospital encounters of adults between January 2016 and June 2021 were used to develop and validate TraumaICD Bidirectional Encoder Representation from Transformers (TraumaICDBERT), an algorithm based on BioLinkBERT. The performance of TraumaICDBERT was compared with Amazon Web Services Comprehend Medical, an existing natural language processing tool., Results: A data set of 3478 tertiary survey notes with 15,762 4-character injury ICD-10 diagnosis codes was analyzed. TraumaICDBERT outperformed Amazon Web Services Comprehend Medical across all evaluated metrics. On average, each tertiary survey note was associated with 3.8 (SD: 2.9) trauma registrar-extracted 4-character injury ICD-10 diagnosis codes., Conclusions: TraumaICDBERT demonstrates promising initial performance in predicting injury ICD-10 diagnosis codes from trauma tertiary survey notes, potentially facilitating the adoption of downstream prediction tools in clinical settings., Competing Interests: The authors report no conflicts of interest., (Copyright © 2023 Wolters Kluwer Health, Inc. All rights reserved.)

Objective: Natural language processing (NLP) algorithms are increasingly being applied to obtain unsupervised representations of electronic health record (EHR) data, but their comparative performance at predicting clinical endpoints remains unclear. Our objective was to compare the performance of unsupervised representations of sequences of disease codes generated by bag-of-words versus sequence-based NLP algorithms at predicting clinically relevant outcomes., Materials and Methods: This cohort study used primary care EHRs from 6 286 233 people with Multiple Long-Term Conditions in England. For each patient, an unsupervised vector representation of their time-ordered sequences of diseases was generated using 2 input strategies (212 disease categories versus 9462 diagnostic codes) and different NLP algorithms (Latent Dirichlet Allocation, doc2vec, and 2 transformer models designed for EHRs). We also developed a transformer architecture, named EHR-BERT, incorporating sociodemographic information. We compared the performance of each of these representations (without fine-tuning) as inputs into a logistic classifier to predict 1-year mortality, healthcare use, and new disease diagnosis., Results: Patient representations generated by sequence-based algorithms performed consistently better than bag-of-words methods in predicting clinical endpoints, with the highest performance for EHR-BERT across all tasks, although the absolute improvement was small. Representations generated using disease categories perform similarly to those using diagnostic codes as inputs, suggesting models can equally manage smaller or larger vocabularies for prediction of these outcomes., Discussion and Conclusion: Patient representations produced by sequence-based NLP algorithms from sequences of disease codes demonstrate improved predictive content for patient outcomes compared with representations generated by co-occurrence-based algorithms. This suggests transformer models may be useful for generating multi-purpose representations, even without fine-tuning., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)

Background: Extracting research of domain criteria (RDoC) from high-risk populations like those with post-traumatic stress disorder (PTSD) is crucial for positive mental health improvements and policy enhancements. The intricacies of collecting, integrating, and effectively leveraging clinical notes for this purpose introduce complexities., Methods: In our study, we created a natural language processing (NLP) workflow to analyze electronic medical record (EMR) data and identify and extract research of domain criteria using a pre-trained transformer-based natural language model, all-mpnet-base-v2. We subsequently built dictionaries from 100,000 clinical notes and analyzed 5.67 million clinical notes from 38,807 PTSD patients from the University of Pittsburgh Medical Center. Subsequently, we showcased the significance of our approach by extracting and visualizing RDoC information in two use cases: (i) across multiple patient populations and (ii) throughout various disease trajectories., Results: The sentence transformer model demonstrated high F1 macro scores across all RDoC domains, achieving the highest performance with a cosine similarity threshold value of 0.3. This ensured an F1 score of at least 80% across all RDoC domains. The study revealed consistent reductions in all six RDoC domains among PTSD patients after psychotherapy. We found that 60.6% of PTSD women have at least one abnormal instance of the six RDoC domains as compared to PTSD men (51.3%), with 45.1% of PTSD women with higher levels of sensorimotor disturbances compared to men (41.3%). We also found that 57.3% of PTSD patients have at least one abnormal instance of the six RDoC domains based on our records. Also, veterans had the higher abnormalities of negative and positive valence systems (60% and 51.9% of veterans respectively) compared to non-veterans (59.1% and 49.2% respectively). The domains following first diagnoses of PTSD were associated with heightened cue reactivity to trauma, suicide, alcohol, and substance consumption., Conclusions: The findings provide initial insights into RDoC functioning in different populations and disease trajectories. Natural language processing proves valuable for capturing real-time, context dependent RDoC instances from extensive clinical notes., (© 2024. The Author(s).)

Introduction: Bidirectional Encoder Representations from Transformers (BERT), introduced in 2018, has revolutionized natural language processing. Its bidirectional understanding of word context has enabled innovative applications, notably in radiology. This study aimed to assess BERT's influence and applications within the radiologic domain., Methods: Adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, we conducted a systematic review, searching PubMed for literature on BERT-based models and natural language processing in radiology from January 1, 2018, to February 12, 2023. The search encompassed keywords related to generative models, transformer architecture, and various imaging techniques., Results: Of 597 results, 30 met our inclusion criteria. The remaining were unrelated to radiology or did not use BERT-based models. The included studies were retrospective, with 14 published in 2022. The primary focus was on classification and information extraction from radiology reports, with x-rays as the prevalent imaging modality. Specific investigations included automatic CT protocol assignment and deep learning applications in chest x-ray interpretation., Conclusion: This review underscores the primary application of BERT in radiology for report classification. It also reveals emerging BERT applications for protocol assignment and report generation. As BERT technology advances, we foresee further innovative applications. Its implementation in radiology holds potential for enhancing diagnostic precision, expediting report generation, and optimizing patient care., (Copyright © 2024 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

Objective: To develop and externally validate models based on neural networks and natural language processing (NLP) to identify suspected serious infections in emergency department (ED) patients afebrile at initial presentation., Methods: This retrospective study included adults who visited the ED afebrile at initial presentation. We developed four models based on artificial neural networks to identify suspected serious infection. Patient demographics, vital signs, laboratory test results and information extracted from initial ED physician notes using term frequency-inverse document frequency were used as model variables. Models were trained and internally validated with data from one hospital and externally validated using data from a different hospital. Model discrimination was evaluated using area under the receiver operating characteristic curve (AUC) and 95% confidence intervals (CIs)., Results: The training, internal validation, and external validation datasets comprised 150,699, 37,675, and 85,098 patients, respectively. The AUCs (95% CIs) for Models 1 (demographics + vital signs), 2 (demographics + vital signs + initial ED physician note), 3 (demographics + vital signs + laboratory tests), and 4 (demographics + vital signs + laboratory tests + initial ED physician note) in the internal validation dataset were 0.789 (0.782-0.796), 0.867 (0.862-0.872), 0.881 (0.876-0.887), and 0.911 (0.906-0.915), respectively. In the external validation dataset, the AUCs (95% CIs) of Models 1, 2, 3, and 4 were 0.824 (0.817-0.830), 0.895 (0.890-0.899), 0.879 (0.873-0.884), and 0.913 (0.909-0.917), respectively. Model 1 can be utilized immediately after ED triage, Model 2 can be utilized after the initial physician notes are recorded (median time from ED triage: 28 min), and Models 3 and 4 can be utilized after the initial laboratory tests are reported (median time from ED triage: 68 min)., Conclusions: We developed and validated models to identify suspected serious infection in the ED. Extracted information from initial ED physician notes using NLP contributed to increased model performance, permitting identification of suspected serious infection at early stages of ED visits., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Natural Language Processing (NLP), a form of Artificial Intelligence, allows free-text based clinical documentation to be integrated in ways that facilitate data analysis, data interpretation and formation of individualized medical and obstetrical care. In this cross-sectional study, we identified all births during the study period carrying the radiology-confirmed diagnosis of fibroid uterus in pregnancy (defined as size of largest diameter of >5 cm) by using an NLP platform and compared it to non-NLP derived data using ICD10 codes of the same diagnosis. We then compared the two sets of data and stratified documentation gaps by race. Using fibroid uterus in pregnancy as a marker, we found that Black patients were more likely to have the diagnosis entered late into the patient's chart or had missing documentation of the diagnosis. With appropriate algorithm definitions, cross referencing and thorough validation steps, NLP can contribute to identifying areas of documentation gaps and improve quality of care., Competing Interests: Declaration of competing interest H.F. and O.S.Y. Are employed by Gynisus Inc. All other authors report no conflict of interest., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: Screening for pancreatic ductal adenocarcinoma (PDAC) is considered in high-risk individuals (HRIs) with established PDAC risk factors, such as family history and germline mutations in PDAC susceptibility genes. Accurate assessment of risk factor status is provider knowledge-dependent and requires extensive manual chart review by experts. Natural Language Processing (NLP) has shown promise in automated data extraction from the electronic health record (EHR). We aimed to use NLP for automated extraction of PDAC risk factors from unstructured clinical notes in the EHR., Methods: We first developed rule-based NLP algorithms to extract PDAC risk factors at the document-level, using an annotated corpus of 2091 clinical notes. Next, we further improved the NLP algorithms using a cohort of 1138 patients through patient-level training, validation, and testing, with comparison against a pre-specified reference standard. To minimize false-negative results we prioritized algorithm recall., Results: In the test set (n = 807), the NLP algorithms achieved a recall of 0.933, precision of 0.790, and F 1 -score of 0.856 for family history of PDAC. For germline genetic mutations, the algorithm had a high recall of 0.851, while precision and F 1 -score were lower at 0.350 and 0.496 respectively. Most false positives for germline mutations resulted from erroneous recognition of tissue mutations., Conclusions: Rule-based NLP algorithms applied to unstructured clinical notes are highly sensitive for automated identification of PDAC risk factors. Further validation in a large primary-care patient population is warranted to assess real-world utility in identifying HRIs for pancreatic cancer screening., Competing Interests: Declaration of competing interest As co-inventor on licensed technologies, S.M. could share potential future royalties to Mayo Clinic from Exact Sciences in accordance with institutional policy and oversight. The other authors of this manuscript have no conflict of interest to declare., (Copyright © 2024 IAP and EPC. Published by Elsevier B.V. All rights reserved.)

Background: Clinical natural language processing (NLP) researchers need access to directly comparable evaluation results for applications such as text deidentification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and the personally identifiable information (PII) categories evaluated are not easily comparable., Objective: This study presents an open-source and extensible end-to-end framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align., Methods: As a use case for this framework, we use 6 off-the-shelf text deidentification systems (ie, CliniDeID, deid from PhysioNet, MITRE Identity Scrubber Toolkit [MIST], NeuroNER, National Library of Medicine [NLM] Scrubber, and Philter) across 3 standard clinical text corpora for the task (2 of which are publicly available) and 1 private corpus (all in English), with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings. Code for running this framework with all settings needed to run all pairs are available via Codeberg and GitHub., Results: From this case study, we found large differences in processing speed between systems. The fastest system (ie, MIST) processed an average of 24.57 (SD 26.23) notes per second, while the slowest (ie, CliniDeID) processed an average of 1.00 notes per second. No system uniformly outperformed the others at identifying PII across corpora and categories. Instead, a rich tapestry of performance trade-offs emerged for PII categories. CliniDeID and Philter prioritize recall over precision (with an average recall 6.9 and 11.2 points higher, respectively, for partially matching spans of text matching any PII category), while the other 4 systems consistently have higher precision (with MIST's precision scoring 20.2 points higher, NLM Scrubber scoring 4.4 points higher, NeuroNER scoring 7.2 points higher, and deid scoring 17.1 points higher). The macroaverage recall across corpora for identifying names, one of the more sensitive PII categories, included deid (48.8%) and MIST (66.9%) at the low end and NeuroNER (84.1%), NLM Scrubber (88.1%), and CliniDeID (95.9%) at the high end. A variety of metrics across categories and corpora are reported with a wider variety (eg, F 2 -score) available via the tool., Conclusions: NLP systems in general and deidentification systems and corpora in our use case tend to be evaluated in stand-alone research articles that only include a limited set of comparators. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. Our open pipeline should reduce barriers to evaluation and system advancement., (©Paul M Heider, Stéphane M Meystre. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 28.05.2024.)

Objective: To develop and validate a natural language processing (NLP) pipeline that detects 18 conditions in French clinical notes, including 16 comorbidities of the Charlson index, while exploring a collaborative and privacy-enhancing workflow., Materials and Methods: The detection pipeline relied both on rule-based and machine learning algorithms, respectively, for named entity recognition and entity qualification, respectively. We used a large language model pre-trained on millions of clinical notes along with annotated clinical notes in the context of 3 cohort studies related to oncology, cardiology, and rheumatology. The overall workflow was conceived to foster collaboration between studies while respecting the privacy constraints of the data warehouse. We estimated the added values of the advanced technologies and of the collaborative setting., Results: The pipeline reached macro-averaged F1-score positive predictive value, sensitivity, and specificity of 95.7 (95%CI 94.5-96.3), 95.4 (95%CI 94.0-96.3), 96.0 (95%CI 94.0-96.7), and 99.2 (95%CI 99.0-99.4), respectively. F1-scores were superior to those observed using alternative technologies or non-collaborative settings. The models were shared through a secured registry., Conclusions: We demonstrated that a community of investigators working on a common clinical data warehouse could efficiently and securely collaborate to develop, validate and use sensitive artificial intelligence models. In particular, we provided an efficient and robust NLP pipeline that detects conditions mentioned in clinical notes., (© The Author(s) 2024. Published by Oxford University Press on behalf of the American Medical Informatics Association.)

Background: There is an urgent need worldwide for qualified health professionals. High attrition rates among health professionals, combined with a predicted rise in life expectancy, further emphasize the need for additional health professionals. Work-related stress is a major concern among health professionals, affecting both the well-being of health professionals and the quality of patient care., Objective: This scoping review aims to identify processes and methods for the automatic detection of work-related stress among health professionals using natural language processing (NLP) and text mining techniques., Methods: This review follows Joanna Briggs Institute Methodology and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. The inclusion criteria for this scoping review encompass studies involving health professionals using NLP for work-related stress detection while excluding studies involving other professions or children. The review focuses on various aspects, including NLP applications for stress detection, criteria for stress identification, technical aspects of NLP, and implications of stress detection through NLP. Studies within health care settings using diverse NLP techniques are considered, including experimental and observational designs, aiming to provide a comprehensive understanding of NLP's role in detecting stress among health professionals. Studies published in English, German, or French from 2013 to present will be considered. The databases to be searched include MEDLINE (via PubMed), CINAHL, PubMed, Cochrane, ACM Digital Library, and IEEE Xplore. Sources of unpublished studies and gray literature to be searched will include ProQuest Dissertations & Theses and OpenGrey. Two reviewers will independently retrieve full-text studies and extract data. The collected data will be organized in tables, graphs, and a qualitative narrative summary. This review will use tables and graphs to present data on studies' distribution by year, country, activity field, and research methods. Results synthesis involves identifying, grouping, and categorizing. The final scoping review will include a narrative written report detailing the search and study selection process, a visual representation using a PRISMA-ScR flow diagram, and a discussion of implications for practice and research., Results: We anticipate the outcomes will be presented in a systematic scoping review by June 2024., Conclusions: This review fills a literature gap by identifying automated work-related stress detection among health professionals using NLP and text mining, providing insights on an innovative approach, and identifying research needs for further systematic reviews. Despite promising outcomes, acknowledging limitations in the reviewed studies, including methodological constraints, sample biases, and potential oversight, is crucial to refining methodologies and advancing automatic stress detection among health professionals., International Registered Report Identifier (irrid): PRR1-10.2196/56267., (©Jannic Stefan Bieri, Catherine Ikae, Souhir Ben Souissi, Thomas Jörg Müller, Margarithe Charlotte Schlunegger, Christoph Golz. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 15.05.2024.)

Background: There is growing concern around the use of sodium nitrite (SN) as an emerging means of suicide, particularly among younger people. Given the limited information on the topic from traditional public health surveillance sources, we studied posts made to an online suicide discussion forum, "Sanctioned Suicide," which is a primary source of information on the use and procurement of SN., Objective: This study aims to determine the trends in SN purchase and use, as obtained via data mining from subscriber posts on the forum. We also aim to determine the substances and topics commonly co-occurring with SN, as well as the geographical distribution of users and sources of SN., Methods: We collected all publicly available from the site's inception in March 2018 to October 2022. Using data-driven methods, including natural language processing and machine learning, we analyzed the trends in SN mentions over time, including the locations of SN consumers and the sources from which SN is procured. We developed a transformer-based source and location classifier to determine the geographical distribution of the sources of SN., Results: Posts pertaining to SN show a rise in popularity, and there were statistically significant correlations between real-life use of SN and suicidal intent when compared to data from the Centers for Disease Control and Prevention (CDC) Wide-Ranging Online Data for Epidemiologic Research (⍴=0.727; P<.001) and the National Poison Data System (⍴=0.866; P=.001). We observed frequent co-mentions of antiemetics, benzodiazepines, and acid regulators with SN. Our proposed machine learning-based source and location classifier can detect potential sources of SN with an accuracy of 72.92% and showed consumption in the United States and elsewhere., Conclusions: Vital information about SN and other emerging mechanisms of suicide can be obtained from online forums., (© Sudeshna Das, Drew Walker, Swati Rajwal, Sahithi Lakamana, Steven A Sumner, Karin A Mack, Wojciech Kaczkowski, Abeed Sarker. Originally published in JMIR Mental Health (https://mental.jmir.org).)

Building clinical registries is an important step in clinical research and improvement of patient care quality. Natural Language Processing (NLP) methods have shown promising results in extracting valuable information from unstructured clinical notes. However, the structure and nature of clinical notes are very different from regular text that state-of-the-art NLP models are trained and tested on, and they have their own set of challenges. In this study, we propose Sentence Extractor with Keywords (SE-K), an efficient and interpretable classification approach for extracting information from clinical notes and show that it outperforms more computationally expensive methods in text classification. Following the Institutional Review Board (IRB) approval, we used SE-K and two embedding based NLP approaches (Sentence Extractor with Embeddings (SE-E) and Bidirectional Encoder Representations from Transformers (BERT)) to develop comprehensive registry of anterior cruciate ligament surgeries from 20 years of unstructured clinical data at a multi-site tertiary-care regional children's hospital. The low-resource approach (SE-K) had better performance (average AUROC of 0.94 ± 0.04) than the embedding-based approaches (SE-E: 0.93 ± 0.04 and BERT: 0.87 ± 0.09) for out of sample validation, in addition to minimum performance drop between test and out-of-sample validation. Moreover, the SE-K approach was at least six times faster (on CPU) than SE-E (on CPU) and BERT (on GPU) and provides interpretability. Our proposed approach, SE-K, can be effectively used to extract relevant variables from clinic notes to build large-scale registries, with consistently better performance compared to the more resource-intensive approaches (e.g., BERT). Such approaches can facilitate information extraction from unstructured notes for registry building, quality improvement and adverse event monitoring., Competing Interests: Declaration of competing interest No conflicts of interests relevant to the topic of this study., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Background: Electronic health records (EHRs) are a valuable resource for data-driven medical research. However, the presence of protected health information (PHI) makes EHRs unsuitable to be shared for research purposes. De-identification, i.e. the process of removing PHI is a critical step in making EHR data accessible. Natural language processing has repeatedly demonstrated its feasibility in automating the de-identification process., Objectives: Our study aims to provide systematic evidence on how the de-identification of clinical free text written in English has evolved in the last thirteen years, and to report on the performances and limitations of the current state-of-the-art systems for the English language. In addition, we aim to identify challenges and potential research opportunities in this field., Methods: A systematic search in PubMed, Web of Science, and the DBLP was conducted for studies published between January 2010 and February 2023. Titles and abstracts were examined to identify the relevant studies. Selected studies were then analysed in-depth, and information was collected on de-identification methodologies, data sources, and measured performance., Results: A total of 2125 publications were identified for the title and abstract screening. 69 studies were found to be relevant. Machine learning (37 studies) and hybrid (26 studies) approaches are predominant, while six studies relied only on rules. The majority of the approaches were trained and evaluated on public corpora. The 2014 i2b2/UTHealth corpus is the most frequently used (36 studies), followed by the 2006 i2b2 (18 studies) and 2016 CEGS N-GRID (10 studies) corpora., Conclusion: Earlier de-identification approaches aimed at English were mainly rule and machine learning hybrids with extensive feature engineering and post-processing, while more recent performance improvements are due to feature-inferring recurrent neural networks. Current leading performance is achieved using attention-based neural models. Recent studies report state-of-the-art F1-scores (over 98 %) when evaluated in the manner usually adopted by the clinical natural language processing community. However, their performance needs to be more thoroughly assessed with different measures to judge their reliability to safely de-identify data in a real-world setting. Without additional manually labeled training data, state-of-the-art systems fail to generalise well across a wide range of clinical sub-domains., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Goran Nenadić reports financial support was provided by Engineering and Physical Sciences Research Council for HealTex—UK Healthcare Text Analytics Research Network (grant no. EP/N027280/1). Aleksandar Kovačević reports financial support was provided by the Republic of Serbia Ministry of Science, Technological Development and Innovation through “Innovative scientific and artistic research from the FTS (activity) domain” (project no. 451-03-47/2023-01/200156)., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Introduction: Electronic health records (EHR) are of great value for clinical research. However, EHR consists primarily of unstructured text which must be analysed by a human and coded into a database before data analysis- a time-consuming and costly process limiting research efficiency. Natural language processing (NLP) can facilitate data retrieval from unstructured text. During AssistMED project, we developed a practical, NLP tool that automatically provides comprehensive clinical characteristics of patients from EHR, that is tailored to clinical researchers needs., Material and Methods: AssistMED retrieves patient characteristics regarding clinical conditions, medications with dosage, and echocardiographic parameters with clinically oriented data structure and provides researcher-friendly database output. We validate the algorithm performance against manual data retrieval and provide critical quantitative and qualitative analysis., Results: AssistMED analysed the presence of 56 clinical conditions, medications from 16 drug groups with dosage and 15 numeric echocardiographic parameters in a sample of 400 patients hospitalized in the cardiology unit. No statistically significant differences between algorithm and human retrieval were noted. Qualitative analysis revealed that disagreements with manual annotation were primarily accounted to random algorithm errors, erroneous human annotation and lack of advanced context awareness of our tool., Conclusions: Current NLP approaches are feasible to acquire accurate and detailed patient characteristics tailored to clinical researchers' needs from EHR. We present an in-depth description of an algorithm development and validation process, discuss obstacles and pinpoint potential solutions, including opportunities arising with recent advancements in the field of NLP, such as large language models., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 Elsevier B.V. All rights reserved.)

Digital therapeutics (DTx) are a somewhat novel class of US Food and Drug Administration-regulated software that help patients prevent, manage, or treat disease. Here, we use natural language processing to characterise registered DTx clinical trials and provide insights into the clinical development landscape for these novel therapeutics. We identified 449 DTx clinical trials, initiated or expected to be initiated between 2010 and 2030, from ClinicalTrials.gov using 27 search terms, and available data were analysed, including trial durations, locations, MeSH categories, enrolment, and sponsor types. Topic modelling of eligibility criteria, done with BERTopic, showed that DTx trials frequently exclude patients on the basis of age, comorbidities, pregnancy, language barriers, and digital determinants of health, including smartphone or data plan access. Our comprehensive overview of the DTx development landscape highlights challenges in designing inclusive DTx clinical trials and presents opportunities for clinicians and researchers to address these challenges. Finally, we provide an interactive dashboard for readers to conduct their own analyses.

Background: Diagnoses entered by general practitioners into electronic medical records have great potential for research and practice, but unfortunately, diagnoses are often in uncoded format, making them of little use. Natural language processing (NLP) could assist in coding free-text diagnoses, but NLP models require local training data to unlock their potential. The aim of this study was to develop a framework of research-relevant diagnostic codes, to test the framework using free-text diagnoses from a Swiss primary care database and to generate training data for NLP modelling., Methods: The framework of diagnostic codes was developed based on input from local stakeholders and consideration of epidemiological data. After pre-testing, the framework contained 105 diagnostic codes, which were then applied by two raters who independently coded randomly drawn lines of free text (LoFT) from diagnosis lists extracted from the electronic medical records of 3000 patients of 27 general practitioners. Coding frequency and mean occurrence rates (n and %) and inter-rater reliability (IRR) of coding were calculated using Cohen's kappa (Κ)., Results: The sample consisted of 26,980 LoFT and in 56.3% no code could be assigned because it was not a specific diagnosis. The most common diagnostic codes were, 'dorsopathies' (3.9%, a code covering all types of back problems, including non-specific lower back pain, scoliosis, and others) and 'other diseases of the circulatory system' (3.1%). Raters were in almost perfect agreement (Κ ≥ 0.81) for 69 of the 105 diagnostic codes, and 28 codes showed a substantial agreement (K between 0.61 and 0.80). Both high coding frequency and almost perfect agreement were found in 37 codes, including codes that are particularly difficult to identify from components of the electronic medical record, such as musculoskeletal conditions, cancer or tobacco use., Conclusion: The coding framework was characterised by a subset of very frequent and highly reliable diagnostic codes, which will be the most valuable targets for training NLP models for automated disease classification based on free-text diagnoses from Swiss general practice., (© 2024. The Author(s).)