Search

Authors: Ye-Jean Park, Abhinav Pillai, Jiawen Deng, Mehul Gupta, Edward Guo, Mike Paget, and Christopher Naugler Large language models (LLMs) – artificial intelligence (AI) systems based on deep learning algorithms– are capable of processing and summarizing vast volumes of text and subsequently generating human-like responses when prompted (1). Through 1) language model pretraining, where the model learns a large corpus of raw, unlabeled text in an unsupervised manner, and 2) fine-tuning, where the model learns to apply knowledge from one task to another particular task using labelled training data, LLMs are ultimately able to understand and create a countless number of text styles and syntaxes (1). Techniques such as transfer learning have enabled LLMs to expand their capabilities beyond pretraining and fine-tuning, and now include techniques such as zero-shot and few-shot learning, allowing them to perform new tasks with little to no additional training data (2, 3). Therefore, LLMs have broad applicability to systems ranging from natural language processors (NLPs) to language translators to virtual assistant chatbots. Several types of LLMs exist, including transformer models, such as Bidirectional Encoder Representations from Transformers (BERT) or Generative Pre-trained Transformer (GPT) (4). Such models are based on what is known as a transformer architecture, a type of neural network model considered the state-of-the-art system used for natural language processing tasks given its self-attention mechanism (used to process input sequences in parallel form and therefore learn context and relationships between words in the given sequences) (4). ChatGPT, an AI model created by Open AI lab in San Francisco that launched in November of 2022, is a variant of GPT and build with the GPT-3.5 large language model (5). As an LLM, ChatGPT is trained on an approximately 570 GB-sized dataset consisting of billions of words and text from multiple languages, including from CommonCrawl and WebText, internet-based book corpora, Wikipedia (8). ChatGPT also possesses a language model head along with a dialogue generation head, used for Reinforcement Learning from Human Feedback (RLHF) to understand and generate human-like text (8). Furthermore, ChatGPT performs “continual/incremental learning”, wherein the model can maintain a memory of previous input and prompts to subsequently improve the accuracy and relevance of its future responses with each iteration of text by strengthening its neural network (5). ChatGPT is also “auto-aggressive”, meaning it can predict and generate future words based on previous sequences presented to it in a uni-directional manner, whereas the comparator BERT has bi-directional architecture and therefore analyzes a word based on the word’s left and right context (4, 7-9). With all these characteristics, ChatGPT has recently emerged as a more promising candidate for original text and dialogue generation, and is a prevalent subject of ongoing discourse and research. Given its potential impact in the real-world context, GPT has been increasingly explored across many contexts and disciplines. For instance, researchers internationally are assessing the model’s performance on business, law, and medical school board exams, its ability to code in Python and JavaScript, and its capabilities of creating original literature and poems (10-13). One particular area of interest is the broader applicability of LLMs like GPT in the realm of biomedicine and healthcare. In fact, BioGPT was launched in late 2022 and was trained on millions of PubMed abstracts to be one of the first biomedical-specific versions of GPT and thereby address limitations of GPT variants in providing evidence-based, tailored responses in biomedical sciences (14). In a similar vein, GPT-4 was launched on March 15, 2023, and said to address several limitations of GPT-3 models (as evidenced by its ranking in some of the highest percentiles on standardized exams rather than in the lower percentiles, possessing greater accuracy across multiple languages, and GPT-4’s capacity to accept prompts consisting of images and diagrams, unlike GPT-3 (15)). Overall, LLMs – particularly GPT and its variants – appear to possess both promising but also concerning features if integrated into the healthcare setting. For instance, several authors have described LLMs’ potential in clinical decision support to provide further information for healthcare teams to make more informed decisions for patient care (16, 18). Others have highlighted the improved speed and efficiency of clinical practice by using programs like ChatGPT in composing patient charts or in creating efficient, standardized discharge notes (18). Furthermore, all of these could be further vetted by the healthcare team to avoid errors while saving time for healthcare providers to focus on delivering optimized patient care (19). Likewise, outside of the immediate clinical setting, individuals may use widely accessible LLMs like ChatGPT in asking their health-related questions. Thus, LLMs may not only have the potential to elevate patient care and optimize hospital administrative tasks, but also in empowering patients with healthcare knowledge that, ideally, is always verified with their healthcare teams. Despite this, concerning elements of using LLMs in healthcare may include the presence of bias and lack of transparency in the training dataset, along with ethical/legal drawbacks when LLMs provide inaccurate recommendations and potentially lead to harm to patients (10, 19, 20). Furthermore, there are also logistical concerns with integrating LLMs into the current system of EMRs or paper charts worldwide, given data privacy issues and security breaches within or between institutions (19, 20). The creation and dissemination of robust, evidence-based research, preferably conducted with a clear and standardized method, should help the scientific community further explore these concerns alongside the potential benefits of LLMs in a clinical context. There is currently a paucity of research around the specific research criteria, recommendations, and methods of evaluating the clinical utility and limitations of GPT models. Thus, the objectives of our scoping review are: Objectives: To elucidate the research landscape regarding the clinical utility of LLMs – with a particular focus on ChatGPT/GPT-3 and GPT-4 and provide a summary of its effectiveness, accuracy, and efficacy (i.e., in the realms of clinical decision-making support, operational efficiency, and patient communication) To explore the ethical, legal, socioeconomic, and other implications of integrating LLMs in the clinical setting To scope practical barriers to the implementation of LLMs in the clinical context (i.e. costs, EMRs integration, data sharing across institutions) and key factors of successful implementation of LLMs in healthcare settings To propose a standardized, more rigorous methodology of investigating and evaluating LLMs’ clinical utility for future research studies To recommend specific clinical contexts that LLMs should be further investigated and may have the highest potential for impacting patient care. References: Yang X, Chen A, PourNejatian N, Shin HC, Smith KE, Parisien C, Compas C, Martin C, Costa AB, Flores MG, Zhang Y, Magoc T, Harle CA, Lipori G, Mitchell DA, Hogan WR, Shenkman EA, Bian J, Wu Y. A large language model for electronic health records. NPJ Digit Med. 2022 Dec 26;5(1):194. doi: 10.1038/s41746-022-00742-2. Weiss, K., Khoshgoftaar, T.M. & Wang, D. A survey of transfer learning. J Big Data 3, 9 (2016). https://doi.org/10.1186/s40537-016-0043-6 Kojima T, Shane SG, Reid M, Matsuo Y, Iwasaw Y. Large Language Models are Zero-Shot Reasoners. arXiv. 2022 May 24: v4. https://doi.org/10.48550/arXiv.2205.11916 Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez A, Kaiser L, Polosukhin I. Attention Is All You Need. arXiv. 2017 June 12;v1. https://doi.org/10.48550/arXiv.1706.03762 Open AI. https://openai.com/blog/chatgpt. Accessed March 17, 2023. Liu S, Wright AP, Patterson BL, Wanderer JP, Turer RW, Nelson SD, McCoy AB, Sittig DF, Wright A. Assessing the Value of ChatGPT for Clinical Decision Support Optimization. medRxiv [Preprint]. 2023 Feb 23:2023.02.21.23286254. doi: 10.1101/2023.02.21.23286254. Invgate. https://blog.invgate.com/gpt-3-vs-bert. Accessed March 17, 2023. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv [Preprint]. 2018 Oct; v1. https://doi.org/10.48550/arXiv.1810.04805 Ji Z, Wei Q, Xu H. BERT-based Ranking for Biomedical Entity Normalization. AMIA Jt Summits Transl Sci Proc. 2020 May 30;2020:269-277. Sallam M. The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations. medRxiv [Pre-print]. 2023 Feb 21: doi: https://doi.org/10.1101/2023.02.19.23286155 Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, Chartash D. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023 Feb 8;9:e45312. doi: 10.2196/45312. PMID: 36753318 Kelly, S. ChatGPT passes exams from law and business schools. https://www.cnn.com/2023/01/26/tech/chatgpt-passes-exams/index.html. Accessed March 17, 2023. Chaarani, J. ChatGPT wrote a poem about winter, but is it truly art? This Waterloo AI ethicist weighs in. https://www.cbc.ca/news/canada/kitchener-waterloo/chatgpt-ai-text-university-waterloo-maura-grossman-1.6703819#:~:text=ChatGPT%20is%20artificial%20intelligence%20chatbot,still%20spotlighting%20concerns%20around%20accuracy.. Accessed March 17, 2023. Luo R, Sun L, Xia Y, Qin T, Zhang S, Poon H, Liu T. BioGPT: Generative Pre-trained Transformer for Biomedical text Generation and Mining. arXiv [Pre-print]. 2023 Jan;v2. https://doi.org/10.48550/arXiv.2210.10341 Open AI. https://openai.com/research/gpt-4. Accessed March 17, 2023. Levine DM, Tuwani R, Kompa B, Varma A, Finlayson SG, Mehrotra A, Beam A. The Diagnostic and Triage Accuracy of the GPT-3 Artificial Intelligence Model. medRxiv [Preprint]. 2023 Feb 1:2023.01.30.23285067. doi: 10.1101/2023.01.30.23285067. Patel SB, Lam K. ChatGPT: the future of discharge summaries? Lancet Digit Health. 2023 Mar;5(3):e107-e108. doi: 10.1016/S2589-7500(23)00021-3. Stewart, Jonathon et al "Applications of Natural Language Processing at Emergency Department Triage: A Systematic Review." medRxiv (2022): 2022.12.20.22283735. Web. 21 Feb. 2023. Stokel-Walker C, Van Noorden R. What ChatGPT and generative AI mean for science. Nature. 2023 Feb;614(7947):214-216. doi: 10.1038/d41586-023-00340-6. Nov, Oded, Nina Singh, and Devin M. Mann. "Putting ChatGPT’s Medical Advice to the (Turing) Test." medRxiv (2023): 2023.01.23.23284735. Web. 22 Feb. 2023.

Parasitic nematodes are major human and agricultural pests, and benzimidazoles are amongst the most important broad-spectrum anthelmintic drug class used for their control. Benzimidazole resistance is now widespread in many species of parasitic nematodes in livestock globally and an emerging concern for the sustainable control of human soil-transmitted helminths. β-tubulin is the major benzimidazole target, although other genes may influence resistance. Among the 6 Caenorhabditis elegans β-tubulin genes, loss of ben-1 causes resistance without other apparent defects. Here, we explored the genetics of C. elegans β-tubulin genes in relation to the response to the benzimidazole derivative albendazole. The most highly expressed β-tubulin isotypes, encoded by tbb-1 and tbb-2, were known to be redundant with each other for viability, and their products are predicted not to bind benzimidazoles. We found that tbb-2 mutants, and to a lesser extent tbb-1 mutants, were hypersensitive to albendazole. The double mutant tbb-2 ben-1 is uncoordinated and short, resembling the wild type exposed to albendazole, but the tbb-1 ben-1 double mutant did not show the same phenotypes. These results suggest that tbb-2 is a modifier of albendazole sensitivity. To better understand how BEN-1 mutates to cause benzimidazole resistance, we isolated mutants resistant to albendazole and found that 15 of 16 mutations occurred in the ben-1 coding region. Mutations ranged from likely nulls to hypomorphs, and several corresponded to residues that cause resistance in other organisms. Null alleles of ben-1 are albendazole-resistant and BEN-1 shows high sequence identity with tubulins from other organisms, suggesting that many amino acid changes could cause resistance. However, our results suggest that missense mutations conferring resistance are not evenly distributed across all possible conserved sites. Independent of their roles in benzimidazole resistance, tbb-1 and tbb-2 may have specialized functions as null mutants of tbb-1 or tbb-2 were cold or heat sensitive, respectively.

Parasitic nematodes are major human and agricultural pests, and benzimidazoles are amongst the most important broad spectrum anthelmintic drug class used for their control. Benzimidazole resistance is now widespread in many species of parasitic nematodes in livestock globally and an emerging concern for the sustainable control of human soil transmitted helminths. β-tubulin is the major benzimidazole target, although other genes may influence resistance. Among the six C. elegans β-tubulin genes, loss of ben-1 causes resistance without other apparent defects. Here, we explored the genetics of C. elegans β-tubulin genes in relation to the response to the benzimidazole derivative albendazole. The most highly expressed β-tubulin isotypes, encoded by tbb-1 and tbb-2, were known to be redundant with each other for viability, and their products are predicted not to bind benzimidazoles. We found that tbb-2 mutants, and to a lesser extent tbb-1 mutants, were hypersensitive to albendazole. The double mutant tbb-2 ben-1 is uncoordinated and dumpy, resembling the wild type exposed to albendazole, but the tbb-1 ben-1 double mutant did not show the same phenotype. These results suggest that tbb-2 is a modifier of ABZ sensitivity. To better understand how BEN-1 mutates to cause benzimidazole resistance, we isolated mutants resistant to albendazole and found that 15 of 16 mutations occurred in ben-1. Mutations ranged from likely nulls to hypomorphs, and several corresponded to residues that cause resistance in other organisms. Null alleles of ben-1 are albendazole-resistant and BEN-1 shows high sequence identity with tubulins from other organisms, suggesting that many amino acid changes could cause resistance. However, our results suggest that missense mutations conferring resistance are not evenly distributed across all possible conserved sites. Independent of their roles in benzimidazole resistance, tbb-1 and tbb-2 may have specialized functions as null mutants of tbb-1 or tbb-2 were cold or heat sensitive, respectively.

Actin and myosin mediate the epidermal cell contractions that elongate the Caenorhabditis elegans embryo from an ovoid to a tubular-shaped worm. Contraction occurs mainly in the lateral epidermal cells, while the dorsoventral epidermis plays a more passive role. Two parallel pathways trigger actinomyosin contraction, one mediated by LET-502/Rho kinase and the other by PAK-1/p21 activated kinase. A number of genes mediating morphogenesis have been shown to be sufficient when expressed either laterally or dorsoventrally. Additional genes show either lateral or dorsoventral phenotypes. This led us to a model where contractile genes have discrete functions in one or the other cell type. We tested this by examining several genes for either lateral or dorsoventral sufficiency. LET-502 expression in the lateral cells was sufficient to drive elongation. MEL-11/Myosin phosphatase, which antagonizes contraction, and PAK-1 were expected to function dorsoventrally, but we could not detect tissue-specific sufficiency. Double mutants of lethal alleles predicted to decrease lateral contraction with those thought to increase dorsoventral force were previously shown to be viable. We hypothesized that these mutant combinations shifted the contractile force from the lateral to the dorsoventral cells and so the embryos would elongate with less lateral cell contraction. This was tested by examining 10 single and double mutant strains. In most cases, elongation proceeded without a noticeable alteration in lateral contraction. We suggest that many embryonic elongation genes likely act in both lateral and dorsoventral cells, even though they may have their primary focus in one or the other cell type.