Search

Due to the cost-prohibitive nature of training Large Language Models (LLMs), fine-tuning has emerged as an attractive alternative for specializing LLMs for specific tasks using limited compute resources in a cost-effective manner. In this paper, we characterize sparse Mixture of Experts (MoE) based LLM fine-tuning to understand their accuracy and runtime performance on a single GPU. Our evaluation provides unique insights into the training efficacy of sparse and dense versions of MoE models, as well as their runtime characteristics, including maximum batch size, execution time breakdown, end-to-end throughput, GPU hardware utilization, and load distribution. Our study identifies the optimization of the MoE layer as crucial for further improving the performance of LLM fine-tuning. Using our profiling results, we also develop and validate an analytical model to estimate the cost of LLM fine-tuning on the cloud. This model, based on parameters of the model and GPU architecture, estimates LLM throughput and the cost of training, aiding practitioners in industry and academia to budget the cost of fine-tuning a specific model., Comment: 10 pages, conference

Cooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.

Electronic Health Records (EHRs) are integral for storing comprehensive patient medical records, combining structured data (e.g., medications) with detailed clinical notes (e.g., physician notes). These elements are essential for straightforward data retrieval and provide deep, contextual insights into patient care. However, they often suffer from discrepancies due to unintuitive EHR system designs and human errors, posing serious risks to patient safety. To address this, we developed EHRCon, a new dataset and task specifically designed to ensure data consistency between structured tables and unstructured notes in EHRs. EHRCon was crafted in collaboration with healthcare professionals using the MIMIC-III EHR dataset, and includes manual annotations of 3,943 entities across 105 clinical notes checked against database entries for consistency. EHRCon has two versions, one using the original MIMIC-III schema, and another using the OMOP CDM schema, in order to increase its applicability and generalizability. Furthermore, leveraging the capabilities of large language models, we introduce CheckEHR, a novel framework for verifying the consistency between clinical notes and database tables. CheckEHR utilizes an eight-stage process and shows promising results in both few-shot and zero-shot settings. The code is available at https://github.com/dustn1259/EHRCon.

Recent advancements in Large Language Models (LLMs) have significantly enhanced the capabilities of conversational agents, making them applicable to various fields (e.g., education). Despite their progress, the evaluation of the agents often overlooks the complexities of real-world conversations, such as real-time interactions, multi-party dialogues, and extended contextual dependencies. To bridge this gap, we introduce DialSim, a real-time dialogue simulator. In this simulator, an agent is assigned the role of a character from popular TV shows, requiring it to respond to spontaneous questions using past dialogue information and to distinguish between known and unknown information. Key features of DialSim include evaluating the agent's ability to respond within a reasonable time limit, handling long-term multi-party dialogues, and managing adversarial settings (e.g., swap character names) to challenge the agent's reliance on pre-trained knowledge. We utilized this simulator to evaluate the latest conversational agents and analyze their limitations. Our experiments highlight both the strengths and weaknesses of these agents, providing valuable insights for future improvements in the field of conversational AI. DialSim is available at https://github.com/jiho283/Simulator.

Large language models (LLMs) are shaping a new user interface (UI) paradigm in writing tools by enabling users to generate text through prompts. This paradigm shifts some creative control from the user to the system, thereby diminishing the user's authorship and autonomy in the writing process. To restore autonomy, we introduce Textfocals, a UI prototype designed to investigate a human-centered approach that emphasizes the user's role in writing. Textfocals supports the writing process by providing LLM-generated summaries, questions, and advice (i.e., LLM views) in a sidebar of a text editor, encouraging reflection and self-driven revision in writing without direct text generation. Textfocals' UI affordances, including contextually adaptive views and scaffolding for prompt selection and customization, offer a novel way to interact with LLMs where users maintain full authorship of their writing. A formative user study with Textfocals showed promising evidence that this approach might help users develop underdeveloped ideas, cater to the rhetorical audience, and clarify their writing. However, the study also showed interaction design challenges related to document navigation and scoping, prompt engineering, and context management. Our work highlights the breadth of the design space of writing support interfaces powered by generative AI that maintain authorship integrity., Comment: 15 pages, 2 figures; Accepted to 5th Workshop on Human-AI Co-Creation with Generative Models (HAI-GEN) at ACM IUI 2024

Free-text radiology reports present a rich data source for various medical tasks, but effectively labeling these texts remains challenging. Traditional rule-based labeling methods fall short of capturing the nuances of diverse free-text patterns. Moreover, models using expert-annotated data are limited by data scarcity and pre-defined classes, impacting their performance, flexibility and scalability. To address these issues, our study offers three main contributions: 1) We demonstrate the potential of GPT as an adept labeler using carefully designed prompts. 2) Utilizing only the data labeled by GPT, we trained a BERT-based labeler, CheX-GPT, which operates faster and more efficiently than its GPT counterpart. 3) To benchmark labeler performance, we introduced a publicly available expert-annotated test set, MIMIC-500, comprising 500 cases from the MIMIC validation set. Our findings demonstrate that CheX-GPT not only excels in labeling accuracy over existing models, but also showcases superior efficiency, flexibility, and scalability, supported by our introduction of the MIMIC-500 dataset for robust benchmarking. Code and models are available at https://github.com/kakaobrain/CheXGPT., Comment: 16 pages, 3 figures

A large-scale image-text pair dataset has greatly contributed to the development of vision-language pre-training (VLP) models, which enable zero-shot or few-shot classification without costly annotation. However, in the medical domain, the scarcity of data remains a significant challenge for developing a powerful VLP model. In this paper, we tackle the lack of image-text data in chest X-ray by expanding image-label pair as image-text pair via general prompt and utilizing multiple images and multiple sections in a radiologic report. We also design two contrastive losses, named ICL and TCL, for learning study-level characteristics of medical images and reports, respectively. Our model outperforms the state-of-the-art models trained under the same conditions. Also, enlarged dataset improve the discriminative power of our pre-trained model for classification, while sacrificing marginal retrieval performance. Code is available at https://github.com/kakaobrain/cxr-clip., Comment: Accepted by MICCAI 2023

While large language models (LLMs) have made considerable advancements in understanding and generating unstructured text, their application in structured data remains underexplored. Particularly, using LLMs for complex reasoning tasks on knowledge graphs (KGs) remains largely untouched. To address this, we propose KG-GPT, a multi-purpose framework leveraging LLMs for tasks employing KGs. KG-GPT comprises three steps: Sentence Segmentation, Graph Retrieval, and Inference, each aimed at partitioning sentences, retrieving relevant graph components, and deriving logical conclusions, respectively. We evaluate KG-GPT using KG-based fact verification and KGQA benchmarks, with the model showing competitive and robust performance, even outperforming several fully-supervised models. Our work, therefore, marks a significant step in unifying structured and unstructured data processing within the realm of LLMs., Comment: Accepted to EMNLP 2023 Findings

In real world applications, knowledge graphs (KG) are widely used in various domains (e.g. medical applications and dialogue agents). However, for fact verification, KGs have not been adequately utilized as a knowledge source. KGs can be a valuable knowledge source in fact verification due to their reliability and broad applicability. A KG consists of nodes and edges which makes it clear how concepts are linked together, allowing machines to reason over chains of topics. However, there are many challenges in understanding how these machine-readable concepts map to information in text. To enable the community to better use KGs, we introduce a new dataset, FactKG: Fact Verification via Reasoning on Knowledge Graphs. It consists of 108k natural language claims with five types of reasoning: One-hop, Conjunction, Existence, Multi-hop, and Negation. Furthermore, FactKG contains various linguistic patterns, including colloquial style claims as well as written style claims to increase practicality. Lastly, we develop a baseline approach and analyze FactKG over these reasoning types. We believe FactKG can advance both reliability and practicality in KG-based fact verification., Comment: Accepted to ACL 2023

Electrocardiogram (ECG) synthesis is the area of research focused on generating realistic synthetic ECG signals for medical use without concerns over annotation costs or clinical data privacy restrictions. Traditional ECG generation models consider a single ECG lead and utilize GAN-based generative models. These models can only generate single lead samples and require separate training for each diagnosis class. The diagnosis classes of ECGs are insufficient to capture the intricate differences between ECGs depending on various features (e.g. patient demographic details, co-existing diagnosis classes, etc.). To alleviate these challenges, we present a text-to-ECG task, in which textual inputs are used to produce ECG outputs. Then we propose Auto-TTE, an autoregressive generative model conditioned on clinical text reports to synthesize 12-lead ECGs, for the first time to our knowledge. We compare the performance of our model with other representative models in text-to-speech and text-to-image. Experimental results show the superiority of our model in various quantitative evaluations and qualitative analysis. Finally, we conduct a user study with three board-certified cardiologists to confirm the fidelity and semantic alignment of generated samples. our code will be available at https://github.com/TClife/text_to_ecg, Comment: Accepted to ICASSP 2023 (5 pages, 3 figures, 4 tables)

As the volume of Electronic Health Records (EHR) sharply grows, there has been emerging interest in learning the representation of EHR for healthcare applications. Representation learning of EHR requires appropriate modeling of the two dominant modalities in EHR: structured data and unstructured text. In this paper, we present MedGTX, a pre-trained model for multi-modal representation learning of the structured and textual EHR data. MedGTX uses a novel graph encoder to exploit the graphical nature of structured EHR data, and a text encoder to handle unstructured text, and a cross-modal encoder to learn a joint representation space. We pre-train our model through four proxy tasks on MIMIC-III, an open-source EHR data, and evaluate our model on two clinical benchmarks and three novel downstream tasks which tackle real-world problems in EHR data. The results consistently show the effectiveness of pre-training the model for joint representation of both structured and unstructured information from EHR. Given the promising performance of MedGTX, we believe this work opens a new door to jointly understanding the two fundamental modalities of EHR data., Comment: To appear in Proceedings of the Conference on Health, Inference, and Learning (CHIL 2022)

An intelligent machine that can answer human questions based on electronic health records (EHR-QA) has a great practical value, such as supporting clinical decisions, managing hospital administration, and medical chatbots. Previous table-based QA studies focusing on translating natural questions into table queries (NLQ2SQL), however, suffer from the unique nature of EHR data due to complex and specialized medical terminology, hence increased decoding difficulty. In this paper, we design UniQA, a unified encoder-decoder architecture for EHR-QA where natural language questions are converted to queries such as SQL or SPARQL. We also propose input masking (IM), a simple and effective method to cope with complex medical terms and various typos and better learn the SQL/SPARQL syntax. Combining the unified architecture with an effective auxiliary training objective, UniQA demonstrated a significant performance improvement against the previous state-of-the-art model for MIMICSQL* (14.2% gain), the most complex NLQ2SQL dataset in the EHR domain, and its typo-ridden versions (approximately 28.8% gain). In addition, we confirmed consistent results for the graph-based EHR-QA dataset, MIMICSPARQL*., Comment: Proc. of Machine Learning for Health (ML4H) 2021 (Oral Spotlight)

We performed broadband optical transmission measurements of Bi2Se3 and In-doped Bi(1-x)In(x)2Se3 thin films, where in the latter the spin-orbit coupling (SOC) strength can be tuned by introducing In. Drude and interband transitions exhibit In-dependent changes that are consistent with evolution from metallic (x=0) to insulating (x=1) nature of the end compounds. Most notably, an optical absorption peak located at hw=1eV in Bi2Se3 is completely quenched at x=0.06, the critical concentration where the phase transition from TI into non-TI takes place. For this x, the surface state (SS) is vanished from the band structure as well. The correlation between the 1eV optical peak and the SS in the x-dependences suggests that the peak is associated with the SS. We further show that when Bi2Se3 is electrically gated, the 1eV-peak becomes stronger(weaker) when electron is depleted from (accumulated into) the SS. These observations combined together demonstrate that under the hw=1eV illumination electron is excited from a bulk band into the topological surface band of Bi2Se3. The optical population of surface band is of significant importance not only for fundamental study but also for TI-based optoelectronic device application., Comment: 9 pages, 5 figures

We performed infrared transmission experiment on ion-gel gated graphene and measured carrier scattering rate g as function of carrier density n over wide range up to n=2E13 cm-2. The g exhibits a rapid decreases along with the gating followed by persistent increases on further carrier doping. This behavior of g(n) demonstrates that carrier is scattered dominantly by the two scattering mechanisms, namely, charged impurity (CI) scattering and short-range disorder (SR) scattering, with additional minor scattering from substrate phonon (SPP). We can determine the absolute strengths of all the scattering channels by fitting the g(n) data and unveils the complete n-dependent map of the scattering mechanisms g(n)=gCI(n)+gSR(n)+gSPP(n). The gCI(n) and gSR(n) are larger than those of SiO2$-gated graphene by 1.8 times, which elucidates the dual role of the ion-gel layer as a CI-scatterer and simultaneously a SR-scatterer to graphene. Additionally we show that freezing of IG at low-T (~200 K) does not cause any change to the carrier scattering., Comment: 5 pages, 4 figures

We report the infrared transmission measurement on electrically gated twisted bilayer graphene. The optical absorption spectrum clearly manifests the dramatic changes such as the splitting of inter-linear-band absorption step, the shift of inter-van Hove singularity transition peak, and the emergence of very strong intra-valence (intra-conduction) band transition. These anomalous optical behaviors demonstrate consistently the non-rigid band structure modification created by the ion-gel gating through the layer-dependent Coulomb screening. We propose that this screening-driven band modification is an universal phenomenon that persists to other bilayer crystals in general, establishing the electrical gating as a versatile technique to engineer the band structures and to create new types of optical absorptions that can be exploited in electro-optical device application., Comment: 13 pages, 4 figures

We synthesize a series of CVD h-BN films and perform critical infrared spectroscopic characterization. For high-temperature (HT, Temp = 1400 degrees) grown h-BN thin film only E1u-mode infrared phonon is activated demonstrating highly aligned 2D h-BN planes over large area, whereas low-temperature (LT, Temp = 1000 degrees) grown film shows two phonon peaks, E1u and A2u, due to stacking of h-BN plane at tilted angle. For CVD graphene transferred on HT h-BN/SiO2/Si substrate, interband transition spectrum s1 shifts strongly to lower energy compared with that on LT h-BN/SiO2/Si and on bare SiO2/Si substrate, revealing that residual carrier density n in graphene is suppressed by use of HT h-BN layer. Also the interband transition width of s1 defined by effective temperature is reduced from 400 K for G/SiO2/Si to 300 K for HT h-BN/SiO2/Si. The behaviors of n and effective temperature show that HT h-BN film can decouple CVD graphene from the impurity and defect of SiO2 leading to large scale free-standing like graphene., Comment: 5 pages, 4 figures

We measured Drude absorption of gated CVD graphene using far-infrared transmission spectroscopy, and determined carrier scattering rate (g) as function of the varied carrier density (n). The n-dependent g(n) was obtained for a series of conditions systematically changed as (10 K, vacuum) -> (300 K, vacuum) -> (300 K, ambient pressure), which reveals that (1) at low-T, charged impurity (=A/sqrt(n)) and short-range defect (=B*sqrt(n)) are the major scattering sources which constitute the total scattering g=A/sqrt(n)+B*sqrt(n), (2) among various kinds of phonons populated at room-T, surface polar phonon of the SiO2 substrate is the dominantly scattering source, (3) in air, the gas molecules adsorbed on graphene play a dual role in carrier scattering as charged impurity center and resonant scattering center. We present the absolute scattering strengths of those individual scattering sources, which provides the complete map of scattering mechanism of CVD graphene for the first time. This scattering map allows us find out practical measures to suppress the individual scatterings, the mobility gains accompanied by them, and finally the ultimate attainable carrier mobility for CVD graphene., Comment: 14 pages, 6 figures