Search

Engagement between instructors and students plays a crucial role in enhancing students'academic performance. However, instructors often struggle to provide timely and personalized support in large classes. To address this challenge, we propose a novel Virtual Teaching Assistant (VTA) named YA-TA, designed to offer responses to students that are grounded in lectures and are easy to understand. To facilitate YA-TA, we introduce the Dual Retrieval-augmented Knowledge Fusion (DRAKE) framework, which incorporates dual retrieval of instructor and student knowledge and knowledge fusion for tailored response generation. Experiments conducted in real-world classroom settings demonstrate that the DRAKE framework excels in aligning responses with knowledge retrieved from both instructor and student sides. Furthermore, we offer additional extensions of YA-TA, such as a Q&A board and self-practice tools to enhance the overall learning experience. Our video is publicly available., Comment: 9 pages, 5 figures

Recently, the demand for psychological counseling has significantly increased as more individuals express concerns about their mental health. This surge has accelerated efforts to improve the accessibility of counseling by using large language models (LLMs) as counselors. To ensure client privacy, training open-source LLMs faces a key challenge: the absence of realistic counseling datasets. To address this, we introduce Cactus, a multi-turn dialogue dataset that emulates real-life interactions using the goal-oriented and structured approach of Cognitive Behavioral Therapy (CBT). We create a diverse and realistic dataset by designing clients with varied, specific personas, and having counselors systematically apply CBT techniques in their interactions. To assess the quality of our data, we benchmark against established psychological criteria used to evaluate real counseling sessions, ensuring alignment with expert evaluations. Experimental results demonstrate that Camel, a model trained with Cactus, outperforms other models in counseling skills, highlighting its effectiveness and potential as a counseling agent. We make our data, model, and code publicly available., Comment: Published at EMNLP 2024 Findings

This work proposes an efficient method to enhance the quality of corrupted speech signals by leveraging both acoustic and visual cues. While existing diffusion-based approaches have demonstrated remarkable quality, their applicability is limited by slow inference speeds and computational complexity. To address this issue, we present FlowAVSE which enhances the inference speed and reduces the number of learnable parameters without degrading the output quality. In particular, we employ a conditional flow matching algorithm that enables the generation of high-quality speech in a single sampling step. Moreover, we increase efficiency by optimizing the underlying U-net architecture of diffusion-based systems. Our experiments demonstrate that FlowAVSE achieves 22 times faster inference speed and reduces the model size by half while maintaining the output quality. The demo page is available at: https://cyongong.github.io/FlowAVSE.github.io/, Comment: INTERSPEECH 2024

The demand for conversational agents that provide mental health care is consistently increasing. In this work, we develop a psychological counseling agent, referred to as CoCoA, that applies Cognitive Behavioral Therapy (CBT) techniques to identify and address cognitive distortions inherent in the client's statements. Specifically, we construct a memory system to efficiently manage information necessary for counseling while extracting high-level insights about the client from their utterances. Additionally, to ensure that the counseling agent generates appropriate responses, we introduce dynamic prompting to flexibly apply CBT techniques and facilitate the appropriate retrieval of information. We conducted dialogues between CoCoA and characters from Character.ai, creating a dataset for evaluation. Then, we asked GPT to evaluate the constructed counseling dataset, and our model demonstrated a statistically significant difference from other models., Comment: 4 pages, 2 figures

The objective of this work is to extract target speaker's voice from a mixture of voices using visual cues. Existing works on audio-visual speech separation have demonstrated their performance with promising intelligibility, but maintaining naturalness remains a challenge. To address this issue, we propose AVDiffuSS, an audio-visual speech separation model based on a diffusion mechanism known for its capability in generating natural samples. For an effective fusion of the two modalities for diffusion, we also propose a cross-attention-based feature fusion mechanism. This mechanism is specifically tailored for the speech domain to integrate the phonetic information from audio-visual correspondence in speech generation. In this way, the fusion process maintains the high temporal resolution of the features, without excessive computational requirements. We demonstrate that the proposed framework achieves state-of-the-art results on two benchmarks, including VoxCeleb2 and LRS3, producing speech with notably better naturalness., Comment: Project page with demo: https://mm.kaist.ac.kr/projects/avdiffuss/

The goal of this work is Active Speaker Detection (ASD), a task to determine whether a person is speaking or not in a series of video frames. Previous works have dealt with the task by exploring network architectures while learning effective representations has been less explored. In this work, we propose TalkNCE, a novel talk-aware contrastive loss. The loss is only applied to part of the full segments where a person on the screen is actually speaking. This encourages the model to learn effective representations through the natural correspondence of speech and facial movements. Our loss can be jointly optimized with the existing objectives for training ASD models without the need for additional supervision or training data. The experiments demonstrate that our loss can be easily integrated into the existing ASD frameworks, improving their performance. Our method achieves state-of-the-art performances on AVA-ActiveSpeaker and ASW datasets.

This paper presents LMStream, which ensures bounded latency while maximizing the throughput on the GPU-enabled micro-batch streaming systems. The main ideas behind LMStream's design can be summarized as two novel mechanisms: (1) dynamic batching and (2) dynamic operation-level query planning. By controlling the micro-batch size, LMStream significantly reduces the latency of individual dataset because it does not perform unconditional buffering only for improving GPU utilization. LMStream bounds the latency to an optimal value according to the characteristics of the window operation used in the streaming application. Dynamic mapping between a query to an execution device based on the data size and dynamic device preference improves both the throughput and latency as much as possible. In addition, LMStream proposes a low-overhead online cost model parameter optimization method without interrupting the real-time stream processing. We implemented LMStream on Apache Spark, which supports micro-batch stream processing. Compared to the previous throughput-oriented method, LMStream showed an average latency improvement up to a maximum of 70.7%, while improving average throughput up to 1.74x., Comment: 11 pages

Near-field scanning optical microscopy has been an indispensable tool for designing, characterizing and understanding the functionalities of diverse nanoscale photonic devices. As the advances in fabrication technology have driven the devices smaller and smaller, the demand has grown steadily for improving its resolving power, which is determined mainly by the size of the probe attached to the scanner. The use of a smaller probe has been a straightforward approach to increase the resolving power, but it cannot be made arbitrarily small in practice due to the steep reduction of the collection efficiency. Here, we develop a method to enhance the resolving power of near-field imaging beyond the limit set by the physical size of the probe aperture. The main working principle is to unveil high-order near-field eigenmodes invisible with conventional near-field microscopy. The destructive interference of near-field waves is induced in these high-order eigenmodes by the locally varying phases, which can reveal subaperture-scale fine structural details. To extract these eigenmodes, we construct a self-interference near-field microscopy system and measure a fully phase-referenced far- to near-field transmission matrix (FNTM) composed of near-field amplitude and phase maps recorded for various angles of far-field illumination. By the singular value decomposition of the measured FNTM, we could extract the antisymmetric mode, quadrupole mode, and other higher-order modes hidden under the lowest-order symmetric mode. This enables us to resolve double and triple nano-slots whose gap size (50 nm) is three times smaller than the diameter of the probe aperture (150 nm). The subaperture near-field mode mapping by the FTNM can be potentially combined with various existing near-field imaging modalities and promote their ability to interrogate local near-field optical waves of nanoscale devices.

The family of hole-doped Pr-based perovskite cobaltites, Pr$_{0.5}$Ca$_{0.5}$CoO$_{3}$ and (Pr$_{1-y}$RE$_{y}$)$_{0.3}$Ca$_{0.7}$CoO$_{3}$ (where RE is rare earth) has recently been found to exhibit simultaneous metal-insulator, spin-state, and valence transitions. We have investigated magnetic-field-induced phase transitions of (Pr$_{1-y}$Y$_{y}$)$_{0.7}$Ca$_{0.3}$CoO$_{3}$ by means of magnetization measurements at 4.2$-$100 K up to an ultrahigh magnetic field of 140 T with the chemical pressure varied by $y$ = 0.0625, 0.075, 0.1. The observed magnetic-field-induced transitions were found to occur simultaneously with the metal-insulator transitions up to 100 T. The obtained magnetic field-temperature ($B$-$T$) phase diagram and magnetization curves are well analyzed by a spin-crossover model of a single ion with interion interactions. On the other hand, the chemical pressure dependence of the experimentally obtained magnetization change during the phase transition disagrees with the single ion model when approaching low temperatures. The significant $y$ dependence of the magnetization change at low temperatures may arise from the itinerant magnetism of Co$^{3+}$ in the paramagnetic metallic phase, where the chemical pressure enhances the exchange splitting by promoting the double-exchange interaction. The observed $B$-$T$ phase diagrams of (Pr$_{1-y}$Y$_{y}$)$_{0.7}$Ca$_{0.3}$CoO$_{3}$ are quite contrary to that of LaCoO$_{3}$, indicating that in (Pr$_{1-y}$Y$_{y}$)$_{0.7}$Ca$_{0.3}$CoO$_{3}$ the high-field phase possesses higher entropy than the low-field phase, whereas it is the other way around in LaCoO$_{3}$., Comment: 8 pages, 6 figures