Search

Audio Description (AD) plays a pivotal role as an application system aimed at guaranteeing accessibility in multimedia content, which provides additional narrations at suitable intervals to describe visual elements, catering specifically to the needs of visually impaired audiences. In this paper, we introduce $\mathrm{CA^3D}$, the pioneering unified Context-Aware Automatic Audio Description system that provides AD event scripts with precise locations in the long cinematic content. Specifically, $\mathrm{CA^3D}$ system consists of: 1) a Temporal Feature Enhancement Module to efficiently capture longer term dependencies, 2) an anchor-based AD event detector with feature suppression module that localizes the AD events and extracts discriminative feature for AD generation, and 3) a self-refinement module that leverages the generated output to tweak AD event boundaries from coarse to fine. Unlike conventional methods which rely on metadata and ground truth AD timestamp for AD detection and generation tasks, the proposed $\mathrm{CA^3D}$ is the first end-to-end trainable system that only uses visual cue. Extensive experiments demonstrate that the proposed $\mathrm{CA^3D}$ improves existing architectures for both AD event detection and script generation metrics, establishing the new state-of-the-art performances in the AD automation., Comment: 10 pages

The topological properties of gapped graphene have been explored for valleytronics applications. Prior transport experiments indicated their topological nature through large nonlocal resistance in Hall-bar devices, but the origin of this resistance was unclear. This study focused on dual-gate bilayer graphene (BLG) devices with naturally cleaved edges, examining how edge-etching with an oxygen plasma process affects electron transport. Before etching, local resistance at the charge neutral point increased exponentially with the displacement field and nonlocal resistance was well explained by ohmic contribution, which is typical of gapped BLG. After-etching, however, local resistance saturated with increasing displacement field, and nonlocal resistance deviated by three orders of magnitude from ohmic contribution. We suggest that these significant changes in local and nonlocal resistance arise from the formation of edge conducting pathways after the edge-etching, rather than from a topological property of gapped BLG that has been claimed in previous literatures., Comment: 22 pages, 6 figures

Graphene's exceptional electronic mobility, gate-tunability, and contact transparency with superconducting materials make it ideal for exploring the superconducting proximity effect. However, the work function difference between graphene and superconductors causes unavoidable doping of graphene near contacts, forming a p-n junction in the hole-doped regime and reducing contact transparency. This challenges the device implementation that exploits graphene's bipolarity. To address this limitation, we developed a new fabrication scheme for two-dimensional superconducting contacts that allows independent control over charge concentration and polarity for both the graphene in contact with superconductors and the graphene channel. Contact transparency, conductance enhancement, and Josephson coupling were measured to confirm transparent contacts to both polarities of graphene. Moreover, we demonstrated the Andreev process in the quantum Hall edge state at a negative filling factor of {\nu} = -2. This scheme will open avenues for realizing various theoretical propositions utilizing the bipolarity of graphene combined with superconductivity., Comment: 20 pages, 10 figures

Multimodal large language models (MLLMs) act as essential interfaces, connecting humans with AI technologies in multimodal applications. However, current MLLMs face challenges in accurately interpreting object orientation in images due to inconsistent orientation annotations in training data, hindering the development of a coherent orientation understanding. To overcome this, we propose egocentric instruction tuning, which aligns MLLMs' orientation understanding with the user's perspective, based on a consistent annotation standard derived from the user's egocentric viewpoint. We first generate egocentric instruction data that leverages MLLMs' ability to recognize object details and applies prior knowledge for orientation understanding. Using this data, we perform instruction tuning to enhance the model's capability for accurate orientation interpretation. In addition, we introduce EgoOrientBench, a benchmark that evaluates MLLMs' orientation understanding across three tasks using images collected from diverse domains. Experimental results on this benchmark show that egocentric instruction tuning significantly improves orientation understanding without compromising overall MLLM performance. The instruction data and benchmark dataset are available on our project page at https://github.com/jhCOR/EgoOrientBench.

Multiply connected electronic networks threaded by flux tubes have been proposed as a platform for adiabatic quantum transport and topological states. Multi-terminal Josephson junction (MTJJ) has been suggested as a pathway to realize this concept. Yet, the manifestations of topology in MTJJ remain open for experimental study. Here, we investigated the artificial topological band structure of three-terminal graphene Josephson junctions. Employing tunnelling spectroscopy and magnetic flux gates, we captured the tomography of the Andreev bound state (ABS) energy spectrum as a function of two independent phase differences. This ABS spectrum exhibits the topological transition from gapped to gapless states, akin to the band structure of nodal-line semimetals. Our results show the potential of graphene-based MTJJs for engineering band topologies in higher dimensions.

Recent advances in quantum information processing with superconducting qubits have fueled a growing demand for scaling and miniaturizing circuit layouts. Despite significant progress, predicting the Hamiltonian of complex circuits remains a challenging task. Here, we propose an improved method for quantizing superconducting circuits that incorporates material- and geometry-dependent kinetic inductance. Our approach models superconducting films as reactive boundary elements, seamlessly integrating into the conventional circuit quantization framework without adding computational complexity. We experimentally validate our method using superconducting devices fabricated with 35 nm-thick disordered niobium films, demonstrating significantly improved accuracy in predicting the Hamiltonian based solely on the device layout and material properties of superconducting films and Josephson junctions. Specifically, conventional methods exhibit an average error of 5.4% in mode frequencies, while our method reduces it to 1.1%. Our method enables systematic studies of superconducting devices with disordered films or compact elements, facilitating precise engineering of superconducting circuits at scale., Comment: 12 pages, 4 figures, 1 table

As the scale of data and models for video understanding rapidly expand, handling long-form video input in transformer-based models presents a practical challenge. Rather than resorting to input sampling or token dropping, which may result in information loss, token merging shows promising results when used in collaboration with transformers. However, the application of token merging for long-form video processing is not trivial. We begin with the premise that token merging should not rely solely on the similarity of video tokens; the saliency of tokens should also be considered. To address this, we explore various video token merging strategies for long-form video classification, starting with a simple extension of image token merging, moving to region-concentrated merging, and finally proposing a learnable video token merging (VTM) algorithm that dynamically merges tokens based on their saliency. Extensive experimental results show that we achieve better or comparable performances on the LVU, COIN, and Breakfast datasets. Moreover, our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms., Comment: 21 pages, NeurIPS 2024

Single photon detectors (SPDs) are essential technology in quantum science, quantum network, biology, and advanced imaging. To detect the small quantum of energy carried in a photon, conventional SPDs rely on energy excitation across either a semiconductor bandgap or superconducting gap. While the energy gap suppresses the false-positive error, it also sets an energy scale that can limit the detection efficiency of lower energy photons and spectral bandwidth of the SPD. Here, we demonstrate an orthogonal approach to detect single near-infrared photons using graphene calorimeters. By exploiting the extremely low heat capacity of the pseudo-relativistic electrons in graphene near its charge neutrality point, we observe an electron temperature rise up to ~2 K using a hybrid Josephson junction. In this proof-of-principle experiment, we achieve an intrinsic quantum efficiency of 87% (75%) with dark count < 1 per second (per hour) at operation temperatures as high as 1.2 K. Our results highlight the potential of electron calorimetric SPDs for detecting lower-energy photons from the mid-IR to microwave regimes, opening pathways to study space science in far-infrared regime, to search for dark matter axions, and to advance quantum technologies across a broader electromagnetic spectrum.

We study noncooperative games, in which each agent's objective is composed of a sequence of ordered-and potentially conflicting-preferences. Problems of this type naturally model a wide variety of scenarios: for example, drivers at a busy intersection must balance the desire to make forward progress with the risk of collision. Mathematically, these problems possess a nested structure, and to behave properly agents must prioritize their most important preference, and only consider less important preferences to the extent that they do not compromise performance on more important ones. We consider multi-agent, noncooperative variants of these problems, and seek generalized Nash equilibria in which each agent's decision reflects both its hierarchy of preferences and other agents' actions. We make two key contributions. First, we develop a recursive approach for deriving the first-order optimality conditions of each agent's nested problem. Second, we propose a sequence of increasingly tight relaxations, each of which can be transcribed as a mixed complementarity problem and solved via existing methods. Experimental results demonstrate that our approach reliably converges to equilibrium solutions that strictly reflect agents' individual ordered preferences.

Recent progress in large language models (LLMs) offers promising new approaches for recommendation system (RecSys) tasks. While the current state-of-the-art methods rely on fine-tuning LLMs to achieve optimal results, this process is costly and introduces significant engineering complexities. Conversely, methods that bypass fine-tuning and use LLMs directly are less resource-intensive but often fail to fully capture both semantic and collaborative information, resulting in sub-optimal performance compared to their fine-tuned counterparts. In this paper, we propose a Simple Training-free Approach for Recommendation (STAR), a framework that utilizes LLMs and can be applied to various recommendation tasks without the need for fine-tuning. Our approach involves a retrieval stage that uses semantic embeddings from LLMs combined with collaborative user information to retrieve candidate items. We then apply an LLM for pairwise ranking to enhance next-item prediction. Experimental results on the Amazon Review dataset show competitive performance for next item prediction, even with our retrieval stage alone. Our full method achieves Hits@10 performance of +23.8% on Beauty, +37.5% on Toys and Games, and -1.8% on Sports and Outdoors relative to the best supervised models. This framework offers an effective alternative to traditional supervised models, highlighting the potential of LLMs in recommendation systems without extensive training or custom architectures.

Missing values are a common problem that poses significant challenges to data analysis and machine learning. This problem necessitates the development of an effective imputation method to fill in the missing values accurately, thereby enhancing the overall quality and utility of the datasets. Existing imputation methods, however, fall short of explicitly considering the `missingness' information in the data during the embedding initialization stage and modeling the entangled feature and sample correlations during the learning process, thus leading to inferior performance. We propose M$^3$-Impute, which aims to explicitly leverage the missingness information and such correlations with novel masking schemes. M$^3$-Impute first models the data as a bipartite graph and uses a graph neural network to learn node embeddings, where the refined embedding initialization process directly incorporates the missingness information. They are then optimized through M$^3$-Impute's novel feature correlation unit (FRU) and sample correlation unit (SRU) that effectively captures feature and sample correlations for imputation. Experiment results on 25 benchmark datasets under three different missingness settings show the effectiveness of M$^3$-Impute by achieving 20 best and 4 second-best MAE scores on average.

The stability of the integrated photonic circuits is of critical importance for many applications that require high frequency precision or robust operation over time, such as optomechanical sensing, frequency conversion, optical communication, and quantum optics. Photonic memory is useful for low-energy optical computing and interconnects. Thin film lithium niobate (TFLN), as an emerging photonic platform, exhibits complex material properties including pyroelectric (PE) and photorefractive (PR) effects which could lead to intra-device drift and excess noise under different environmental or operating conditions as well as be utilized for building photonic memory. However, the long-term stability and memory effect of its optical properties has not been explored. In this paper, we discovered a long-lived change of optical refractive index as a result of light excitation and temporal temperature variation using Z-cut TFLN microresonators and reveal a strong dependence of the instability with the crystal orientation of the thin film form. The recovery time are measured to be over 10 hours. Leveraging the photonic memory with a long relaxation time, we realize optical trimming of the cavity resonance frequencies. Our result offers insights towards understanding the fundamental noise properties and dynamic behavior of the integrated TFLN material and devices., Comment: 9 pages, 5 figures

Pain is a more intuitive and user-friendly way of communicating problems, making it especially useful in rehabilitation nurse training robots. While most previous methods have focused on classifying or recognizing pain expressions, these approaches often result in unnatural, jiggling robot faces. We introduce PainDiffusion, a model that generates facial expressions in response to pain stimuli, with controllable pain expressiveness and emotion status. PainDiffusion leverages diffusion forcing to roll out predictions over arbitrary lengths using a conditioned temporal U-Net. It operates as a latent diffusion model within EMOCA's facial expression latent space, ensuring a compact data representation and quick rendering time. For training data, we process the BioVid Heatpain Database, extracting expression codes and subject identity configurations. We also propose a novel set of metrics to evaluate pain expressions, focusing on expressiveness, diversity, and the appropriateness of model-generated outputs. Finally, we demonstrate that PainDiffusion outperforms the autoregressive method, both qualitatively and quantitatively. Code, videos, and further analysis are available at: \href{https://damtien444.github.io/paindf/}{https://damtien444.github.io/paindf/}., Comment: Under reviewing

Incremental Learning (IL) aims to accumulate knowledge from sequential input tasks while overcoming catastrophic forgetting. Existing IL methods typically assume that an incoming task has only increments of classes or domains, referred to as Class IL (CIL) or Domain IL (DIL), respectively. In this work, we consider a more challenging and realistic but under-explored IL scenario, named Versatile Incremental Learning (VIL), in which a model has no prior of which of the classes or domains will increase in the next task. In the proposed VIL scenario, the model faces intra-class domain confusion and inter-domain class confusion, which makes the model fail to accumulate new knowledge without interference with learned knowledge. To address these issues, we propose a simple yet effective IL framework, named Incremental Classifier with Adaptation Shift cONtrol (ICON). Based on shifts of learnable modules, we design a novel regularization method called Cluster-based Adaptation Shift conTrol (CAST) to control the model to avoid confusion with the previously learned knowledge and thereby accumulate the new knowledge more effectively. Moreover, we introduce an Incremental Classifier (IC) which expands its output nodes to address the overwriting issue from different domains corresponding to a single class while maintaining the previous knowledge. We conducted extensive experiments on three benchmarks, showcasing the effectiveness of our method across all the scenarios, particularly in cases where the next task can be randomly altered. Our implementation code is available at https://github.com/KHU-AGI/VIL., Comment: 17 pages, 6 figures, 6 tables, ECCV 2024 Poster

In Huntington's disease research, a current goal is to understand how symptoms change prior to a clinical diagnosis. Statistically, this entails modeling symptom severity as a function of the covariate 'time until diagnosis', which is often heavily right-censored in observational studies. Existing estimators that handle right-censored covariates have varying statistical efficiency and robustness to misspecified models for nuisance distributions (those of the censored covariate and censoring variable). On one extreme, complete case estimation, which utilizes uncensored data only, is free of nuisance distribution models but discards informative censored observations. On the other extreme, maximum likelihood estimation is maximally efficient but inconsistent when the covariate's distribution is misspecified. We propose a semiparametric estimator that is robust and efficient. When the nuisance distributions are modeled parametrically, the estimator is doubly robust, i.e., consistent if at least one distribution is correctly specified, and semiparametric efficient if both models are correctly specified. When the nuisance distributions are estimated via nonparametric or machine learning methods, the estimator is consistent and semiparametric efficient. We show empirically that the proposed estimator, implemented in the R package sparcc, has its claimed properties, and we apply it to study Huntington's disease symptom trajectories using data from the Enroll-HD study.

We study the communication complexity of multiplying $k\times t$ elements from the group $H=\text{SL}(2,q)$ in the number-on-forehead model with $k$ parties. We prove a lower bound of $(t\log H)/c^{k}$. This is an exponential improvement over previous work, and matches the state-of-the-art in the area. Relatedly, we show that the convolution of $k^{c}$ independent copies of a 3-uniform distribution over $H^{m}$ is close to a $k$-uniform distribution. This is again an exponential improvement over previous work which needed $c^{k}$ copies. The proofs are remarkably simple; the results extend to other quasirandom groups. We also show that for any group $H$, any distribution over $H^{m}$ whose weight-$k$ Fourier coefficients are small is close to a $k$-uniform distribution. This generalizes previous work in the abelian setting, and the proof is simpler.

Drawing tests like the Rey Complex Figure Test (RCFT) are widely used to assess cognitive functions such as visuospatial skills and memory, making them valuable tools for detecting mild cognitive impairment (MCI). Despite their utility, existing predictive models based on these tests often suffer from limitations like small sample sizes and lack of external validation, which undermine their reliability. We developed a multi-stream deep learning framework that integrates two distinct processing streams: a multi-head self-attention based spatial stream using raw RCFT images and a scoring stream employing a previously developed automated scoring system. Our model was trained on data from 1,740 subjects in the Korean cohort and validated on an external hospital dataset of 222 subjects from Korea. The proposed multi-stream model demonstrated superior performance over baseline models (AUC = 0.872, Accuracy = 0.781) in external validation. The integration of both spatial and scoring streams enables the model to capture intricate visual details from the raw images while also incorporating structured scoring data, which together enhance its ability to detect subtle cognitive impairments. This dual approach not only improves predictive accuracy but also increases the robustness of the model, making it more reliable in diverse clinical settings. Our model has practical implications for clinical settings, where it could serve as a cost-effective tool for early MCI screening., Comment: 20 pages, 3 figures, 2 tables

We prove several new results on the Hamming weight of bounded uniform and small-bias distributions. We exhibit bounded-uniform distributions whose weight is anti-concentrated, matching existing concentration inequalities. This construction relies on a recent result in approximation theory due to Erd\'eyi (Acta Arithmetica 2016). In particular, we match the classical tail bounds, generalizing a result by Bun and Steinke (RANDOM 2015). Also, we improve on a construction by Benjamini, Gurel-Gurevich, and Peled (2012). We give a generic transformation that converts any bounded uniform distribution to a small-bias distribution that almost preserves its weight distribution. Applying this transformation in conjunction with the above results and others, we construct small-bias distributions with various weight restrictions. In particular, we match the concentration that follows from that of bounded uniformity and the generic closeness of small-bias and bounded-uniform distributions, answering a question by Bun and Steinke (RANDOM 2015). Moreover, these distributions are supported on only a constant number of Hamming weights. We further extend the anti-concentration constructions to small-bias distributions perturbed with noise, a class that has received much attention recently in derandomization. Our results imply (but are not implied by) a recent result of the authors (CCC 2024), and are based on different techniques. In particular, we prove that the standard Gaussian distribution is far from any mixture of Gaussians with bounded variance.

The goal of trace reconstruction is to reconstruct an unknown $n$-bit string $x$ given only independent random traces of $x$, where a random trace of $x$ is obtained by passing $x$ through a deletion channel. A Statistical Query (SQ) algorithm for trace reconstruction is an algorithm which can only access statistical information about the distribution of random traces of $x$ rather than individual traces themselves. Such an algorithm is said to be $\ell$-local if each of its statistical queries corresponds to an $\ell$-junta function over some block of $\ell$ consecutive bits in the trace. Since several -- but not all -- known algorithms for trace reconstruction fall under the local statistical query paradigm, it is interesting to understand the abilities and limitations of local SQ algorithms for trace reconstruction. In this paper we establish nearly-matching upper and lower bounds on local Statistical Query algorithms for both worst-case and average-case trace reconstruction. For the worst-case problem, we show that there is an $\tilde{O}(n^{1/5})$-local SQ algorithm that makes all its queries with tolerance $\tau \geq 2^{-\tilde{O}(n^{1/5})}$, and also that any $\tilde{O}(n^{1/5})$-local SQ algorithm must make some query with tolerance $\tau \leq 2^{-\tilde{\Omega}(n^{1/5})}$. For the average-case problem, we show that there is an $O(\log n)$-local SQ algorithm that makes all its queries with tolerance $\tau \geq 1/\mathrm{poly}(n)$, and also that any $O(\log n)$-local SQ algorithm must make some query with tolerance $\tau \leq 1/\mathrm{poly}(n).$, Comment: RANDOM 2024

Harmful suicide content on the Internet is a significant risk factor inducing suicidal thoughts and behaviors among vulnerable populations. Despite global efforts, existing resources are insufficient, specifically in high-risk regions like the Republic of Korea. Current research mainly focuses on understanding negative effects of such content or suicide risk in individuals, rather than on automatically detecting the harmfulness of content. To fill this gap, we introduce a harmful suicide content detection task for classifying online suicide content into five harmfulness levels. We develop a multi-modal benchmark and a task description document in collaboration with medical professionals, and leverage large language models (LLMs) to explore efficient methods for moderating such content. Our contributions include proposing a novel detection task, a multi-modal Korean benchmark with expert annotations, and suggesting strategies using LLMs to detect illegal and harmful content. Owing to the potential harm involved, we publicize our implementations and benchmark, incorporating an ethical verification process., Comment: 30 pages, 7 figures

The tetrahedron algebra $\boxtimes$ is an infinite-dimensional Lie algebra defined by generators $\{x_{ij} \mid i, j \in \{0, 1, 2, 3\}, i \neq j\}$ and some relations, including the Dolan-Grady relations. These twelve generators are called standard. We introduce a type of element in $\boxtimes$ that "looks like" a standard generator. For mutually distinct $h, i, j, k \in \{0, 1, 2, 3\}$, consider the standard generator $x_{ij}$ of $\boxtimes$. An element $\xi \in \boxtimes$ is called $x_{ij}$-like whenever both (i) $\xi$ commutes with $x_{ij}$; (ii) $\xi$ and $x_{hk}$ satisfy a Dolan-Grady relation. Pick mutually distinct $i,j,k \in \{0,1,2,3\}$. In our main result, we find an attractive basis for $\boxtimes$ with the property that every basis element is either $x_{ij}$-like or $x_{jk}$-like or $x_{ki}$-like. We discuss this basis from multiple points of view., Comment: 30 pages