Search

Inference-time intervention (ITI) has emerged as a promising method for steering large language model (LLM) behavior in a particular direction (e.g., improving helpfulness) by intervening on token representations without costly updates to the LLM's parameters. However, existing ITI approaches fail to scale to multi-attribute settings with conflicts, such as enhancing helpfulness while also reducing toxicity. To address this, we introduce Multi-Attribute Targeted Steering (MAT-Steer), a novel steering framework designed for selective token-level intervention across multiple attributes. MAT-Steer learns steering vectors using an alignment objective that shifts the model's internal representations of undesirable outputs closer to those of desirable ones while enforcing sparsity and orthogonality among vectors for different attributes, thereby reducing inter-attribute conflicts. We evaluate MAT-Steer in two distinct settings: (i) on question answering (QA) tasks where we balance attributes like truthfulness, bias, and toxicity; (ii) on generative tasks where we simultaneously improve attributes like helpfulness, correctness, and coherence. MAT-Steer outperforms existing ITI and parameter-efficient finetuning approaches across both task types (e.g., 3% average accuracy gain across QA tasks and 55.82% win rate against the best ITI baseline)., Comment: 15 pages, code link: https://github.com/duykhuongnguyen/MAT-Steer

In this paper, we would like to examine whether the S\'aez-Ballester theory admits stable and attractive Bianchi type I inflationary solutions in the presence of a non-minimal coupling between scalar and vector fields such as $f^2(\phi)F_{\mu\nu}F^{\mu\nu}$. As a result, such a solution will be shown to exist within this theory for a suitable setup of fields. However, the corresponding tensor-to-scalar ratio of this solution turns out to be higher than the latest observational value of the Planck satellite (Planck 2018) due to the fact that $c_s$, the corresponding speed of sound of scalar perturbations of the S\'aez-Ballester theory, turns out to be one. This result indicates an important hint that the speed of sound, $c_s$, could play an important role in making the corresponding non-canonical anisotropic inflation cosmologically viable in the light of the Planck 2018 data. To be more specific, we will point out that any modifications of the S\'aez-Ballester theory having $c_s \sim 0.1$ will have a great potential to be highly consistent with the Planck 2018 data., Comment: 30 pages, 5 figures. Comments are welcome

Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels, hindering model generalization. This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification. We first extend the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology image tiles, into a vision-language model by adding adaptors and aligning it with medical text encoders via contrastive learning on 923K image-text pairs. The model is then used to extract visual features and text embeddings from few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike prior methods that combine prompts with frozen features using prefix embeddings or self-attention, we propose multi-granular attention that compares interactions between learnable prompts with individual image patches and groups of them. This approach improves the model's ability to capture both fine-grained details and broader context, enhancing its recognition of complex patterns across sub-regions. To further improve accuracy, we leverage (unbalanced) optimal transport-based visual-text distance to secure model robustness by mitigating perturbations that might occur during the data augmentation process. Empirical experiments on lung, kidney, and breast pathology modalities validate the effectiveness of our approach; thereby, we surpass several of the latest competitors and consistently improve performance across diverse architectures, including CLIP, PLIP, and Prov-GigaPath integrated PLIP. We release our implementations and pre-trained models at this MGPATH., Comment: first version

The LLaMA-Adapter has recently emerged as an efficient fine-tuning technique for LLaMA models, leveraging zero-initialized attention to stabilize training and enhance performance. However, despite its empirical success, the theoretical foundations of zero-initialized attention remain largely unexplored. In this paper, we provide a rigorous theoretical analysis, establishing a connection between zero-initialized attention and mixture-of-expert models. We prove that both linear and non-linear prompts, along with gating functions, can be optimally estimated, with non-linear prompts offering greater flexibility for future applications. Empirically, we validate our findings on the open LLM benchmarks, demonstrating that non-linear prompts outperform linear ones. Notably, even with limited training data, both prompt types consistently surpass vanilla attention, highlighting the robustness and adaptability of zero-initialized attention., Comment: 43 pages, 5 tables, 6 figures

Query rewriting (QR) is a critical technique in e-commerce search, addressing the lexical gap between user queries and product descriptions to enhance search performance. Existing QR approaches typically fall into two categories: discriminative models and generative methods leveraging large language models (LLMs). Discriminative models often struggle with natural language understanding and offer limited flexibility in rewriting, while generative LLMs, despite producing high-quality rewrites, face high inference latency and cost in online settings. These limitations force offline deployment, making them vulnerable to issues like information staleness and semantic drift. To overcome these challenges, we propose a novel hybrid pipeline for QR that balances efficiency and effectiveness. Our approach combines offline knowledge distillation to create a lightweight but efficient student model with online reinforcement learning (RL) to refine query rewriting dynamically using real-time feedback. A key innovation is the use of LLMs as simulated human feedback, enabling scalable reward signals and cost-effective evaluation without manual annotations. Experimental results on Amazon ESCI dataset demonstrate significant improvements in query relevance, diversity, and adaptability, as well as positive feedback from the LLM simulation. This work contributes to advancing LLM capabilities for domain-specific applications, offering a robust solution for dynamic and complex e-commerce search environments.

Language models are prone to occasionally undesirable generations, such as harmful or toxic content, despite their impressive capability to produce texts that appear accurate and coherent. This paper presents a new two-stage approach to detect and mitigate undesirable content generations by rectifying activations. First, we train an ensemble of layerwise classifiers to detect undesirable content using activations by minimizing a smooth surrogate of the risk-aware score. Then, for contents that are detected as undesirable, we propose layerwise distributional intervention policies that perturb the attention heads minimally while guaranteeing probabilistically the effectiveness of the intervention. Benchmarks on several language models and datasets show that our method outperforms baselines in reducing the generation of undesirable output., Comment: 3 figures

The torsional mode of atomic force microscope (AFM) cantilevers plays a crucial role in a wide range of sensitive measurements. Despite their importance, the use of approximated frequencies and mode shapes for width-varying cantilevers often results in discrepancies between theoretical models and experimental observations. In this study, we present an analytical approach to accurately calculate the mode shapes and resonance frequencies of these cantilevers, including higher-order modes, then we derive the modal sensitivity. Our results reveal distinct changes in mode shapes and frequencies as the overhang length increases, with the mode shapes showing multiple maxima. Furthermore, we demonstrate that tuning the overhang length provides effective control over the resonant frequency. The relationship between modal sensitivity and the coupling strength between the cantilever and the surface is also established, aligning with previous experimental findings. This work offers valuable insights for optimizing cantilever geometry to achieve desired frequency responses in AFM applications.

Dynamics of icrocantilevers are of important interest in micro-mechanical systems for enhancing the functionality and applicable range of the cantilevers in vibration transducing and highly sensitive measurement. In this study, using the semi-classical Hamiltonian formalism, we study in detail the modification of the mechanical frequency and damping rate taking into account both the linear and quadratic coupling between the mechanical oscillator and the laser field in an opto-mechanical system. It has been seen that, the linear coupling greatly enhances the modification of the effective mechanical frequency and the effective damping rate while the quadratic coupling reduces these quantities. For a MHz-frequency oscillator, the damping rate could be 10^5 times increased and the frequency is several times modified. These results help clarifying the origin of the modification of the susceptibility function for cooling of the mechanical mode

The popularity of e-commerce has given rise to fake advertisements that can expose users to financial and data risks while damaging the reputation of these e-commerce platforms. For these reasons, detecting and removing such fake advertisements are important for the success of e-commerce websites. In this paper, we propose FADAML, a novel end-to-end machine learning system to detect and filter out fake online advertisements. Our system combines techniques in multimodal machine learning and automated machine learning to achieve a high detection rate. As a case study, we apply FADAML to detect fake advertisements on popular Vietnamese real estate websites. Our experiments show that we can achieve 91.5% detection accuracy, which significantly outperforms three different state-of-the-art fake news detection systems.

This paper presents the design a Proportional-Integral-Derivative (PID) controller with optimized parameters for a two-degree-of-freedom robotic arm. A genetic algorithm (GA) is proposed to optimize the controller parameters, addressing the challenges in determining PID controller parameters for highly nonlinear systems like robotic arms compared to traditional methods. The GA-optimized PID controller significantly improves control accuracy and performance over traditional control methods. Simulation results demonstrate that the robotic arm system operates with high precision and stability. Additionally, the shortened trajectory tracking response time enhances the feasibility of applying this control algorithm in realworld scenarios. This research not only confirms the suitability of PID-GA for robotic arms and similar systems but also opens new avenues for applying this algorithm to real physical systems.

The emergence of artificial intelligence (AI), particularly deep learning (DL), has marked a new era in the realm of ophthalmology, offering transformative potential for the diagnosis and treatment of posterior segment eye diseases. This review explores the cutting-edge applications of DL across a range of ocular conditions, including diabetic retinopathy, glaucoma, age-related macular degeneration, and retinal vessel segmentation. We provide a comprehensive overview of foundational ML techniques and advanced DL architectures, such as CNNs, attention mechanisms, and transformer-based models, highlighting the evolving role of AI in enhancing diagnostic accuracy, optimizing treatment strategies, and improving overall patient care. Additionally, we present key challenges in integrating AI solutions into clinical practice, including ensuring data diversity, improving algorithm transparency, and effectively leveraging multimodal data. This review emphasizes AI's potential to improve disease diagnosis and enhance patient care while stressing the importance of collaborative efforts to overcome these barriers and fully harness AI's impact in advancing eye care., Comment: First version

We analyzed theoretically the nonlinear dynamics of a strong magnetic pendulum consisting of a cylindrical neodymium magnet swinging into a metal plane. The heavy damping of oscillations of the pendulum is caused by eddy currents induced in the metal plate. In this paper, we proposed a mathematical model to describe the braking force acting on the magnetic pendulum and its nonlinear motion.

In a reinforcement learning (RL) framework, we study the exploratory version of the continuous time expected utility (EU) maximization problem with a portfolio constraint that includes widely-used financial regulations such as short-selling constraints and borrowing prohibition. The optimal feedback policy of the exploratory unconstrained classical EU problem is shown to be Gaussian. In the case where the portfolio weight is constrained to a given interval, the corresponding constrained optimal exploratory policy follows a truncated Gaussian distribution. We verify that the closed form optimal solution obtained for logarithmic utility and quadratic utility for both unconstrained and constrained situations converge to the non-exploratory expected utility counterpart when the exploration weight goes to zero. Finally, we establish a policy improvement theorem and devise an implementable reinforcement learning algorithm by casting the optimal problem in a martingale framework. Our numerical examples show that exploration leads to an optimal wealth process that is more dispersedly distributed with heavier tail compared to that of the case without exploration. This effect becomes less significant as the exploration parameter is smaller. Moreover, the numerical implementation also confirms the intuitive understanding that a broader domain of investment opportunities necessitates a higher exploration cost. Notably, when subjected to both short-selling and money borrowing constraints, the exploration cost becomes negligible compared to the unconstrained case.

Given a simple graph $G$, the artinian monomial algebra associated to $G$, denoted by $A(G)$, is defined by the edge ideal of $G$ and the squares of the variables. In this article, we classify some tadpole graphs $G$ for which $A(G)$ has or fails the weak Lefschetz property., Comment: 12 pages. arXiv admin note: substantial text overlap with arXiv:2310.14368 by other authors

We show that the existing generalized separation statements including the conventional extremal principle and its extensions differ {in the ways norms on product spaces are defined}. We prove a general separation statement with arbitrary product norms covering the existing results of this kind. The proof is divided into a series of claims and exposes the key steps and arguments used when proving generalized separation statements. As an application, we prove dual necessary (sufficient) conditions for an abstract product norm extension of the approximate stationarity (transversality) property., Comment: 21 pages

Early detection of skin abnormalities plays a crucial role in diagnosing and treating skin cancer. Segmentation of affected skin regions using AI-powered devices is relatively common and supports the diagnostic process. However, achieving high performance remains a significant challenge due to the need for high-resolution images and the often unclear boundaries of individual lesions. At the same time, medical devices require segmentation models to have a small memory foot-print and low computational cost. Based on these requirements, we introduce a novel lightweight model called MambaU-Lite, which combines the strengths of Mamba and CNN architectures, featuring just over 400K parameters and a computational cost of more than 1G flops. To enhance both global context and local feature extraction, we propose the P-Mamba block, a novel component that incorporates VSS blocks along-side multiple pooling layers, enabling the model to effectively learn multiscale features and enhance segmentation performance. We evaluate the model's performance on two skin datasets, ISIC2018 and PH2, yielding promising results. Our source code will be made publicly available at: https://github.com/nqnguyen812/MambaU-Lite., Comment: 10 pages, 3 figures, 2 tables

In this work, we derive three recursive formulas for the integrals of products of Hermite polynomials. The derivation is notably straightforward, relying solely on the well-established properties of Hermite polynomials and the technique of integration by parts. These results hold broad relevance across various fields of physics and mathematics. Specifically, they would be applied to accurately compute two- and three-body matrix elements in ab initio simulations of one-dimensional few-body systems confined in harmonic traps. Additionally, we provide a numerical subroutine that implements these recursive formulas, which accompanies this work., Comment: 12 pages, comments are welcome

Using the Green's function approach, we investigate the resonance energy transfer (RET) rate between two parallel, identical two-level atoms in the presence of three types of cylindrical system: a distributed Bragg reflector (DBR), a perfectly reflecting wall (PRW), and a two-layer silicon fiber. Our analysis, incorporating the cut-off frequency condition, reveals significant suppression of the RET rate for atoms positioned along the axis of the cylinder with the PRW. In contrast, for atoms located within the DBR, the RET rate is enhanced in the far zone. Additionally, we find that for atoms oriented radially are placed inside or near the surface of the silicon fiber, the RET rate is entirely inhibited. We also investigate the Casimir-Polder (CP) interaction between a cut-off-frequency DBR and an excited atom, discovering a fully attractive potential towards the surface for the atom within the waveguide., Comment: 20 pages, 5 figures

Polariton lasing is a promising phenomenon with potential applications in next-generation lasers that operate without the need for population inversion. Applying a perpendicular magnetic field to a quantum well (QW) significantly alters the properties of exciton-polaritons. In this theoretical study, we investigate how the lasing threshold of QW exciton-polaritons depends on the magnetic field. By modifying the exciton's effective mass and Rabi splitting, the magnetic field induces notable changes in the relaxation kinetics, which directly affect the lasing threshold. For low-energy pumping, an increase in the magnetic field delays the lasing threshold, while for high-energy pumping, the threshold is reached at much lower pump intensities. Furthermore, increasing both the pump energy and the magnetic field enhances relaxation efficiency, leading to a substantially larger number of condensed polaritons. Our result gives insights into the modulation of exciton-polariton condensation through magnetic fields, with potential implications for the design of low-threshold polariton lasers., Comment: 5 pages, 3 figures

Gcd-graphs over the ring of integers modulo $n$ are a simple and elegant class of integral graphs. The study of these graphs connects multiple areas of mathematics, including graph theory, number theory, and ring theory. In a recent work, inspired by the analogy between number fields and function fields, we define and study gcd-graphs over polynomial rings with coefficients in finite fields. We discover that, in both cases, gcd-graphs share many similar and analogous properties. In this article, we extend this line of research further. Among other topics, we explore an analog of a conjecture of So and a weaker version of Sander-Sander, concerning the conditions under which two gcd-graphs are isomorphic or isospectral. We also provide several constructions showing that, unlike the case over $\mathbb{Z}$, it is not uncommon for two gcd-graphs over polynomial rings to be isomorphic., Comment: Comments are welcome!

A graph is called integral if its eigenvalues are integers. In this article, we provide the necessary and sufficient conditions for a Cayley graph over a finite symmetric algebra $R$ to be integral. This generalizes the work of So who studies the case where $R$ is the ring of integers modulo $n.$ We also explain some number-theoretic constructions of finite symmetric algebras arising from global fields, which we hope could pave the way for future studies on Paley graphs associated with a finite Hecke character., Comment: To appear in Archiv der Mathematik

The spatial Sigma-Delta ($\Sigma\Delta$) architecture can be leveraged to reduce the quantization noise and enhance the effective resolution of few-bit analog-to-digital converters (ADCs) at certain spatial frequencies of interest. Utilizing the variational Bayesian (VB) inference framework, this paper develops novel data detection algorithms tailored for massive multiple-input multiple-output (MIMO) systems with few-bit $\Sigma\Delta$ ADCs and angular channel models, where uplink signals are confined to a specific angular sector. We start by modeling the corresponding Bayesian networks for the $1^{\mathrm{st}}$- and $2^{\mathrm{nd}}$-order $\Sigma\Delta$ receivers. Next, we propose an iterative algorithm, referred to as Sigma-Delta variational Bayes (SD-VB), for MIMO detection, offering low-complexity updates through closed-form expressions of the variational densities of the latent variables. Simulation results show that the proposed $2^{\mathrm{nd}}$-order SD-VB algorithm delivers the best symbol error rate (SER) performance while maintaining the same computational complexity as in unquantized systems, matched-filtering VB with conventional quantization, and linear minimum mean-squared error (LMMSE) methods. Moreover, the $1^{\mathrm{st}}$- and $2^{\mathrm{nd}}$-order SD-VB algorithms achieve their lowest SER at an antenna separation of one-fourth wavelength for a fixed number of antenna elements. The effects of the steering angle of the $\Sigma\Delta$ architecture, the number of ADC resolution bits, and the number of antennas and users are also extensively analyzed., Comment: 13 pages, 12 figures, under submission for a journal publication

State-of-the-art medical multi-modal large language models (med-MLLM), like LLaVA-Med or BioMedGPT, leverage instruction-following data in pre-training. However, those models primarily focus on scaling the model size and data volume to boost performance while mainly relying on the autoregressive learning objectives. Surprisingly, we reveal that such learning schemes might result in a weak alignment between vision and language modalities, making these models highly reliant on extensive pre-training datasets - a significant challenge in medical domains due to the expensive and time-consuming nature of curating high-quality instruction-following instances. We address this with LoGra-Med, a new multi-graph alignment algorithm that enforces triplet correlations across image modalities, conversation-based descriptions, and extended captions. This helps the model capture contextual meaning, handle linguistic variability, and build cross-modal associations between visuals and text. To scale our approach, we designed an efficient end-to-end learning scheme using black-box gradient estimation, enabling faster LLaMa 7B training. Our results show LoGra-Med matches LLAVA-Med performance on 600K image-text pairs for Medical VQA and significantly outperforms it when trained on 10% of the data. For example, on VQA-RAD, we exceed LLAVA-Med by 20.13% and nearly match the 100% pre-training score (72.52% vs. 72.64%). We also surpass SOTA methods like BiomedGPT on visual chatbots and RadFM on zero-shot image classification with VQA, highlighting the effectiveness of multi-graph alignment., Comment: First version, fixed typo

Reward Models (RMs) play a crucial role in aligning LLMs with human preferences, enhancing their performance by ranking outputs during inference or iterative training. However, the degree to which an RM generalizes to new tasks is often not known a priori (e.g. some RMs may excel at scoring creative writing vs. math reasoning). Therefore, using only one fixed RM while training LLMs can be suboptimal. Moreover, optimizing LLMs with multiple RMs simultaneously can be prohibitively computationally-intensive and challenging due to conflicting signals from different RMs, potentially degrading performance. To address these challenges, we introduce LASeR (Learning to Adaptively Select Rewards), which iteratively trains LLMs using multiple RMs, selecting and utilizing the most well-suited RM for each instance to rank outputs and generate preference data, framed as a multi-armed bandit problem. Our results on commonsense and math reasoning tasks demonstrate that LASeR can boost iterative LLM optimization by optimizing for multiple RMs, improving the absolute average accuracy of Llama-3-8B over three datasets by 2.67% over training with ensemble RM scores while also showing superior training efficiency (e.g., a 2x speedup). Moreover, on WildChat, a benchmark of instruction-following prompts, we find that using Llama-3-8B LASeR leads to a 71.45% AlpacaEval win rate over sequentially optimizing multiple RMs. Extending to long-context generation tasks, we find that on Llama-3-8B, LASeR achieves an average improvement of 2.64 F1 and 2.42 F1 on single- and multi-document QA over random RM selection when used with best-of-n sampling. LASeR is robust to noisy rewards and generalizes to multiple settings. Finally, LASeR's RM selection changes depending on the underlying task or instance and we verify the presence of conflicting preferences from multiple RMs that can be mitigated using LASeR., Comment: 20 pages; First two authors contributed equally. Code: https://github.com/duykhuongnguyen/LASeR-MAB

While robust optimal control theory provides a rigorous framework to compute robot control policies that are provably safe, it struggles to scale to high-dimensional problems, leading to increased use of deep learning for tractable synthesis of robot safety. Unfortunately, existing neural safety synthesis methods often lack convergence guarantees and solution interpretability. In this paper, we present Minimax Actors Guided by Implicit Critic Stackelberg (MAGICS), a novel adversarial reinforcement learning (RL) algorithm that guarantees local convergence to a minimax equilibrium solution. We then build on this approach to provide local convergence guarantees for a general deep RL-based robot safety synthesis algorithm. Through both simulation studies on OpenAI Gym environments and hardware experiments with a 36-dimensional quadruped robot, we show that MAGICS can yield robust control policies outperforming the state-of-the-art neural safety synthesis methods., Comment: Algorithmic Foundations of Robotics (WAFR) XVI

Gcd-graphs over the ring of integers modulo $n$ are a natural generalization of unitary Cayley graphs. The study of these graphs has foundations in various mathematical fields, including number theory, ring theory, and representation theory. Using the theory of Ramanujan sums, it is known that these gcd-graphs have integral spectra; i.e., all their eigenvalues are integers. In this work, inspired by the analogy between number fields and function fields, we define and study gcd-graphs over polynomial rings with coefficients in finite fields. We establish some fundamental properties of these graphs, emphasizing their analogy to their counterparts over $\mathbb{Z}.$, Comment: Comments are welcome!

Due to their elegant and simple nature, unitary Cayley graphs have been an active research topic in the literature. These graphs are naturally connected to several branches of mathematics, including number theory, finite algebra, representation theory, and graph theory. In this article, we study the perfectness property of these graphs. More precisely, we provide a complete classification of perfect unitary Cayley graphs associated with finite rings., Comment: Comments are welcome!

Let $n\geq 3$. We show that for every number field $K$ with $\zeta_p \notin K$, the absolute and tame Galois groups of $K$ satisfy the strong $n$-fold Massey property relative to $p$. Our work is based on an adapted version of the proof of the Theorem of Scholz-Reichardt.

Increasing the throughput of the Transformer architecture, a foundational component used in numerous state-of-the-art models for vision and language tasks (e.g., GPT, LLaVa), is an important problem in machine learning. One recent and effective strategy is to merge token representations within Transformer models, aiming to reduce computational and memory requirements while maintaining accuracy. Prior works have proposed algorithms based on Bipartite Soft Matching (BSM), which divides tokens into distinct sets and merges the top k similar tokens. However, these methods have significant drawbacks, such as sensitivity to token-splitting strategies and damage to informative tokens in later layers. This paper presents a novel paradigm called PiToMe, which prioritizes the preservation of informative tokens using an additional metric termed the energy score. This score identifies large clusters of similar tokens as high-energy, indicating potential candidates for merging, while smaller (unique and isolated) clusters are considered as low-energy and preserved. Experimental findings demonstrate that PiToMe saved from 40-60\% FLOPs of the base models while exhibiting superior off-the-shelf performance on image classification (0.5\% average performance drop of ViT-MAE-H compared to 2.6\% as baselines), image-text retrieval (0.3\% average performance drop of CLIP on Flickr30k compared to 4.5\% as others), and analogously in visual questions answering with LLaVa-7B. Furthermore, PiToMe is theoretically shown to preserve intrinsic spectral properties of the original token space under mild conditions, Comment: Accepted at NeurIPS 2024

The paper explores a new extremality model involving collections of arbitrary families of sets. We demonstrate its applicability to set-valued optimization problems with general preferences, weakening the assumptions of the known results and streamlining their proofs., Comment: 15 pages. Formerly appeared as part of arXiv:2403.16511v2

Businesses need to query visually rich documents (VRDs) like receipts, medical records, and insurance forms to make decisions. Existing techniques for extracting entities from VRDs struggle with new layouts or require extensive pre-training data. We introduce VRDSynth, a program synthesis method to automatically extract entity relations from multilingual VRDs without pre-training data. To capture the complexity of VRD domain, we design a domain-specific language (DSL) to capture spatial and textual relations to describe the synthesized programs. Along with this, we also derive a new synthesis algorithm utilizing frequent spatial relations, search space pruning, and a combination of positive, negative, and exclusive programs to improve coverage. We evaluate VRDSynth on the FUNSD and XFUND benchmarks for semantic entity linking, consisting of 1,592 forms in 8 languages. VRDSynth outperforms state-of-the-art pre-trained models (LayoutXLM, InfoXLMBase, and XLMRobertaBase) in 5, 6, and 7 out of 8 languages, respectively, improving the F1 score by 42% over LayoutXLM in English. To test the extensibility of the model, we further improve VRDSynth with automated table recognition, creating VRDSynth(Table), and compare it with extended versions of the pre-trained models, InfoXLM(Large) and XLMRoberta(Large). VRDSynth(Table) outperforms these baselines in 4 out of 8 languages and in average F1 score. VRDSynth also significantly reduces memory footprint (1M and 380MB vs. 1.48GB and 3GB for LayoutXLM) while maintaining similar time efficiency., Comment: Accepted in ISSTA'24

Prompt learning methods are gaining increasing attention due to their ability to customize large vision-language models to new domains using pre-trained contextual knowledge and minimal training data. However, existing works typically rely on optimizing unified prompt inputs, often struggling with fine-grained classification tasks due to insufficient discriminative attributes. To tackle this, we consider a new framework based on a dual context of both domain-shared and class-specific contexts, where the latter is generated by Large Language Models (LLMs) such as GPTs. Such dual prompt methods enhance the model's feature representation by joining implicit and explicit factors encoded in LLM knowledge. Moreover, we formulate the Unbalanced Optimal Transport (UOT) theory to quantify the relationships between constructed prompts and visual tokens. Through partial matching, UOT can properly align discrete sets of visual tokens and prompt embeddings under different mass distributions, which is particularly valuable for handling irrelevant or noisy elements, ensuring that the preservation of mass does not restrict transport solutions. Furthermore, UOT's characteristics integrate seamlessly with image augmentation, expanding the training sample pool while maintaining a reasonable distance between perturbed images and prompt inputs. Extensive experiments across few-shot classification and adapter settings substantiate the superiority of our model over current state-of-the-art baselines., Comment: Version 1

This paper presents a novel system, named AHMsys, designed to automate the process of generating 3D Heating, Ventilation, and Air Conditioning (HVAC) models from 2D Computer-Aided Design (CAD) drawings, a key component of Building Information Modeling (BIM). By automatically preprocessing and extracting essential HVAC object information then creating detailed 3D models, our proposed AHMsys significantly reduced the 20 percent work schedule of the BIM process in Akila. This advancement highlights the essential impact of integrating AI technologies in managing the lifecycle of a digital representation of the building.

Cardiovascular diseases (CVDs) are notably prevalent among patients with obstructive sleep apnea (OSA), posing unique challenges in predicting CVD progression due to the intricate interactions of comorbidities. Traditional models typically lack the necessary dynamic and longitudinal scope to accurately forecast CVD trajectories in OSA patients. This study introduces a novel multi-level phenotypic model to analyze the progression and interplay of these conditions over time, utilizing data from the Wisconsin Sleep Cohort, which includes 1,123 participants followed for decades. Our methodology comprises three advanced steps: (1) Conducting feature importance analysis through tree-based models to underscore critical predictive variables like total cholesterol, low-density lipoprotein (LDL), and diabetes. (2) Developing a logistic mixed-effects model (LGMM) to track longitudinal transitions and pinpoint significant factors, which displayed a diagnostic accuracy of 0.9556. (3) Implementing t-distributed Stochastic Neighbor Embedding (t-SNE) alongside Gaussian Mixture Models (GMM) to segment patient data into distinct phenotypic clusters that reflect varied risk profiles and disease progression pathways. This phenotypic clustering revealed two main groups, with one showing a markedly increased risk of major adverse cardiovascular events (MACEs), underscored by the significant predictive role of nocturnal hypoxia and sympathetic nervous system activity from sleep data. Analysis of transitions and trajectories with t-SNE and GMM highlighted different progression rates within the cohort, with one cluster progressing more slowly towards severe CVD states than the other. This study offers a comprehensive understanding of the dynamic relationship between CVD and OSA, providing valuable tools for predicting disease onset and tailoring treatment approaches., Comment: 30 pages, 5 figure, 5 tables

This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias -- locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a token). We mainly showcase the effectiveness of pixels-as-tokens across three well-studied tasks in computer vision: supervised learning for object classification, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although directly operating on individual pixels is less computationally practical, we believe the community must be aware of this surprising piece of knowledge when devising the next generation of neural architectures for computer vision., Comment: Technical report, 23 pages

Comprehending how humans process visual information in dynamic settings is crucial for psychology and designing user-centered interactions. While mobile eye-tracking systems combining egocentric video and gaze signals can offer valuable insights, manual analysis of these recordings is time-intensive. In this work, we present a novel human-centered learning algorithm designed for automated object recognition within mobile eye-tracking settings. Our approach seamlessly integrates an object detector with a spatial relation-aware inductive message-passing network (I-MPN), harnessing node profile information and capturing object correlations. Such mechanisms enable us to learn embedding functions capable of generalizing to new object angle views, facilitating rapid adaptation and efficient reasoning in dynamic contexts as users navigate their environment. Through experiments conducted on three distinct video sequences, our interactive-based method showcases significant performance improvements over fixed training/testing algorithms, even when trained on considerably smaller annotated samples collected through user feedback. Furthermore, we demonstrate exceptional efficiency in data annotation processes and surpass prior interactive methods that use complete object detectors, combine detectors with convolutional networks, or employ interactive video segmentation., Comment: Updated version

Rating elicitation is a success element for recommender systems to perform well at cold-starting, in which the systems need to recommend items to a newly arrived user with no prior knowledge about the user's preference. Existing elicitation methods employ a fixed set of items to learn the user's preference and then infer the users' preferences on the remaining items. Using a fixed seed set can limit the performance of the recommendation system since the seed set is unlikely optimal for all new users with potentially diverse preferences. This paper addresses this challenge using a 2-phase, personalized elicitation scheme. First, the elicitation scheme asks users to rate a small set of popular items in a ``burn-in'' phase. Second, it sequentially asks the user to rate adaptive items to refine the preference and the user's representation. Throughout the process, the system represents the user's embedding value not by a point estimate but by a region estimate. The value of information obtained by asking the user's rating on an item is quantified by the distance from the region center embedding space that contains with high confidence the true embedding value of the user. Finally, the recommendations are successively generated by considering the preference region of the user. We show that each subproblem in the elicitation scheme can be efficiently implemented. Further, we empirically demonstrate the effectiveness of the proposed method against existing rating-elicitation methods on several prominent datasets., Comment: Accepted at UAI 2024