Search

The remarkable multimodal capabilities and interactive experience of GPT-4o underscore their necessity in practical applications, yet open-source models rarely excel in both areas. In this paper, we introduce VITA, the first-ever open-source Multimodal Large Language Model (MLLM) adept at simultaneous processing and analysis of Video, Image, Text, and Audio modalities, and meanwhile has an advanced multimodal interactive experience. Starting from Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary followed by bilingual instruction tuning. We further endow the language model with visual and audio capabilities through two-stage multi-task learning of multimodal alignment and instruction tuning. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. Beyond foundational capabilities, we have made considerable progress in enhancing the natural multimodal human-computer interaction experience. To the best of our knowledge, we are the first to exploit non-awakening interaction and audio interrupt in MLLM. VITA is the first step for the open-source community to explore the seamless integration of multimodal understanding and interaction. While there is still lots of work to be done on VITA to get close to close-source counterparts, we hope that its role as a pioneer can serve as a cornerstone for subsequent research. Project Page: https://vita-home.github.io., Comment: Project Page: https://vita-home.github.io

Carrollian amplitude is the natural object that defines the correlator of the boundary Carrollian field theory. In this work, we will elaborate on its proper definition in general dimensions. We use the vielbein field on the unit sphere to define the fundamental field with non-vanishing helicity in the local Cartesian frame which is the building block of the Carrollian amplitude. In general dimensions, the Carrollian amplitude is related to the momentum space scattering matrix by a modified Fourier transform. The Poincar\'e transformation law of the Carrollian amplitude in this definition has been discussed. We also find an isomorphism between the local rotation of the vielbein field and the superduality transformation., Comment: 56 pages

We perform the first lattice calculation on the semileptonic decay of $J/\psi$ using the (2+1)-flavor Wilson-clover gauge ensembles generated by CLQCD collaboration. Three gauge ensembles with different lattice spacings, from 0.0519 fm to 0.1053 fm, and pion masses, $m_{\pi}\sim$ 300 MeV, are utilized. After a naive continuum extrapolation using three lattice spacings, we obtain $\operatorname{Br}(J/\psi\rightarrow D_s e\nu_e)=1.90(6)(5)_{V_{cs}}\times 10^{-10}$ and $\operatorname{Br}(J/\psi\rightarrow D e\nu_e)=1.21(6)(9)_{V_{cd}}\times 10^{-11}$, where the first errors are statistical, and the second come from the uncertainties of CKM matrix element $V_{cs(d)}$. The ratios of the branching fractions between lepton $\mu$ and $e$ are also calculated as $R_{J/\psi}(D_s)=0.97002(8)$ and $R_{J/\psi}(D)=0.97423(15)$ after performing a continuum limit including only $a^2$ term. The ratios provide necessary theoretical support for the future experimental test of lepton flavor universality., Comment: 13 pages, 7 figures

Automated security protocol verifiers such as ProVerif and Tamarin have been increasingly applied to verify large scale complex real-world protocols. While their ability to automate difficult reasoning processes required to handle protocols at that scale is impressive, there remains a gap in the modeling languages used. In particular, providing support for writing and maintaining large protocol specifications. This work attempts to fill this gap by introducing a high-level protocol modeling language, called Tamgram, with a formal semantics that can be translated to the multiset rewriting semantics of Tamarin. Tamgram supports writing native Tamarin code directly, but also allows for easier structuring of large specifications through various high-level constructs, in particular those needed to manipulate states in protocols. We prove the soundness and the completeness of Tamgram with respect to the trace semantics of Tamarin, discuss different translation strategies, and identify an optimal strategy that yields performance comparable to manually coded Tamarin specifications. Finally we show the practicality of Tamgram with a set of small case studies and one large scale case study.

Adversarial training is one of the most effective methods for enhancing model robustness. Recent approaches incorporate adversarial distillation in adversarial training architectures. However, we notice two scenarios of defense methods that limit their performance: (1) Previous methods primarily use static ground truth for adversarial training, but this often causes robust overfitting; (2) The loss functions are either Mean Squared Error or KL-divergence leading to a sub-optimal performance on clean accuracy. To solve those problems, we propose a dynamic label adversarial training (DYNAT) algorithm that enables the target model to gradually and dynamically gain robustness from the guide model's decisions. Additionally, we found that a budgeted dimension of inner optimization for the target model may contribute to the trade-off between clean accuracy and robust accuracy. Therefore, we propose a novel inner optimization method to be incorporated into the adversarial training. This will enable the target model to adaptively search for adversarial examples based on dynamic labels from the guiding model, contributing to the robustness of the target model. Extensive experiments validate the superior performance of our approach.

This paper formulates a new Best-Arm Identification problem in the non-stationary stochastic bandits setting, where the means of all arms are shifted in the same way due to a global influence of the environment. The aim is to identify the unique best arm across environmental change given a fixed total budget. While this setting can be regarded as a special case of Adversarial Bandits or Corrupted Bandits, we demonstrate that existing solutions tailored to those settings do not fully utilise the nature of this global influence, and thus, do not work well in practice (despite their theoretical guarantees). To overcome this issue, in this paper we develop a novel selection policy that is consistent and robust in dealing with global environmental shifts. We then propose an allocation policy, LinLUCB, which exploits information about global shifts across all arms in each environment. Empirical tests depict a significant improvement in our policies against other existing methods., Comment: Extended version of the paper accepted at the 27th European Conference on Artificial Intelligence (ECAI 2024); Paper ID: M1125

This paper explores the recent advancements in enhancing Computational Fluid Dynamics (CFD) tasks through Machine Learning (ML) techniques. We begin by introducing fundamental concepts, traditional methods, and benchmark datasets, then examine the various roles ML plays in improving CFD. The literature systematically reviews papers in recent five years and introduces a novel classification for forward modeling: Data-driven Surrogates, Physics-Informed Surrogates, and ML-assisted Numerical Solutions. Furthermore, we also review the latest ML methods in inverse design and control, offering a novel classification and providing an in-depth discussion. Then we highlight real-world applications of ML for CFD in critical scientific and engineering disciplines, including aerodynamics, combustion, atmosphere & ocean science, biology fluid, plasma, symbolic regression, and reduced order modeling. Besides, we identify key challenges and advocate for future research directions to address these challenges, such as multi-scale representation, physical knowledge encoding, scientific foundation model and automatic scientific discovery. This review serves as a guide for the rapidly expanding ML for CFD community, aiming to inspire insights for future advancements. We draw the conclusion that ML is poised to significantly transform CFD research by enhancing simulation accuracy, reducing computational time, and enabling more complex analyses of fluid dynamics. The paper resources can be viewed at https://github.com/WillDreamer/Awesome-AI4CFD., Comment: 22 pages, 6 figures

The international collaboration designing and constructing the Deep Underground Neutrino Experiment (DUNE) at the Long-Baseline Neutrino Facility (LBNF) has developed a two-phase strategy toward the implementation of this leading-edge, large-scale science project. The 2023 report of the US Particle Physics Project Prioritization Panel (P5) reaffirmed this vision and strongly endorsed DUNE Phase I and Phase II, as did the European Strategy for Particle Physics. While the construction of the DUNE Phase I is well underway, this White Paper focuses on DUNE Phase II planning. DUNE Phase-II consists of a third and fourth far detector (FD) module, an upgraded near detector complex, and an enhanced 2.1 MW beam. The fourth FD module is conceived as a "Module of Opportunity", aimed at expanding the physics opportunities, in addition to supporting the core DUNE science program, with more advanced technologies. This document highlights the increased science opportunities offered by the DUNE Phase II near and far detectors, including long-baseline neutrino oscillation physics, neutrino astrophysics, and physics beyond the standard model. It describes the DUNE Phase II near and far detector technologies and detector design concepts that are currently under consideration. A summary of key R&D goals and prototyping phases needed to realize the Phase II detector technical designs is also provided. DUNE's Phase II detectors, along with the increased beam power, will complete the full scope of DUNE, enabling a multi-decadal program of groundbreaking science with neutrinos.

Rapid progress in quantum technology has transformed quantum computing and quantum information science from theoretical possibilities into tangible engineering challenges. Breakthroughs in quantum algorithms, quantum simulations, and quantum error correction are bringing useful quantum computation closer to fruition. These remarkable achievements have been facilitated by advances in quantum characterization, verification, and validation (QCVV). QCVV methods and protocols enable scientists and engineers to scrutinize, understand, and enhance the performance of quantum information-processing devices. In this Tutorial, we review the fundamental principles underpinning QCVV, and introduce a diverse array of QCVV tools used by quantum researchers. We define and explain QCVV's core models and concepts -- quantum states, measurements, and processes -- and illustrate how these building blocks are leveraged to examine a target system or operation. We survey and introduce protocols ranging from simple qubit characterization to advanced benchmarking methods. Along the way, we provide illustrated examples and detailed descriptions of the protocols, highlight the advantages and disadvantages of each, and discuss their potential scalability to future large-scale quantum computers. This Tutorial serves as a guidebook for researchers unfamiliar with the benchmarking and characterization of quantum computers, and also as a detailed reference for experienced practitioners.

In order to explore how the ubiquitous stochastic low-frequency (SLF) variability of O-type stars is related to various stellar characteristics, we compiled a sample of 150 O-type stars observed via ground-based spectroscopic surveys, alongside photometric data obtained from the Transiting Exoplanet Survey Satellite (TESS). We analyzed 298 light curves obtained from TESS sectors 1-65 for the stars in our sample. Leveraging the spectroscopic parameters, we used Bonnsai to determine masses, radii, fractional main sequence ages, and mass-loss rates for stars of our sample. Subsequently, we identified possible correlations between the fitted parameters of SLF variability and stellar properties. Our analysis unveiled four significant correlations between the amplitude and stellar parameters, including mass, radius, fractional main sequence ages, and mass-loss rate. For stars with $M \gtrsim 30 M_{\odot}$, we observed a decrease in characteristic frequency and steepness with increasing radius. Finally, we compared various physical processes that may account for the SLF variability with our results. The observed SLF variability may arise from the combined effects of FeCZ and IGWs, with IGWs potentially more dominant in the early stages of stellar evolution, and the contribution of FeCZ becoming more significant as stars evolve. Meanwhile, our results indicate that the SLF variability of O-type stars bears certain signatures of the line-driven wind instability and granulation., Comment: 40 pages, 12 figures,accepted by Astrophysical Journal Supplement Series

We introduce SimBench, a benchmark designed to evaluate the proficiency of student large language models (S-LLMs) in generating digital twins (DTs) that can be used in simulators for virtual testing. Given a collection of S-LLMs, this benchmark enables the ranking of the S-LLMs based on their ability to produce high-quality DTs. We demonstrate this by comparing over 20 open- and closed-source S-LLMs. Using multi-turn interactions, SimBench employs a rule-based judge LLM (J-LLM) that leverages both predefined rules and human-in-the-loop guidance to assign scores for the DTs generated by the S-LLM, thus providing a consistent and expert-inspired evaluation protocol. The J-LLM is specific to a simulator, and herein the proposed benchmarking approach is demonstrated in conjunction with the Chrono multi-physics simulator. Chrono provided the backdrop used to assess an S-LLM in relation to the latter's ability to create digital twins for multibody dynamics, finite element analysis, vehicle dynamics, robotic dynamics, and sensor simulations. The proposed benchmarking principle is broadly applicable and enables the assessment of an S-LLM's ability to generate digital twins for other simulation packages. All code and data are available at https://github.com/uwsbel/SimBench.

The detection and analysis of infant cry and snoring events are crucial tasks within the field of audio signal processing. While existing datasets for general sound event detection are plentiful, they often fall short in providing sufficient, strongly labeled data specific to infant cries and snoring. To provide a benchmark dataset and thus foster the research of infant cry and snoring detection, this paper introduces the Infant Cry and Snoring Detection (ICSD) dataset, a novel, publicly available dataset specially designed for ICSD tasks. The ICSD comprises three types of subsets: a real strongly labeled subset with event-based labels annotated manually, a weakly labeled subset with only clip-level event annotations, and a synthetic subset generated and labeled with strong annotations. This paper provides a detailed description of the ICSD creation process, including the challenges encountered and the solutions adopted. We offer a comprehensive characterization of the dataset, discussing its limitations and key factors for ICSD usage. Additionally, we conduct extensive experiments on the ICSD dataset to establish baseline systems and offer insights into the main factors when using this dataset for ICSD research. Our goal is to develop a dataset that will be widely adopted by the community as a new open benchmark for future ICSD research., Comment: 11 pages, 6 figures

Rhombohedral multilayer graphene, with its flat electronic bands and concentrated Berry curvature, is a promising material for the realization of correlated topological phases of matter. When aligned to an adjacent hexagonal boron nitride (hBN) layer, the graphene develops narrow minibands with non-trivial topology. By tuning an externally-applied electric displacement field, the conduction electrons can either be pushed towards or away from the moir\'e superlattice. Motivated by the recent observation of the fractional quantum anomalous Hall effect (FQAHE) in the moir\'e-distant case, we study the opposite moir\'e-proximal case, where the superlattice potential is considerably stronger. We explore the physics within the moir\'e conduction bands through capacitance measurements that allow us to determine the inverse electronic compressibility and extract energy gaps of incompressible states. We observe integer and fractional Chern insulator states at superlattice filling factors v = 1, 2/3, and 1/3 with Streda slopes of -1, -2/3, and -1/3, respectively. Remarkably, the v = 1/3 state persists down to a magnetic field of 0.2 T. In addition, we also observe numerous trivial and topological charge density waves. We map out a phase diagram that is highly sensitive to both displacement and magnetic fields, which tune the system between various ground states by modifying the band dispersion and the structure of the electronic wavefunctions. This work demonstrates displacement field control of topological phase transitions in the moir\'e-proximal limit of rhombohedral pentalayer graphene, creating a highly-tunable platform for studying the interplay between intrinsic band topology and strong lattice effects.

Countless disasters have resulted from climate change, causing severe damage to infrastructure and the economy. These disasters have significant societal impacts, necessitating mental health services for the millions affected. To prepare for and respond effectively to such events, it is important to understand people's emotions and the life incidents they experience before and after a disaster strikes. In this case study, we collected a dataset of approximately 400,000 public tweets related to the storm. Using a BERT-based model, we predicted the emotions associated with each tweet. To efficiently identify these topics, we utilized the Latent Dirichlet Allocation (LDA) technique for topic modeling, which allowed us to bypass manual content analysis and extract meaningful patterns from the data. However, rather than stopping at topic identification like previous methods \cite{math11244910}, we further refined our analysis by integrating Graph Neural Networks (GNN) and Large Language Models (LLM). The GNN was employed to generate embeddings and construct a similarity graph of the tweets, which was then used to optimize clustering. Subsequently, we used an LLM to automatically generate descriptive names for each event cluster, offering critical insights for disaster preparedness and response strategies.

Large language models (LLMs) have advanced financial applications, yet they often lack sufficient financial knowledge and struggle with tasks involving multi-modal inputs like tables and time series data. To address these limitations, we introduce \textit{Open-FinLLMs}, a series of Financial LLMs. We begin with FinLLaMA, pre-trained on a 52 billion token financial corpus, incorporating text, tables, and time-series data to embed comprehensive financial knowledge. FinLLaMA is then instruction fine-tuned with 573K financial instructions, resulting in FinLLaMA-instruct, which enhances task performance. Finally, we present FinLLaVA, a multimodal LLM trained with 1.43M image-text instructions to handle complex financial data types. Extensive evaluations demonstrate FinLLaMA's superior performance over LLaMA3-8B, LLaMA3.1-8B, and BloombergGPT in both zero-shot and few-shot settings across 19 and 4 datasets, respectively. FinLLaMA-instruct outperforms GPT-4 and other Financial LLMs on 15 datasets. FinLLaVA excels in understanding tables and charts across 4 multimodal tasks. Additionally, FinLLaMA achieves impressive Sharpe Ratios in trading simulations, highlighting its robust financial application capabilities. We will continually maintain and improve our models and benchmarks to support ongoing innovation in academia and industry., Comment: 33 pages, 13 figures

Currently, Audio Language Models (ALMs) are rapidly advancing due to the developments in large language models and audio neural codecs. These ALMs have significantly lowered the barrier to creating deepfake audio, generating highly realistic and diverse types of deepfake audio, which pose severe threats to society. Consequently, effective audio deepfake detection technologies to detect ALM-based audio have become increasingly critical. This paper investigate the effectiveness of current countermeasure (CM) against ALM-based audio. Specifically, we collect 12 types of the latest ALM-based deepfake audio and utilizing the latest CMs to evaluate. Our findings reveal that the latest codec-trained CM can effectively detect ALM-based audio, achieving 0% equal error rate under most ALM test conditions, which exceeded our expectations. This indicates promising directions for future research in ALM-based deepfake audio detection.

Buildup and switching mechanisms of solitons in complex nonlinear systems are fundamentally important dynamical regimes. Using a novel strongly nonlinear optical system,the work reveals a new buildup scenario for soliton molecules , which includes a long-duration stage dominated by the emergence of transient NLPs modes to withstand strong disturbances arising from turbulence and extreme nonlinearity in the optical cavity. Systematic simulations reveal effects of the PC rotation angle and intra-cavity nonlinearity on the periodic phase transitions between the different soliton states, and accurately reproduce the experimentally observed buildup and switching mechanisms. These findings could enhance our fundamental study and points to potential uses in designing information encoding systems., Comment: To be published in LASER & PHOTONICS REVIEWS

Manipulating valley-polarized currents at optical frequencies is the key to petahertz valleytronics, yet it remains intractable. To tackle this challenge, we propose an all-optical scheme using non-resonant bichromatic optical fields, which allow for the control of sub-cycle electron dynamics. The combined effect of the helical and asymmetric waveforms of the optical fields leads to the valley-polarization and displacement of the excited electrons concurrently, thereby inducing the valleypolarized currents, on the sub-optical-cycle timescale. This scheme inherently possesses remarkable resilience to decoherence, making it particularly suitable for materials with short decoherence times. Moreover, the direction of the currents can be precisely controlled by adjusting the relative phase of the bichromatic components. Our scheme offers a promising avenue for generating and modulating valley-polarized currents at the femtosecond timescale, opening the door to the realm of petahertz valleytronics.

The detection of gravitational waves has ushered in a new era of observing the universe. Quantum resource advantages offer significant enhancements to the sensitivity of gravitational wave observatories. While squeezed states for ground-based gravitational wave detection have received marked attention, the generation of squeezed states suitable for mid-to-low-frequency detection has remained unexplored. To address the gap in squeezed state optical fields at ultra-low frequencies, we report on the first direct observation of a squeezed vacuum field until Fourier frequency of 4 millihertz with the quantum noise reduction of up to 8 dB, by the employment of a multiple noise suppression scheme. Our work provides quantum resources for future gravitational wave observatories, facilitating the development of quantum precision measurement., Comment: 17 pages,4 figures,1 table and 50 Refs. To be published in Light: Science & Applications

Spectral clustering requires the time-consuming decomposition of the Laplacian matrix of the similarity graph, thus limiting its applicability to large datasets. To improve the efficiency of spectral clustering, a top-down approach was recently proposed, which first divides the data into several micro-clusters (granular-balls), then splits these micro-clusters when they are not "compact'', and finally uses these micro-clusters as nodes to construct a similarity graph for more efficient spectral clustering. However, this top-down approach is challenging to adapt to unevenly distributed or structurally complex data. This is because constructing micro-clusters as a rough ball struggles to capture the shape and structure of data in a local range, and the simplistic splitting rule that solely targets ``compactness'' is susceptible to noise and variations in data density and leads to micro-clusters with varying shapes, making it challenging to accurately measure the similarity between them. To resolve these issues, this paper first proposes to start from local structures to obtain micro-clusters, such that the complex structural information inside local neighborhoods is well captured by them. Moreover, by noting that Euclidean distance is more suitable for convex sets, this paper further proposes a data splitting rule that couples local density and data manifold structures, so that the similarities of the obtained micro-clusters can be easily characterized. A novel similarity measure between micro-clusters is then proposed for the final spectral clustering. A series of experiments based on synthetic and real-world datasets demonstrate that the proposed method has better adaptability to structurally complex data than granular-ball based methods.

Nonlocality, evidenced by the violation of Bell inequalities, not only signifies entanglement but also highlights measurement incompatibility in quantum systems. Utilizing the generalized Clauser-Horne-Shimony-Holt (CHSH) Bell inequality, our high-efficiency optical setup achieves a loophole-free violation of $2.0132$. This result provides a device-independent lower bound on entanglement, quantified as the entanglement of formation at $0.0159$. Moreover, by tuning the parameters of the generalized Bell inequality, we enhance the estimation of measurement incompatibility, which is quantified by an effective overlap of $4.3883 \times 10^{-5}$. To explore the intricate interplay among nonlocality, entanglement, and measurement incompatibility, we generate mixed states, allowing for flexible modulation of entanglement via fast switching among the four Bell states using Pockels cells, achieving a fidelity above $99.10\%$. Intriguingly, our results reveal a counterintuitive relationship where increasing incompatibility initially boosts nonlocality but eventually leads to its reduction. Typically, maximal nonlocality does not coincide with maximal incompatibility. This experimental study sheds light on the optimal management of quantum resources for Bell-inequality-based quantum information processing., Comment: 15 pages, 9 figures

We present the properties of two candidate massive ($M_\star\sim10^{11}M_\odot$) and dusty ($A_{\rm v}>2.5$ mag) galaxies at $z=5-7$ in the first 0.28 deg$^2$ of the COSMOS-Web survey. One object is spectroscopically confirmed at $z_{\rm spec}=5.051$, while the other has a robust $z_{\rm phot}=6.7\pm0.3$. Thanks to their extremely red colors ($F277W-F444W\sim1.7$ mag), these galaxies satisfy the nominal color-selection for the widely-studied ``little red dot" (LRD) population with the exception of their spatially-resolved morphologies. The morphology of our targets allows us to conclude that their red continuum is dominated by highly obscured stellar emission and not by reddened nuclear activity. Using a variety of SED-fitting tools and star formation histories, we estimate the stellar masses to be $\log(M_\star)=11.32^{+0.07}_{-0.15}$ $M_\odot$ and $\log(M_\star)=11.2^{+0.1}_{-0.2}$ $M_\odot$, respectively, with a red continuum emission dominated by a recent episode of star formation. We then compare their number density to the halo mass function to infer stellar baryon fractions of $\epsilon_\star\sim0.25$ and $\epsilon_\star\sim0.5$. Both are significantly higher than what is commonly observed in lower-z galaxies or more dust-obscured galaxies at similar redshifts. With very bright ultra-high-z Lyman-Break Galaxies and some non-AGN dominated LRDs, such ``extended" LRDs represent another population that may require very efficient star formation at early times., Comment: 11 pages, 4 figures, 1 table. Accepted for publication in ApJL

Electrons in topological flat bands can form novel topological states driven by the correlation effects. The penta-layer rhombohedral graphene/hBN moire superlattice has been shown to host fractional quantum anomalous Hall effect (FQAHE) at ~400 mK, triggering discussions around the underlying mechanism and the role of moire effects. In particular, novel electron crystal states with non-trivial topology have been proposed. Here we report DC electrical transport measurement in rhombohedral penta- and tetra-layer graphene/hBN moire superlattices at electronic temperatures down to ~40 mK. We observed two more FQAH states in the penta-layer devices than previously reported. In a new tetra-layer device, we observed FQAHE at filling factors v = 3/5 and 2/3 at 300 mK. With a small bias current and the lowest temperature, we observed a new extended quantum anomalous Hall (EQAH) state and magnetic hysteresis, where Rxy = h/e2 and vanishing Rxx span a wide range of moire filling factor v from 0.5 to up to 1.3. By increasing the temperature or current, FQAHE can be recovered -- suggesting the break-down of the EQAH states and a phase transition into the fractional quantum Hall liquid. Furthermore, we observed displacement field-induced quantum phase transitions from the EQAH states to Fermi liquid, FQAH liquid and the likely composite Fermi liquid. Our observation establishes a new topological phase of electrons with quantized Hall resistance at zero magnetic field, and enriches the emergent quantum phenomena in materials with topological flat bands.

Image restoration endeavors to reconstruct a high-quality, detail-rich image from a degraded counterpart, which is a pivotal process in photography and various computer vision systems. In real-world scenarios, different types of degradation can cause the loss of image details at various scales and degrade image contrast. Existing methods predominantly rely on CNN and Transformer to capture multi-scale representations. However, these methods are often limited by the high computational complexity of Transformers and the constrained receptive field of CNN, which hinder them from achieving superior performance and efficiency in image restoration. To address these challenges, we propose a novel Multi-Scale State-Space Model-based (MS-Mamba) for efficient image restoration that enhances the capacity for multi-scale representation learning through our proposed global and regional SSM modules. Additionally, an Adaptive Gradient Block (AGB) and a Residual Fourier Block (RFB) are proposed to improve the network's detail extraction capabilities by capturing gradients in various directions and facilitating learning details in the frequency domain. Extensive experiments on nine public benchmarks across four classic image restoration tasks, image deraining, dehazing, denoising, and low-light enhancement, demonstrate that our proposed method achieves new state-of-the-art performance while maintaining low computational complexity. The source code will be publicly available.

Density-based clustering methods by mode-seeking usually achieve clustering by using local density estimation to mine structural information, such as local dependencies from lower density points to higher neighbors. However, they often rely too heavily on \emph{local} structures and neglect \emph{global} characteristics, which can lead to significant errors in peak selection and dependency establishment. Although introducing more hyperparameters that revise dependencies can help mitigate this issue, tuning them is challenging and even impossible on real-world datasets. In this paper, we propose a new algorithm (TANGO) to establish local dependencies by exploiting a global-view \emph{typicality} of points, which is obtained by mining further the density distributions and initial dependencies. TANGO then obtains sub-clusters with the help of the adjusted dependencies, and characterizes the similarity between sub-clusters by incorporating path-based connectivity. It achieves final clustering by employing graph-cut on sub-clusters, thus avoiding the challenging selection of cluster centers. Moreover, this paper provides theoretical analysis and an efficient method for the calculation of typicality. Experimental results on several synthetic and $16$ real-world datasets demonstrate the effectiveness and superiority of TANGO.

Engineering non-Abelian quasiparticles by combining superconductivity and topological states have been proposed as a route to realize topological quantum computation. Rhombohedral multilayer graphene with layer number N>=3 has been shown as a promising platform, as it hosts integer and fractional quantum anomalous Hall effects when proximitized by transition metal dichalcogenide (TMD) and a moire potential. However, superconductivity in similar devices have remained largely unexplored, although proximitized spin-orbit-coupling (SOC) effect has been shown to strengthen or induce superconductivity in both crystalline and twisted graphene. Here we report electron transport measurements of TMD-proximitized rhombohedral trilayer graphene (RTG) at temperatures down to 40 mK. We observed a new hole-doped superconducting state SC4 with a transition temperature Tc of 230 mK. On the electron-doped side, we identified a new isospin-symmetry breaking three-quarter-metal (TQM) phase. Near this three-quarter-metal state, the state SC3, very weak in bare RTG, is fully developed into a superconducting state at 110 mK. By performing fermiology analysis based on the quantum oscillation measurement, we showed that the SC3 and SC4 states reside at the phase boundaries between different isospin-symmetry-breaking states. These observations are aligned with the existing understanding that SOC enhances graphene superconductivity. Surprisingly, the original superconducting state SC1 in bare RTG is strongly suppressed in the presence of TMD, and we cannot find it down to the base temperature of our measurement. Our observations form the basis of exploring superconductivity and non-Abelian quasiparticles in rhombohedral graphene devices, and provide experimental evidence that challenges the understanding of the impacts of SOC on graphene superconductivity., Comment: 35 pages; 4 figures, 1 table, 13 extended data figures

Integrated sensing and communication (ISAC) is an emerging technology in next-generation communication networks. However, the communication performance of the ISAC system may be severely affected by interference from the radar system if the sensing task has demanding performance requirements. In this paper, we exploit device-to-device communication (D2D) to improve system communication capacity. The ISAC system in a single cell D2D assisted-network is investigated, where the base station (BS) performs target sensing and communication with multiple celluar user equipments (CUEs) as well as D2D user equipments (DUEs) simultaneously communicating with other DUEs by multiplexing the same frequency resource. To achieve the optimal communication performance in the D2D-assisted ISAC system, a joint beamforming and power control problem is formulated with the goal to maximize the sum rate of the system while guaranteeing the performance requirements of radar sensing. Due to the non-convexity of the problem, we propose the algorithm to transform the origin problem into a relaxation form and obtain the solution. We also proposed the zero-forcing (ZF) beamforming scheme to acquire the solution that can eliminate the interference of the BS on DUEs. Extensive numerical simulations demonstrated that with the assistance of the D2D communications, our proposed algorithm significantly outperforms the baseline schemes in the system sum rate.

The previous advancements in pathology image understanding primarily involved developing models tailored to specific tasks. Recent studies has demonstrated that the large vision-language model can enhance the performance of various downstream tasks in medical image understanding. In this study, we developed a domain-specific large language-vision assistant (PA-LLaVA) for pathology image understanding. Specifically, (1) we first construct a human pathology image-text dataset by cleaning the public medical image-text data for domain-specific alignment; (2) Using the proposed image-text data, we first train a pathology language-image pretraining (PLIP) model as the specialized visual encoder for pathology image, and then we developed scale-invariant connector to avoid the information loss caused by image scaling; (3) We adopt two-stage learning to train PA-LLaVA, first stage for domain alignment, and second stage for end to end visual question \& answering (VQA) task. In experiments, we evaluate our PA-LLaVA on both supervised and zero-shot VQA datasets, our model achieved the best overall performance among multimodal models of similar scale. The ablation experiments also confirmed the effectiveness of our design. We posit that our PA-LLaVA model and the datasets presented in this work can promote research in field of computational pathology. All codes are available at: https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA}{https://github.com/ddw2AIGROUP2CQUPT/PA-LLaVA, Comment: 8 pages, 4 figs

Audio-LLM introduces audio modality into a large language model (LLM) to enable a powerful LLM to recognize, understand, and generate audio. However, during speech recognition in noisy environments, we observed the presence of illusions and repetition issues in audio-LLM, leading to substitution and insertion errors. This paper proposes a transcription prompt-based audio-LLM by introducing an ASR expert as a transcription tokenizer and a hybrid Autoregressive (AR) Non-autoregressive (NAR) decoding approach to solve the above problems. Experiments on 10k-hour WenetSpeech Mandarin corpus show that our approach decreases 12.2% and 9.6% CER relatively on Test_Net and Test_Meeting evaluation sets compared with baseline. Notably, we reduce the decoding repetition rate on the evaluation set to zero, showing that the decoding repetition problem has been solved fundamentally.

Vision-based motion estimation methods show promise in accurately and unobtrusively estimating human body motion for healthcare purposes. However, these methods are not specifically designed for healthcare purposes and face challenges in real-world applications. Human pose estimation methods often lack the accuracy needed for detecting fine-grained, subtle body movements, while optical flow-based methods struggle with poor lighting conditions and unseen real-world data. These issues result in human body motion estimation errors, particularly during critical medical situations where the body is motionless, such as during unconsciousness. To address these challenges and improve the accuracy of human body motion estimation for healthcare purposes, we propose the OPPH operator designed to enhance current vision-based motion estimation methods. This operator, which considers human body movement and noise properties, functions as a multi-stage filter. Results tested on two real-world and one synthetic human motion dataset demonstrate that the operator effectively removes real-world noise, significantly enhances the detection of motionless states, maintains the accuracy of estimating active body movements, and maintains long-term body movement trends. This method could be beneficial for analyzing both critical medical events and chronic medical conditions.

Unbreakable decomposition, introduced by Cygan et al. (SICOMP'19) and Cygan et al. (TALG'20), has proven to be one of the most powerful tools for parameterized graph cut problems in recent years. Unfortunately, all known constructions require at least $\Omega_k\left(mn^2\right)$ time, given an undirected graph with $n$ vertices, $m$ edges, and cut-size parameter $k$. In this work, we show the first close-to-linear time parameterized algorithm that computes an unbreakable decomposition. More precisely, for any $0<\epsilon\leq 1$, our algorithm runs in time $2^{O(\frac{k}{\epsilon} \log \frac{k}{\epsilon})}m^{1 + \epsilon}$ and computes a $(O(k/\epsilon), k)$ unbreakable tree decomposition of $G$, where each bag has adhesion at most $O(k/\epsilon)$. This immediately opens up possibilities for obtaining close-to-linear time algorithms for numerous problems whose only known solution is based on unbreakable decomposition., Comment: 37 pages

Accurate delineation of the boundaries between the renal cortex and medulla is crucial for subsequent functional structural analysis and disease diagnosis. Training high-quality deep-learning models for layer segmentation relies on the availability of large amounts of annotated data. However, due to the patient's privacy of medical data and scarce clinical cases, constructing pathological datasets from clinical sources is relatively difficult and expensive. Moreover, using external natural image datasets introduces noise during the domain generalization process. Cross-species homologous data, such as mouse kidney data, which exhibits high structural and feature similarity to human kidneys, has the potential to enhance model performance on human datasets. In this study, we incorporated the collected private Periodic Acid-Schiff (PAS) stained mouse kidney dataset into the human kidney dataset for joint training. The results showed that after introducing cross-species homologous data, the semantic segmentation models based on CNN and Transformer architectures achieved an average increase of 1.77% and 1.24% in mIoU, and 1.76% and 0.89% in Dice score for the human renal cortex and medulla datasets, respectively. This approach is also capable of enhancing the model's generalization ability. This indicates that cross-species homologous data, as a low-noise trainable data source, can help improve model performance under conditions of limited clinical samples. Code is available at https://github.com/hrlblab/layer_segmentation.

Tracking by detection has been the prevailing paradigm in the field of Multi-object Tracking (MOT). These methods typically rely on the Kalman Filter to estimate the future locations of objects, assuming linear object motion. However, they fall short when tracking objects exhibiting nonlinear and diverse motion in scenarios like dancing and sports. In addition, there has been limited focus on utilizing learning-based motion predictors in MOT. To address these challenges, we resort to exploring data-driven motion prediction methods. Inspired by the great expectation of state space models (SSMs), such as Mamba, in long-term sequence modeling with near-linear complexity, we introduce a Mamba-based motion model named Mamba moTion Predictor (MTP). MTP is designed to model the complex motion patterns of objects like dancers and athletes. Specifically, MTP takes the spatial-temporal location dynamics of objects as input, captures the motion pattern using a bi-Mamba encoding layer, and predicts the next motion. In real-world scenarios, objects may be missed due to occlusion or motion blur, leading to premature termination of their trajectories. To tackle this challenge, we further expand the application of MTP. We employ it in an autoregressive way to compensate for missing observations by utilizing its own predictions as inputs, thereby contributing to more consistent trajectories. Our proposed tracker, MambaTrack, demonstrates advanced performance on benchmarks such as Dancetrack and SportsMOT, which are characterized by complex motion and severe occlusion., Comment: Accepted by ACM Multimedia 2024

Accurate traffic forecasting is crucial for effective urban planning and transportation management, enabling efficient resource allocation and enhanced travel experiences. However, existing models often face limitations in generalization, struggling with zero-shot prediction on unseen regions and cities, as well as diminished long-term accuracy. This is primarily due to the inherent challenges in handling the spatial and temporal heterogeneity of traffic data, coupled with the significant distribution shift across time and space. In this work, we aim to unlock new possibilities for building versatile, resilient and adaptive spatio-temporal foundation models for traffic prediction. To achieve this goal, we introduce a novel foundation model, named OpenCity, that can effectively capture and normalize the underlying spatio-temporal patterns from diverse data characteristics, facilitating zero-shot generalization across diverse urban environments. OpenCity integrates the Transformer architecture with graph neural networks to model the complex spatio-temporal dependencies in traffic data. By pre-training OpenCity on large-scale, heterogeneous traffic datasets, we enable the model to learn rich, generalizable representations that can be seamlessly applied to a wide range of traffic forecasting scenarios. Experimental results demonstrate that OpenCity exhibits exceptional zero-shot predictive performance. Moreover, OpenCity showcases promising scaling laws, suggesting the potential for developing a truly one-for-all traffic prediction solution that can adapt to new urban contexts with minimal overhead. We made our proposed OpenCity model open-source and it is available at the following link: https://github.com/HKUDS/OpenCity., Comment: 12 pages

Federated Collaborative Filtering (FedCF) is an emerging field focused on developing a new recommendation framework with preserving privacy in a federated setting. Existing FedCF methods typically combine distributed Collaborative Filtering (CF) algorithms with privacy-preserving mechanisms, and then preserve personalized information into a user embedding vector. However, the user embedding is usually insufficient to preserve the rich information of the fine-grained personalization across heterogeneous clients. This paper proposes a novel personalized FedCF method by preserving users' personalized information into a latent variable and a neural model simultaneously. Specifically, we decompose the modeling of user knowledge into two encoders, each designed to capture shared knowledge and personalized knowledge separately. A personalized gating network is then applied to balance personalization and generalization between the global and local encoders. Moreover, to effectively train the proposed framework, we model the CF problem as a specialized Variational AutoEncoder (VAE) task by integrating user interaction vector reconstruction with missing value prediction. The decoder is trained to reconstruct the implicit feedback from items the user has interacted with, while also predicting items the user might be interested in but has not yet interacted with. Experimental results on benchmark datasets demonstrate that the proposed method outperforms other baseline methods, showcasing superior performance., Comment: 10 pages, 3 figures, 4 tables, conference

Achieving human-level intelligence requires refining cognitive distinctions between System 1 and System 2 thinking. While contemporary AI, driven by large language models, demonstrates human-like traits, it falls short of genuine cognition. Transitioning from structured benchmarks to real-world scenarios presents challenges for visual agents, often leading to inaccurate and overly confident responses. To address the challenge, we introduce FaST, which incorporates the Fast and Slow Thinking mechanism into visual agents. FaST employs a switch adapter to dynamically select between System 1/2 modes, tailoring the problem-solving approach to different task complexity. It tackles uncertain and unseen objects by adjusting model confidence and integrating new contextual data. With this novel design, we advocate a flexible system, hierarchical reasoning capabilities, and a transparent decision-making pipeline, all of which contribute to its ability to emulate human-like cognitive processes in visual intelligence. Empirical results demonstrate that FaST outperforms various well-known baselines, achieving 80.8% accuracy over VQA^{v2} for visual question answering and 48.7% GIoU score over ReasonSeg for reasoning segmentation, demonstrate FaST's superior performance. Extensive testing validates the efficacy and robustness of FaST's core components, showcasing its potential to advance the development of cognitive visual agents in AI systems.

Burst super-resolution aims to reconstruct high-resolution images with higher quality and richer details by fusing the sub-pixel information from multiple burst low-resolution frames. In BusrtSR, the key challenge lies in extracting the base frame's content complementary sub-pixel details while simultaneously suppressing high-frequency noise disturbance. Existing methods attempt to extract sub-pixels by modeling inter-frame relationships frame by frame while overlooking the mutual correlations among multi-current frames and neglecting the intra-frame interactions, leading to inaccurate and noisy sub-pixels for base frame super-resolution. Further, existing methods mainly employ static upsampling with fixed parameters to improve spatial resolution for all scenes, failing to perceive the sub-pixel distribution difference across multiple frames and cannot balance the fusion weights of different frames, resulting in over-smoothed details and artifacts. To address these limitations, we introduce a novel Query Mamba Burst Super-Resolution (QMambaBSR) network, which incorporates a Query State Space Model (QSSM) and Adaptive Up-sampling module (AdaUp). Specifically, based on the observation that sub-pixels have consistent spatial distribution while random noise is inconsistently distributed, a novel QSSM is proposed to efficiently extract sub-pixels through inter-frame querying and intra-frame scanning while mitigating noise interference in a single step. Moreover, AdaUp is designed to dynamically adjust the upsampling kernel based on the spatial distribution of multi-frame sub-pixel information in the different burst scenes, thereby facilitating the reconstruction of the spatial arrangement of high-resolution details. Extensive experiments on four popular synthetic and real-world benchmarks demonstrate that our method achieves a new state-of-the-art performance.

We present the first systematic evaluation examining format bias in performance of large language models (LLMs). Our approach distinguishes between two categories of an evaluation metric under format constraints to reliably and accurately assess performance: one measures performance when format constraints are adhered to, while the other evaluates performance regardless of constraint adherence. We then define a metric for measuring the format bias of LLMs and establish effective strategies to reduce it. Subsequently, we present our empirical format bias evaluation spanning four commonly used categories -- multiple-choice question-answer, wrapping, list, and mapping -- covering 15 widely-used formats. Our evaluation on eight generation tasks uncovers significant format bias across state-of-the-art LLMs. We further discover that improving the format-instruction following capabilities of LLMs across formats potentially reduces format bias. Based on our evaluation findings, we study prompting and fine-tuning with synthesized format data techniques to mitigate format bias. Our methods successfully reduce the variance in ChatGPT's performance among wrapping formats from 235.33 to 0.71 (%$^2$).

Photo-Sharing Multi-modal dialogue generation requires a dialogue agent not only to generate text responses but also to share photos at the proper moment. Using image text caption as the bridge, a pipeline model integrates an image caption model, a text generation model, and an image generation model to handle this complex multi-modal task. However, representing the images with text captions may loss important visual details and information and cause error propagation in the complex dialogue system. Besides, the pipeline model isolates the three models separately because discrete image text captions hinder end-to-end gradient propagation. We propose the first end-to-end model for photo-sharing multi-modal dialogue generation, which integrates an image perceptron and an image generator with a large language model. The large language model employs the Q-Former to perceive visual images in the input end. For image generation in the output end, we propose a dynamic vocabulary transformation matrix and use straight-through and gumbel-softmax techniques to align the large language model and stable diffusion model and achieve end-to-end gradient propagation. We perform experiments on PhotoChat and DialogCC datasets to evaluate our end-to-end model. Compared with pipeline models, the end-to-end model gains state-of-the-art performances on various metrics of text and image generation. More analysis experiments also verify the effectiveness of the end-to-end model for photo-sharing multi-modal dialogue generation., Comment: Work in progress

Viscosity is the resistance of a liquid to flow, governed by atomic-scale friction between its constituent atoms. While viscosity can be directly computed using the Green-Kubo formalism, this method does not fully elucidate the physical mechanisms underlying such momentum transport coefficient. In fact, the microscopic origin of liquid viscosity remains poorly understood. In this work, we calculate the viscosity of a $\mathrm{Cu_{50}Zr_{50}}$ metallic liquid and a Kob-Andersen model in a large range of temperatures and compare the results with a theoretical formula based on nonaffine linear response and instantaneous normal mode theory. This analysis reveals that only unstable localized instantaneous normal modes (ULINMs) contribute to viscosity, providing a microscopic definition of viscosity as diffusive momentum transport facilitated by local structural excitations mediated by ULINMs as precursors. Leveraging on the concept of configurational entropy, a quantitative model to connect viscosity with ULINMs is also proposed and validated in both the Arrhenius and non-Arrhenius regimes. In summary, our work provides a microscopic definition of liquid viscosity and highlights the fundamental role of ULINMs as its building blocks, ultimately opening the path to an atomic-scale prediction of viscosity in liquids and glasses., Comment: v1: comments welcome!

The Learning-With-Errors (LWE) problem is a crucial computational challenge with significant implications for post-quantum cryptography and computational learning theory. Here we propose a quantum-classical hybrid algorithm with Ising model (HAWI) to address the LWE problem. Our approach involves transforming the LWE problem into the Shortest Vector Problem (SVP), using variable qubits to encode lattice vectors into an Ising Hamiltonian. We then identify the low-energy levels of the Hamiltonian to extract the solution, making it suitable for implementation on current noisy intermediate-scale quantum (NISQ) devices. We prove that the number of qubits required is less than $m(3m-1)/2$, where $m$ is the number of samples in the algorithm. Our algorithm is heuristic, and its time complexity depends on the specific quantum algorithm employed to find the Hamiltonian's low-energy levels. If the Quantum Approximate Optimization Algorithm (QAOA) is used to solve the Ising Hamiltonian problem, and the number of iterations satisfies $y < O\left(m\log m\cdot 2^{0.2972k}/pk^2\right)$, our algorithm will outperform the classical Block Korkine-Zolotarev (BKZ) algorithm, where $k$ is the block size related to problem parameters, and $p$ is the number of layers in QAOA. We demonstrate the algorithm by solving a $2$-dimensional LWE problem on a real quantum device with $5$ qubits, showing its potential for solving meaningful instances of the LWE problem in the NISQ era.

The Maximum Minimal Cut Problem (MMCP), a NP-hard combinatorial optimization (CO) problem, has not received much attention due to the demanding and challenging bi-connectivity constraint. Moreover, as a CO problem, it is also a daunting task for machine learning, especially without labeled instances. To deal with these problems, this work proposes an unsupervised learning framework combined with heuristics for MMCP that can provide valid and high-quality solutions. As far as we know, this is the first work that explores machine learning and heuristics to solve MMCP. The unsupervised solver is inspired by a relaxation-plus-rounding approach, the relaxed solution is parameterized by graph neural networks, and the cost and penalty of MMCP are explicitly written out, which can train the model end-to-end. A crucial observation is that each solution corresponds to at least one spanning tree. Based on this finding, a heuristic solver that implements tree transformations by adding vertices is utilized to repair and improve the solution quality of the unsupervised solver. Alternatively, the graph is simplified while guaranteeing solution consistency, which reduces the running time. We conduct extensive experiments to evaluate our framework and give a specific application. The results demonstrate the superiority of our method against two techniques designed.

This paper proposes a control design approach for stabilizing nonlinear control systems. Our key observation is that the set of points where the decrease condition of a control Lyapunov function (CLF) is feasible can be regarded as a safe set. By leveraging a nonsmooth version of control barrier functions (CBFs) and a weaker notion of CLF, we develop a control design that forces the system to converge to and remain in the region where the CLF decrease condition is feasible. We characterize the conditions under which our controller asymptotically stabilizes the origin or a small neighborhood around it, even in the cases where it is discontinuous. We illustrate our design in various examples.

JWST has enabled detecting and spatially resolving the heavily dust-attenuated stellar populations of sub-millimeter galaxies, revealing detail that was previously inaccessible. In this work we construct a sample of 289 sub-millimeter galaxies with detailed joint ALMA and JWST constraints in the COSMOS field. Sources are originally selected using the SCUBA-2 instrument and have archival ALMA observations from various programs. Their JWST NIRCam imaging is from COSMOS-Web and PRIMER. We extract multi-wavelength photometry in a manner that leverages the unprecedented near-infrared spatial resolution of JWST, and fit the data with spectral energy distribution models to derive photometric redshifts, stellar masses, star-formation rates and optical attenuation. The sample has an average z=2.6, A_V=2.5, SFR=270 and log(M*)=11.1. There are 81 (30%) galaxies that have no previous optical/near-infrared detections, including 75% of the z>4 sub-sample (n=28). The faintest observed near-infrared sources have the highest redshifts and largest A_V=4. In a preliminary morphology analysis we find that ~10% of our sample exhibit spiral arms and 5% host stellar bars, with one candidate bar found at z>3. Finally, we find that the clustering of JWST galaxies within 10 arcseconds of a sub-mm galaxy is a factor of 2 greater than what is expected based on either random clustering or the distribution of sources around any red galaxy irrespective of a sub-mm detection., Comment: 37 pages (15 for RGBs + references), 14 figures, submitted to ApJ

Estimation of cardiovascular model parameters from electronic health records (EHR) poses a significant challenge primarily due to lack of identifiability. Structural non-identifiability arises when a manifold in the space of parameters is mapped to a common output, while practical non-identifiability can result due to limited data, model misspecification, or noise corruption. To address the resulting ill-posed inverse problem, optimization-based or Bayesian inference approaches typically use regularization, thereby limiting the possibility of discovering multiple solutions. In this study, we use inVAErt networks, a neural network-based, data-driven framework for enhanced digital twin analysis of stiff dynamical systems. We demonstrate the flexibility and effectiveness of inVAErt networks in the context of physiological inversion of a six-compartment lumped parameter hemodynamic model from synthetic data to real data with missing components.

Large Language Models (LLMs) have showcased exceptional ability in causal reasoning from textual information. However, will these causalities remain straightforward for Vision Large Language Models (VLLMs) when only visual hints are provided? Motivated by this, we propose a novel Multimodal Causal Reasoning benchmark, namely MuCR, to challenge VLLMs to infer semantic cause-and-effect relationship when solely relying on visual cues such as action, appearance, clothing, and environment. Specifically, we introduce a prompt-driven image synthesis approach to create siamese images with embedded semantic causality and visual cues, which can effectively evaluate VLLMs' causal reasoning capabilities. Additionally, we develop tailored metrics from multiple perspectives, including image-level match, phrase-level understanding, and sentence-level explanation, to comprehensively assess VLLMs' comprehension abilities. Our extensive experiments reveal that the current state-of-the-art VLLMs are not as skilled at multimodal causal reasoning as we might have hoped. Furthermore, we perform a comprehensive analysis to understand these models' shortcomings from different views and suggest directions for future research. We hope MuCR can serve as a valuable resource and foundational benchmark in multimodal causal reasoning research. The project is available at: https://github.com/Zhiyuan-Li-John/MuCR, Comment: 20 pages