Search

In recent years, mobile phone data has been widely used for human mobility analytics. Identifying individual activity locations is the fundamental step for mobile phone data processing. Current methods typically aggregate spatially adjacent location records over multiple days to identify activity locations. However, only considering spatial relationships while overlooking temporal ones may lead to inaccurate activity location identification, and also affect activity pattern analysis. In this study, we propose a spatiotemporal knowledge graph-based (STKG) method for identifying activity locations from mobile phone data. An STKG is designed and constructed to describe individual mobility characteristics. The spatial and temporal relationships of individual stays are inferred and transformed into a spatiotemporal graph. The modularity-optimization community detection algorithm is applied to identify stays with dense spatiotemporal relationships, which are considering as activity locations. A case study in Shanghai was conducted to verify the performance of the proposed method. The results show that compared with two baseline methods, the STKG-based method can limit an additional 45% of activity locations with the longest daytime stay within a reasonable spatial range; In addition, the STKG-based method exhibit lower variance in the start and end times of activities across different days, performing approximately 10% to 20% better than the two baseline methods. Moreover, the STKG-based method effectively distinguishes between locations that are geographically close but exhibit different temporal patterns. These findings demonstrate the effectiveness of STKG-based method in enhancing both spatial precision and temporal consistency., Comment: 24 pages, 10 figures, 1 table

In this work, we investigate a particular implicit bias in the gradient descent training process, which we term "Feature Averaging", and argue that it is one of the principal factors contributing to non-robustness of deep neural networks. Despite the existence of multiple discriminative features capable of classifying data, neural networks trained by gradient descent exhibit a tendency to learn the average (or certain combination) of these features, rather than distinguishing and leveraging each feature individually. In particular, we provide a detailed theoretical analysis of the training dynamics of gradient descent in a two-layer ReLU network for a binary classification task, where the data distribution consists of multiple clusters with orthogonal cluster center vectors. We rigorously prove that gradient descent converges to the regime of feature averaging, wherein the weights associated with each hidden-layer neuron represent an average of the cluster centers (each center corresponding to a distinct feature). It leads the network classifier to be non-robust due to an attack that aligns with the negative direction of the averaged features. Furthermore, we prove that, with the provision of more granular supervised information, a two-layer multi-class neural network is capable of learning individual features, from which one can derive a binary classifier with the optimal robustness under our setting. Besides, we also conduct extensive experiments using synthetic datasets, MNIST and CIFAR-10 to substantiate the phenomenon of feature averaging and its role in adversarial robustness of neural networks. We hope the theoretical and empirical insights can provide a deeper understanding of the impact of the gradient descent training on feature learning process, which in turn influences the robustness of the network, and how more detailed supervision may enhance model robustness., Comment: 78 pages, 10 figures

Let $\sigma_i$ be the braid actions on infinite Grassmannian cluster algebras induced from Fraser's braid group actions. Let $\mathsf{T}_i$ be the braid group actions on (quantum) Grothendieck rings of Hernandez-Leclerc category ${\mathscr C}_\mathfrak{g}^0$ of affine type $A_n^{(1)}$, and $\mathsf{R}_i$ the braid group actions on the corresponding extended crystals. In the paper, we prove that the actions $\sigma_i$ coincide with the braid group actions $\mathsf{T}_i$ and $\mathsf{R}_i$., Comment: 33 pages

In the field of industrial inspection, Multimodal Large Language Models (MLLMs) have a high potential to renew the paradigms in practical applications due to their robust language capabilities and generalization abilities. However, despite their impressive problem-solving skills in many domains, MLLMs' ability in industrial anomaly detection has not been systematically studied. To bridge this gap, we present MMAD, the first-ever full-spectrum MLLMs benchmark in industrial Anomaly Detection. We defined seven key subtasks of MLLMs in industrial inspection and designed a novel pipeline to generate the MMAD dataset with 39,672 questions for 8,366 industrial images. With MMAD, we have conducted a comprehensive, quantitative evaluation of various state-of-the-art MLLMs. The commercial models performed the best, with the average accuracy of GPT-4o models reaching 74.9%. However, this result falls far short of industrial requirements. Our analysis reveals that current MLLMs still have significant room for improvement in answering questions related to industrial anomalies and defects. We further explore two training-free performance enhancement strategies to help models improve in industrial scenarios, highlighting their promising potential for future research., Comment: The code and data are available at https://github.com/jam-cc/MMAD

Although granular materials are the second most processed in industry after water, the theoretical study of granules-structure interactions is not as advanced as that of fluid-structure interactions due to the lack of a unified view of the constitutive relation of granular materials. In the previous work, the theoretical model of granules-beam coupled vibration was developed and verified by experiments. However, it was also found that the system exhibits significant stiffness-softening Duffing characteristics even under micro-vibration, which implies that chaotic responses may be reached under certain conditions. To reveal the route and critical conditions for the system to enter chaos, in the present work, the chaotic dynamics of the system are studied. In qualitative analysis, Melnikov method is applied to analyze the instability behavior of the perturbed heteroclinic orbit of the system, thus the critical condition for the system to enter Smale horseshoe chaos, i.e., the Melnikov criterion, is obtained. The validity of the criterion is verified numerically. In experimental studies, the existence of chaotic responses and the route to chaos for granules-beam coupled vibrations is revealed. The experimental results suggest that the system response first experiences symmetry-breaking then undergoes a complete period-doubling cascade, and finally enters single-scroll chaos. In addition, although the Additional Dynamic Load (ADL) generated by granular media is highly complicated, a general and simple evolution pattern of the chaos threshold is found by parametric experiments, which is also supported by the Melnikov criterion. In short, the chaotic dynamics of granules-beam coupled vibration is revealed, which is a contribution to the engineering vibration aspect. On the theoretical side, an impressive result lies in that for the first time, the Melnikov criterion of such fractional-order..., Comment: Accepted

This paper is dedicated to clarifying and introducing the correct application of Melnikov method in fractional dynamics. Attention to the complex dynamics of hyperbolic orbits and to fractional calculus can be, respectively, traced back to Poincar\'es attack on the three-body problem a century ago and to the early days of calculus three centuries ago. Nowadays, fractional calculus has been widely applied in modeling dynamic problems across various fields due to its advantages in describing problems with non-locality. Some of these models have also been confirmed to exhibit hyperbolic orbit dynamics, and recently, they have been extensively studied based on Melnikov method, an analytical approach for homoclinic and heteroclinic orbit dynamics. Despite its decade-long application in fractional dynamics, there is a universal problem in these applications that remains to be clarified, i.e., defining fractional-order systems within finite memory boundaries leads to the neglect of perturbation calculation for parts of the stable and unstable manifolds in Melnikov analysis. After clarifying and redefining the problem, a rigorous analytical case is provided for reference. Unlike existing results, the Melnikov criterion here is derived in a globally closed form, which was previously considered unobtainable due to difficulties in the analysis of fractional-order perturbations characterized by convolution integrals with power-law type singular kernels. Finally, numerical methods are employed to verify the derived Melnikov criterion. Overall, the clarification for the problem and the presented case are expected to provide insights for future research in this topic., Comment: Accepted

For each simple Lie algebra $\mathfrak{g}$ of simply-laced type, Hernandez and Leclerc introduced a certain category $\mathcal{C}_{\mathbb{Z}}$ of finite-dimensional representations of the quantum affine algebra of $\mathfrak{g}$, as well as certain subcategories $\mathcal{C}_{\mathbb{Z}}^{\leq \xi}$ depending on a choice of height function adapted to an orientation of the Dynkin graph of $\mathfrak{g}$. In our previous work we constructed an algebra homomorphism $\widetilde{D}_{\xi}$ whose domain contains the image of the Grothendieck ring of $\mathcal{C}_{\mathbb{Z}}^{\leq \xi}$ under the truncated $q$-character morphism $\widetilde{\chi}_q$ corresponding to $\xi$. We exhibited a close relationship between the composition of $\widetilde{D}_{\xi}$ with $\widetilde{\chi}_q$ and the morphism $\overline{D}$ recently introduced by Baumann, Kamnitzer and Knutson in their study of the equivariant homology of Mirkovi\'c-Vilonen cycles. In this paper, we extend $\widetilde{D}_{\xi}$ in order to investigate its composition with Frenkel-Reshetikhin's original $q$-character morphism. Our main result consists in proving that the $q$-characters of all standard modules in $\mathcal{C}_{\mathbb{Z}}$ lie in the kernel of $\widetilde{D}_{\xi}$. This provides a large family of new non-trivial rational identities suggesting possible geometric interpretations., Comment: 26 pages, 2 figures. Comments welcome

Restless multi-armed bandits (RMAB) has been widely used to model constrained sequential decision making problems, where the state of each restless arm evolves according to a Markov chain and each state transition generates a scalar reward. However, the success of RMAB crucially relies on the availability and quality of reward signals. Unfortunately, specifying an exact reward function in practice can be challenging and even infeasible. In this paper, we introduce Pref-RMAB, a new RMAB model in the presence of preference signals, where the decision maker only observes pairwise preference feedback rather than scalar reward from the activated arms at each decision epoch. Preference feedback, however, arguably contains less information than the scalar reward, which makes Pref-RMAB seemingly more difficult. To address this challenge, we present a direct online preference learning (DOPL) algorithm for Pref-RMAB to efficiently explore the unknown environments, adaptively collect preference data in an online manner, and directly leverage the preference feedback for decision-makings. We prove that DOPL yields a sublinear regret. To our best knowledge, this is the first algorithm to ensure $\tilde{\mathcal{O}}(\sqrt{T\ln T})$ regret for RMAB with preference feedback. Experimental results further demonstrate the effectiveness of DOPL.

We report the detection of an extended very-high-energy (VHE) gamma-ray source coincident with the locations of middle-aged (62.4~\rm kyr) pulsar PSR J0248+6021, by using the LHAASO-WCDA data of live 796 days and LHAASO-KM2A data of live 1216 days. A significant excess of \gray induced showers is observed both by WCDA in energy bands of 1-25~\rm TeV and KM2A in energy bands of $>$ 25~\rm TeV with 7.3 $\sigma$ and 13.5 $\sigma$, respectively. The best-fit position derived through WCDA data is R.A. = 42.06$^\circ \pm$ 0.12$^\circ$ and Dec. = 60.24$^\circ \pm $ 0.13$^\circ$ with an extension of 0.69$^\circ\pm$0.15$^\circ$ and that of the KM2A data is R.A.= 42.29$^\circ \pm $ 0.13$^\circ$ and Dec. = 60.38$^\circ \pm$ 0.07$^\circ$ with an extension of 0.37$^\circ\pm$0.07$^\circ$. No clear extended multiwavelength counterpart of this LHAASO source has been found from the radio band to the GeV band. The most plausible explanation of the VHE \gray emission is the inverse Compton process of highly relativistic electrons and positrons injected by the pulsar. These electrons/positrons are hypothesized to be either confined within the pulsar wind nebula or to have already escaped into the interstellar medium, forming a pulsar halo., Comment: 12 pages, 10 figures, Accepted by Sci. China-Phys. Mech. Astron

Building on work of Derksen-Fei and Plamondon, we formulate a conjectural correspondence between additive and monoidal categorifications of cluster algebras, which reveals a new connection between the additive reachability conjecture and the multiplicative reachability conjecture. Evidence for this conjecture is provided by results on Grassmannian cluster algebras and categories in the tame types. Moreover, we give a construction of the generic kernels introduced by Hernandez and Leclerc for type $\mathbb{A}$ via the Grassmannian cluster categories. As an application of the correspondence, we construct rigid indecomposable modules and indecomposable non-rigid rigid modules in Grassmannian cluster categories., Comment: 47 pages, many figures

Employing aberration-corrected scanning transmission electron microscopy (STEM), we meticulously investigated the intrinsic chemical heterogeneity of Fe1+yTe, Fe1+yTe0.8Se0.2, and Fe1+yTe0.5Se0.5. Comprehensive analysis reveals the presence of interstitial iron atoms (Feint) across all samples, pre-dominantly occupying the 2c site of the P4/nmm space group. Moreover, a superstructure phase characterized by a wave vector q = 2/5a + 1/2c, originating from the ordering of Feint, is distinctly observable in the parent compound Fe1+yTe. In this scenario, the Feint atoms interact with adjacent Fe atoms, forming iron polycomplex and leading to an evident distortion of the FeTe4 tetrahedral. Experimental results further demonstrate effective suppression of Feint concentration and ordering through appropriate Se substitution; notably, Fe1+yTe0.5Se0.5 manifests the lowest concentration of Feint atoms. Our findings additionally indicate that Se substitution is random, and nanoscale phase separation induced by Te/Se chemical heterogeneity is commonly observed within Fe1+yTe1-xSex crystals., Comment: 14 pages, 5 figures

In some recent work, we provided a quantitative explanation for the number asymmetry of Jupiter Trojans by hypothesizing a free-floating planet (FFP) flyby into the Solar System. In support of that explanation, this paper examines the influence of the same FFP flyby on the Hilda asteroids, which orbit stably in the 3:2 mean motion resonance with Jupiter. The observed Hilda population exhibits two distinct resonant patterns: (1) a lack of Hildas with resonant amplitudes < 40 deg. at eccentricities < 0.1; (2) a nearly complete absence of Hildas with amplitudes < 20 deg., regardless of eccentricity. Previous models of Jupiter migration and resonance capture could account for the eccentricity distribution of Hildas but have failed to replicate the unusual absence of those with the smallest resonant amplitudes, which theoretically should be the most stable. Here we report that the FFP flyby can trigger an extremely rapid outward migration of Jupiter, causing a sudden shift in the 3:2 Jovian resonance. Consequently, Hildas with varying eccentricities would have their resonant amplitudes changed by different degrees, leading to the observed resonant patterns. We additionally show that, in our FFP flyby scenario, these patterns are consistently present across different resonant amplitude distributions of primordial Hildas arising from various formation models. We also place constraints on the potential parameters of the FFP, suggesting it should have an eccentricity of 1-1.3 or larger, an inclination up to 30 deg. or higher, and a minimum mass of about 50 Earth masses., Comment: 22 pages, 10 figures, accepted for publication in Icarus

Recently, there has been significant interest in replacing the reward model in Reinforcement Learning with Human Feedback (RLHF) methods for Large Language Models (LLMs), such as Direct Preference Optimization (DPO) and its variants. These approaches commonly use a binary cross-entropy mechanism on pairwise samples, i.e., minimizing and maximizing the loss based on preferred or dis-preferred responses, respectively. However, while this training strategy omits the reward model, it also overlooks the varying preference degrees within different responses. We hypothesize that this is a key factor hindering LLMs from sufficiently understanding human preferences. To address this problem, we propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss, thereby helping LLMs improve their ability to understand the degree of preference. Extensive experiments are conducted on two widely used datasets of different tasks. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods and significantly boost their performance to achieve state-of-the-art performance. We also conduct detailed analyses to offer comprehensive insights into SPO, which verifies its effectiveness. The code is available at https://github.com/lijian16/SPO., Comment: Accepted at EMNLP 2024 Findings

To obtain the optimal constraints in complex environments, Inverse Constrained Reinforcement Learning (ICRL) seeks to recover these constraints from expert demonstrations in a data-driven manner. Existing ICRL algorithms collect training samples from an interactive environment. However, the efficacy and efficiency of these sampling strategies remain unknown. To bridge this gap, we introduce a strategic exploration framework with guaranteed efficiency. Specifically, we define a feasible constraint set for ICRL problems and investigate how expert policy and environmental dynamics influence the optimality of constraints. Motivated by our findings, we propose two exploratory algorithms to achieve efficient constraint inference via 1) dynamically reducing the bounded aggregate error of cost estimation and 2) strategically constraining the exploration policy. Both algorithms are theoretically grounded with tractable sample complexity. We empirically demonstrate the performance of our algorithms under various environments.

This paper introduces MCTrack, a new 3D multi-object tracking method that achieves state-of-the-art (SOTA) performance across KITTI, nuScenes, and Waymo datasets. Addressing the gap in existing tracking paradigms, which often perform well on specific datasets but lack generalizability, MCTrack offers a unified solution. Additionally, we have standardized the format of perceptual results across various datasets, termed BaseVersion, facilitating researchers in the field of multi-object tracking (MOT) to concentrate on the core algorithmic development without the undue burden of data preprocessing. Finally, recognizing the limitations of current evaluation metrics, we propose a novel set that assesses motion information output, such as velocity and acceleration, crucial for downstream tasks. The source codes of the proposed method are available at this link: https://github.com/megvii-research/MCTrack}{https://github.com/megvii-research/MCTrack, Comment: 14 pages, 7 figures

Recently, methods like Zero-1-2-3 have focused on single-view based 3D reconstruction and have achieved remarkable success. However, their predictions for unseen areas heavily rely on the inductive bias of large-scale pretrained diffusion models. Although subsequent work, such as DreamComposer, attempts to make predictions more controllable by incorporating additional views, the results remain unrealistic due to feature entanglement in the vanilla latent space, including factors such as lighting, material, and structure. To address these issues, we introduce the Visual Isotropy 3D Reconstruction Model (VI3DRM), a diffusion-based sparse views 3D reconstruction model that operates within an ID consistent and perspective-disentangled 3D latent space. By facilitating the disentanglement of semantic information, color, material properties and lighting, VI3DRM is capable of generating highly realistic images that are indistinguishable from real photographs. By leveraging both real and synthesized images, our approach enables the accurate construction of pointmaps, ultimately producing finely textured meshes or point clouds. On the NVS task, tested on the GSO dataset, VI3DRM significantly outperforms state-of-the-art method DreamComposer, achieving a PSNR of 38.61, an SSIM of 0.929, and an LPIPS of 0.027. Code will be made available upon publication.

We used the Five-hundred-meter Aperture Spherical radio Telescope (FAST) to search for the molecular emissions in the L-band between 1.0 and 1.5 GHz toward four comets, C/2020 F3 (NEOWISE), C/2020 R4 (ATLAS), C/2021 A1 (Leonard), and 67P/Churyumov-Gerasimenko during or after their perihelion passages. Thousands of molecular transition lines fall in this low-frequency range, many attributed to complex organic or prebiotic molecules. We conducted a blind search for the possible molecular lines in this frequency range in those comets and could not identify clear signals of molecular emissions in the data. Although several molecules have been detected at high frequencies of great than 100 GHz in comets, our results confirm that it is challenging to detect molecular transitions in the L-band frequency ranges. The non-detection of L-band molecular lines in the cometary environment could rule out the possibility of unusually strong lines, which could be caused by the masers or non-LTE effects. Although the line strengths are predicted to be weak, for FAST, using the ultra-wide bandwidth receiver and improving the radio frequency interference environments would enhance the detectability of those molecular transitions at low frequencies in the future., Comment: 14 pages, 6 figures, 6 tables, accepted for publication in RAA

In quantum field theory study, Grassmannian manifolds $\text{Gr}(4,n)$ are closely related to $D{=}4$ kinematics input for $n$-particle scattering processes, whose combinatorial and geometrical structures have been widely applied in studying conformal invariant physical theories and their scattering amplitudes. Recently, \cite{HLY21} observed that constraining $D{=}4$ kinematics input to its $D{=}3$ subspace can be interpreted as folding Grassmannian cluster algebras $\mathbb{C}[\text{Gr}(4,n)]$. In this paper, we deduce general expressions for these constraints in terms of Pl\"ucker variables of $\text{Gr}(4,n)$ directly from $D{=}3$ subspace definition, and propose a series of initial quivers for algebra $\mathbb{C}[\text{Gr}(4,n)]$ whose folding conditions exactly meet the constraints, which proves the observation finally., Comment: 12 pages, 2 figures

With the development of urbanization, the scale of urban road network continues to expand, especially in some Asian countries. Short-term traffic state prediction is one of the bases of traffic management and control. Constrained by the space-time cost of computation, the short-term traffic state prediction of large-scale urban road network is difficult. One way to solve this problem is to partition the whole network into multiple sub-networks to predict traffic state separately. In this study, a deep learning method is proposed for road network partitioning. The method mainly includes three steps. First, the daily speed series for roads are encoded into the matrix. Second, a convolutional autoencoder (AE) is built to extract series features and compress data. Third, the spatial hierarchical clustering method with adjacency relationship is applied in the road network. The proposed method was verified by the road network of Shenzhen which contains more than 5000 links. The results show that AE-hierarchical clustering distinguishes the tidal traffic characteristics and reflects the process of congestion propagation. Furthermore, two indicators are designed to make a quantitative comparison with spectral clustering: intra homogeneity increases by about 9% on average while inter heterogeneity about 9.5%. Compared with past methods, the time cost also decreases. The results may suggest ways to improve the management and control of urban road network in other metropolitan cities. The proposed method is expected to be extended to other problems related to large-scale networks.

Nonreciprocal superconductivity, also called as superconducting diode effect that spontaneously breaks time-reversal symmetry, is characterized by asymmetric critical currents under opposite applied current directions. This distinct state unveils a rich ore of intriguing physical properties, particularly in the realm of nanoscience application of superconductors. Towards the experimental realization of superconducting diode effect, the construction of two-dimensional heterostructures of magnets and $s$-wave superconductors is considered to be a promising pathway. In this study, we present our findings of superconducting diode effect manifested in the magnetic Mott insulator $\alpha$-RuCl$_3$. This phenomenon is induced by the proximity effect within a van der Waals heterostructure, consisting of thin $\alpha$-RuCl$_3$/NbSe$_2$ flakes. Through transport property measurements, we have confirmed a weak superconducting gap of 0.2 meV, which is significantly lower than the intrinsic gap of NbSe$_2$(1.2 meV). Upon the application of a weak magnetic field below 70 mT, we observed an asymmetry in the critical currents under positive and negative applied currents. This observation demonstrates a typical superconducting diode effect in the superconducting $\alpha$-RuCl$_3$. The superconducting diode effect and nonreciprocal resistance are observed exclusively when the magnetic field is aligned out-of-plane. This suggests that an Ising-type spin-orbit coupling in the superconducting $\alpha$-RuCl$_3$ may be responsible for the mechanism. Our findings furnish a platform for the exploration of superconducting diode effect via the artificial construction of heterostructures.

We study cluster structures on spinor helicity and momentum twistor varieties which describe the kinematic spaces of massless scattering (with or without dual conformal symmetry). Both are instances of partial flag varieties. We exhibit embeddings of the corresponding cluster algebras into cluster algebras of sufficiently large Grassmannians and show how the former is obtained from the latter by freezing certain cluster variables and specializing some cluster variables to one., Comment: 26 pages, 9 figures, comments are welcome

Federated Learning (FL) aims to train a shared model using data and computation power on distributed agents coordinated by a central server. Decentralized FL (DFL) utilizes local model exchange and aggregation between agents to reduce the communication and computation overheads on the central server. However, when agents are mobile, the communication opportunity between agents can be sporadic, largely hindering the convergence and accuracy of DFL. In this paper, we study delay-tolerant model spreading and aggregation enabled by model caching on mobile agents. Each agent stores not only its own model, but also models of agents encountered in the recent past. When two agents meet, they exchange their own models as well as the cached models. Local model aggregation works on all models in the cache. We theoretically analyze the convergence of DFL with cached models, explicitly taking into account the model staleness introduced by caching. We design and compare different model caching algorithms for different DFL and mobility scenarios. We conduct detailed case studies in a vehicular network to systematically investigate the interplay between agent mobility, cache staleness, and model convergence. In our experiments, cached DFL converges quickly, and significantly outperforms DFL without caching., Comment: 27 pages

The low elastic modulus of soft materials, combined with geometric nonlinearity and rate dependence, presents significant challenges in the characterization of their mechanical response. We introduce a novel method for measuring the mechanical properties of soft materials under large deformations via cylindrical cavity expansion. In this method, a cylindrical cavity is fabricated in the material and expanded by volume-controlled injection of an incompressible fluid with simultaneous measurement of the applied pressure at the cavity wall. The relationship between applied pressure and deformation at the cavity wall is then employed to characterize the nonlinear mechanical properties. We demonstrate the feasibility of the proposed method and validate it by measuring the mechanical properties of synthetic polydimethylsiloxane (PDMS) and comparing with reported values in the literature. Results indicate that the cylindrical cavitation method effectively captures the response of PDMS over a wide range of stiffness (shear modulus ranging from 5 kPa to 300 kPa) and exhibit high repeatability. The proposed method overcomes limitations in characterization of ultra-soft materials using traditional testing methods, such as challenges with fabrication and clamping in unaxial tension testing and friction and adhesion effects in compression and indentation testing, thus enabling accurate and precise characterization. It also offers improved accuracy and repeatability over other needle induced cavity expansion methods due to precise control over the initial cavity dimension and shape at the cost of increased invasiveness of testing.

Multimodal Large Language Models (MLLMs) are gaining increasing popularity in both academia and industry due to their remarkable performance in various applications such as visual question answering, visual perception, understanding, and reasoning. Over the past few years, significant efforts have been made to examine MLLMs from multiple perspectives. This paper presents a comprehensive review of 200 benchmarks and evaluations for MLLMs, focusing on (1)perception and understanding, (2)cognition and reasoning, (3)specific domains, (4)key capabilities, and (5)other modalities. Finally, we discuss the limitations of the current evaluation methods for MLLMs and explore promising future directions. Our key argument is that evaluation should be regarded as a crucial discipline to support the development of MLLMs better. For more details, please visit our GitHub repository: https://github.com/swordlidev/Evaluation-Multimodal-LLMs-Survey.

Relativistic spin-polarized electron beams are important for fundamental research and the industry, but their generation currently requires conventional accelerators or ultrastrong laser facilities, limiting their accessibility and broad applications. Here, we put forward a novel method for constructing a compact efficient "polarizer" that achieves direct ultrafast conversion of relativistic dense electron beams into polarized ones, based on the beam "self-polarization" mechanism via simple beam-target interactions. In this scheme, as the electron beam grazes through the polarizer (a double-layer solid target), it ionizes the target and excites an asymmetric plasma field due to the plasma backflows. This field then reacts on the beam itself, triggering spontaneous radiative polarization and reflection of the beam, and ultimately yielding a dense polarized electron beam. Moreover, the double-layer target setup induces a plasma bubble that focuses the polarized beam and reshapes its polarization distribution. Our method is robust with respect to the beam and target parameters, and opens a new avenue for relativistic beam polarization with compact accessible devices, which would facilitate their broad applications and the development of related experiments, such as in strong-field QED studies, and polarized electron-positron and electron-ion colliders.

In this paper, we present MooER, a LLM-based large-scale automatic speech recognition (ASR) / automatic speech translation (AST) model of Moore Threads. A 5000h pseudo labeled dataset containing open source and self collected speech data is used for training. We achieve performance comparable to other open source models trained with up to hundreds of thousands of hours of labeled speech data. Meanwhile, experiments conducted on Covost2 Zh2en testset suggest that our model outperforms other open source Speech LLMs. A BLEU score of 25.2 can be obtained. The main contributions of this paper are summarized as follows. First, this paper presents a training strategy for encoders and LLMs on speech related tasks (including ASR and AST) using a small size of pseudo labeled data without any extra manual annotation and selection. Second, we release our ASR and AST models and plan to open-source our training code and strategy in the near future. Moreover, a model trained on 8wh scale training data is planned to be released later on.

Visual Spatial Description (VSD) aims to generate texts that describe the spatial relationships between objects within images. Traditional visual spatial relationship classification (VSRC) methods typically output the spatial relationship between two objects in an image, often neglecting world knowledge and lacking general language capabilities. In this paper, we propose a Large Language-and-Vision Assistant for Visual Spatial Description, named LLaVA-VSD, which is designed for the classification, description, and open-ended description of visual spatial relationships. Specifically, the model first constructs a VSD instruction-following dataset using given figure-caption pairs for the three tasks. It then employs LoRA to fine-tune a Large Language and Vision Assistant for VSD, which has 13 billion parameters and supports high-resolution images. Finally, a large language model (Qwen-2) is used to refine the generated sentences, enhancing their diversity and accuracy. LLaVA-VSD demonstrates excellent multimodal conversational capabilities and can follow open-ended instructions to assist with inquiries about object relationships in images., Comment: We have discovered a significant error in the paper that affects the main conclusions. To ensure the accuracy of our research, we have decided to withdraw this paper and will resubmit it after making the necessary corrections

Investigating vortex electron interactions with electromagnetic fields is essential for advancing particle acceleration techniques, scattering theory in background fields, and obtaining novel electron beams for material diagnostics. A systematic investigation into the dynamics of vortex electrons in external laser fields and the exploration of laser-induced vortex modes remains lacking. In this work, we study the propagation of vortex electrons in linearly polarized (LP) and circularly polarized (CP) laser pulses, both separately and in their combined form in two-mode laser pulses. The theoretical formalism is developed by utilizing Volkov-Bessel wave functions, and the four-current density is obtained as a crucial observable quantity. Numerical results illustrate the dynamics of vortex electrons in external lasers, showing that the beam center of the vortex electron follows the classical motion of a point charge electron, while maintaining the probability distribution structure for both vortex eigenstates and superposition modes. The combined effect of LP and CP laser pulses in the two-mode laser field allows for the versatile control of vortex electrons, which is absent with LP or CP lasers alone, at femtosecond and sub-nanometer scales. Our findings demonstrate the versatile control over vortex electrons via laser pulses, with our formalism providing a reference for vortex scattering in laser backgrounds and inspiring the laser-controlled achievement of novel vortex modes as targeted diagnostic probes for specialized materials., Comment: 4 figures

Robotic datasets are important for scientific benchmarking and developing algorithms, for example for Simultaneous Localization and Mapping (SLAM). Modern robotic datasets feature video data of high resolution and high framerates. Storing and sharing those datasets becomes thus very costly, especially if more than one camera is used for the datasets. It is thus essential to store this video data in a compressed format. This paper investigates the use of modern video encoders for robotic datasets. We provide a software that can replay mp4 videos within ROS 1 and ROS 2 frameworks, supporting the synchronized playback in simulated time. Furthermore, the paper evaluates different encoders and their settings to find optimal configurations in terms of resulting size, quality and encoding time. Through this work we show that it is possible to store and share even highest quality video datasets within reasonable storage constraints.

In the e-commerce realm, compelling advertising images are pivotal for attracting customer attention. While generative models automate image generation, they often produce substandard images that may mislead customers and require significant labor costs to inspect. This paper delves into increasing the rate of available generated images. We first introduce a multi-modal Reliable Feedback Network (RFNet) to automatically inspect the generated images. Combining the RFNet into a recurrent process, Recurrent Generation, results in a higher number of available advertising images. To further enhance production efficiency, we fine-tune diffusion models with an innovative Consistent Condition regularization utilizing the feedback from RFNet (RFFT). This results in a remarkable increase in the available rate of generated images, reducing the number of attempts in Recurrent Generation, and providing a highly efficient production process without sacrificing visual appeal. We also construct a Reliable Feedback 1 Million (RF1M) dataset which comprises over one million generated advertising images annotated by human, which helps to train RFNet to accurately assess the availability of generated images and faithfully reflect the human feedback. Generally speaking, our approach offers a reliable solution for advertising image generation., Comment: ECCV2024

Motivated by very recent experimental observations of the 1/9 magnetization plateaus in YCu$_3$(OH)$_{6+x}$Br$_{3-x}$ and YCu$_3$(OD)$_{6+x}$Br$_{3-x}$, our study delves into the magnetic field-induced phase transitions in the nearest-neighbor antiferromagnetic Heisenberg model on the kagome lattice using the variational Monte Carlo technique. We uncover a phase transition from a zero-field Dirac spin liquid to a field-induced magnetically disordered phase that exhibits the 1/9 magnetization plateau. Through a comprehensive analysis encompassing the magnetization distribution, spin correlations, chiral order parameter, topological entanglement entropy, ground-state degeneracy, Chern number and excitation spectrum, we pinpoint the phase associated with this magnetization plateau as a chiral $\mathbb{Z}_3$ topological quantum spin liquid and elucidate its diverse physical properties., Comment: Accepted by Phys. Rev. Lett

We investigate the spin-$\frac{1}{2}$ antiferromagnetic Heisenberg model with a Dzyaloshinskii-Moriya interaction on kagome lattice, making use of the variational Monte Carlo technique. An exotic quantum spin state is found to arise from a melting of the $\boldsymbol{Q} = 0$ long-range magnetic order by a topological transition, when a small anisotropic third nearest-neighbor antiferromagnetic Heisenberg interaction is turned on. This novel state is a gapped quantum spin liquid, characterized by a topological order with ground-state degeneracy $n_g = 4$ and topological entanglement entropy $\gamma = \ln 2$, suggesting it is an Abelian topological phase. Furthermore, the Chern numbers of the spin-up (-down) spinon occupied bands of this state are $C_{\uparrow \downarrow} = \pm 1$, respectively. From this perspective, this state is also a time-reversal symmetric (total Chern number $C_{total} = 0$) topological insulator with spinons as the chiral edge states, which carry opposite spin and move in the opposite direction. It is analogous to the quantum spin Hall state but the spin current is carried by deconfined spinons in a quantum spin liquid, so is dubbed as the spinon quantum spin Hall state.

IGR J17091-3624, in addition to GRS 1915+105, is the only black hole X-ray binary that displays ``heartbeat"-like variability characterized by structured flares with high amplitudes. In this study, we conduct a detailed phase-resolved analysis of the recently identified ``heartbeat"-like Class X variability in IGR J17091-3624 during its 2022 outburst, utilizing data from NICER and NuSTAR observations. A shortage in the high-energy (>20 keV) X-ray flux is detected at peak phases of the soft X-ray flare at a ~15 sigma confidence level from the phase-folded light curves. Furthermore, our phase-resolved spectral analysis reveals variations in the spectral shape, particularly showing significant synchronous variations in the disk temperature and flux with the count rate. These findings imply that the flare is primarily driven by instabilities within the accretion disk, consistent with previous studies on the well-known Class rho variability in GRS 1915+105. However, we also observe a positive correlation between the disk temperature and flux over the flare cycle, which differs from a loop relation between the two parameters found in the Class rho variability. This could suggest differences in underlying physical processes between the two variability classes. Variations in the Componization component during flares are also observed: the electron temperature and covering fraction show anti-correlations with the disk flux, revealing potential interactions between the accretion disk and the corona during these flares., Comment: Accepted for publication in The Astrophysical Journal

Low-frequency quasi-periodic oscillations (LFQPOs) are commonly observed in X-ray light curves of black hole X-ray binaries (BHXRBs); however, their origin remains a topic of debate. In order to thoroughly investigate variations in spectral properties on the QPO timescale, we utilized the Hilbert-Huang transform technique to conduct phase-resolved spectroscopy across a broad energy band for LFQPOs in the newly discovered BHXRB Swift J1727.8-1613. This is achieved through quasi-simultaneous observations from Neutron star Interior Composition ExploreR (NICER), Nuclear Spectroscopic Telescope ARray (NuSTAR), and Hard X-ray Modulation Telescope (Insight-HXMT). Our analysis reveals that both the non-thermal and disk-blackbody components exhibit variations on the QPO timescale, with the former dominating the QPO variability. For the spectral parameters, we observe modulation of the disk temperature, spectral indices, and reflection fraction with the QPO phase with high statistical significance (>5\sigma). Notably, the variation in the disk temperature is found to precede the variations in the non-thermal and disk fluxes by ~0.4-0.5 QPO cycles. We suggest that these findings offer further evidence that the type-C QPO variability is a result of geometric effects of the accretion flow., Comment: Accepted for pulication in The Astrophysical Journal

We investigate the nonlinear magneto-optical response in non-centrosymmetric magnetic Weyl semimetals featuring a quadratic tilt, focusing particularly on the influence of the van Hove singularity (VHS). In the absence of a magnetic field, the second-order nonlinear Drude conductivity components exhibit inflection or dip behavior across the VHS. In contrast, the second-order nonlinear anomalous Hall conductivity, primarily governed by the Berry curvature dipole, manifests a subtle plateau-like structure. As the tilt strength increases, the VHS energy escalates, thereby amplifying the VHS-induced characteristics within these second-order conductivity components. However, in the presence of a magnetic field, we show that the resultant magnetic moment suppresses nonlinear electron transport while enhancing nonlinear hole transport. %both suppresses and notably enhances nonlinear magnetic-optical transport in the electron and hole regions, respectively. This effect serves to mitigate the impact of the VHS, resulting specifically in an asymmetric peak or a kinked-like structure in the magnetic field-induced contribution to the second-order nonlinear conductivity near the Weyl nodes. These findings provide new insights into the intricate interplay among the VHS, Berry curvature, and magnetic moment in nonlinear magneto-optical transport through non-centrosymmetric magnetic Weyl semimetals., Comment: 10 pages, 5 figures. Accepted for publication in Phys. Rev. B

In this paper, a third-order time adaptive algorithm with less computation, low complexity is provided for shale reservoir model based on coupled fluid flow with porous media flow. The algorithm combines the three-step linear time filters method for simple post-processing and the second-order backward differential formula (BDF2), is third-order accurate and provides, at no extra computational complexity. At the same time, the time filter method can also be used to damp non-physical oscillations inherent in the BDF2 method, ensuring stability. We proves the variable time stepsize second-order backward differential formula plus time filter (BDF2-TF) algorithm's stability and the convergence properties of the fluid velocity u and hydraulic head $\phi$ in the $L^2$ norm with an order of $O(k_{n+1}^3 + h^3)$. In the experiments, the adaptive algorithm automatically adjusts the time step in response to the varying characteristics of different models, ensuring that errors are maintained within acceptable limits. This algorithm addresses the issue that high-order algorithms may select inappropriate time steps, resulting in instability or reduced precision of the numerical solution, thereby enhancing calculation accuracy and efficiency. We perform three-dimensional numerical experiments to verify the BDF2-TF algorithm's effectiveness, stability, and third-order convergence. Simultaneously, a simplified model is employed to simulate the process of shale oil extraction from reservoirs, further demonstrating the algorithm's practical applicability.

Semantic communication (SemCom) has been a transformative paradigm, emphasizing the precise exchange of meaningful information over traditional bit-level transmissions. However, existing SemCom research, primarily centered on simplified scenarios like single-pair transmissions with direct wireless links, faces significant challenges when applied to real-world radio access networks (RANs). This article introduces a Semantic-aware Radio Access Network (S-RAN), offering a holistic systematic view of SemCom beyond single-pair transmissions. We begin by outlining the S-RAN architecture, introducing new physical components and logical functions along with key design challenges. We then present transceiver design for end-to-end transmission to overcome conventional SemCom transceiver limitations, including static channel conditions, oversimplified background knowledge models, and hardware constraints. Later, we delve into the discussion on radio resource management for multiple users, covering semantic channel modeling, performance metrics, resource management algorithms, and a case study, to elaborate distinctions from resource management for legacy RANs. Finally, we highlight open research challenges and potential solutions. The objective of this article is to serve as a basis for advancing SemCom research into practical wireless systems.

Large Language Models (LLMs) have demonstrated exceptional proficiency in mathematical reasoning tasks due to their extensive parameter counts and training on vast datasets. Despite these capabilities, deploying LLMs is hindered by their computational demands. Distilling LLM mathematical reasoning into Smaller Language Models (SLMs) has emerged as a solution to this challenge, although these smaller models often suffer from errors in calculation and semantic understanding. Prior work has proposed Program-of-Thought Distillation (PoTD) to avoid calculation error. To further address semantic understanding errors, we propose Key-Point-Driven Mathematical Reasoning Distillation (KPDD). KPDD enhances the reasoning performance of SLMs by breaking down the problem-solving process into three stages: Core Question Extraction, Problem-Solving Information Extraction, and Step-by-Step Solution. This method is further divided into KPDD-CoT, which generates Chain-of-Thought rationales, and KPDD-PoT, which creates Program-of-Thought rationales. The experiment results show that KPDD-CoT significantly improves reasoning abilities, while KPDD-PoT achieves state-of-the-art performance in mathematical reasoning tasks. Our approach effectively mitigates misunderstanding errors, advancing the deployment of efficient and capable SLMs., Comment: Major Updates:1.fix faults in the error analysis, 2. improve our method, 3. use ChatGPT as teacher LLMs to ensure fairness in performance comparisons

We obtain a trichotomy for the orbits of a hypercyclic operator $T$ on a separable Banach space $X$: (1) every vector is mean asymptotic to zero; (2) generic vectors are absolutely mean irregular; (3) every hypercyclic vector is mean divergent to infinity. Examples of weighted backward shifts on $\ell^p$ show that all three cases can happen., Comment: 12 papers,to appear in Proc. Amer. Math. Soc

Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-bench, GSM8K, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance. Code is available at https://github.com/Outsider565/LoRA-GA.

Polymer-related materials exhibit rich glassy behaviors at different length scales due to their various molecular structures and topological constraints. Recent studies have identified transient interpenetration of the long-chain rings contributing to dynamic arrest on the center-of-mass level. Interpenetration of rings is proposed as an approach to facilitate glass formation in polymer melts. In this work, inspired by recent advances in the synthesis of mechanically interlocked polymers, we investigate glass transition on the nanometer-scale segments influenced by permanent interpenetration of rings using molecular dynamics simulations. We find that decreasing chain length in the mechanically interlocked system is equivalent to inducing an effective chain stiffness on the sub-rings. The induced stiffness provides a unified explanation for these unique structural features and transient dynamic arrest in the system of interlocked rings with rather short chains. Further, a crossover is observed in the scaling relation between localization and glassy depth upon cooling. Our work reveals a dynamic transition from weak to strong caging at the crossover temperature. According to the localization model, we demonstrate that the chain stiffness increases the critical temperature and oscillation distance, therefore leads to more fragile dynamics and deeper glassy state. These findings are consistent with the predictions of molecular simulations and theories for polymers with real local stiffness. Our work deepens the understanding of the role of induced stiffness on glass transition, and opens up a new direction to design rich glass materials by manipulating stiffness through mechanical bonds., Comment: 35 pages, 8 figures

6-DoF grasp detection of small-scale grasps is crucial for robots to perform specific tasks. This paper focuses on enhancing the recognition capability of small-scale grasping, aiming to improve the overall accuracy of grasping prediction results and the generalization ability of the network. We propose an enhanced receptive field method that includes a multi-radii cylinder grouping module and a passive attention module. This method enhances the receptive field area within the graspable space and strengthens the learning of graspable features. Additionally, we design a graspable balance sampling module based on a segmentation network, which enables the network to focus on features of small objects, thereby improving the recognition capability of small-scale grasping. Our network achieves state-of-the-art performance on the GraspNet-1Billion dataset, with an overall improvement of approximately 10% in average precision@k (AP). Furthermore, we deployed our grasp detection model in pybullet grasping platform, which validates the effectiveness of our method.

This study evaluates whether large language models (LLMs) exhibit biases towards medical professionals. Fictitious candidate resumes were created to control for identity factors while maintaining consistent qualifications. Three LLMs (GPT-4, Claude-3-haiku, and Mistral-Large) were tested using a standardized prompt to evaluate resumes for specific residency programs. Explicit bias was tested by changing gender and race information, while implicit bias was tested by changing names while hiding race and gender. Physician data from the Association of American Medical Colleges was used to compare with real-world demographics. 900,000 resumes were evaluated. All LLMs exhibited significant gender and racial biases across medical specialties. Gender preferences varied, favoring male candidates in surgery and orthopedics, while preferring females in dermatology, family medicine, obstetrics and gynecology, pediatrics, and psychiatry. Claude-3 and Mistral-Large generally favored Asian candidates, while GPT-4 preferred Black and Hispanic candidates in several specialties. Tests revealed strong preferences towards Hispanic females and Asian males in various specialties. Compared to real-world data, LLMs consistently chose higher proportions of female and underrepresented racial candidates than their actual representation in the medical workforce. GPT-4, Claude-3, and Mistral-Large showed significant gender and racial biases when evaluating medical professionals for residency selection. These findings highlight the potential for LLMs to perpetuate biases and compromise healthcare workforce diversity if used without proper bias mitigation strategies., Comment: 36 pages, 6 figures

Let $(\mathbf{U}, \mathbf{U}^\imath)$ be a split affine quantum symmetric pair of type $\mathsf{B}_n^{(1)}, \mathsf{C}_n^{(1)}$ or $\mathsf{D}_n^{(1)}$. We prove factorization and coproduct formulae for the Drinfeld-Cartan operators $\Theta_i(z)$ in the Lu-Wang Drinfeld-type presentation, generalizing the type $\mathsf{A}_n^{(1)}$ result from [Prz23]. As an application, we show that a boundary analogue of the $q$-character map, defined via the spectra of these operators, is compatible with the usual $q$-character map. As an auxiliary result, we also produce explicit reduced expressions for the fundamental weights in the extended affine Weyl groups of classical types., Comment: 38 pages

Let $G$ be a countable infinite discrete group. We show that a subset $F$ of $G$ contains a return time set of some piecewise syndetic recurrent point $x$ in a compact Hausdorff space $X$ with a $G$-action if and only if $F$ is a quasi-central set. As an application, we show that if a nonempty closed subsemigroup $S$ of the Stone-\v{C}ech compactification $\beta G$ contains the smallest ideal $K(\beta G)$ of $\beta G$ then $S$-product recurrent is equivalent to distality, which partially answers a question of Auslander and Furstenberg (Trans. Amer. Math. Soc. 343, 1994, 221--232)., Comment: 22 pages

This report presents our team's 'PCIE_EgoHandPose' solution for the EgoExo4D Hand Pose Challenge at CVPR2024. The main goal of the challenge is to accurately estimate hand poses, which involve 21 3D joints, using an RGB egocentric video image provided for the task. This task is particularly challenging due to the subtle movements and occlusions. To handle the complexity of the task, we propose the Hand Pose Vision Transformer (HP-ViT). The HP-ViT comprises a ViT backbone and transformer head to estimate joint positions in 3D, utilizing MPJPE and RLE loss function. Our approach achieved the 1st position in the Hand Pose challenge with 25.51 MPJPE and 8.49 PA-MPJPE. Code is available at https://github.com/KanokphanL/PCIE_EgoHandPose

Twisted transition metal dichalcogenides are a new platform for realizing strongly correlated physics with high tunability. Recent transport experiments [A. Ghiotto et al. Nature 597, 345 (2021)] have reported the bandwidth-driven evolution of a Mott insulator to a strange metal behavior via the tuning of a displacement field in twisted $\mathrm{WSe_2}$ fixed at half filling. However, the nature of the correlated states and the related Mott physics involved in the whole process remain to be determined. Here, we unveil theoretically the evolution of the ground state of the half-filled $\mathrm{moir\acute{e}}$ Hubbard model as applied to $\mathrm{tWSe_2}$, transiting from a pseudogap state with Fermi arcs to a $120^\circ$ Ne$\acute{\mathrm{e}}$l ordered Mott insulator, then to another pseudogap state with Fermi pockets, and eventually to a Fermi liquid via a Lifshitz transition. The pseudogap phases are definitely identified by the vanishing of quasiparticle weights over parts of the Fermi surface, with the remaining parts forming disconnected Fermi arcs or pockets with well-defined quasiparticles. We demonstrate that the Fermi arc/pocket results from the electronic band structure reconstruction driven by electron correlations, marked by the coexistence of the poles and zeros of the single-particle Green's function. This work reveals the fundamental aspects of the Mottness in $\mathrm{moir\acute{e}}$ system and will stimulate the direct probes of the underling physice beyond transports via the angle-resolved photoemission spectroscopy and scanning tunneling microscopy., Comment: 13 pages, 10 figures

Multimodal Large Language Models (MLLMs) have exhibited impressive capabilities in visual understanding and reasoning, providing sightly reasonable answers, such as image descriptions. This has spurred extensive research on the evaluation of MLLMs. Most evaluation benchmarks assume that incorrect answers indicate a lack of understanding of the visual content. However, our findings reveal that, in many cases, MLLMs answer questions incorrectly despite correctly understanding the visual content. This suggests that incorrect answers do not necessarily imply a lack of comprehension but may instead result from lacking robustness to leading questions. To comprehensively measure MLLMs' understanding capability and robustness to leading questions, we introduce a MultiModal Robustness benchmark (MMR). MMR contains paired positive and negative questions across 12 categories, meticulously annotated by humans. We evaluate 18 leading MLLMs on the MMB benchmark, revealing that MLLMs suffer from fragility to leading questions despite understanding the visual content. To enhance MLLMs' understanding capability and robustness, we further present a training set with paired positive and negative visual question-answer samples. Experiments verify that MLLMs' robustness can be significantly enhanced by tuning on this new training set. The benchmark, training set, and code can be found at https://github.com/BAAI-DCAI/Multimodal-Robustness-Benchmark.

As a robust and large-scale multilingual speech recognition model, Whisper has demonstrated impressive results in many low-resource and out-of-distribution scenarios. However, its encoder-decoder structure hinders its application to streaming speech recognition. In this paper, we introduce Simul-Whisper, which uses the time alignment embedded in Whisper's cross-attention to guide auto-regressive decoding and achieve chunk-based streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we observe the negative effect of the truncated words at the chunk boundaries on the decoding results and propose an integrate-and-fire-based truncation detection model to address this issue. Experiments on multiple languages and Whisper architectures show that Simul-Whisper achieves an average absolute word error rate degradation of only 1.46% at a chunk size of 1 second, which significantly outperforms the current state-of-the-art baseline., Comment: Accepted by INTERSPEECH 2024