Search

Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable perceptual and reasoning abilities, typically comprising a Vision Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as the critical bridge between the visual and language components. However, training adapters with image-level supervision often results in significant misalignment, undermining the LLMs' capabilities and limiting the potential of Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment (SEA), a token-level alignment method that leverages vision-language pre-trained models, such as CLIP, to align visual tokens with the LLM's embedding space through contrastive learning. This approach ensures a more coherent integration of visual and language representations, enhancing the performance and interpretability of multimodal LLMs while preserving their inherent capabilities. Extensive experiments show that SEA effectively improves MLLMs, particularly for smaller models, without adding extra data or inference computation. SEA also lays the groundwork for developing more general and adaptable solutions to enhance multimodal systems.

In this paper, we consider a new nonlocal approximation to the linear Stokes system with periodic boundary conditions in two and three dimensional spaces . A relaxation term is added to the equation of nonlocal divergence free equation, which is reminiscent to the relaxation of local Stokes equation with small artificial compressibility. Our analysis shows that the well-posedness of the nonlocal system can be established under some mild assumptions on the kernel of nonlocal interactions. Furthermore, the new nonlocal system converges to the conventional, local Stokes system in second order as the horizon parameter of the nonlocal interaction goes to zero. The study provides more theoretical understanding to some numerical methods, such as smoothed particle hydrodynamics, for simulating incompressible viscous flows.

In this paper, we propose a class of nonlocal models to approximate the Poisson model on manifolds with homogeneous Neumann boundary condition, where the manifolds are assumed to be embedded in high dimensional Euclid spaces. In comparison to the existing nonlocal approximation of Poisson models with Neumann boundary, we optimize the truncation error of model by adding an augmented term along the $2\delta$ layer of boundary, with $2\delta$ be the nonlocal interaction horizon. Such term is formulated by the integration of the second order normal derivative of solution through the boundary, while the second order normal derivative is expressed as the difference between the interior Laplacian and the boundary Laplacian. The concentration of our paper is on the construction of nonlocal model, the well-posedness of model, and its second-order convergence rate to its local counterpart. The localization rate of our nonlocal model is currently optimal among all related works even for the case of high dimensional Euclid spaces., Comment: arXiv admin note: text overlap with arXiv:2203.02120

Bresch--Desjardins--Gisclon--Sart had derived that the capillarity slows down the growth rate of Rayleigh--Taylor (RT) instability in an inhomogeneous incompressible fluid endowed with internal capillarity based on a linearized incompressible Navier--Stokes--Korteweg (NSK) equations in 2008. Later Li--Zhang further obtained another result that the capillarity inhibits RT instability also based on the linearized equations in (SIAM J. Math. Anal. 3287--3315, 2023), if the capillarity coefficient is bigger than some threshold. In this paper, we further rigorously prove such phenomenon of capillarity inhibiting the RT instability in the \emph{nonlinear} incompressible NSK equations in a horizontally periodic slab domain with Navier (slip) boundary conditions. The key idea in the proof is to capture the dissipative estimates of the tangential derivatives of density. Such dissipative estimates result in the decay-in-time of both the velocity and the perturbation density which is very useful to overcome the difficulties arising from the nonlinear terms., Comment: arXiv admin note: text overlap with arXiv:2302.01013

Telescope arrays are receiving increasing attention due to their promise of higher resource utilization, greater sky survey area, and higher frequency of full space-time monitoring than single telescopes. Compared with the ordinary coordinated operation of several telescopes, the new astronomical observation mode has an order of magnitude difference in the number of telescopes. It requires efficient coordinated observation by large-domain telescopes distributed at different sites. Coherent modeling of various abstract environmental constraints is essential for responding to multiple complex science goals. Also, due to competing science priorities and field visibility, how the telescope arrays are scheduled for observations can significantly affect observation efficiency. This paper proposes a multilevel scheduling model oriented toward the problem of telescope-array scheduling for time-domain surveys. A flexible framework is developed with basic functionality encapsulated in software components implemented on hierarchical architectures. An optimization metric is proposed to self-consistently weight contributions from time-varying observation conditions to maintain uniform coverage and efficient time utilization from a global perspective. The performance of the scheduler is evaluated through simulated instances. The experimental results show that our scheduling framework performs correctly and provides acceptable solutions considering the percentage of time allocation efficiency and sky coverage uniformity in a feasible amount of time. Using a generic version of the telescope-array scheduling framework, we also demonstrate its scalability and its potential to be applied to other astronomical applications., Comment: Accepted for publication in the Astronomical Journal

Cross-matching operation, which is to find corresponding data for the same celestial object or region from multiple catalogues,is indispensable to astronomical data analysis and research. Due to the large amount of astronomical catalogues generated by the ongoing and next-generation large-scale sky surveys, the time complexity of the cross-matching is increasing dramatically. Heterogeneous computing environments provide a theoretical possibility to accelerate the cross-matching, but the performance advantages of heterogeneous computing resources have not been fully utilized. To meet the challenge of cross-matching for substantial increasing amount of astronomical observation data, this paper proposes Heterogeneous-computing-enabled Large Catalogue Cross-matcher (HLC2), a high-performance cross-matching framework based on spherical position deviation on CPU-GPU heterogeneous computing platforms. It supports scalable and flexible cross-matching and can be directly applied to the fusion of large astronomical cataloguesfrom survey missions and astronomical data centres. A performance estimation model is proposed to locate the performance bottlenecks and guide the optimizations. A two-level partitioning strategy is designed to generate an optimized data placement according to the positions of celestial objects to increase throughput. To make HLC2 a more adaptive solution, the architecture-aware task splitting, thread parallelization, and concurrent scheduling strategies are designed and integrated. Moreover, a novel quad-direction strategy is proposed for the boundary problem to effectively balance performance and completeness. We have experimentally evaluated HLC2 using public released catalogue data. Experiments demonstrate that HLC2 scales well on different sizes of catalogues and the cross-matching speed is significantly improved compared to the state-of-the-art cross-matchers., Comment: Accepted for publication in Monthly Notices of the Royal Astronomical Society

Spin properties of organic molecules have attracted great interest for their potential applications in spintronic devices and quantum computing. Fe-tetraphenyl porphyrin (FeTPP) is of particular interest for its robust magnetic properties on metallic substrates. FeTPP is prepared in vacuum via on-surface synthesis. Molecular structure and spin-related transport properties are characterized by low-temperature scanning tunneling microscope and spectroscopy at 0.5 K. Density functional theory calculations are performed to understand molecular adsorption and spin distribution on Au(111). The molecular structure of FeTPP is distorted upon adsorption on the substrate. Spin excitations of FeTPP are observed on the Fe atom and high pyrrole groups in differential conductance spectra. The calculated spin density distribution indicates that the electron spin of FeTPP is mainly distributed on the Fe atom. The atomic transmission calculation indicates that electrons transport to substrate is mediated through Fe atom, when the tip is above the high pyrrole group.

Investigation of intermolecular electron spin interaction is of fundamental importance in both science and technology.Here, radical pairs of all-trans retinoic acid molecules on Au(111) are created using an ultra-low temperature scanning tunneling microscope. Antiferromagnetic coupling between two radicals is identified by magnetic-field dependent spectroscopy.The measured exchange energies are from 0.1 to 1.0 meV. The biradical spin coupling is mediated through O-H$\cdots$O hydrogen bonds, as elucidated from analysis combining density functional theory calculation and a modern version of valence bond theory., Comment: 10 pages with supplemental materials, PRL accepted

In this work, we introduced a class of nonlocal models to accurately approximate the Poisson model on manifolds that are embedded in high dimensional Euclid spaces with Dirichlet boundary. In comparison to the existing nonlocal Poisson models, instead of utilizing volumetric boundary constraint to reduce the truncation error to its local counterpart, we rely on the Poisson equation itself along the boundary to explicitly express the second order normal derivative by some geometry-based terms, so that to create a new model with $\mathcal{O}(\delta)$ truncation error along the $2\delta-$boundary layer and $\mathcal{O}(\delta^2)$ at interior, with $\delta$ be the nonlocal interaction horizon. Our concentration is on the construction and the truncation error analysis of such nonlocal model. The control on the truncation error is currently optimal among all nonlocal models, and is sufficient to attain second order localization rate that will be derived in our subsequent work., Comment: arXiv admin note: text overlap with arXiv:2101.01016

Recently, we constructed a class of nonlocal Poisson model on manifold under Dirichlet boundary with global $\mathcal{O}(\delta^2)$ truncation error to its local counterpart, where $\delta$ denotes the nonlocal horizon parameter. In this paper, the well-posedness of such manifold model is studied. We utilize Poincare inequality to control the lower order terms along the $2\delta$-boundary layer in the weak formulation of model. The second order localization rate of model is attained by combining the well-posedness argument and the truncation error analysis. Such rate is currently optimal among all nonlocal models. Besides, we implement the point integral method(PIM) to our nonlocal model through 2 specific numerical examples to illustrate the quadratic rate of convergence on the other side.

Single image deraining is an urgent task because the degraded rainy image makes many computer vision systems fail to work, such as video surveillance and autonomous driving. So, deraining becomes important and an effective deraining algorithm is needed. In this paper, we propose a novel network based on physical model guided learning for single image deraining, which consists of three sub-networks: rain streaks network, rain-free network, and guide-learning network. The concatenation of rain streaks and rain-free image that are estimated by rain streaks network, rain-free network, respectively, is input to the guide-learning network to guide further learning and the direct sum of the two estimated images is constrained with the input rainy image based on the physical model of rainy image. Moreover, we further develop the Multi-Scale Residual Block (MSRB) to better utilize multi-scale information and it is proved to boost the deraining performance. Quantitative and qualitative experimental results demonstrate that the proposed method outperforms the state-of-the-art deraining methods. The source code will be available at \url{https://supercong94.wixsite.com/supercong94}., Comment: IEEE ICME2020