Search

A reliable knowledge structure is a prerequisite for building effective adaptive learning systems and intelligent tutoring systems. Pursuing an explainable and trustworthy knowledge structure, we propose a method for constructing causal knowledge networks. This approach leverages Bayesian networks as a foundation and incorporates causal relationship analysis to derive a causal network. Additionally, we introduce a dependable knowledge-learning path recommendation technique built upon this framework, improving teaching and learning quality while maintaining transparency in the decision-making process., Comment: 8 pages, 3 figures, Educational Data Mining 2024, Human-Centric eXplainable AI in Education

This work is concerned with the exponential turnpike property for optimal control problems of particle systems and their mean-field limit. Under the assumption of the strict dissipativity of the cost function, exponential estimates for both optimal states and optimal control are proven. Moreover, we show that all the results for particle systems can be preserved under the limit in the case of infinitely many particles., Comment: 24 pages

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs), and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. The code and models will be made open-source.

Most multi-modal tasks can be formulated into problems of either generation or embedding. Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding. To explore the minimalism of multi-modal paradigms, we attempt to achieve only one model per modality in this work. We propose a Multi-Modal Generative Embedding Model (MM-GEM), whereby the generative and embedding objectives are encapsulated in one Large Language Model. We also propose a PoolAggregator to boost efficiency and enable the ability of fine-grained embedding and generation. A surprising finding is that these two objectives do not significantly conflict with each other. For example, MM-GEM instantiated from ViT-Large and TinyLlama shows competitive performance on benchmarks for multimodal embedding models such as cross-modal retrieval and zero-shot classification, while has good ability of image captioning. Additionally, MM-GEM can seamlessly execute region-level image caption generation and retrieval tasks. Besides, the advanced text model in MM-GEM brings over 5% improvement in Recall@1 for long text and image retrieval.

Humans constantly interact with their surrounding environments. Current human-centric generative models mainly focus on synthesizing humans plausibly interacting with static scenes and objects, while the dynamic human action-reaction synthesis for ubiquitous causal human-human interactions is less explored. Human-human interactions can be regarded as asymmetric with actors and reactors in atomic interaction periods. In this paper, we comprehensively analyze the asymmetric, dynamic, synchronous, and detailed nature of human-human interactions and propose the first multi-setting human action-reaction synthesis benchmark to generate human reactions conditioned on given human actions. To begin with, we propose to annotate the actor-reactor order of the interaction sequences for the NTU120, InterHuman, and Chi3D datasets. Based on them, a diffusion-based generative model with a Transformer decoder architecture called ReGenNet together with an explicit distance-based interaction loss is proposed to predict human reactions in an online manner, where the future states of actors are unavailable to reactors. Quantitative and qualitative results show that our method can generate instant and plausible human reactions compared to the baselines, and can generalize to unseen actor motions and viewpoint changes., Comment: Accepted by CVPR 2024, Project Page: https://liangxuy.github.io/ReGenNet/

The analysis of the ubiquitous human-human interactions is pivotal for understanding humans as social beings. Existing human-human interaction datasets typically suffer from inaccurate body motions, lack of hand gestures and fine-grained textual descriptions. To better perceive and generate human-human interactions, we propose Inter-X, a currently largest human-human interaction dataset with accurate body movements and diverse interaction patterns, together with detailed hand gestures. The dataset includes ~11K interaction sequences and more than 8.1M frames. We also equip Inter-X with versatile annotations of more than 34K fine-grained human part-level textual descriptions, semantic interaction categories, interaction order, and the relationship and personality of the subjects. Based on the elaborate annotations, we propose a unified benchmark composed of 4 categories of downstream tasks from both the perceptual and generative directions. Extensive experiments and comprehensive analysis show that Inter-X serves as a testbed for promoting the development of versatile human-human interaction analysis. Our dataset and benchmark will be publicly available for research purposes., Comment: Project page: https://liangxuy.github.io/inter-x/

Quantum Spin Hall (QSH) insulators possess distinct helical in-gap states, enabling their edge states to act as one-dimensional conducting channels when backscattering is prohibited by time-reversal symmetry. However, it remains challenging to achieve high-performance combinations of nontrivial topological QSH states with superconductivity for applications and requires understanding of the complicated underlying mechanisms. Here, our experimental observations for a novel superconducting phase in the pressurized QSH insulator Ta2Pd3Te5 is reported, and the high-pressure phase maintains its original ambient pressure lattice symmetry up to 45 GPa. Our in-situ high-pressure synchrotron X-ray diffraction, electrical transport, infrared reflectance, and Raman spectroscopy measurements, in combination with rigorous theoretical calculations, provide compelling evidence for the association between the superconducting behavior and the abnormal densified phase. The isostructural transition was found to modify the topology of the Fermi surface directly, accompanied by a fivefold amplification of the density of states at 20 GPa compared to ambient pressure, which synergistically fosters the emergence of robust superconductivity. A profound comprehension of the fascinating properties exhibited by the compressed Ta2Pd3Te5 phase is achieved, highlighting the extraordinary potential of van der Waals (vdW) QSH insulators for exploring and investigating high-performance electronic advanced devices under extreme conditions., Comment: 16pages,4figures

Image captioning requires numerous annotated image-text pairs, resulting in substantial annotation costs. Recently, large models (e.g. diffusion models and large language models) have excelled in producing high-quality images and text. This potential can be harnessed to create synthetic image-text pairs for training captioning models. Synthetic data can improve cost and time efficiency in data collection, allow for customization to specific domains, bootstrap generalization capability for zero-shot performance, and circumvent privacy concerns associated with real-world data. However, existing methods struggle to attain satisfactory performance solely through synthetic data. We identify the issue as generated images from simple descriptions mostly capture a solitary perspective with limited context, failing to align with the intricate scenes prevalent in real-world imagery. To tackle this, we present an innovative pipeline that introduces multi-context data generation. Beginning with an initial text corpus, our approach employs a large language model to extract multiple sentences portraying the same scene from diverse viewpoints. These sentences are then condensed into a single sentence with multiple contexts. Subsequently, we generate intricate images using the condensed captions through diffusion models. Our model is exclusively trained on synthetic image-text pairs crafted through this process. The effectiveness of our pipeline is validated through experimental results in both the in-domain and cross-domain settings, where it achieves state-of-the-art performance on well-known datasets such as MSCOCO, Flickr30k, and NoCaps., Comment: Accepted by AAAI 2024

This work concerns with boundary conditions (BCs) of the linearized moment system for rarefied gases. As the Knudsen number is sufficiently small, we analyze the boundary-layer behaviors of the moment system by resorting to a three-scale asymptotic expansion. The asymptotic analysis casts the flows into the outer solution, the viscous layer and the Knudsen layer. Starting from the BCs of the moment system, we propose a matching requirement and construct BCs for the Navier-Stokes equations. The obtained BCs contain the effect of second-order terms on the velocity slip and temperature jump. For the illustrative case of the Couette flow, we prove the validity of the constructed BCs through the error estimates. Meanwhile, numerical tests are presented to show the performance of the constructed BCs.

This work is concerned with coupling conditions for linear hyperbolic relaxation systems with multiple relaxation times. In the region with small relaxation time, an equilibrium system can be used for computational efficiency. Under the assumption that the relaxation system satisfies the structural stability condition and the interface is non-characteristic, we derive a coupling condition at the interface to couple the two systems in a domain decomposition setting. We prove the validity by the energy estimate and Laplace transform, which shows how the error of the domain decomposition method depends on the smaller relaxation time and the boundary layer effects. In addition, we propose a discontinuous Galerkin (DG) scheme for solving the interface problem with the derived coupling condition and prove the L2 stability. We validate our analysis on the linearized Carleman model and the linearized Grad's moment system and show the effectiveness of the DG scheme., Comment: 30 pages, 2 figures

Inspired by the fact that human eyes continue to develop tracking ability in early and middle childhood, we propose to use tracking as a proxy task for a computer vision system to learn the visual representations. Modelled on the Catch game played by the children, we design a Catch-the-Patch (CtP) game for a 3D-CNN model to learn visual representations that would help with video-related tasks. In the proposed pretraining framework, we cut an image patch from a given video and let it scale and move according to a pre-set trajectory. The proxy task is to estimate the position and size of the image patch in a sequence of video frames, given only the target bounding box in the first frame. We discover that using multiple image patches simultaneously brings clear benefits. We further increase the difficulty of the game by randomly making patches invisible. Extensive experiments on mainstream benchmarks demonstrate the superior performance of CtP against other video pretraining methods. In addition, CtP-pretrained features are less sensitive to domain gaps than those trained by a supervised action recognition task. When both trained on Kinetics-400, we are pleasantly surprised to find that CtP-pretrained representation achieves much higher action classification accuracy than its fully supervised counterpart on Something-Something dataset. Code is available online: github.com/microsoft/CtP., Comment: To appear in CVPR'21. Code available at github.com/microsoft/CtP

Predicting future frames of video sequences is challenging due to the complex and stochastic nature of the problem. Video prediction methods based on variational auto-encoders (VAEs) have been a great success, but they require the training data to contain multiple possible futures for an observed video sequence. This is hard to be fulfilled when videos are captured in the wild where any given observation only has a determinate future. As a result, training a vanilla VAE model with these videos inevitably causes posterior collapse. To alleviate this problem, we propose a novel VAE structure, dabbed VAE-in-VAE or VAE$^2$. The key idea is to explicitly introduce stochasticity into the VAE. We treat part of the observed video sequence as a random transition state that bridges its past and future, and maximize the likelihood of a Markov Chain over the video sequence under all possible transition states. A tractable lower bound is proposed for this intractable objective function and an end-to-end optimization algorithm is designed accordingly. VAE$^2$ can mitigate the posterior collapse problem to a large extent, as it breaks the direct dependence between future and observation and does not directly regress the determinate future provided by the training data. We carry out experiments on a large-scale dataset called Cityscapes, which contains videos collected from a number of urban cities. Results show that VAE$^2$ is capable of predicting diverse futures and is more resistant to posterior collapse than the other state-of-the-art VAE-based approaches. We believe that VAE$^2$ is also applicable to other stochastic sequence prediction problems where training data are lack of stochasticity., Comment: 17 pages, 26 figures

In this paper, we propose a data-driven method to discover multiscale chemical reactions governed by the law of mass action. First, we use a single matrix to represent the stoichiometric coefficients for both the reactants and products in a system without catalysis reactions. The negative entries in the matrix denote the stoichiometric coefficients for the reactants and the positive ones for the products. Second, we find that the conventional optimization methods usually get stuck in the local minima and could not find the true solution in learning the multiscale chemical reactions. To overcome this difficulty, we propose a partial-parameters-freezing (PPF) technique to progressively determine the network parameters by using the fact that the stoichiometric coefficients are integers. With such a technique, the dimension of the searching space is gradually reduced in the training process and the global mimina can be eventually obtained. Several numerical experiments including the classical Michaelis-Menten kinetics, the hydrogen oxidation reactions, and the simplified GRI-3.0 mechanism verify the good performance of our algorithm in learning the multiscale chemical reactions. The code is available at \url{https://github.com/JuntaoHuang/multiscale-chemical-reaction}.

This work is concerned with boundary conditions for one-dimensional hyperbolic relaxation systems with characteristic boundaries. We assume that the relaxation system satisfies the structural stability condition proposed by the second author previously and the boundary is characteristic of type I (characteristic for the relaxation system but non-characteristic for the corresponding equilibrium system). For this kind of characteristic initial-boundary-value problems, we propose a modified Generalized Kreiss condition (GKC). This extends the GKC proposed by the second author for the non-characteristic boundaries to the present characteristic case. Under this modified GKC, we derive the reduced boundary condition and verify its validity by combining an energy estimate with the Laplace transform. Moreover, we show the existence of boundary-layers for nonlinear problems.

In this work, we develop a method for learning interpretable, thermodynamically stable and Galilean invariant partial differential equations (PDEs) based on the Conservation-dissipation Formalism of irreversible thermodynamics. As governing equations for non-equilibrium flows in one dimension, the learned PDEs are parameterized by fully-connected neural networks and satisfy the conservation-dissipation principle automatically. In particular, they are hyperbolic balance laws and Galilean invariant. The training data are generated from a kinetic model with smooth initial data. Numerical results indicate that the learned PDEs can achieve good accuracy in a wide range of Knudsen numbers. Remarkably, the learned dynamics can give satisfactory results with randomly sampled discontinuous initial data and Sod's shock tube problem although it is trained only with smooth initial data.