Search

With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench., Comment: Work on progress

Video understanding is a crucial next step for multimodal large language models (MLLMs). To probe specific aspects of video understanding ability, existing video benchmarks typically require careful video selection based on the target capability, along with laborious annotation of query-response pairs to match the specific video content. This process is both challenging and resource-intensive. In this paper, we propose VideoNIAH (Video Needle In A Haystack), a benchmark construction framework through synthetic video generation. VideoNIAH decouples test video content from their query-responses by inserting unrelated image/text 'needles' into original videos. It generates annotations solely from these needles, ensuring diversity in video sources and a variety of query-responses. Additionally, by inserting multiple needles, VideoNIAH rigorously evaluates the temporal understanding capabilities of models. We utilized VideoNIAH to compile a video benchmark VNBench, including tasks such as retrieval, ordering, and counting. VNBench can efficiently evaluate the fine-grained understanding ability and spatio-temporal modeling ability of a video model, while also supporting the long-context evaluation. Additionally, we evaluated recent video-centric multimodal large language models (MLLMs), both open-source and proprietary, providing a comprehensive analysis. We found that although proprietary models have significant advantages over open-source models, all existing video models still perform poorly on long-distance dependency tasks. VideoNIAH is a simple yet highly scalable benchmark construction framework, and we believe it will inspire future video benchmark works. The code and data are available at https://github.com/joez17/VideoNIAH.

Visual instruction tuning is an essential approach to improving the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs). A surge of visual instruction datasets with various focuses and characteristics have been proposed recently, enabling MLLMs to achieve surprising results on evaluation benchmarks. To develop more capable MLLMs, in this paper, we aim to investigate a more fundamental question: ``what makes for good visual instructions?''. By conducting a comprehensive empirical study, we find that instructions focused on complex visual reasoning tasks are particularly effective in improving the performance of MLLMs on evaluation benchmarks. Building upon this finding, we design a systematic approach to automatically creating high-quality complex visual reasoning instructions. Our approach employs a synthesis-complication-reformulation paradigm, leveraging multiple stages to gradually increase the complexity of the instructions while guaranteeing quality. Based on this approach, we create the synthetic visual reasoning instruction dataset consisting of 32K examples, namely ComVint, and fine-tune four MLLMs on it. Experimental results demonstrate that our dataset consistently enhances the performance of all the compared MLLMs, e.g., improving the performance of MiniGPT-4 and BLIP-2 on MME-Cognition by 32.6% and 28.8%, respectively. Our code and data are publicly available at the link: https://github.com/RUCAIBox/ComVint., Comment: Work in progress

In this paper, we propose a novel language model guided captioning approach, LAMOC, for knowledge-based visual question answering (VQA). Our approach employs the generated captions by a captioning model as the context of an answer prediction model, which is a Pre-trained Language model (PLM). As the major contribution, we leverage the guidance and feedback of the prediction model to improve the capability of the captioning model. In this way, the captioning model can become aware of the task goal and information need from the PLM. To develop our approach, we design two specific training stages, where the first stage adapts the captioning model to the prediction model (selecting more suitable caption propositions for training) and the second stage tunes the captioning model according to the task goal (learning from feedback of the PLM). Extensive experiments demonstrate the effectiveness of the proposed approach on the knowledge-based VQA task. Specifically, on the challenging A-OKVQA dataset, LAMOC outperforms several competitive zero-shot methods and even achieves comparable results to a fine-tuned VLP model. Our code is publicly available at https://github.com/RUCAIBox/LAMOC., Comment: Accepted by ACL2023 findings

People often imagine relevant scenes to aid in the writing process. In this work, we aim to utilize visual information for composition in the same manner as humans. We propose a method, LIVE, that makes pre-trained language models (PLMs) Learn to Imagine for Visuallyaugmented natural language gEneration. First, we imagine the scene based on the text: we use a diffusion model to synthesize high-quality images conditioned on the input texts. Second, we use CLIP to determine whether the text can evoke the imagination in a posterior way. Finally, our imagination is dynamic, and we conduct synthesis for each sentence rather than generate only one image for an entire paragraph. Technically, we propose a novel plug-and-play fusion layer to obtain visually-augmented representations for each text. Our vision-text fusion layer is compatible with Transformerbased architecture. We have conducted extensive experiments on four generation tasks using BART and T5, and the automatic results and human evaluation demonstrate the effectiveness of our proposed method. We will release the code, model, and data at the link: https://github.com/RUCAIBox/LIVE., Comment: Accepted by ACL 2023

Inspired by the superior language abilities of large language models (LLM), large vision-language models (LVLM) have been recently explored by integrating powerful LLMs for improving the performance on complex multimodal tasks. Despite the promising progress on LVLMs, we find that LVLMs suffer from the hallucination problem, i.e. they tend to generate objects that are inconsistent with the target images in the descriptions. To investigate it, this work presents the first systematic study on object hallucination of LVLMs. We conduct the evaluation experiments on several representative LVLMs, and show that they mostly suffer from severe object hallucination issue. We further discuss that the visual instructions may influence the hallucination, and find that: objects that frequently occur in the visual instructions or co-occur with the image objects, are obviously prone to be hallucinated by LVLMs. Besides, we find that existing evaluation methods might be affected by the input instructions and generation styles of LVLMs. Thus, we further design an improved evaluation method for object hallucination by proposing a polling-based query method called POPE. Experiment results demonstrate that our POPE can evaluate the object hallucination in a more stable and flexible way. Our codes and data are publicly available at https://github.com/RUCAIBox/POPE., Comment: Accepted to EMNLP 2023

Language is essentially a complex, intricate system of human expressions governed by grammatical rules. It poses a significant challenge to develop capable AI algorithms for comprehending and grasping a language. As a major approach, language modeling has been widely studied for language understanding and generation in the past two decades, evolving from statistical language models to neural language models. Recently, pre-trained language models (PLMs) have been proposed by pre-training Transformer models over large-scale corpora, showing strong capabilities in solving various NLP tasks. Since researchers have found that model scaling can lead to performance improvement, they further study the scaling effect by increasing the model size to an even larger size. Interestingly, when the parameter scale exceeds a certain level, these enlarged language models not only achieve a significant performance improvement but also show some special abilities that are not present in small-scale language models. To discriminate the difference in parameter scale, the research community has coined the term large language models (LLM) for the PLMs of significant size. Recently, the research on LLMs has been largely advanced by both academia and industry, and a remarkable progress is the launch of ChatGPT, which has attracted widespread attention from society. The technical evolution of LLMs has been making an important impact on the entire AI community, which would revolutionize the way how we develop and use AI algorithms. In this survey, we review the recent advances of LLMs by introducing the background, key findings, and mainstream techniques. In particular, we focus on four major aspects of LLMs, namely pre-training, adaptation tuning, utilization, and capacity evaluation. Besides, we also summarize the available resources for developing LLMs and discuss the remaining issues for future directions., Comment: ongoing work; 124 pages, 946 citations

The state of turbulent, minimal-channel flow is estimated from spatio-temporal sparse observations of the velocity, using both a physics-informed neural network (PINN) and adjoint-variational data assimilation (4DVar). The performance of PINN is assessed against the benchmark results from 4DVar. The PINN is efficient to implement, takes advantage of automatic differentiation to evaluate the governing equations, and does not require the development of an adjoint model. In addition, the flow evolution is expressed in terms of the network parameters which have a far smaller dimension than the predicted trajectory in state space or even just the initial condition of the flow. Provided adequate observations, network architecture and training, the PINN can yield satisfactory estimates of the the flow field, both for the missing velocity data and the entirely unobserved pressure field. However, accuracy depends on the network architecture, and the dependence is not known a priori. In comparison to 4DVar estimation which becomes progressively more accurate over the observation horizon, the PINN predictions are generally less accurate and maintain the same level of errors throughout the assimilation time window. Another notable distinction is the capacity to accurately forecast the flow evolution: while the 4DVar prediction depart from the true flow state gradually and according to the Lyapunov exponent, the PINN is entirely inaccurate immediately beyond the training time horizon unless re-trained. Most importantly, while 4DVar satisfies the discrete form of the governing equations point-wise to machine precision, in PINN the equations are only satisfied in an $L^2$ sense.

As transformer evolves, pre-trained models have advanced at a breakneck pace in recent years. They have dominated the mainstream techniques in natural language processing (NLP) and computer vision (CV). How to adapt pre-training to the field of Vision-and-Language (V-L) learning and improve downstream task performance becomes a focus of multimodal learning. In this paper, we review the recent progress in Vision-Language Pre-Trained Models (VL-PTMs). As the core content, we first briefly introduce several ways to encode raw images and texts to single-modal embeddings before pre-training. Then, we dive into the mainstream architectures of VL-PTMs in modeling the interaction between text and image representations. We further present widely-used pre-training tasks, and then we introduce some common downstream tasks. We finally conclude this paper and present some promising research directions. Our survey aims to provide researchers with synthesis and pointer to related research., Comment: Accepted by IJCAI-2022 survey track

A non-intrusive data assimilation methodology is developed to improve the statistical predictions of large-eddy simulations (LES). The ensemble-variational (EnVar) approach aims to minimize a cost function that is defined as the discrepancy between LES predictions and reference statistics from experiments or, in the present demonstration, independent direct numerical simulations (DNS). This methodology is applied to adjust the Smagorinsky subgrid model and obtain data assimilated LES (DA-LES) which accurately estimate the statistics of turbulent channel flow. To separately control the mean and fluctuations of the modeled subgrid tensor, and ultimately the first- and second-order flow statistics, two types of model corrections are considered. The first one optimizes the wall-normal profile of the Smagorinsky coefficient, while the second one introduces an adjustable steady forcing in the momentum equations to independently act on the mean flow. Using these two elements, the data assimilation procedure can satisfactorily modify the subgrid model and accurately recover reference flow statistics. The retrieved subgrid model significantly outperforms more elaborate baseline models such as dynamic and mixed models, in a posteriori testing. The robustness of the present data assimilation methodology is assessed by changing the Reynolds number and considering grid resolutions that are away from usual recommendations. Taking advantage of the stochastic formulation of EnVar, the developed framework also provides the uncertainty of the retrieved model.

The notion of an Evolutional Deep Neural Network (EDNN) is introduced for the solution of partial differential equations (PDE). The parameters of the network are trained to represent the initial state of the system only, and are subsequently updated dynamically, without any further training, to provide an accurate prediction of the evolution of the PDE system. In this framework, the network parameters are treated as functions with respect to the appropriate coordinate and are numerically updated using the governing equations. By marching the neural network weights in the parameter space, EDNN can predict state-space trajectories that are indefinitely long, which is difficult for other neural network approaches. Boundary conditions of the PDEs are treated as hard constraints, are embedded into the neural network, and are therefore exactly satisfied throughout the entire solution trajectory. Several applications including the heat equation, the advection equation, the Burgers equation, the Kuramoto Sivashinsky equation and the Navier-Stokes equations are solved to demonstrate the versatility and accuracy of EDNN. The application of EDNN to the incompressible Navier-Stokes equation embeds the divergence-free constraint into the network design so that the projection of the momentum equation to solenoidal space is implicitly achieved. The numerical results verify the accuracy of EDNN solutions relative to analytical and benchmark numerical solutions, both for the transient dynamics and statistics of the system.

The discovery of Partial Differential Equations (PDEs) is an essential task for applied science and engineering. However, data-driven discovery of PDEs is generally challenging, primarily stemming from the sensitivity of the discovered equation to noise and the complexities of model selection. In this work, we propose an advanced Bayesian sparse learning algorithm for PDE discovery with variable coefficients, predominantly when the coefficients are spatially or temporally dependent. Specifically, we apply threshold Bayesian group Lasso regression with a spike-and-slab prior (tBGL-SS) and leverage a Gibbs sampler for Bayesian posterior estimation of PDE coefficients. This approach not only enhances the robustness of point estimation with valid uncertainty quantification but also relaxes the computational burden from Bayesian inference through the integration of coefficient thresholds as an approximate MCMC method. Moreover, from the quantified uncertainties, we propose a Bayesian total error bar criteria for model selection, which outperforms classic metrics including the root mean square and the Akaike information criterion. The capability of this method is illustrated by the discovery of several classical benchmark PDEs with spatially or temporally varying coefficients from solution data obtained from the reference simulations. In the experiments, we show that the tBGL-SS method is more robust than the baseline methods under noisy environments and provides better model selection criteria along the regularization path.

In the recent application of scientific modeling, machine learning models are largely applied to facilitate computational simulations of fluid systems. Rotation symmetry is a general property for most symmetric fluid systems. However, in general, current machine learning methods have no theoretical way to guarantee rotational symmetry. By observing an important property of contraction and rotation operation on high-order symmetric tensors, we prove that the rotation operation is preserved via tensor contraction. Based on this theoretical justification, in this paper, we introduce Rotation-Equivariant Network (RotEqNet) to guarantee the property of rotation-equivariance for high-order tensors in fluid systems. We implement RotEqNet and evaluate our claims through four case studies on various fluid systems. The property of error reduction and rotation-equivariance is verified in these case studies. Results from the comparative study show that our method outperforms conventional methods, which rely on data augmentation., Comment: Preprint submitted to Journal of Computational Physics

Long Range Wide Area Network (LoRaWAN) has an open specification that defines the Medium Access Control (MAC) protocol for LoRa networks, mainly based on pure-Aloha with beacon option. The LoRaWAN has a star of star topology where end-devices are directly linked to the gateway and up-link packets are primary traffic. This paper~contributes an approach of multi-hop in LoRaWAN. The potential benefits of multi-hop in LoRaWAN are investigated. A RSSI-based trilateration algorithm for graph construction is proposed. The shortest path for multi-hop and forwarding mechanism without local beacon is designed and implemented in simulation. This novel approach is tested in ns-3 simulator and different scenarios are run. Results indicate that multi-hop can decrease the energy consumption of LoRaWAN, by reducing Time on Air and TX power., Comment: There are error and dispute in this report, current approach is withdrawn

Long Range (LoRa) is a wireless communication technology for the Internet of Things (IoT), able to provide wide coverage networking to devices with low power consumption and low data-rate. This paper~proposes and describes a simulation module of LoRa wireless networks, based on models of the LoRa physical and data-link layers, in order to enable simulation-based evaluation of LoRa performance in large-scale networks. The physical layer model uses path loss parameters estimated by way of real measurements performed at different scenarios. The data-link layer model relies on an Aloha-like medium access protocol. Both models for the physical and data-link layers are implemented in a ns-3 module. Behavior and performance of this module are studied by way of a set of experiments that show the consistency of ns-3 simulation and measurements on a real LoRa testbed. Finally, an evaluation of the simulation module scope shows that the time complexity of this module is exponential., Comment: There are error and dispute in this report, current approach is withdrawn

Long Range (LoRa) is a modem technology for wireless communication in the Internet of Things (IoT), which trades off low data-rate for low power consumption. Long Range Wide Area Network (LoRaWAN) has an open specification that determines the Medium Access Control (MAC) protocol for LoRa networks. This paper contributes a simulation module in Network Simulator 3 (ns-3) to evaluate the limitation of LoRaWAN in various metrics. The result shows the performance of LoRaWAN in a simulation environment and a large scale scenario, indicating the feature of flexible propagation range on physical layer, and limited channel capacity on the data-link layer. The feasibility of optimized multi-hop in LoRaWAN is analyzed. Based on the simulation result, a multi-hop mechanism driven by spreading factor is proposed, not only to extend the network coverage but to reduce the channel load. The novel approach takes benefit of the dynamic feature of LoRa and estimates optimized forwarding hops with parameter configuration. The result shows that Time on Air can be significantly decreased., Comment: There are error and dispute in this report, current approach is withdrawn