Search

Recently, multimodal large language models have made significant advancements in video understanding tasks. However, their ability to understand unprocessed long videos is very limited, primarily due to the difficulty in supporting the enormous memory overhead. Although existing methods achieve a balance between memory and information by aggregating frames, they inevitably introduce the severe hallucination issue. To address this issue, this paper constructs a comprehensive hallucination mitigation pipeline based on existing MLLMs. Specifically, we use the CLIP Score to guide the frame sampling process with questions, selecting key frames relevant to the question. Then, We inject question information into the queries of the image Q-former to obtain more important visual features. Finally, during the answer generation stage, we utilize chain-of-thought and in-context learning techniques to explicitly control the generation of answers. It is worth mentioning that for the breakpoint mode, we found that image understanding models achieved better results than video understanding models. Therefore, we aggregated the answers from both types of models using a comparison mechanism. Ultimately, We achieved 84.2\% and 62.9\% for the global and breakpoint modes respectively on the MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%. Moreover the proposed method won the third place in the CVPR LOVEU 2024 Long-Term Video Question Answering Challenge. The code is avaiable at https://github.com/lntzm/CVPR24Track-LongVideo

Quantum (Poincar\'e) recurrence theorem are known for closed quantum (classical) systems. Can recurrence happen in open systems? We provide the recurrence theorem for open quantum systems via non-Hermitian (NH) description. We find that PT symmetry and pseudo-Hermitian symmetry protect recurrence for NH open quantum systems and the recurrence fails with the symmetry breaking. Applying our theorem to PT-symmetric systems, we reveal why quantum recurrence happens in PT-unbroken phase but fails in PT-broken phase, which was misunderstood before. A contradiction emerges when we apply our theorem to anti-PT symmetric systems and we settle it, revealing that distinguishability and von Neumann entropy are generally not effective to describe the information dynamics in NH systems. A new approach is developed to investigate the information dynamics of NH systems. For anti-PT symmetric systems in PT-broken phase, we find there are three information-dynamics patterns: oscillations with an overall decrease (increase) , and periodic oscillations. The periodic oscillations (information complete retrieval) happen only if the spectrum of NH Hamiltonian is real. The three patterns degenerate to the periodic oscillation using distinguishability or von Neumann entropy because normalization of non-unitary evolved states leads to loss of information. We conclude with a discussion of the physical meaning behind the recurrence in open systems and give the direction of recurrence theorem not limited to conservative systems in classical mechanics.

Video Moment Retrieval (VMR) aims to retrieve temporal segments in untrimmed videos corresponding to a given language query by constructing cross-modal alignment strategies. However, these existing strategies are often sub-optimal since they ignore the modality imbalance problem, \textit{i.e.}, the semantic richness inherent in videos far exceeds that of a given limited-length sentence. Therefore, in pursuit of better alignment, a natural idea is enhancing the video modality to filter out query-irrelevant semantics, and enhancing the text modality to capture more segment-relevant knowledge. In this paper, we introduce Modal-Enhanced Semantic Modeling (MESM), a novel framework for more balanced alignment through enhancing features at two levels. First, we enhance the video modality at the frame-word level through word reconstruction. This strategy emphasizes the portions associated with query words in frame-level features while suppressing irrelevant parts. Therefore, the enhanced video contains less redundant semantics and is more balanced with the textual modality. Second, we enhance the textual modality at the segment-sentence level by learning complementary knowledge from context sentences and ground-truth segments. With the knowledge added to the query, the textual modality thus maintains more meaningful semantics and is more balanced with the video modality. By implementing two levels of MESM, the semantic information from both modalities is more balanced to align, thereby bridging the modality gap. Experiments on three widely used benchmarks, including the out-of-distribution settings, show that the proposed framework achieves a new start-of-the-art performance with notable generalization ability (e.g., 4.42% and 7.69% average gains of R1@0.7 on Charades-STA and Charades-CG). The code will be available at https://github.com/lntzm/MESM., Comment: Accepted to AAAI 2024

We reveal the continuous phase transition in anyonic-PT symmetric systems, contrasting with the discontinuous phase transition corresponding to the discrete (anti-) PT symmetry. The continuous phase transition originates from the continuity of anyonic-PT symmetry. We find there are three information-dynamics patterns for anyonic-PT symmetric systems: damped oscillations with an overall decrease (increase) and asymptotically stable damped oscillations, which are three-fold degenerate and distorted using the Hermitian quantum R\'enyi entropy or distinguishability. It is the normalization of the non-unitary evolved density matrix causes the degeneracy and distortion. We give a justification for non-Hermitian quantum R\'enyi entropy being negative. By exploring the mathematics and physical meaning of the negative entropy in open quantum systems, we connect the negative non-Hermitian quantum R\'enyi entropy and negative quantum conditional entropy, opening up a new journey to rigorously investigate the negative entropy in open quantum systems.

The rich-club phenomenon, which provides information about the association between nodes, is a useful method to study the hierarchy structure of networks. In this work, we explore the behavior of rich-club coefficient (RCC) in scale-free networks, and find that the degree-based RCC is a power function of degree centrality with power exponent \b{eta} and the betweenness-based RCC is a linear function of betweenness centrality with slope {\theta}. The findings from the New York metro network further validate our conclusions. Moreover, we calculate the value of RCC in a BA network by deleting nodes and obtain a general expression for RCC as a function of node sequence. On this basis, the solution of RCC for centrality is also obtained, which shows that the curve of RCC is determined by the centrality distribution. In order to find out what affects \b{eta} and {\theta}, we conduct numerical simulations in scale-free networks, and observe: \b{eta} and {\gamma} (the degree distribution exponent) increase together, {\theta} increases with m (the number of new edges) and decreases to convergence as {\gamma} increases., Comment: 7pages,7 figures