Search

Recently, multimodal large language models have made significant advancements in video understanding tasks. However, their ability to understand unprocessed long videos is very limited, primarily due to the difficulty in supporting the enormous memory overhead. Although existing methods achieve a balance between memory and information by aggregating frames, they inevitably introduce the severe hallucination issue. To address this issue, this paper constructs a comprehensive hallucination mitigation pipeline based on existing MLLMs. Specifically, we use the CLIP Score to guide the frame sampling process with questions, selecting key frames relevant to the question. Then, We inject question information into the queries of the image Q-former to obtain more important visual features. Finally, during the answer generation stage, we utilize chain-of-thought and in-context learning techniques to explicitly control the generation of answers. It is worth mentioning that for the breakpoint mode, we found that image understanding models achieved better results than video understanding models. Therefore, we aggregated the answers from both types of models using a comparison mechanism. Ultimately, We achieved 84.2\% and 62.9\% for the global and breakpoint modes respectively on the MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%. Moreover the proposed method won the third place in the CVPR LOVEU 2024 Long-Term Video Question Answering Challenge. The code is avaiable at https://github.com/lntzm/CVPR24Track-LongVideo

Quantum (Poincar\'e) recurrence theorem are known for closed quantum (classical) systems. Can recurrence happen in open systems? We provide the recurrence theorem for open quantum systems via non-Hermitian (NH) description. We find that PT symmetry and pseudo-Hermitian symmetry protect recurrence for NH open quantum systems and the recurrence fails with the symmetry breaking. Applying our theorem to PT-symmetric systems, we reveal why quantum recurrence happens in PT-unbroken phase but fails in PT-broken phase, which was misunderstood before. A contradiction emerges when we apply our theorem to anti-PT symmetric systems and we settle it, revealing that distinguishability and von Neumann entropy are generally not effective to describe the information dynamics in NH systems. A new approach is developed to investigate the information dynamics of NH systems. For anti-PT symmetric systems in PT-broken phase, we find there are three information-dynamics patterns: oscillations with an overall decrease (increase) , and periodic oscillations. The periodic oscillations (information complete retrieval) happen only if the spectrum of NH Hamiltonian is real. The three patterns degenerate to the periodic oscillation using distinguishability or von Neumann entropy because normalization of non-unitary evolved states leads to loss of information. We conclude with a discussion of the physical meaning behind the recurrence in open systems and give the direction of recurrence theorem not limited to conservative systems in classical mechanics.