Search

We present HealthGPT, a powerful Medical Large Vision-Language Model (Med-LVLM) that integrates medical visual comprehension and generation capabilities within a unified autoregressive paradigm. Our bootstrapping philosophy is to progressively adapt heterogeneous comprehension and generation knowledge to pre-trained large language models (LLMs). This is achieved through a novel heterogeneous low-rank adaptation (H-LoRA) technique, which is complemented by a tailored hierarchical visual perception approach and a three-stage learning strategy. To effectively learn the HealthGPT, we devise a comprehensive medical domain-specific comprehension and generation dataset called VL-Health. Experimental results demonstrate exceptional performance and scalability of HealthGPT in medical visual unified tasks. Our project can be accessed at https://github.com/DCDmllm/HealthGPT.

Galactic cosmic rays (CRs) generally share common propagation features, leading to consistent spectral observations of secondary nuclei such as Li, Be, and B. However, the Li spectrum predicted by the CR diffusion coefficient inferred from B/C is significantly lower than the latest measurement of AMS-02. This anomaly may be attributed to the missing contributions from the heavy nuclei components in cosmic rays. By including these missing contributions the excess of the Li spectrum disappears. However, another inconsistency still exists since the calculated Li spectrum is now overestimated compared to the data. In this work, we update the cross-section model used to calculate the Li production according to more cross-section measurements. We find that the cross sections of these added reactions are systematically overestimated, and should be renormalized to the interpolations of available data. As a result, our prediction of the total Li spectrum is consistent with the measurement without discrepancy, and our prediction of the $\rm^6Li$ and $\rm^7Li$ spectra are consistent with the preliminary measurements of AMS-02 within the cross-section uncertainties., Comment: 12 pages, 8 figures

Despite the groundbreaking success of diffusion models in generating high-fidelity images, their latent space remains relatively under-explored, even though it holds significant promise for enabling versatile and interpretable image editing capabilities. The complicated denoising trajectory and high dimensionality of the latent space make it extremely challenging to interpret. Existing methods mainly explore the feature space of U-Net in Diffusion Models (DMs) instead of the latent space itself. In contrast, we directly investigate the latent space via Singular Value Decomposition (SVD) and discover three useful properties that can be used to control generation results without the requirements of data collection and maintain identity fidelity generated images. Based on these properties, we propose a novel image editing framework that is capable of learning arbitrary attributes from one pair of latent codes destined by text prompts in Stable Diffusion Models. To validate our approach, extensive experiments are conducted to demonstrate its effectiveness and flexibility in image editing. We will release our codes soon to foster further research and applications in this area.

Reputation serves as a powerful mechanism for promoting cooperation in multi-agent systems, as agents are more inclined to cooperate with those of good social standing. While existing multi-agent reinforcement learning methods typically rely on predefined social norms to assign reputations, the question of how a population reaches a consensus on judgement when agents hold private, independent views remains unresolved. In this paper, we propose a novel bottom-up reputation learning method, Learning with Reputation Reward (LR2), designed to promote cooperative behaviour through rewards shaping based on assigned reputation. Our agent architecture includes a dilemma policy that determines cooperation by considering the impact on neighbours, and an evaluation policy that assigns reputations to affect the actions of neighbours while optimizing self-objectives. It operates using local observations and interaction-based rewards, without relying on centralized modules or predefined norms. Our findings demonstrate the effectiveness and adaptability of LR2 across various spatial social dilemma scenarios. Interestingly, we find that LR2 stabilizes and enhances cooperation not only with reward reshaping from bottom-up reputation but also by fostering strategy clustering in structured populations, thereby creating environments conducive to sustained cooperation., Comment: Accepted by AAMAS 2025 (24th International Conference on Autonomous Agents and Multiagent Systems)

Oscillations between axion-like particles (ALPs) and photons in astrophysical magnetic fields can lead to irregularities in the high energy gamma ray spectra of blazars. The magnetic field within the blazar jet plays a crucial role in shaping these effects, with its strength in the emission region being an important parameter determined by multi-wavelength observations. However, the origin of the high energy bump observed in the spectral energy distribution of some blazars is a topic of debate, with both leptonic and hadronic scenarios providing plausible explanations that result in different magnetic field strengths in the emission region. In this study, we investigate the impact of magnetic field configurations on the constraints of ALP parameters. We consider both leptonic and hadronic emission scenarios for the blazar Mrk 501 and derive the corresponding jet magnetic field strengths. Additionally, we explore two jet magnetic field models: one with a toroidal component and the other with helical and tangled components. By analyzing the spectra of Mrk 501 observed by MAGIC and Fermi-LAT from 2017-06-17 to 2019-07-23, which are well-described by both emission scenarios, we derive constraints on the ALP parameters. Our results demonstrate that both the emission scenario and the magnetic field structure play a significant role in deriving these constraints, with the hadronic model leading to much more stringent limits compared to the leptonic model., Comment: 9 pages, 5 figures

The observed spectral bump in cosmic-ray (CR) proton and helium spectra, along with the phase and amplitude evolution of CR dipole anisotropy, provide plausible yet indirect evidence for the presence of a nearby CR source. This study investigates the potential of giant molecular clouds (GMCs) located near the solar system to act as natural probes of CRs from the nearby source, with their gamma-ray emissions serving as indicators of spatial variations in CR flux within the solar neighborhood resulting from this source. We show that a nearby source, accounting for the CR data, could imprint distinct features on the $\gamma$-ray spectra of different GMCs. We expect that these features are detectable by LHAASO and upcoming high-energy $\gamma$-ray observatories, providing a powerful test for the hypothesized nearby source. Notably, we find that determining the energy dependence of the $\gamma$-ray spectral index offers a promising approach to investigate the nearby source and constrain its distance. Conversely, if the spectral bump is a widespread Galactic phenomenon, the energy dependence would exhibit uniformity across all GMCs, distinguishing the underlying mechanism accounting for the spectral bump from the scenario involving a nearby source., Comment: 10 pages, 6 figures

3D Gaussian Splatting (3DGS) has shown remarkable performance in novel view synthesis. However, its rendering quality deteriorates with sparse inphut views, leading to distorted content and reduced details. This limitation hinders its practical application. To address this issue, we propose a sparse-view 3DGS method. Given the inherently ill-posed nature of sparse-view rendering, incorporating prior information is crucial. We propose a semantic regularization technique, using features extracted from the pretrained DINO-ViT model, to ensure multi-view semantic consistency. Additionally, we propose local depth regularization, which constrains depth values to improve generalization on unseen views. Our method outperforms state-of-the-art novel view synthesis approaches, achieving up to 0.4dB improvement in terms of PSNR on the LLFF dataset, with reduced distortion and enhanced visual quality., Comment: 5 pages, 5 figures, has been accepted by the ICASSP 2025

Group-agent reinforcement learning (GARL) is a newly arising learning scenario, where multiple reinforcement learning agents study together in a group, sharing knowledge in an asynchronous fashion. The goal is to improve the learning performance of each individual agent. Under a more general heterogeneous setting where different agents learn using different algorithms, we advance GARL by designing novel and effective group-learning mechanisms. They guide the agents on whether and how to learn from action choices from the others, and allow the agents to adopt available policy and value function models sent by another agent if they perform better. We have conducted extensive experiments on a total of 43 different Atari 2600 games to demonstrate the superior performance of the proposed method. After the group learning, among the 129 agents examined, 96% are able to achieve a learning speed-up, and 72% are able to learn over 100 times faster. Also, around 41% of those agents have achieved a higher accumulated reward score by learning in less than 5% of the time steps required by a single agent when learning on its own.

Cosmological simulations of fuzzy dark matter (FDM) are computationally expensive, and the resulting halos lack flexibility in parameter adjustments, such as virial mass, density profile, and global velocity. Previous studies have introduced a method for constructing FDM halos with predefined density profiles. In this study, we investigate the initial global velocity of these constructed halos and find that it is non-zero. We provide the theoretical formula for this velocity and illustrate that it arises from the interference between states of odd and even parity. Our calculated results closely match simulation outcomes. Additionally, we showcase how to counteract this velocity and create a halo with a customizable initial global velocity. Our study presents a practical method for adjusting the initial global velocity of halos in controlled FDM simulations, facilitating investigations into tidal effects, galaxy collisions, and other scenarios., Comment: 10 pages, 3 figures

Audio-visual video segmentation (AVVS) aims to generate pixel-level maps of sound-producing objects that accurately align with the corresponding audio. However, existing methods often face temporal misalignment, where audio cues and segmentation results are not temporally coordinated. Audio provides two critical pieces of information: i) target object-level details and ii) the timing of when objects start and stop producing sounds. Current methods focus more on object-level information but neglect the boundaries of audio semantic changes, leading to temporal misalignment. To address this issue, we propose a Collaborative Hybrid Propagator Framework~(Co-Prop). This framework includes two main steps: Preliminary Audio Boundary Anchoring and Frame-by-Frame Audio-Insert Propagation. To Anchor the audio boundary, we employ retrieval-assist prompts with Qwen large language models to identify control points of audio semantic changes. These control points split the audio into semantically consistent audio portions. After obtaining the control point lists, we propose the Audio Insertion Propagator to process each audio portion using a frame-by-frame audio insertion propagation and matching approach. We curated a compact dataset comprising diverse source conversion cases and devised a metric to assess alignment rates. Compared to traditional simultaneous processing methods, our approach reduces memory requirements and facilitates frame alignment. Experimental results demonstrate the effectiveness of our approach across three datasets and two backbones. Furthermore, our method can be integrated with existing AVVS approaches, offering plug-and-play functionality to enhance their performance.

Terahertz (THz) technology is critical for quantum material physics, biomedical imaging, ultrafast electronics, and next-generation wireless communications. However, standing in the way of widespread applications is the scarcity of efficient ultrafast THz sources with on-demand fast modulation and easy on-chip integration capability. Here we report the discovery of colossal THz emission from a van der Waals (vdW) ferroelectric semiconductor NbOI$_2$. Using THz emission spectroscopy, we observe a THz generation efficiency an order of magnitude higher than that of ZnTe, a standard nonlinear crystal for ultrafast THz generation. We further uncover the underlying generation mechanisms associated with its large ferroelectric polarization by studying the THz emission dependence on excitation wavelength, incident polarization and fluence. Moreover, we demonstrate the ultrafast coherent amplification and annihilation of the THz emission and associated coherent phonon oscillations by employing a double-pump scheme. These findings combined with first-principles calculations, inform new understanding of the THz light-matter interaction in emergent vdW ferroelectrics and pave the way to develop high-performance THz devices on them for quantum materials sensing and ultrafast electronics.

The p16/Ki-67 dual staining method is a new approach for cervical cancer screening with high sensitivity and specificity. However, there are issues of mis-detection and inaccurate recognition when the YOLOv5s algorithm is directly applied to dual-stained cell images. This paper Proposes a novel cervical cancer dual-stained image recognition (DSIR-YOLO) model based on an YOLOv5. By fusing the Swin-Transformer module, GAM attention mechanism, multi-scale feature fusion, and EIoU loss function, the detection performance is significantly improved, with mAP@0.5 and mAP@0.5:0.95 reaching 92.6% and 70.5%, respectively. Compared with YOLOv5s in five-fold cross-validation, the accuracy, recall, mAP@0.5, and mAP@0.5:0.95 of the improved algorithm are increased by 2.3%, 4.1%, 4.3%, and 8.0%, respectively, with smaller variances and higher stability. Compared with other detection algorithms, DSIR-YOLO in this paper sacrifices some performance requirements to improve the network recognition effect. In addition, the influence of dataset quality on the detection results is studied. By controlling the sealing property of pixels, scale difference, unlabelled cells, and diagonal annotation, the model detection accuracy, recall, mAP@0.5, and mAP@0.5:0.95 are improved by 13.3%, 15.3%, 18.3%, and 30.5%, respectively.

Recent observations have revealed an unusual stellar distribution within the almost dark dwarf galaxy Nube. The galaxy exhibits a remarkably flat stellar distribution, with an effective radius of approximately 6.9 kpc, exceeding the typical size of dwarf galaxies and even surpassing that of ultra-diffuse galaxies (UDGs) with similar stellar masses. The dynamical heating effect of fuzzy dark matter (FDM) may offer an explanation for this extremely diffuse stellar distribution in Nube. In this research, we utilize simulation techniques to investigate this issue and find that a particle mass $\mathcal{O} (1)\times 10^{-23}$ eV offers a plausible explanation for this peculiar stellar distribution anomaly., Comment: 8 pages, 4 figures

Point cloud processing (PCP) encompasses tasks like reconstruction, denoising, registration, and segmentation, each often requiring specialized models to address unique task characteristics. While in-context learning (ICL) has shown promise across tasks by using a single model with task-specific demonstration prompts, its application to PCP reveals significant limitations. We identify inter-task and intra-task sensitivity issues in current ICL methods for PCP, which we attribute to inflexible sampling strategies lacking context adaptation at the point and prompt levels. To address these challenges, we propose MICAS, an advanced ICL framework featuring a multi-grained adaptive sampling mechanism tailored for PCP. MICAS introduces two core components: task-adaptive point sampling, which leverages inter-task cues for point-level sampling, and query-specific prompt sampling, which selects optimal prompts per query to mitigate intra-task sensitivity. To our knowledge, this is the first approach to introduce adaptive sampling tailored to the unique requirements of point clouds within an ICL framework. Extensive experiments show that MICAS not only efficiently handles various PCP tasks but also significantly outperforms existing methods. Notably, it achieves a remarkable $4.1\%$ improvement in the part segmentation task and delivers consistent gains across various PCP applications., Comment: 15 pages, 6 figures, 3 tables

With the advance of diffusion models, today's video generation has achieved impressive quality. To extend the generation length and facilitate real-world applications, a majority of video diffusion models (VDMs) generate videos in an autoregressive manner, i.e., generating subsequent clips conditioned on the last frame(s) of the previous clip. However, existing autoregressive VDMs are highly inefficient and redundant: The model must re-compute all the conditional frames that are overlapped between adjacent clips. This issue is exacerbated when the conditional frames are extended autoregressively to provide the model with long-term context. In such cases, the computational demands increase significantly (i.e., with a quadratic complexity w.r.t. the autoregression step). In this paper, we propose Ca2-VDM, an efficient autoregressive VDM with Causal generation and Cache sharing. For causal generation, it introduces unidirectional feature computation, which ensures that the cache of conditional frames can be precomputed in previous autoregression steps and reused in every subsequent step, eliminating redundant computations. For cache sharing, it shares the cache across all denoising steps to avoid the huge cache storage cost. Extensive experiments demonstrated that our Ca2-VDM achieves state-of-the-art quantitative and qualitative video generation results and significantly improves the generation speed. Code is available at https://github.com/Dawn-LX/CausalCache-VDM, Comment: Technical Report. Code is available at https://github.com/Dawn-LX/CausalCache-VDM

Blind image restoration remains a significant challenge in low-level vision tasks. Recently, denoising diffusion models have shown remarkable performance in image synthesis. Guided diffusion models, leveraging the potent generative priors of pre-trained models along with a differential guidance loss, have achieved promising results in blind image restoration. However, these models typically consider data consistency solely in the spatial domain, often resulting in distorted image content. In this paper, we propose a novel frequency-aware guidance loss that can be integrated into various diffusion models in a plug-and-play manner. Our proposed guidance loss, based on 2D discrete wavelet transform, simultaneously enforces content consistency in both the spatial and frequency domains. Experimental results demonstrate the effectiveness of our method in three blind restoration tasks: blind image deblurring, imaging through turbulence, and blind restoration for multiple degradations. Notably, our method achieves a significant improvement in PSNR score, with a remarkable enhancement of 3.72\,dB in image deblurring. Moreover, our method exhibits superior capability in generating images with rich details and reduced distortion, leading to the best visual quality., Comment: 17 pages, 6 figures, has been accepted by the ECCV 2024: AIM workshop

Euclidean representation learning methods have achieved commendable results in image fusion tasks, which can be attributed to their clear advantages in handling with linear space. However, data collected from a realistic scene usually have a non-Euclidean structure, where Euclidean metric might be limited in representing the true data relationships, degrading fusion performance. To address this issue, a novel SPD (symmetric positive definite) manifold learning framework is proposed for multi-modal image fusion, named SPDFusion, which extends the image fusion approach from the Euclidean space to the SPD manifolds. Specifically, we encode images according to the Riemannian geometry to exploit their intrinsic statistical correlations, thereby aligning with human visual perception. Actually, the SPD matrix underpins our network learning, with a cross-modal fusion strategy employed to harness modality-specific dependencies and augment complementary information. Subsequently, an attention module is designed to process the learned weight matrix, facilitating the weighting of spatial global correlation semantics via SPD matrix multiplication. Based on this, we design an end-to-end fusion network based on cross-modal manifold learning. Extensive experiments on public datasets demonstrate that our framework exhibits superior performance compared to the current state-of-the-art methods., Comment: 14 pages, 12 figures

The stylization of 3D scenes is an increasingly attractive topic in 3D vision. Although image style transfer has been extensively researched with promising results, directly applying 2D style transfer methods to 3D scenes often fails to preserve the structural and multi-view properties of 3D environments, resulting in unpleasant distortions in images from different viewpoints. To address these issues, we leverage the remarkable generative prior of diffusion-based models and propose a novel style transfer method, OSDiffST, based on a pre-trained one-step diffusion model (i.e., SD-Turbo) for rendering diverse styles in multi-view images of 3D scenes. To efficiently adapt the pre-trained model for multi-view style transfer on small datasets, we introduce a vision condition module to extract style information from the reference style image to serve as conditional input for the diffusion model and employ LoRA in diffusion model for adaptation. Additionally, we consider color distribution alignment and structural similarity between the stylized and content images using two specific loss functions. As a result, our method effectively preserves the structural information and multi-view consistency in stylized images without any 3D information. Experiments show that our method surpasses other promising style transfer methods in synthesizing various styles for multi-view images of 3D scenes. Stylized images from different viewpoints generated by our method achieve superior visual quality, with better structural integrity and less distortion. The source code is available at https://github.com/YushenZuo/OSDiffST., Comment: Accepted by ECCV 2024 AI for Visual Arts Workshop and Challenges, 18 pages, 7 figures

In this paper, we investigate the strong gravitational lensing effects around a spherically symmetric regular black hole, whose metric is derived from a non-singular collapsing dust ball model in asymptotically safe gravity. In this regular black hole spacetime, we obtain the analytical expression of the light deflection angle via calculating the strong field limit coefficients, and then evaluate the lensing observables in strong field regime by supposing the regular black holes as the candidate of $M87^*$ and $SgrA*$ supermassive black holes, respectively. In addition, we also in detail analyze the effects of the scale parameter $\xi$ on the strong deflection angle and the lensing observables. We expect our results will be useful in the future to distinguish these non-singular black holes from their classical singular counterparts., Comment: 14 pages, 22 figures, corrected some minor error fixes and added time delays to the content. Comments are welcome!

Superluminal propagation is an intrinsic problem in the diffusion equation and has not been effectively addressed for a long time. In this work, a rigorous solution to this issue is obtained under the assumption that particles undergo a random flight process, where they move isotropically at a constant speed while experiencing random scatterings. We validate this solution by comparing it with comprehensive simulations of the random flight process and find that it significantly deviates from the solution derived from the J\"{u}ttner propagator. This solution is broadly applicable to various diffusion phenomena, such as cosmic-ray propagation. We emphasize that our rigorous solution is particularly crucial in scenarios involving burst-like particle injection, where previous phenomenological approaches to the superluminal diffusion problem may not yield accurate results., Comment: 8 pages, 4 figures

Image style transfer has attracted widespread attention in the past few years. Despite its remarkable results, it requires additional style images available as references, making it less flexible and inconvenient. Using text is the most natural way to describe the style. More importantly, text can describe implicit abstract styles, like styles of specific artists or art movements. In this paper, we propose a Contrastive Learning for Artistic Style Transfer (CLAST) that leverages advanced image-text encoders to control arbitrary style transfer. We introduce a supervised contrastive training strategy to effectively extract style descriptions from the image-text model (i.e., CLIP), which aligns stylization with the text description. To this end, we also propose a novel and efficient adaLN based state space models that explore style-content fusion. Finally, we achieve a text-driven image style transfer. Extensive experiments demonstrate that our approach outperforms the state-of-the-art methods in artistic style transfer. More importantly, it does not require online fine-tuning and can render a 512x512 image in 0.03s., Comment: 18 pages, 8 figures. arXiv admin note: substantial text overlap with arXiv:2202.13562

Customized Image Generation, generating customized images with user-specified concepts, has raised significant attention due to its creativity and novelty. With impressive progress achieved in subject customization, some pioneer works further explored the customization of action and interaction beyond entity (i.e., human, animal, and object) appearance. However, these approaches only focus on basic actions and interactions between two entities, and their effects are limited by insufficient ''exactly same'' reference images. To extend customized image generation to more complex scenes for general real-world applications, we propose a new task: event-customized image generation. Given a single reference image, we define the ''event'' as all specific actions, poses, relations, or interactions between different entities in the scene. This task aims at accurately capturing the complex event and generating customized images with various target entities. To solve this task, we proposed a novel training-free event customization method: FreeEvent. Specifically, FreeEvent introduces two extra paths alongside the general diffusion denoising process: 1) Entity switching path: it applies cross-attention guidance and regulation for target entity generation. 2) Event transferring path: it injects the spatial feature and self-attention maps from the reference image to the target image for event generation. To further facilitate this new task, we collected two evaluation benchmarks: SWiG-Event and Real-Event. Extensive experiments and ablations have demonstrated the effectiveness of FreeEvent.

Multimodal Industrial Anomaly Detection (MIAD), utilizing 3D point clouds and 2D RGB images to identify the abnormal region of products, plays a crucial role in industrial quality inspection. However, the conventional MIAD setting presupposes that all 2D and 3D modalities are paired, overlooking the fact that multimodal data collected from the real world is often imperfect due to missing modalities. Consequently, MIAD models that demonstrate robustness against modal-incomplete data are highly desirable in practice. To address this practical challenge, we introduce a first-of-its-kind study that comprehensively investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD), to consider the imperfect learning environment in which the multimodal information may be incomplete. Not surprisingly, we discovered that most existing MIAD approaches are inadequate for addressing MIIAD challenges, leading to significant performance degradation on the MIIAD benchmark we developed. In this paper, we propose a novel two-stage Robust modAlity-imcomplete fusing and Detecting frAmewoRk, abbreviated as RADAR. Our bootstrapping philosophy is to enhance two stages in MIIAD, improving the robustness of the Multimodal Transformer: i) In feature fusion, we first explore learning modality-incomplete instruction, guiding the pre-trained Multimodal Transformer to robustly adapt to various modality-incomplete scenarios, and implement adaptive parameter learning based on a HyperNetwork; ii) In anomaly detection, we construct a real-pseudo hybrid module to highlight the distinctiveness of modality combinations, further enhancing the robustness of the MIIAD model. Our experimental results demonstrate that the proposed RADAR significantly surpasses conventional MIAD methods in terms of effectiveness and robustness on our newly created MIIAD dataset, underscoring its practical application value.

Recent advances in Vision-Language Models (VLMs) and the scarcity of high-quality multi-modal alignment data have inspired numerous researches on synthetic VLM data generation. The conventional norm in VLM data construction uses a mixture of specialists in caption and OCR, or stronger VLM APIs and expensive human annotation. In this paper, we present World to Code (W2C), a meticulously curated multi-modal data construction pipeline that organizes the final generation output into a Python code format. The pipeline leverages the VLM itself to extract cross-modal information via different prompts and filter the generated outputs again via a consistency filtering strategy. Experiments have demonstrated the high quality of W2C by improving various existing visual question answering and visual grounding benchmarks across different VLMs. Further analysis also demonstrates that the new code parsing ability of VLMs presents better cross-modal equivalence than the commonly used detail caption ability. Our code is available at https://github.com/foundation-multimodal-models/World2Code., Comment: Accepted at EMNLP 2024 Main Conference, 16pages

This paper reviews the challenge on Sparse Neural Rendering that was part of the Advances in Image Manipulation (AIM) workshop, held in conjunction with ECCV 2024. This manuscript focuses on the competition set-up, the proposed methods and their respective results. The challenge aims at producing novel camera view synthesis of diverse scenes from sparse image observations. It is composed of two tracks, with differing levels of sparsity; 3 views in Track 1 (very sparse) and 9 views in Track 2 (sparse). Participants are asked to optimise objective fidelity to the ground-truth images as measured via the Peak Signal-to-Noise Ratio (PSNR) metric. For both tracks, we use the newly introduced Sparse Rendering (SpaRe) dataset and the popular DTU MVS dataset. In this challenge, 5 teams submitted final results to Track 1 and 4 teams submitted final results to Track 2. The submitted models are varied and push the boundaries of the current state-of-the-art in sparse neural rendering. A detailed description of all models developed in the challenge is provided in this paper., Comment: Part of Advances in Image Manipulation workshop at ECCV 2024

In most existing multi-view modeling scenarios, cross-view correspondence (CVC) between instances of the same target from different views, like paired image-text data, is a crucial prerequisite for effortlessly deriving a consistent representation. Nevertheless, this premise is frequently compromised in certain applications, where each view is organized and transmitted independently, resulting in the view-unaligned problem (VuP). Restoring CVC of unaligned multi-view data is a challenging and highly demanding task that has received limited attention from the research community. To tackle this practical challenge, we propose to integrate the permutation derivation procedure into the bipartite graph paradigm for view-unaligned clustering, termed Probabilistically Aligned View-unaligned Clustering with Adaptive Template Selection (PAVuC-ATS). Specifically, we learn consistent anchors and view-specific graphs by the bipartite graph, and derive permutations applied to the unaligned graphs by reformulating the alignment between two latent representations as a 2-step transition of a Markov chain with adaptive template selection, thereby achieving the probabilistic alignment. The convergence of the resultant optimization problem is validated both experimentally and theoretically. Extensive experiments on six benchmark datasets demonstrate the superiority of the proposed PAVuC-ATS over the baseline methods., Comment: 12 pages, 6 figures

The rapid evolution of Multimodal Large Language Models (MLLMs) has brought substantial advancements in artificial intelligence, significantly enhancing the capability to understand and generate multimodal content. While prior studies have largely concentrated on model architectures and training methodologies, a thorough analysis of the benchmarks used for evaluating these models remains underexplored. This survey addresses this gap by systematically reviewing 211 benchmarks that assess MLLMs across four core domains: understanding, reasoning, generation, and application. We provide a detailed analysis of task designs, evaluation metrics, and dataset constructions, across diverse modalities. We hope that this survey will contribute to the ongoing advancement of MLLM research by offering a comprehensive overview of benchmarking practices and identifying promising directions for future work. An associated GitHub repository collecting the latest papers is available., Comment: Ongoing project

In this paper, we investigate the gravitational lensing effect and the shadow around a Schwarzschild-like black hole in metric-affine bumblebee gravity, which leads to the Lorentz symmetry breaking. We first present a generalized formalism for calculating higher-order corrections to light weak bending angle in a static, spherically symmetric and not asymptotically flat spacetime, and then applying this general formalism to the metric-affine bumblebee gravity. Moreover, we derive the light deflection angle and the size of the Einstein ring within the weak field in this scenario. In addition, we analyze the black hole shadow in this theory framework. By using observational data from the Einstein's ring of the galaxy ESO325-G004 and the black hole shadow of the ${\rm M}87$ galaxy, we estimate the upper bounds of the Lorentz symmetry breaking coefficient $\ell$, respectively., Comment: 12 pages, 4 figures, Accepted for publication in European Physical Journal C

The recently discovered stellar system, Ursa Major III/UNIONS 1, may be the faintest and densest dwarf spheroidal satellite galaxy of the Milky Way. Owing to its close proximity and substantial dark matter (DM) component, Ursa Major III emerges as a highly promising target for DM indirect detection. It is known that electrons and positrons originating from DM annihilation can generate a broad radio spectrum through the processes of synchrotron radiation and inverse Compton scattering within galaxies. In this study, we investigate the potential of the Square Kilometre Array (SKA) in detecting radio signatures arising from DM annihilation in Ursa Major III over a 100 hour observation period. Our analysis indicates that the SKA has strong capabilities in detecting these signatures. For instance, the SKA sensitivity to the DM annihilation cross section is estimated to reach $\mathcal{O}(10^{-30})-\mathcal{O}(10^{-28})\; \rm cm^{3} s^{-1}$ in the DM mass range from several GeV to $\sim100$ GeV for the $e^+e^-$ and $\mu^+\mu^-$ annihilation channels. The precise results are significantly influenced by various astrophysical factors, such as the strength of magnetic field, the diffusion coefficient, and the DM density profile in the dwarf galaxy. We discuss the impact of the uncertainties associated with these factors, and find that the SKA sensitivities have the potential to surpass the current constraints, even when considering these uncertainties., Comment: 20 pages, 6 figures

Extract-then-Abstract is a naturally coherent paradigm to conduct abstractive summarization with the help of salient information identified by the extractive model. Previous works that adopt this paradigm train the extractor and abstractor separately and introduce extra parameters to highlight the extracted salients to the abstractor, which results in error accumulation and additional training costs. In this paper, we first introduce a parameter-free highlight method into the encoder-decoder framework: replacing the encoder attention mask with a saliency mask in the cross-attention module to force the decoder to focus only on salient parts of the input. A preliminary analysis compares different highlight methods, demonstrating the effectiveness of our saliency mask. We further propose the novel extract-and-abstract paradigm, ExtAbs, which jointly and seamlessly performs Extractive and Abstractive summarization tasks within single encoder-decoder model to reduce error accumulation. In ExtAbs, the vanilla encoder is augmented to extract salients, and the vanilla decoder is modified with the proposed saliency mask to generate summaries. Built upon BART and PEGASUS, experiments on three datasets show that ExtAbs can achieve superior performance than baselines on the extractive task and performs comparable, or even better than the vanilla models on the abstractive task.

The recent AMS-02 measurements of cosmic-ray (CR) deuteron fluxes suggest the presence of primary deuterons in quantities far exceeding predictions from Big Bang nucleosynthesis. This poses a significant challenge to modern astrophysics, as no known processes can account for such large amounts of deuterons without violating existing constraints~\cite{Epstein:1976hq}. In contrast, it is recently proposed that the AMS-02 measurements can be explained by a purely secondary origin when contributions from heavier nuclei are considered. In this study, we recalculate the secondary deuteron flux using production cross sections updated with the latest collider data. We find that some of the deuteron production cross sections are overestimated in the widely-used calculation tools for CR propagation, and a primary deuteron component is still necessary. We then propose a novel process for generating primary deuterons at CR sources through a fusion mechanism, which is naturally unique to deuterons. This model could explain the observed deuteron excess while maintaining consistency with other CR measurements., Comment: 8 pages, 8 figures

In recent years, Multi-Modality Image Fusion (MMIF) has been applied to many fields, which has attracted many scholars to endeavour to improve the fusion performance. However, the prevailing focus has predominantly been on the architecture design, rather than the training strategies. As a low-level vision task, image fusion is supposed to quickly deliver output images for observation and supporting downstream tasks. Thus, superfluous computational and storage overheads should be avoided. In this work, a lightweight Distilled Mini-Model with a Dynamic Refresh strategy (MMDRFuse) is proposed to achieve this objective. To pursue model parsimony, an extremely small convolutional network with a total of 113 trainable parameters (0.44 KB) is obtained by three carefully designed supervisions. First, digestible distillation is constructed by emphasising external spatial feature consistency, delivering soft supervision with balanced details and saliency for the target network. Second, we develop a comprehensive loss to balance the pixel, gradient, and perception clues from the source images. Third, an innovative dynamic refresh training strategy is used to collaborate history parameters and current supervision during training, together with an adaptive adjust function to optimise the fusion network. Extensive experiments on several public datasets demonstrate that our method exhibits promising advantages in terms of model efficiency and complexity, with superior performance in multiple image fusion tasks and downstream pedestrian detection application. The code of this work is publicly available at https://github.com/yanglinDeng/MMDRFuse., Comment: 10 pages, 8 figures, accpeted by ACM International Conference on Multimedia 2024(Oral)

Recent advancements in the collection and analysis of sequential educational data have brought time series analysis to a pivotal position in educational research, highlighting its essential role in facilitating data-driven decision-making. However, there is a lack of comprehensive summaries that consolidate these advancements. To the best of our knowledge, this paper is the first to provide a comprehensive review of time series analysis techniques specifically within the educational context. We begin by exploring the landscape of educational data analytics, categorizing various data sources and types relevant to education. We then review four prominent time series methods-forecasting, classification, clustering, and anomaly detection-illustrating their specific application points in educational settings. Subsequently, we present a range of educational scenarios and applications, focusing on how these methods are employed to address diverse educational tasks, which highlights the practical integration of multiple time series methods to solve complex educational problems. Finally, we conclude with a discussion on future directions, including personalized learning analytics, multimodal data fusion, and the role of large language models (LLMs) in educational time series. The contributions of this paper include a detailed taxonomy of educational data, a synthesis of time series techniques with specific educational applications, and a forward-looking perspective on emerging trends and future research opportunities in educational analysis. The related papers and resources are available and regularly updated at the project page., Comment: 24 pages, 3 figures, 6 tables, project page: see https://github.com/ai-for-edu/time-series-analysis-for-education

Observations of the velocity dispersion indicate a severe dark matter (DM) deficit in the ultra-diffuse galaxy, NGC1052-DF4 (DF4). The ultra-deep images obtained with the Gemini telescope, which has the deepest imaging data till now, confirm the presence of tidal tails in DF4, suggesting its tidal formation. To enhance tidal effects, we consider the self-interaction among DM particles. Using an N-body simulation in the scenario of self-interacting dark matter (SIDM), we reproduce a DM-deficient galaxy that is consistent with all observational data of DF4. Specifically, our simulation result yields an extremely low DM-to-star mass ratio and a radial surface brightness profile very similar to that from deep images, showing accurate tidal features. By performing simulations with similar tidal effects and various cross-sections of SIDM, we show a significant impact of SIDM on the DM-to-star mass ratio in the central region of the galaxy. Our work confirms the tidal formation of DF4 in theory., Comment: 9 pages, 6 figures

The interaction between axion-like particles (ALPs) and photons induces ALP-photon oscillations in astrophysical magnetic fields, leading to spectral distortions in the $\gamma$-ray spectrum of blazars. The primary uncertainty of this phenomenon may originate from the magnetic field within the jet of the blazar. While many studies have explored the effects of ALP-photon oscillations using typical values for jet magnetic field parameters, it is important to recognize that these parameters can be constrained by multi-wavelength observations. In this study, we utilize the high energy $\gamma$-ray spectrum of Mrk 421 obtained from MAGIC and Fermi-LAT observations. By employing multi-wavelength fitting with a one-zone synchrotron self-Compton model, we derive the parameters characterizing the magnetic field model within the jet, and investigate their impacts on the ALP constraints., Comment: 9 pages, 7 figures

Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations. GCP typically performs classification of the covariance matrices by applying matrix function normalization, such as matrix logarithm or power, followed by a Euclidean classifier. However, covariance matrices inherently lie in a Riemannian manifold, known as the Symmetric Positive Definite (SPD) manifold. The current literature does not provide a satisfactory explanation of why Euclidean classifiers can be applied directly to Riemannian features after the normalization of the matrix power. To mitigate this gap, this paper provides a comprehensive and unified understanding of the matrix logarithm and power from a Riemannian geometry perspective. The underlying mechanism of matrix functions in GCP is interpreted from two perspectives: one based on tangent classifiers (Euclidean classifiers on the tangent space) and the other based on Riemannian classifiers. Via theoretical analysis and empirical validation through extensive experiments on fine-grained and large-scale visual classification datasets, we conclude that the working mechanism of the matrix functions should be attributed to the Riemannian classifiers they implicitly respect. The code is available at https://github.com/GitZH-Chen/RiemGCP.git., Comment: Accepted to ICLR 2025

Panoptic Scene Graph Generation (PSG) aims to generate a comprehensive graph-structure representation based on panoptic segmentation masks. Despite remarkable progress in PSG, almost all existing methods neglect the importance of shape-aware features, which inherently focus on the contours and boundaries of objects. To bridge this gap, we propose a model-agnostic Curricular shApe-aware FEature (CAFE) learning strategy for PSG. Specifically, we incorporate shape-aware features (i.e., mask features and boundary features) into PSG, moving beyond reliance solely on bbox features. Furthermore, drawing inspiration from human cognition, we propose to integrate shape-aware features in an easy-to-hard manner. To achieve this, we categorize the predicates into three groups based on cognition learning difficulty and correspondingly divide the training process into three stages. Each stage utilizes a specialized relation classifier to distinguish specific groups of predicates. As the learning difficulty of predicates increases, these classifiers are equipped with features of ascending complexity. We also incorporate knowledge distillation to retain knowledge acquired in earlier stages. Due to its model-agnostic nature, CAFE can be seamlessly incorporated into any PSG model. Extensive experiments and ablations on two PSG tasks under both robust and zero-shot PSG have attested to the superiority and robustness of our proposed CAFE, which outperforms existing state-of-the-art methods by a large margin., Comment: Accepted by IJCV

Compositional actions consist of dynamic (verbs) and static (objects) concepts. Humans can easily recognize unseen compositions using the learned concepts. For machines, solving such a problem requires a model to recognize unseen actions composed of previously observed verbs and objects, thus requiring so-called compositional generalization ability. To facilitate this research, we propose a novel Zero-Shot Compositional Action Recognition (ZS-CAR) task. For evaluating the task, we construct a new benchmark, Something-composition (Sth-com), based on the widely used Something-Something V2 dataset. We also propose a novel Component-to-Composition (C2C) learning method to solve the new ZS-CAR task. C2C includes an independent component learning module and a composition inference module. Last, we devise an enhanced training strategy to address the challenges of component variations between seen and unseen compositions and to handle the subtle balance between learning seen and unseen actions. The experimental results demonstrate that the proposed framework significantly surpasses the existing compositional generalization methods and sets a new state-of-the-art. The new Sth-com benchmark and code are available at https://github.com/RongchangLi/ZSCAR_C2C., Comment: Accepted by ECCV2024

The waveguide-integrated metasurface introduces a novel photonic chip capable of converting guided modes into free-space light. This enables functions such as off-chip beam focusing, steering, and imaging. The challenge lies in achieving hypermultiplexing across diverse parameters, including guided-wave mode type, direction, polarization, and notably, multiple wavelengths. Here, we introduce a comprehensive end-to-end inverse design framework, rooted in a physical model, for the multifunctional design of on-chip metasurfaces. This framework allows for metasurface optimization through a target-field-driven iteration process. We demonstrate a hypermultiplexed on-chip metasurface capable of generating red-green-blue holograms at multiple target planes, with both independent and cooperative control over guided-wave direction. Significantly, the proposed method streamlines the design process utilizing only the positions of meta-atoms as the design variable. We demonstrate 9 independent holographic channels through a combination of wavelength and distance multiplexing. Moreover, by incorporating the excitation direction into the design, the metasurface produces a total of 36 distinct holograms. The robustness of these results against fabrication discrepancies is validated through 3D full-wave electromagnetic simulations, aligning well with advanced manufacturing techniques. Our research presents a universal design framework for the development of multifunctional on-chip metasurfaces, opening up new avenues for a wide range of applications.

This paper presents two new metrics on the Symmetric Positive Definite (SPD) manifold via the Cholesky manifold, i.e., the space of lower triangular matrices with positive diagonal elements. We first unveil that the existing popular Riemannian metric on the Cholesky manifold can be generally characterized as the product metric of a Euclidean metric and a Riemannian metric on the space of n-dimensional positive vectors. Based on this analysis, we propose two novel metrics on the Cholesky manifolds, i.e., Diagonal Power Euclidean Metric and Diagonal Generalized Bures-Wasserstein Metric, which are numerically stabler than the existing Cholesky metric. We also discuss the gyro structures and deformed metrics associated with our metrics. The gyro structures connect the linear and geometric properties, while the deformed metrics interpolate between our proposed metrics and the existing metric. Further, by Cholesky decomposition, the proposed deformed metrics and gyro structures are pulled back to SPD manifolds. Compared with existing Riemannian metrics on SPD manifolds, our metrics are easy to use, computationally efficient, and numerically stable., Comment: 25 pages, 1 figures

The recently discovered satellite dwarf galaxy Ursa Major III provides a promising opportunity to explore the signatures resulting from dark matter (DM) annihilation, due to its proximity and large J-factor. Owing to the absence of an excess of $\gamma$-ray signatures originating from Ursa Major III, observations of $\gamma$-rays, such as those from Fermi-LAT, can be utilized to set constraints on the DM annihilation cross section. In this study, we determine the DM density profile, and consider the relationship between DM density and velocity dispersion at different locations within Ursa Major III through Jeans analysis. We calculate the J-factor of Ursa Major III for s-wave annihilation, along with the effective J-factors for p-wave and Sommerfeld enhanced annihilation scenarios. Utilizing these derived J-factors, we set stringent constraints on DM annihilation cross sections in three scenarios. Given the substantial impact of member star identification on the J-factor of Ursa Major III, we further calculate J-factors with the condition of excluding the largest velocity outlier. Our analysis reveals a notable reduction in the median value and an increase in the deviation of J-factors, thereby leading to considerably weaker constraints.

With the advance of diffusion models, today's video generation has achieved impressive quality. But generating temporal consistent long videos is still challenging. A majority of video diffusion models (VDMs) generate long videos in an autoregressive manner, i.e., generating subsequent clips conditioned on last frames of previous clip. However, existing approaches all involve bidirectional computations, which restricts the receptive context of each autoregression step, and results in the model lacking long-term dependencies. Inspired from the huge success of large language models (LLMs) and following GPT (generative pre-trained transformer), we bring causal (i.e., unidirectional) generation into VDMs, and use past frames as prompt to generate future frames. For Causal Generation, we introduce causal temporal attention into VDM, which forces each generated frame to depend on its previous frames. For Frame as Prompt, we inject the conditional frames by concatenating them with noisy frames (frames to be generated) along the temporal axis. Consequently, we present Video Diffusion GPT (ViD-GPT). Based on the two key designs, in each autoregression step, it is able to acquire long-term context from prompting frames concatenated by all previously generated frames. Additionally, we bring the kv-cache mechanism to VDMs, which eliminates the redundant computation from overlapped frames, significantly boosting the inference speed. Extensive experiments demonstrate that our ViD-GPT achieves state-of-the-art performance both quantitatively and qualitatively on long video generation. Code will be available at https://github.com/Dawn-LX/Causal-VideoGen., Comment: Code will be available at https://github.com/Dawn-LX/Causal-VideoGen

As virtual and augmented reality applications gain popularity, omnidirectional image (ODI) super-resolution has become increasingly important. Unlike 2D plain images that are formed on a plane, ODIs are projected onto spherical surfaces. Applying established image super-resolution methods to ODIs, therefore, requires performing equirectangular projection (ERP) to map the ODIs onto a plane. ODI super-resolution needs to take into account geometric distortion resulting from ERP. However, without considering such geometric distortion of ERP images, previous deep-learning-based methods only utilize a limited range of pixels and may easily miss self-similar textures for reconstruction. In this paper, we introduce a novel Geometric Distortion Guided Transformer for Omnidirectional image Super-Resolution (GDGT-OSR). Specifically, a distortion modulated rectangle-window self-attention mechanism, integrated with deformable self-attention, is proposed to better perceive the distortion and thus involve more self-similar textures. Distortion modulation is achieved through a newly devised distortion guidance generator that produces guidance by exploiting the variability of distortion across latitudes. Furthermore, we propose a dynamic feature aggregation scheme to adaptively fuse the features from different self-attention modules. We present extensive experimental results on public datasets and show that the new GDGT-OSR outperforms methods in existing literature., Comment: 13 pages, 12 figures, journal

Multimodal visual information fusion aims to integrate the multi-sensor data into a single image which contains more complementary information and less redundant features. However the complementary information is hard to extract, especially for infrared and visible images which contain big similarity gap between these two modalities. The common cross attention modules only consider the correlation, on the contrary, image fusion tasks need focus on complementarity (uncorrelation). Hence, in this paper, a novel cross attention mechanism (CAM) is proposed to enhance the complementary information. Furthermore, a two-stage training strategy based fusion scheme is presented to generate the fused images. For the first stage, two auto-encoder networks with same architecture are trained for each modality. Then, with the fixed encoders, the CAM and a decoder are trained in the second stage. With the trained CAM, features extracted from two modalities are integrated into one fused feature in which the complementary information is enhanced and the redundant features are reduced. Finally, the fused image can be generated by the trained decoder. The experimental results illustrate that our proposed fusion method obtains the SOTA fusion performance compared with the existing fusion networks. The codes are available at https://github.com/hli1221/CrossFuse, Comment: 11 pages, 16 fuigures

For better explore the relations of inter-modal and inner-modal, even in deep learning fusion framework, the concept of decomposition plays a crucial role. However, the previous decomposition strategies (base \& detail or low-frequency \& high-frequency) are too rough to present the common features and the unique features of source modalities, which leads to a decline in the quality of the fused images. The existing strategies treat these relations as a binary system, which may not be suitable for the complex generation task (e.g. image fusion). To address this issue, a continuous decomposition-based fusion framework (Conti-Fuse) is proposed. Conti-Fuse treats the decomposition results as few samples along the feature variation trajectory of the source images, extending this concept to a more general state to achieve continuous decomposition. This novel continuous decomposition strategy enhances the representation of complementary information of inter-modal by increasing the number of decomposition samples, thus reducing the loss of critical information. To facilitate this process, the continuous decomposition module (CDM) is introduced to decompose the input into a series continuous components. The core module of CDM, State Transformer (ST), is utilized to efficiently capture the complementary information from source modalities. Furthermore, a novel decomposition loss function is also designed which ensures the smooth progression of the decomposition process while maintaining linear growth in time complexity with respect to the number of decomposition samples. Extensive experiments demonstrate that our proposed Conti-Fuse achieves superior performance compared to the state-of-the-art fusion methods.