Search

As one of the tasks in Image Fusion, Infrared and Visible Image Fusion aims to integrate complementary information captured by sensors of different modalities into a single image. The Selective State Space Model (SSSM), known for its ability to capture long-range dependencies, has demonstrated its potential in the field of computer vision. However, in image fusion, current methods underestimate the potential of SSSM in capturing the global spatial information of both modalities. This limitation prevents the simultaneous consideration of the global spatial information from both modalities during interaction, leading to a lack of comprehensive perception of salient targets. Consequently, the fusion results tend to bias towards one modality instead of adaptively preserving salient targets. To address this issue, we propose the Saliency-aware Selective State Space Fusion Model (S4Fusion). In our S4Fusion, the designed Cross-Modal Spatial Awareness Module (CMSA) can simultaneously focus on global spatial information from both modalities while facilitating their interaction, thereby comprehensively capturing complementary information. Additionally, S4Fusion leverages a pre-trained network to perceive uncertainty in the fused images. By minimizing this uncertainty, S4Fusion adaptively highlights salient targets from both images. Extensive experiments demonstrate that our approach produces high-quality images and enhances performance in downstream tasks.

Generative models are widely utilized to model the distribution of fused images in the field of infrared and visible image fusion. However, current generative models based fusion methods often suffer from unstable training and slow inference speed. To tackle this problem, a novel fusion method based on consistency model is proposed, termed as CoMoFusion, which can generate the high-quality images and achieve fast image inference speed. In specific, the consistency model is used to construct multi-modal joint features in the latent space with the forward and reverse process. Then, the infrared and visible features extracted by the trained consistency model are fed into fusion module to generate the final fused image. In order to enhance the texture and salient information of fused images, a novel loss based on pixel value selection is also designed. Extensive experiments on public datasets illustrate that our method obtains the SOTA fusion performance compared with the existing fusion methods.

We consider an embedded general complex torus $C_n$ into a complex manifold $M_{n+d}$ with a unitary flat normal bundle $N_C$. We show the existence of (non-singular) holomorphic foliation in a neighborhood of $C$ in $M$ having $C$ as leaf under some conditions.

Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}, fostering further research and collaboration in this domain., Comment: Taiyi-Diffusion-XL Tech Report

In a scenario where multi-modal cameras are operating together, the problem of working with non-aligned images cannot be avoided. Yet, existing image fusion algorithms rely heavily on strictly registered input image pairs to produce more precise fusion results, as a way to improve the performance of downstream high-level vision tasks. In order to relax this assumption, one can attempt to register images first. However, the existing methods for registering multiple modalities have limitations, such as complex structures and reliance on significant semantic information. This paper aims to address the problem of image registration and fusion in a single framework, called BusRef. We focus on Infrared-Visible image registration and fusion task (IVRF). In this framework, the input unaligned image pairs will pass through three stages: Coarse registration, Fine registration and Fusion. It will be shown that the unified approach enables more robust IVRF. We also propose a novel training and evaluation strategy, involving the use of masks to reduce the influence of non-reconstructible regions on the loss functions, which greatly improves the accuracy and robustness of the fusion task. Last but not least, a gradient-aware fusion network is designed to preserve the complementary information. The advanced performance of this algorithm is demonstrated by

With the open-sourcing of text-to-image models (T2I) such as stable diffusion (SD) and stable diffusion XL (SD-XL), there is an influx of models fine-tuned in specific domains based on the open-source SD model, such as in anime, character portraits, etc. However, there are few specialized models in certain domains, such as interior design, which is attributed to the complex textual descriptions and detailed visual elements inherent in design, alongside the necessity for adaptable resolution. Therefore, text-to-image models for interior design are required to have outstanding prompt-following capabilities, as well as iterative collaboration with design professionals to achieve the desired outcome. In this paper, we collect and optimize text-image data in the design field and continue training in both English and Chinese on the basis of the open-source CLIP model. We also proposed a fine-tuning strategy with curriculum learning and reinforcement learning from CLIP feedback to enhance the prompt-following capabilities of our approach so as to improve the quality of image generation. The experimental results on the collected dataset demonstrate the effectiveness of the proposed approach, which achieves impressive results and outperforms strong baselines.

3D object detection in Bird's-Eye-View (BEV) space has recently emerged as a prevalent approach in the field of autonomous driving. Despite the demonstrated improvements in accuracy and velocity estimation compared to perspective view methods, the deployment of BEV-based techniques in real-world autonomous vehicles remains challenging. This is primarily due to their reliance on vision-transformer (ViT) based architectures, which introduce quadratic complexity with respect to the input resolution. To address this issue, we propose an efficient BEV-based 3D detection framework called BEVENet, which leverages a convolutional-only architectural design to circumvent the limitations of ViT models while maintaining the effectiveness of BEV-based methods. Our experiments show that BEVENet is 3$\times$ faster than contemporary state-of-the-art (SOTA) approaches on the NuScenes challenge, achieving a mean average precision (mAP) of 0.456 and a nuScenes detection score (NDS) of 0.555 on the NuScenes validation dataset, with an inference speed of 47.6 frames per second. To the best of our knowledge, this study stands as the first to achieve such significant efficiency improvements for BEV-based methods, highlighting their enhanced feasibility for real-world autonomous driving applications.

Various large language models (LLMs) have been proposed in recent years, including closed- and open-source ones, continually setting new records on multiple benchmarks. However, the development of LLMs still faces several issues, such as high cost of training models from scratch, and continual pre-training leading to catastrophic forgetting, etc. Although many such issues are addressed along the line of research on LLMs, an important yet practical limitation is that many studies overly pursue enlarging model sizes without comprehensively analyzing and optimizing the use of pre-training data in their learning process, as well as appropriate organization and leveraging of such data in training LLMs under cost-effective settings. In this work, we propose Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation model, and further pre-trained on 700 billion tokens, where we focus on pre-training techniques and use data-centric optimization to enhance the learning process of Ziya2 on different stages. We define three data attributes and firstly establish data-centric scaling laws to illustrate how different data impacts LLMs. Experiments show that Ziya2 significantly outperforms other models in multiple benchmarks especially with promising results compared to representative open-source ones. Ziya2 (Base) is released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.

Recent advancements enlarge the capabilities of large language models (LLMs) in zero-shot image-to-text generation and understanding by integrating multi-modal inputs. However, such success is typically limited to English scenarios due to the lack of large-scale and high-quality non-English multi-modal resources, making it extremely difficult to establish competitive counterparts in other languages. In this paper, we introduce the Ziya-Visual series, a set of bilingual large-scale vision-language models (LVLMs) designed to incorporate visual semantics into LLM for multi-modal dialogue. Composed of Ziya-Visual-Base and Ziya-Visual-Chat, our models adopt the Querying Transformer from BLIP-2, further exploring the assistance of optimization schemes such as instruction tuning, multi-stage training and low-rank adaptation module for visual-language alignment. In addition, we stimulate the understanding ability of GPT-4 in multi-modal scenarios, translating our gathered English image-text datasets into Chinese and generating instruction-response through the in-context learning method. The experiment results demonstrate that compared to the existing LVLMs, Ziya-Visual achieves competitive performance across a wide range of English-only tasks including zero-shot image-text retrieval, image captioning, and visual question answering. The evaluation leaderboard accessed by GPT-4 also indicates that our models possess satisfactory image-text understanding and generation capabilities in Chinese multi-modal scenario dialogues. Code, demo and models are available at ~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}.

We study the structure of the Albanese map for K\"ahler manifolds with nef anticanonical bundle. First, we give a result for fourfolds whose Albanse torus is an elliptic curve. In the general case of any dimension, we look at two cases: The general fiber of the Albanese map is a Calabi-Yau manifold or a projective space. In the first case, we show that the manifold itself must be Calabi-Yau. In the second case, we give a more topological proof of a result by Cao and H\"oring which says that the manifold must be a projectivization of a numerically flat vector bundle., Comment: 25 pages; we added a section about the uniruled case when fibers are surfaces

The development of two-dimensional (2D) room temperature magnets is of great significance to the practical application of spintronic devices. However, the number of synthesized intrinsic 2D magnets is limited and the performances of them are not satisfactory, e.g. typically with low Curie temperature and poor environmental stability. Magnetic modulation based on developed 2D materials, especially non-magnetic 2D materials, can bring us new breakthroughs. Herein, we report room temperature ferromagnetism in halogenated MoS2 monolayer under the synergistic effect of strain engineering and charge injection, and the combined implementation of these two processes is based on the halogenation of MoS2. The adsorbed halogen atoms X (X = F, Cl, and Br) on the surface leads to lattice superstretching and hole injection, resulting in MoS2 monolayer exhibiting half-metallic properties, with one spin channel being gapless in the band structure. The Curie temperature of halogenated MoS2 monolayer is 513~615 K, which is much higher than the room temperature. In addition, large magnetic anisotropy energy and good environmental stability make halogenated MoS2 display great advantages in practical spintronic nanodevices.

In the field of remote sensing, we often utilize oriented bounding boxes (OBB) to bound the objects. This approach significantly reduces the overlap among dense detection boxes and minimizes the inclusion of background content within the bounding boxes. To enhance the detection accuracy of oriented objects, we propose a unique loss function based on edge gradients, inspired by the similarity measurement function used in template matching task. During this process, we address the issues of non-differentiability of the function and the semantic alignment between gradient vectors in ground truth (GT) boxes and predicted boxes (PB). Experimental results show that our proposed loss function achieves $0.6\%$ mAP improvement compared to the commonly used Smooth L1 loss in the baseline algorithm. Additionally, we design an edge-based self-attention module to encourage the detection network to focus more on the object edges. Leveraging these two innovations, we achieve a mAP increase of 1.3% on the DOTA dataset., Comment: 9 pages, 8 figures, 1 algorithm

The ability to generate, detect, and control coherent terahertz (THz) spin currents with femtosecond temporal and nanoscale spatial resolution has significant ramifications. The diffraction limit of concentrated THz radiation, which has a wavelength range of 5 {\mu}m-1.5 mm, has impeded the accumulation of nanodomain data of magnetic structures and spintronic dynamics despite its potential benefits. Contemporary spintronic optoelectronic apparatuses with dimensions 100 nm presented a challenge for researchers due to this restriction. In this study, we demonstrate the use of spintronic THz emission nanoscopy (STEN), which allows for the efficient injection and precise coherent detection of ultrafast THz spin currents at the nanoscale. Furthermore, STEN is an effective method that does not require invasion for characterising and etching nanoscale spintronic heterostructures. The cohesive integration of nanophotonics, nanospintronics, and THz-nano technology into a single platform is poised to accelerate the development of high-frequency spintronic optoelectronic nanodevices and their revolutionary technical applications.

Deep neural networks for learning Symmetric Positive Definite (SPD) matrices are gaining increasing attention in machine learning. Despite the significant progress, most existing SPD networks use traditional Euclidean classifiers on an approximated space rather than intrinsic classifiers that accurately capture the geometry of SPD manifolds. Inspired by Hyperbolic Neural Networks (HNNs), we propose Riemannian Multinomial Logistics Regression (RMLR) for the classification layers in SPD networks. We introduce a unified framework for building Riemannian classifiers under the metrics pulled back from the Euclidean space, and showcase our framework under the parameterized Log-Euclidean Metric (LEM) and Log-Cholesky Metric (LCM). Besides, our framework offers a novel intrinsic explanation for the most popular LogEig classifier in existing SPD networks. The effectiveness of our method is demonstrated in three applications: radar recognition, human action recognition, and electroencephalography (EEG) classification. The code is available at https://github.com/GitZH-Chen/SPDMLR.git., Comment: Accepted to CVPR 2024

In this article we pursue the following main goals. In the first place, we establish the existence of "estimable" Hermite--Einstein metrics for stable reflexive coherent sheaves on compact normal K\"ahler spaces. If moreover the background variety has klt singularities, we obtain a much more precise result., Comment: A few more references and comments added

In this paper, we prove that a non-projective compact K\"ahler $3$-fold with nef anti-canonical bundle is, up to a finite \'etale cover, one of the following: a manifold with vanishing first Chern class; the product of a K3 surface and the projective line; the projective space bundle of a numerically flat vector bundle over a torus. This result extends Cao-H\"oring's structure theorem for projective manifolds to compact K\"ahler manifolds in dimension three. For the proof, we study the Minimal Model Program for compact K\"ahler $3$-folds with nef anti-canonical bundle by using the positivity of direct image sheaves, $\mathbb{Q}$-conic bundles, and orbifold vector bundles., Comment: 33 pages; comments are welcome

In this paper, we study compact complex orbifolds. In the first part, we shows the equivalence of two notions of compact K\"ahler orbifold. In the second part, we shows various versions of Demailly's regularisation theorems for compact orbifold and study the positivity of orbifold vector bundle. In the last section, we give a version of equivariant GAGA communicated to us by Brion., Comment: 30 pages

Recently, prompt-based learning has gained popularity across many natural language processing (NLP) tasks by reformulating them into a cloze-style format to better align pre-trained language models (PLMs) with downstream tasks. However, applying this approach to relation classification poses unique challenges. Specifically, associating natural language words that fill the masked token with semantic relation labels (\textit{e.g.} \textit{``org:founded\_by}'') is difficult. To address this challenge, this paper presents a novel prompt-based learning method, namely LabelPrompt, for the relation classification task. Motivated by the intuition to ``GIVE MODEL CHOICES!'', we first define additional tokens to represent relation labels, which regard these tokens as the verbaliser with semantic initialisation and explicitly construct them with a prompt template method. Then, to mitigate inconsistency between predicted relations and given entities, we implement an entity-aware module with contrastive learning. Last, we conduct an attention query strategy within the self-attention layer to differentiates prompt tokens and sequence tokens. Together, these strategies enhance the adaptability of prompt-based learning, especially when only small labelled datasets is available. Comprehensive experiments on benchmark datasets demonstrate the superiority of our method, particularly in the few-shot scenario., Comment: 20 pages, 5 figures

Let $(X, \omega)$ be an n-dimensional compact K\"ahler manifold. Let $D=\sum (1-\beta_j) Y_j=\sum (1-\beta_j) [s_j=0]$ a divisor with simple normal crossings with $\beta_j \in ]0,1[$ such that $-(K_X+D)$ is nef. We show that its Albanese map is submersion outside an analytic set of codimension larger than two with connected fibres., Comment: 11 pages

The coherent many-body interaction at the organic-inorganic interface can give rise to intriguing hybrid excitons that combine the advantages of the Wannier-Mott and Frenkel excitons simultaneously. Unlike the 2D inorganic heterostructures that suffer from moment mismatch, the hybrid excitons formed at the organic-inorganic interface have a momentum-direct nature, which have yet to be explored. Here, we report hybrid excitons at the copper phthalocyanine/molybdenum diselenide (CuPc/MoSe2) interface with strong molecular orientation dependence using low-temperature photoluminescence spectroscopy. The new emission peaks observed in the CuPc/MoSe2 heterostructure indicate the formation of interfacial hybrid excitons. The density functional theory (DFT) calculation confirms the strong hybridization between the lowest unoccupied molecular orbital (LUMO) of CuPc and the conduction band minimum (CBM) of MoSe2, suggesting that the hybrid excitons consist of electrons extended in both layers and holes confined in individual layers. The temperature-dependent measurements show that the hybrid excitons can gain the signatures of the Frenkel excitons of CuPc and the Wannier-Mott excitons of MoSe2 simultaneously. The out-of-plane molecular orientation is used to tailor the interfacial hybrid exciton states. Our results reveal the hybrid excitons at the CuPc/MoSe2 interface with tunability by molecular orientation, which suggests that the emerging organic-inorganic heterostructure can be a promising platform for many-body exciton physics.

The task of Compositional Zero-Shot Learning (CZSL) is to recognize images of novel state-object compositions that are absent during the training stage. Previous methods of learning compositional embedding have shown effectiveness in closed-world CZSL. However, in Open-World CZSL (OW-CZSL), their performance tends to degrade significantly due to the large cardinality of possible compositions. Some recent works separately predict simple primitives (i.e., states and objects) to reduce cardinality. However, they consider simple primitives as independent probability distributions, ignoring the heavy dependence between states, objects, and compositions. In this paper, we model the dependence of compositions via feasibility and contextuality. Feasibility-dependence refers to the unequal feasibility relations between simple primitives, e.g., \textit{hairy} is more feasible with \textit{cat} than with \textit{building} in the real world. Contextuality-dependence represents the contextual variance in images, e.g., \textit{cat} shows diverse appearances under the state of \textit{dry} and \textit{wet}. We design Semantic Attention (SA) and generative Knowledge Disentanglement (KD) to learn the dependence of feasibility and contextuality, respectively. SA captures semantics in compositions to alleviate impossible predictions, driven by the visual similarity between simple primitives. KD disentangles images into unbiased feature representations, easing contextual bias in predictions. Moreover, we complement the current compositional probability model with feasibility and contextuality in a compatible format. Finally, we conduct comprehensive experiments to analyze and validate the superior or competitive performance of our model, Semantic Attention and knowledge Disentanglement guided Simple Primitives (SAD-SP), on three widely-used benchmark OW-CZSL datasets.

Nowadays, foundation models become one of fundamental infrastructures in artificial intelligence, paving ways to the general intelligence. However, the reality presents two urgent challenges: existing foundation models are dominated by the English-language community; users are often given limited resources and thus cannot always use foundation models. To support the development of the Chinese-language community, we introduce an open-source project, called Fengshenbang, which leads by the research center for Cognitive Computing and Natural Language (CCNL). Our project has comprehensive capabilities, including large pre-trained models, user-friendly APIs, benchmarks, datasets, and others. We wrap all these in three sub-projects: the Fengshenbang Model, the Fengshen Framework, and the Fengshen Benchmark. An open-source roadmap, Fengshenbang, aims to re-evaluate the open-source community of Chinese pre-trained large-scale models, prompting the development of the entire Chinese large-scale model community. We also want to build a user-centered open-source ecosystem to allow individuals to access the desired models to match their computing resources. Furthermore, we invite companies, colleges, and research institutions to collaborate with us to build the large-scale open-source model-based ecosystem. We hope that this project will be the foundation of Chinese cognitive intelligence., Comment: Added the Chinese version and is now a bilingual paper

The theory of the Oukounkov body is a useful tool for studying the asymptotic behaviour of the canonical ring of a line bundle over a projective manifold. In this note, combined with the algebraic reduction, we study the asymptotic behaviour of the canonical ring of a line bundle over an arbitrary compact normal irreducible complex space., Comment: 7 pages

In this note, we show various versions of holomorphic Morse inequalities tensoring with a coherent sheaf., Comment: 10 pages

Excitonic insulators are long-sought-after quantum materials predicted to spontaneously open a gap by the Bose condensation of bound electron-hole pairs, namely, excitons, in their ground state. Since the theoretical conjecture, extensive efforts have been devoted to pursuing excitonic insulator platforms for exploring macroscopic quantum phenomena in real materials. Reliable evidences of excitonic character have been obtained in layered chalcogenides as promising candidates. However, owing to the interference of intrinsic lattice instabilities, it is still debatable whether those features, such as charge density wave and gap opening, are primarily driven by the excitonic effect or by the lattice transition. Herein, we develop a novel charge-transfer excitonic insulator in organic-inorganic superlattice interfaces, which serves as an ideal platform to decouple the excitonic effect from the lattice effect. In this system, we observe the narrow gap opening and the formation of a charge density wave without periodic lattice distortion, providing visualized evidence of exciton condensation occurring in thermal equilibrium. Our findings identify spontaneous interfacial charge transfer as a new strategy for developing novel excitonic insulators and investigating their correlated many-body physics.

Designing anode materials with high lithium specific capacity is crucial to the development of high energy-density lithium ion batteries. Herein, a distinctive lithium growth mechanism, namely, the restricted multilayered growth for lithium, and a strategy for lithium storage are proposed to achieve the balance between the ultra-high specific capacity and the need to avert uncontrolled dendritic growth of lithium. In particular, based on first-principles computation, we show that the Al2C monolayer with planar tetracoordinate carbon structure can be an ideal platform for realizing the restricted multilayered growth mechanism as a 2D anode material. Furthermore, the Al2C monolayer exhibits ultra-high specific capacity of lithium of 4059 mAh/g, yet with a low dif-fusion barrier of 0.039-0.17 eV as well as low open circuit voltage in the range of 0.002-0.34 V. These novel properties endow the Al2C monolayer a promising anode material for future lithium ion batteries. Our study offers a new way to design promising 2D anode materials with high specific capacity, fast lithium-ion diffusion, and safe lithium storage mechanism.