Search

We have used high-resolution zoom-in simulations of direct collapse to supermassive black hole (SMBH) seeds within dark mater (DM) halos in the presence of magnetic fields generated during the collapse, down to $10^{-5}$ pc or 2 AU. We confirm an efficient amplification of magnetic field during collapse, formation of geometrically-thick self-gravitating accretion disk inside 0.1 pc, and damping of fragmentation in the disk by the field. This disk differs profoundly from SMBH accretion disks. We find that (1) accretion disk is subject to the magneto-rotational instability which further amplifies the field to near-equipartition. No artificial seeding of the disk field has been used; (2) Equipartition toroidal field changes its polarity in the mid-plane; (3) The nonlinear Parker instability develops, accompanied by the vertical buckling of the field lines which injects material above the disk, leading to an increase in the disk scaleheight; (4) With the Coriolis force producing a coherent helicity above the disk, vertical poloidal field has been generated and amplified; (5) We estimate that the associated outflow will be most probably squashed by accretion. The resulting configuration consists of a magnetized disk with $\beta \geq 0.1$ and its magnetosphere with $\beta << 1$, where $\beta = P_{\rm th}/P_{\rm B}$ is the ratio of thermal to magnetic energy density; (6) The disk is highly variable due to feeding by variable accretion flow, and strong vortical motions are present. (7) Finally, the negative gradient of the total vertical stress drives an equatorial outflow sandwiched by an inward accretion flow., Comment: 17 pages, 15 figures, accepted for publication in ApJ

The emergence of text-to-image generation models has led to the recognition that image enhancement, performed as post-processing, would significantly improve the visual quality of the generated images. Exploring diffusion models to enhance the generated images nevertheless is not trivial and necessitates to delicately enrich plentiful details while preserving the visual appearance of key content in the original image. In this paper, we propose a novel framework, namely FreeEnhance, for content-consistent image enhancement using the off-the-shelf image diffusion models. Technically, FreeEnhance is a two-stage process that firstly adds random noise to the input image and then capitalizes on a pre-trained image diffusion model (i.e., Latent Diffusion Models) to denoise and enhance the image details. In the noising stage, FreeEnhance is devised to add lighter noise to the region with higher frequency to preserve the high-frequent patterns (e.g., edge, corner) in the original image. In the denoising stage, we present three target properties as constraints to regularize the predicted noise, enhancing images with high acutance and high visual quality. Extensive experiments conducted on the HPDv2 dataset demonstrate that our FreeEnhance outperforms the state-of-the-art image enhancement models in terms of quantitative metrics and human preference. More remarkably, FreeEnhance also shows higher human preference compared to the commercial image enhancement solution of Magnific AI., Comment: ACM Multimedia 2024

In multi-label classification, machine learning encounters the challenge of domain generalization when handling tasks with distributions differing from the training data. Existing approaches primarily focus on vision object recognition and neglect the integration of natural language. Recent advancements in vision-language pre-training leverage supervision from extensive visual-language pairs, enabling learning across diverse domains and enhancing recognition in multi-modal scenarios. However, these approaches face limitations in loss function utilization, generality across backbones, and class-aware visual fusion. This paper proposes solutions to these limitations by inferring the actual loss, broadening evaluations to larger vision-language backbones, and introducing Mixup-CLIPood, which incorporates a novel mix-up loss for enhanced class-aware visual fusion. Our method demonstrates superior performance in domain generalization across multiple datasets., Comment: 6 pages, 2 figures. This is a preprint version of the article. The final version will be published in the proceedings of the IEEE conference

The surging demand for cloud computing resources, driven by the rapid growth of sophisticated large-scale models and data centers, underscores the critical importance of efficient and adaptive resource allocation. As major tech enterprises deploy massive infrastructures with thousands of GPUs, existing cloud platforms still struggle with low resource utilization due to key challenges: capturing hierarchical indicator structures, modeling non-Gaussian distributions, and decision-making under uncertainty. To address these challenges, we propose HRAMONY, an adaptive Hierarchical Attention-based Resource Modeling and Decision-Making System. HARMONY combines hierarchical multi-indicator distribution forecasting and uncertainty-aware Bayesian decision-making. It introduces a novel hierarchical attention mechanism that comprehensively models complex inter-indicator dependencies, enabling accurate predictions that can adapt to evolving environment states. By transforming Gaussian projections into adaptive non-Gaussian distributions via Normalizing Flows. Crucially, HARMONY leverages the full predictive distributions in an adaptive Bayesian process, proactively incorporating uncertainties to optimize resource allocation while robustly meeting SLA constraints under varying conditions. Extensive evaluations across four large-scale cloud datasets demonstrate HARMONY's state-of-the-art performance, significantly outperforming nine established methods. A month-long real-world deployment validated HARMONY's substantial practical impact, realizing over 35,000 GPU hours in savings and translating to $100K+ in cost reduction, showcasing its remarkable economic value through adaptive, uncertainty-aware scaling. Our code is available at https://github.com/Floating-LY/HARMONY1.

In order to develop robots that can effectively serve as versatile and capable home assistants, it is crucial for them to reliably perceive and interact with a wide variety of objects across diverse environments. To this end, we proposed Open Vocabulary Mobile Manipulation as a key benchmark task for robotics: finding any object in a novel environment and placing it on any receptacle surface within that environment. We organized a NeurIPS 2023 competition featuring both simulation and real-world components to evaluate solutions to this task. Our baselines on the most challenging version of this task, using real perception in simulation, achieved only an 0.8% success rate; by the end of the competition, the best participants achieved an 10.8\% success rate, a 13x improvement. We observed that the most successful teams employed a variety of methods, yet two common threads emerged among the best solutions: enhancing error detection and recovery, and improving the integration of perception with decision-making processes. In this paper, we detail the results and methodologies used, both in simulation and real-world settings. We discuss the lessons learned and their implications for future research. Additionally, we compare performance in real and simulated environments, emphasizing the necessity for robust generalization to novel settings.

Multimodal learning robust to missing modality has attracted increasing attention due to its practicality. Existing methods tend to address it by learning a common subspace representation for different modality combinations. However, we reveal that they are sub-optimal due to their implicit constraint on intra-class representation. Specifically, the sample with different modalities within the same class will be forced to learn representations in the same direction. This hinders the model from capturing modality-specific information, resulting in insufficient learning. To this end, we propose a novel Decoupled Multimodal Representation Network (DMRNet) to assist robust multimodal learning. Specifically, DMRNet models the input from different modality combinations as a probabilistic distribution instead of a fixed point in the latent space, and samples embeddings from the distribution for the prediction module to calculate the task loss. As a result, the direction constraint from the loss minimization is blocked by the sampled representation. This relaxes the constraint on the inference representation and enables the model to capture the specific information for different modality combinations. Furthermore, we introduce a hard combination regularizer to prevent DMRNet from unbalanced training by guiding it to pay more attention to hard modality combinations. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that the proposed DMRNet outperforms the state-of-the-art significantly., Comment: ECCV2024 17 pages

Kagome metal CsV$_3$Sb$_5$ has attracted much recent attention due to the coexistence of multiple exotic orders and the associated proposals to mimic unconventional high temperature superconductors. Nevertheless, magnetism and strong electronic correlations -- two essential ingredients for unconventional superconductivity, are absent in this V-based Kagome metal. CsCr$_3$Sb$_5$ is a newly discovered Cr-based parallel of CsV$_3$Sb$_5$, in which magnetism appears with charge density wave and superconductivity at different temperature and pressure regions. Enhanced electronic correlations are also suggested by theoretical proposals due to the calculated flat bands. Here, we report angle-resolved photoemission measurements and first-principles calculations on this new material system. Electron energy bands and the associated orbitals are resolved. Flat bands are observed near the Fermi level. Doping dependent measurements on Cs(Cr$_x$V$_{1-x}$)$_3$Sb$_5$ reveal a gradually enhanced band renormalization from CsV$_3$Sb$_5$ to CsCr$_3$Sb$_5$, accompanied by distinct spatial symmetry breaking states in the phase diagram.

Deep State Space Models (SSMs) have proven effective in numerous task scenarios but face significant security challenges due to Adversarial Perturbations (APs) in real-world deployments. Adversarial Training (AT) is a mainstream approach to enhancing Adversarial Robustness (AR) and has been validated on various traditional DNN architectures. However, its effectiveness in improving the AR of SSMs remains unclear. While many enhancements in SSM components, such as integrating Attention mechanisms and expanding to data-dependent SSM parameterizations, have brought significant gains in Standard Training (ST) settings, their potential benefits in AT remain unexplored. To investigate this, we evaluate existing structural variants of SSMs with AT to assess their AR performance. We observe that pure SSM structures struggle to benefit from AT, whereas incorporating Attention yields a markedly better trade-off between robustness and generalization for SSMs in AT compared to other components. Nonetheless, the integration of Attention also leads to Robust Overfitting (RO) issues. To understand these phenomena, we empirically and theoretically analyze the output error of SSMs under AP. We find that fixed-parameterized SSMs have output error bounds strictly related to their parameters, limiting their AT benefits, while input-dependent SSMs may face the problem of error explosion. Furthermore, we show that the Attention component effectively scales the output error of SSMs during training, enabling them to benefit more from AT, but at the cost of introducing RO due to its high model complexity. Inspired by this, we propose a simple and effective Adaptive Scaling (AdS) mechanism that brings AT performance close to Attention-integrated SSMs without introducing the issue of RO.

The increase in parameter size of multimodal large language models (MLLMs) introduces significant capabilities, particularly in-context learning, where MLLMs enhance task performance without updating pre-trained parameters. This effectiveness, however, hinges on the appropriate selection of in-context examples, a process that is currently biased towards visual data, overlooking textual information. Furthermore, the area of supervised retrievers for MLLMs, crucial for optimal in-context example selection, continues to be uninvestigated. Our study offers an in-depth evaluation of the impact of textual information on the unsupervised selection of in-context examples in multimodal contexts, uncovering a notable sensitivity of retriever performance to the employed modalities. Responding to this, we introduce a novel supervised MLLM-retriever MSIER that employs a neural network to select examples that enhance multimodal in-context learning efficiency. This approach is validated through extensive testing across three distinct tasks, demonstrating the method's effectiveness. Additionally, we investigate the influence of modalities on our supervised retrieval method's training and pinpoint factors contributing to our model's success. This exploration paves the way for future advancements, highlighting the potential for refined in-context learning in MLLMs through the strategic use of multimodal data.

Images suffer from heavy spatial redundancy because pixels in neighboring regions are spatially correlated. Existing approaches strive to overcome this limitation by reducing less meaningful image regions. However, current leading methods rely on supervisory signals. They may compel models to preserve content that aligns with labeled categories and discard content belonging to unlabeled categories. This categorical inductive bias makes these methods less effective in real-world scenarios. To address this issue, we propose a self-supervised framework for image redundancy reduction called Learning to Rank Patches (LTRP). We observe that image reconstruction of masked image modeling models is sensitive to the removal of visible patches when the masking ratio is high (e.g., 90\%). Building upon it, we implement LTRP via two steps: inferring the semantic density score of each patch by quantifying variation between reconstructions with and without this patch, and learning to rank the patches with the pseudo score. The entire process is self-supervised, thus getting out of the dilemma of categorical inductive bias. We design extensive experiments on different datasets and tasks. The results demonstrate that LTRP outperforms both supervised and other self-supervised methods due to the fair assessment of image content., Comment: Accepted by CVPR 2024

Due to the complementary nature of visible light and thermal infrared modalities, object tracking based on the fusion of visible light images and thermal images (referred to as RGB-T tracking) has received increasing attention from researchers in recent years. How to achieve more comprehensive fusion of information from the two modalities at a lower cost has been an issue that researchers have been exploring. Inspired by visual prompt learning, we designed a novel two-stream RGB-T tracking architecture based on cross-modal mutual prompt learning, and used this model as a teacher to guide a one-stream student model for rapid learning through knowledge distillation techniques. Extensive experiments have shown that, compared to similar RGB-T trackers, our designed teacher model achieved the highest precision rate, while the student model, with comparable precision rate to the teacher model, realized an inference speed more than three times faster than the teacher model.(Codes will be available if accepted.)

Logit knowledge distillation attracts increasing attention due to its practicality in recent studies. However, it often suffers inferior performance compared to the feature knowledge distillation. In this paper, we argue that existing logit-based methods may be sub-optimal since they only leverage the global logit output that couples multiple semantic knowledge. This may transfer ambiguous knowledge to the student and mislead its learning. To this end, we propose a simple but effective method, i.e., Scale Decoupled Distillation (SDD), for logit knowledge distillation. SDD decouples the global logit output into multiple local logit outputs and establishes distillation pipelines for them. This helps the student to mine and inherit fine-grained and unambiguous logit knowledge. Moreover, the decoupled knowledge can be further divided into consistent and complementary logit knowledge that transfers the semantic information and sample ambiguity, respectively. By increasing the weight of complementary parts, SDD can guide the student to focus more on ambiguous samples, improving its discrimination ability. Extensive experiments on several benchmark datasets demonstrate the effectiveness of SDD for wide teacher-student pairs, especially in the fine-grained classification task. Code is available at: https://github.com/shicaiwei123/SDD-CVPR2024, Comment: Accepted to CVPR2024 10 pages 6figure

This letter aims to provide a fundamental analytical comparison for the two major types of relaying methods: intelligent reflecting surfaces and full-duplex relays, particularly focusing on unmanned aerial vehicle communication scenarios. Both amplify-and-forward and decode-and-forward relaying schemes are included in the comparison. In addition, optimal 3D UAV deployment and minimum transmit power under the quality of service constraint are derived. Our numerical results show that IRSs of medium size exhibit comparable performance to AF relays, meanwhile outperforming DF relays under extremely large surface size and high data rates.

The capability to excite, probe, and manipulate vibrational modes is essential for understanding and controlling chemical reactions at the molecular level. Recent advancements in tip-enhanced Raman spectroscopies have enabled the probing of vibrational fingerprints in a single molecule with Angstrom-scale spatial resolution. However, achieving controllable excitation of specific vibrational modes in individual molecules remains challenging. Here, we demonstrate the selective excitation and probing of vibrational modes in single deprotonated phthalocyanine molecules utilizing resonance Raman spectroscopy in a scanning tunneling microscope. Selective excitation is achieved by finely tuning the excitation wavelength of the laser to be resonant with the vibronic transitions between the molecular ground electronic state and the vibrational levels in the excited electronic state, resulting in the state-selective enhancement of the resonance Raman signal. Our approach sets the stage for steering chemical transformations in molecules on surfaces by selective excitation of molecular vibrations.

The massive generation of time-series data by largescale Internet of Things (IoT) devices necessitates the exploration of more effective models for multivariate time-series forecasting. In previous models, there was a predominant use of the Channel Dependence (CD) strategy (where each channel represents a univariate sequence). Current state-of-the-art (SOTA) models primarily rely on the Channel Independence (CI) strategy. The CI strategy treats all channels as a single channel, expanding the dataset to improve generalization performance and avoiding inter-channel correlation that disrupts long-term features. However, the CI strategy faces the challenge of interchannel correlation forgetting. To address this issue, we propose an innovative Mixed Channels strategy, combining the data expansion advantages of the CI strategy with the ability to counteract inter-channel correlation forgetting. Based on this strategy, we introduce MCformer, a multivariate time-series forecasting model with mixed channel features. The model blends a specific number of channels, leveraging an attention mechanism to effectively capture inter-channel correlation information when modeling long-term features. Experimental results demonstrate that the Mixed Channels strategy outperforms pure CI strategy in multivariate time-series forecasting tasks.

This commentary dedicates to envision what role THz is going to play in the coming human-centric 6G era. Three distinct THz network types including outdoor, indoor, and body area networks are discussed, with an emphasis on their capabilities in human body detection. Synthesizing these networks will unlock a bunch of fascinating applications across industrial, biomedical and entertainment fields, significantly enhancing the quality of human life.

Kagome superconductors AV$_3$Sb$_5$ (A = K, Rb and Cs) have attracted much recent attention due to the coexistence of multiple exotic orders. Among them, the charge density wave (CDW) order has been shown to host various unconventional behaviors. Here, we investigate the CDW order by a combination of both bulk and surface doping methods. While element substitutions in bulk doping change both carriers and the crystal lattice, the surface doping primarily tunes the carrier concentration. As such, our results reveal a two-dimensional phase diagram of the CDW in doped CsV$_3$Sb$_5$. In the lightly bulk doped regime, the existence of CDW order is reversible by tuning the carrier concentration. But excessive bulk doping permanently destroys the CDW, regardless of the carrier doping level. These results provide insights to the origin of the CDW from both electronic and structural degrees of freedom. They also open an avenue for manipulating the exotic CDW order in Kagome superconductors., Comment: 14 pages, 4 figures

We present a novel, deep-learning based method -- dubbed Galactic-Seismology Substructures and Streams Hunter, or GS$^{3}$ Hunter for short, to search for substructures and streams in stellar kinematics data. GS$^{3}$ Hunter relies on a combined application of Siamese Neural Networks to transform the phase space information and the K-means algorithm for the clustering. As a validation test, we apply GS$^{3}$ Hunter to a subset of the Feedback in Realistic Environments (FIRE) cosmological simulations. The stellar streams and substructures thus identified are in good agreement with corresponding results reported earlier by the FIRE team. In the same vein, we apply our method to a subset of local halo stars from the Gaia Early Data Release 3 and GALAH DR3 datasets, and recover several, previously known dynamical groups, such as Thamnos 1+2, Hot Thick Disk, ED-1, L-RL3, Helmi 1+2, and Gaia-Sausage-Enceladus, Sequoia, VRM, Cronus, Nereus. Finally, we apply our method without fine-tuning to a subset of K-giant stars located in the inner halo region, obtained from the LAMOST Data Release 5 (DR5) dataset. We recover three, previously known structures (Sagittarius, Hercules-Aquila Cloud, and the Virgo Overdensity), but we also discover a number of new substructures. We anticipate that GS$^{3}$ Hunter will become a useful tool for the community dedicated to the search of stellar streams and structures in the Milky Way (MW) and the Local group, thus helping advance our understanding of the stellar inner and outer halos, and of the assembly and tidal stripping history in and around the MW., Comment: 41 pages, 34 figures, 3 tables. Accepted for Publication in the Astrophysical Journal

The application of asymmetric drift (AD) tomography across different populations provides valuable insights into the kinematics, dynamics, and rotation curves of the Galactic disk. By leveraging common stars identified in both the LAMOST and Gaia surveys, alongside Gaia DR3's circular velocity curve, we conducted a qualitative exploration of asymmetric drift distributions within the Galactic disk spanning distances from 8 to 16 kpc. In the R-Z plane, we observed that the asymmetric drift is minimal near the mid-plane of the Galactic disk and gradually increases with vertical distance, resulting in a distinctive ``horn" shape. Additionally, our analysis revealed that populations with higher [$\alpha$/Fe] ratios exhibit greater asymmetric drift compared to those with lower [$\alpha$/Fe] ratios. Specifically, we found the asymmetric drift around the solar location to be approximately 6 km s$^{-1}$, with a median value of 16 km s$^{-1}$ across the entire sample. Notably, the median asymmetric drift in the northern region of the Galactic disk (20 km s$^{-1}$) surpasses that in the southern region (13 km s$^{-1}$), with errors remaining within 2 km s$^{-1}$. Furthermore, our investigation into mono-age stellar populations unveiled that older stellar populations tend to exhibit larger asymmetric drift and velocity dispersion, aligning closely with predictions from previous numerical models. Finally, based on chemical compositions, we observed that the median asymmetric drift of the thick disk significantly exceeds that of the thin disk and found that star formation within the thick disk primarily occurred earlier than 8-10 billion years, whereas the thin disk's predominant star formation period spanned 6-8 billion years ago., Comment: 14 pages and 17 figures. The Open Journal of Astrophysics. AD analysis is the key target and the time scale picture of the milky way disk are also revealed here in another point of view, see the Fig. 9

Learning based on multimodal data has attracted increasing interest recently. While a variety of sensory modalities can be collected for training, not all of them are always available in development scenarios, which raises the challenge to infer with incomplete modality. To address this issue, this paper presents a one-stage modality distillation framework that unifies the privileged knowledge transfer and modality information fusion into a single optimization procedure via multi-task learning. Compared with the conventional modality distillation that performs them independently, this helps to capture the valuable representation that can assist the final model inference directly. Specifically, we propose the joint adaptation network for the modality transfer task to preserve the privileged information. This addresses the representation heterogeneity caused by input discrepancy via the joint distribution adaptation. Then, we introduce the cross translation network for the modality fusion task to aggregate the restored and available modality features. It leverages the parameters-sharing strategy to capture the cross-modal cues explicitly. Extensive experiments on RGB-D classification and segmentation tasks demonstrate the proposed multimodal inheritance framework can overcome the problem of incomplete modality input in various scenes and achieve state-of-the-art performance.

Object tracking based on the fusion of visible and thermal im-ages, known as RGB-T tracking, has gained increasing atten-tion from researchers in recent years. How to achieve a more comprehensive fusion of information from the two modalities with fewer computational costs has been a problem that re-searchers have been exploring. Recently, with the rise of prompt learning in computer vision, we can better transfer knowledge from visual large models to downstream tasks. Considering the strong complementarity between visible and thermal modalities, we propose a tracking architecture based on mutual prompt learning between the two modalities. We also design a lightweight prompter that incorporates attention mechanisms in two dimensions to transfer information from one modality to the other with lower computational costs, embedding it into each layer of the backbone. Extensive ex-periments have demonstrated that our proposed tracking ar-chitecture is effective and efficient, achieving state-of-the-art performance while maintaining high running speeds., Comment: 9 pages, 5 figures, 5 tables

Adaptive gradient methods, such as Adam and LAMB, have demonstrated excellent performance in the training of large language models. Nevertheless, the need for adaptivity requires maintaining second-moment estimates of the per-parameter gradients, which entails a high cost of extra memory overheads. To solve this problem, several memory-efficient optimizers (e.g., Adafactor) have been proposed to obtain a drastic reduction in auxiliary memory usage, but with a performance penalty. In this paper, we first study a confidence-guided strategy to reduce the instability of existing memory efficient optimizers. Based on this strategy, we propose CAME to simultaneously achieve two goals: fast convergence as in traditional adaptive methods, and low memory usage as in memory-efficient methods. Extensive experiments demonstrate the training stability and superior performance of CAME across various NLP tasks such as BERT and GPT-2 training. Notably, for BERT pre-training on the large batch size of 32,768, our proposed optimizer attains faster convergence and higher accuracy compared with the Adam optimizer. The implementation of CAME is publicly available., Comment: Accepted by ACL 2023

Van Hove singularity (VHS) has been considered as a driving source for unconventional superconductivity. A VHS in two-dimensional (2D) materials consists of a saddle point connecting electron-like and hole-like bands. In a rare case, when a VHS appears at Fermi level, both electron-like and hole-like conduction can coexist, giving rise to an enhanced density of states as well as an attractive component of Coulomb interaction for unconventional electronic pairing. However, this van Hove scenario is often destroyed by an incorrect chemical potential or competing instabilities. Here, by using angle-resolved photoemission measurements, we report the observation of a VHS perfectly aligned with the Fermi level in a kagome superconductor CsV$_{3-x}$Ta$_x$Sb$_5$ (x~0.4), in which a record-high superconducting transition temperature is achieved among all the current variants of AV$_3$Sb$_5$ (A=Cs, Rb, K) at ambient pressure. Doping dependent measurements reveal the important role of van Hove scenario in boosting superconductivity, and spectroscopic-imaging scanning tunneling microscopy measurements indicate a distinct superconducting state in this system., Comment: 20 pages, 4 figures

Large language models (LLMs) have revolutionized the field of AI, demonstrating unprecedented capacity across various tasks. However, the inference process for LLMs comes with significant computational costs. In this paper, we propose an efficient LLM inference pipeline that harnesses the power of LLMs. Our approach begins by tapping into the potential of LLMs to accurately perceive and predict the response length with minimal overhead. By leveraging this information, we introduce an efficient sequence scheduling technique that groups queries with similar response lengths into micro-batches. We evaluate our approach on real-world instruction datasets using the LLaMA-based model, and our results demonstrate an impressive 86% improvement in inference throughput without compromising effectiveness. Notably, our method is orthogonal to other inference acceleration techniques, making it a valuable addition to many existing toolkits (e.g., FlashAttention, Quantization) for LLM inference.

The three-dimensional electronic structure and the nature of Ce 4f electrons of the Kondo insulator CeRu4Sn6 are investigated by angle-resolved photoemission spectroscopy, utilizing tunable photon energies. Our results reveal (i) the three-dimensional k-space nature of the Fermi surface, (ii) the localized-to-itinerant transition of f electrons occurs at a much high temperature than the hybridization gap opening temperature, and (iii) the "relocalization" of itinerant f electrons below 25 K, which could be the precursor to the establishment of magnetic order., Comment: 6 pages, 4 figures

Multimodal learning has shown great potentials in numerous scenes and attracts increasing interest recently. However, it often encounters the problem of missing modality data and thus suffers severe performance degradation in practice. To this end, we propose a general framework called MMANet to assist incomplete multimodal learning. It consists of three components: the deployment network used for inference, the teacher network transferring comprehensive multimodal information to the deployment network, and the regularization network guiding the deployment network to balance weak modality combinations. Specifically, we propose a novel margin-aware distillation (MAD) to assist the information transfer by weighing the sample contribution with the classification uncertainty. This encourages the deployment network to focus on the samples near decision boundaries and acquire the refined inter-class margin. Besides, we design a modality-aware regularization (MAR) algorithm to mine the weak modality combinations and guide the regularization network to calculate prediction loss for them. This forces the deployment network to improve its representation ability for the weak modality combinations adaptively. Finally, extensive experiments on multimodal classification and segmentation tasks demonstrate that our MMANet outperforms the state-of-the-art significantly. Code is available at: https://github.com/shicaiwei123/MMANet, Comment: 10 pages, 3 figures, CVPR2023

RGB-T tracking involves the use of images from both visible and thermal modalities. The primary objective is to adaptively leverage the relatively dominant modality in varying conditions to achieve more robust tracking compared to single-modality tracking. An RGB-T tracker based on mixed attention mechanism to achieve complementary fusion of modalities (referred to as MACFT) is proposed in this paper. In the feature extraction stage, we utilize different transformer backbone branches to extract specific and shared information from different modalities. By performing mixed attention operations in the backbone to enable information interaction and self-enhancement between the template and search images, it constructs a robust feature representation that better understands the high-level semantic features of the target. Then, in the feature fusion stage, a modality-adaptive fusion is achieved through a mixed attention-based modality fusion network, which suppresses the low-quality modality noise while enhancing the information of the dominant modality. Evaluation on multiple RGB-T public datasets demonstrates that our proposed tracker outperforms other RGB-T trackers on general evaluation metrics while also being able to adapt to longterm tracking scenarios., Comment: 14 pages, 10 figures