Search

Despite the superior capabilities of Multimodal Large Language Models (MLLMs) across diverse tasks, they still face significant trustworthiness challenges. Yet, current literature on the assessment of trustworthy MLLMs remains limited, lacking a holistic evaluation to offer thorough insights into future improvements. In this work, we establish MultiTrust, the first comprehensive and unified benchmark on the trustworthiness of MLLMs across five primary aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark employs a rigorous evaluation strategy that addresses both multimodal risks and cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets. Extensive experiments with 21 modern MLLMs reveal some previously unexplored trustworthiness issues and risks, highlighting the complexities introduced by the multimodality and underscoring the necessity for advanced methodologies to enhance their reliability. For instance, typical proprietary models still struggle with the perception of visually confusing images and are vulnerable to multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to disclose privacy in text and reveal ideological and cultural biases even when paired with irrelevant images in inference, indicating that the multimodality amplifies the internal risks from base LLMs. Additionally, we release a scalable toolbox for standardized trustworthiness research, aiming to facilitate future advancements in this important field. Code and resources are publicly available at: https://multi-trust.github.io/., Comment: 100 pages, 84 figures, 33 tables

Vision-Language Pre-training (VLP) models like CLIP have achieved remarkable success in computer vision and particularly demonstrated superior robustness to distribution shifts of 2D images. However, their robustness under 3D viewpoint variations is still limited, which can hinder the development for real-world applications. This paper successfully addresses this concern while keeping VLPs' original performance by breaking through two primary obstacles: 1) the scarcity of training data and 2) the suboptimal fine-tuning paradigms. To combat data scarcity, we build the Multi-View Caption (MVCap) dataset -- a comprehensive collection of over four million multi-view image-text pairs across more than 100K objects, providing more potential for VLP models to develop generalizable viewpoint-invariant representations. To address the limitations of existing paradigms in performance trade-offs and training efficiency, we design a novel fine-tuning framework named Omniview-Tuning (OVT). Specifically, OVT introduces a Cross-Viewpoint Alignment objective through a minimax-like optimization strategy, which effectively aligns representations of identical objects from diverse viewpoints without causing overfitting. Additionally, OVT fine-tunes VLP models in a parameter-efficient manner, leading to minimal computational cost. Extensive experiments on various VLP models with different architectures validate that OVT significantly improves the models' resilience to viewpoint shifts and keeps the original performance, establishing a pioneering standard for boosting the viewpoint invariance of VLP models., Comment: 20 pages

In this study, we investigate the electronic structure of a spin-triplet superconductor candidate CeRh$_2$As$_2$ using high-resolution angle-resolved photoemission spectroscopy and density functional theory calculations. Notably, Fermi surface nesting hints at connections to magnetic excitation or quadrupole density wave phenomena, elucidating the superconducting mechanisms. Measured band structures reveal primarily localized 4f electrons, with minor itinerant contributions. Additionally, a transition from localized to itinerant behavior and significant c-f hybridization anisotropy underscore the role of f-electrons in shaping electronic properties. These findings deepen our understanding of CeRh$_2$As$_2$'s unconventional superconductivity and magnetism. Further exploration promises advances in superconductivity research., Comment: 6 pages, 3 figures

The optical Ramsey technique is used to obtain precise measurements of the hyperfine splittings in the $2\,^3\!S_1$ and $2\,^3\!P_J$ states of $^7$Li$^+$. Together with bound-state quantum electrodynamic theory, the Zemach radius and quadrupole moment of the $^7$Li nucleus are determined to be $3.35(1)$~fm and $-3.86(5)$~fm$^2$ respectively, with the quadrupole moment deviating from the recommended value of $-4.00(3)$~fm$^2$ by $1.75\sigma$. Furthermore, we determine the quadrupole moment ratio of $^6$Li to $^7$Li as $0.101(13)$, exhibiting a $6\sigma$ deviation from the previous measured value of $0.020161(13)$ by LiF molecular spectroscopy. The results taken together provide a sensitive test of nuclear structure models.

Federated Contrastive Learning (FCL) represents a burgeoning approach for learning from decentralized unlabeled data while upholding data privacy. In FCL, participant clients collaborate in learning a global encoder using unlabeled data, which can serve as a versatile feature extractor for diverse downstream tasks. Nonetheless, FCL is susceptible to privacy risks, such as membership information leakage, stemming from its distributed nature, an aspect often overlooked in current solutions. This study delves into the feasibility of executing a membership inference attack on FCL and proposes a robust attack methodology. The attacker's objective is to determine if the data signifies training member data by accessing the model's inference output. Specifically, we concentrate on attackers situated within a client framework, lacking the capability to manipulate server-side aggregation methods or discern the training status of other clients. We introduce two membership inference attacks tailored for FCL: the \textit{passive membership inference attack} and the \textit{active membership inference attack}, contingent on the attacker's involvement in local model training. Experimental findings across diverse datasets validate the effectiveness of our attacks and underscore the inherent privacy risks associated with the federated contrastive learning paradigm.

Compared with transferable untargeted attacks, transferable targeted adversarial attacks could specify the misclassification categories of adversarial samples, posing a greater threat to security-critical tasks. In the meanwhile, 3D adversarial samples, due to their potential of multi-view robustness, can more comprehensively identify weaknesses in existing deep learning systems, possessing great application value. However, the field of transferable targeted 3D adversarial attacks remains vacant. The goal of this work is to develop a more effective technique that could generate transferable targeted 3D adversarial examples, filling the gap in this field. To achieve this goal, we design a novel framework named TT3D that could rapidly reconstruct from few multi-view images into Transferable Targeted 3D textured meshes. While existing mesh-based texture optimization methods compute gradients in the high-dimensional mesh space and easily fall into local optima, leading to unsatisfactory transferability and distinct distortions, TT3D innovatively performs dual optimization towards both feature grid and Multi-layer Perceptron (MLP) parameters in the grid-based NeRF space, which significantly enhances black-box transferability while enjoying naturalness. Experimental results show that TT3D not only exhibits superior cross-model transferability but also maintains considerable adaptability across different renders and vision tasks. More importantly, we produce 3D adversarial examples with 3D printing techniques in the real world and verify their robust performance under various scenarios., Comment: Accepted by CVPR 2024

As physical adversarial attacks become extensively applied in unearthing the potential risk of security-critical scenarios, especially in dynamic scenarios, their vulnerability to environmental variations has also been brought to light. The non-robust nature of physical adversarial attack methods brings less-than-stable performance consequently. Although methods such as EOT have enhanced the robustness of traditional contact attacks like adversarial patches, they fall short in practicality and concealment within dynamic environments such as traffic scenarios. Meanwhile, non-contact laser attacks, while offering enhanced adaptability, face constraints due to a limited optimization space for their attributes, rendering EOT less effective. This limitation underscores the necessity for developing a new strategy to augment the robustness of such practices. To address these issues, this paper introduces the Embodied Laser Attack (ELA), a novel framework that leverages the embodied intelligence paradigm of Perception-Decision-Control to dynamically tailor non-contact laser attacks. For the perception module, given the challenge of simulating the victim's view by full-image transformation, ELA has innovatively developed a local perspective transformation network, based on the intrinsic prior knowledge of traffic scenes and enables effective and efficient estimation. For the decision and control module, ELA trains an attack agent with data-driven reinforcement learning instead of adopting time-consuming heuristic algorithms, making it capable of instantaneously determining a valid attack strategy with the perceived information by well-designed rewards, which is then conducted by a controllable laser emitter. Experimentally, we apply our framework to diverse traffic scenarios both in the digital and physical world, verifying the effectiveness of our method under dynamic successive scenes., Comment: 9 pages, 7 figures, Accepted by ACM MM 2024

Although the origin of the fat-tail characteristic of the degree distribution in complex networks has been extensively researched, the underlying cause of the degree distribution characteristic across the complete range of degrees remains obscure. Here, we propose an evolution model that incorporates only two factors: the node's weight, reflecting its innate attractiveness (nature), and the node's degree, reflecting the external influences (nurture). The proposed model provides a good fit for degree distributions and degree ratio distributions of numerous real-world networks and reproduces their evolution processes. Our results indicate that the nurture factor plays a dominant role in the evolution of social networks. In contrast, the nature factor plays a dominant role in the evolution of non-social networks, suggesting that whether nodes are people determines the dominant factor influencing the evolution of real-world networks., Comment: 8pages, 4 figures

Synchrotron-based angle-resolved photoemission spectroscopy was employed to investigate the temperature evolution of the Yb 4f spectral for surface and bulk in the Kondo lattice YbCu$_2$Si$_2$.Our study quantitatively distinguishes between the surface and bulk hybridization processes, revealing that the onset temperatures for both surface and bulk hybridization processes are significantly higher than the Kondo temperature. Additionally, we found that the effective surface Kondo temperature is much lower than that of the bulk. These findings offer valuable insights into the understanding of heavy fermion physics., Comment: 7 pages, 4 figures

Physical adversarial attacks have put a severe threat to DNN-based object detectors. To enhance security, a combination of visible and infrared sensors is deployed in various scenarios, which has proven effective in disabling existing single-modal physical attacks. To further demonstrate the potential risks in such cases, we design a unified adversarial patch that can perform cross-modal physical attacks, achieving evasion in both modalities simultaneously with a single patch. Given the different imaging mechanisms of visible and infrared sensors, our work manipulates patches' shape features, which can be captured in different modalities when they undergo changes. To deal with challenges, we propose a novel boundary-limited shape optimization approach that aims to achieve compact and smooth shapes for the adversarial patch, making it easy to implement in the physical world. And a score-aware iterative evaluation method is also introduced to balance the fooling degree between visible and infrared detectors during optimization, which guides the adversarial patch to iteratively reduce the predicted scores of the multi-modal sensors. Furthermore, we propose an Affine-Transformation-based enhancement strategy that makes the learnable shape robust to various angles, thus mitigating the issue of shape deformation caused by different shooting angles in the real world. Our method is evaluated against several state-of-the-art object detectors, achieving an Attack Success Rate (ASR) of over 80%. We also demonstrate the effectiveness of our approach in physical-world scenarios under various settings, including different angles, distances, postures, and scenes for both visible and infrared sensors., Comment: 13 pages, 16 figures. arXiv admin note: substantial text overlap with arXiv:2307.07859

Recently, physical adversarial attacks have been presented to evade DNNs-based object detectors. To ensure the security, many scenarios are simultaneously deployed with visible sensors and infrared sensors, leading to the failures of these single-modal physical attacks. To show the potential risks under such scenes, we propose a unified adversarial patch to perform cross-modal physical attacks, i.e., fooling visible and infrared object detectors at the same time via a single patch. Considering different imaging mechanisms of visible and infrared sensors, our work focuses on modeling the shapes of adversarial patches, which can be captured in different modalities when they change. To this end, we design a novel boundary-limited shape optimization to achieve the compact and smooth shapes, and thus they can be easily implemented in the physical world. In addition, to balance the fooling degree between visible detector and infrared detector during the optimization process, we propose a score-aware iterative evaluation, which can guide the adversarial patch to iteratively reduce the predicted scores of the multi-modal sensors. We finally test our method against the one-stage detector: YOLOv3 and the two-stage detector: Faster RCNN. Results show that our unified patch achieves an Attack Success Rate (ASR) of 73.33% and 69.17%, respectively. More importantly, we verify the effective attacks in the physical world when visible and infrared sensors shoot the objects under various settings like different angles, distances, postures, and scenes., Comment: 10 pages, 8 figures, accepted by ICCV2023

Hybrid Optimization Software Suite (HOSS), which is a combined finite-discrete element method (FDEM), is one of the advanced approaches to simulating high-fidelity fracture and fragmentation processes but the application of pure HOSS simulation is computationally expensive. At the same time, machine learning methods, shown tremendous success in several scientific problems, are increasingly being considered promising alternatives to physics-based models in the scientific domains. Thus, our goal in this work is to build a new data-driven methodology to reconstruct the crack fracture accurately in the spatial and temporal fields. We leverage physical constraints to regularize the fracture propagation in the long-term reconstruction. In addition, we introduce perceptual loss and several extra pure machine learning optimization approaches to improve the reconstruction performance of fracture data further. We demonstrate the effectiveness of our proposed method through both extrapolation and interpolation experiments. The results confirm that our proposed method can reconstruct high-fidelity fracture data over space and time in terms of pixel-wise reconstruction error and structural similarity. Visual comparisons also show promising results in long-term, Comment: 12 pages

A comprehensive understanding of molecular clumps is essential for investigating star formation. We present an algorithm for molecular clump detection, called FacetClumps. This algorithm uses a morphological approach to extract signal regions from the original data. The Gaussian Facet model is employed to fit the signal regions, which enhances the resistance to noise and the stability of the algorithm in diverse overlapping areas. The introduction of the extremum determination theorem of multivariate functions offers theoretical guidance for automatically locating clump centers. To guarantee that each clump is continuous, the signal regions are segmented into local regions based on gradient, and then the local regions are clustered into the clump centers based on connectivity and minimum distance to identify the regional information of each clump. Experiments conducted with both simulated and synthetic data demonstrate that FacetClumps exhibits great recall and precision rates, small location error and flux loss, a high consistency between the region of detected clump and that of simulated clump, and is generally stable in various environments. Notably, the recall rate of FacetClumps in the synthetic data, which comprises $^{13}CO$ ($J = 1-0$) emission line of the MWISP within $11.7^{\circ} \leq l \leq 13.4^{\circ}$, $0.22^{\circ} \leq b \leq 1.05^{\circ}$ and 5 km s$^{-1}$ $\leq v \leq$ 35 km s$^{-1}$ and simulated clumps, reaches 90.2%. Additionally, FacetClumps demonstrates satisfactory performance when applied to observational data., Comment: Accepted for ApJS

Deep learning models have become the mainstream method for medical image segmentation, but they require a large manually labeled dataset for training and are difficult to extend to unseen categories. Few-shot segmentation(FSS) has the potential to address these challenges by learning new categories from a small number of labeled samples. The majority of the current methods employ a prototype learning architecture, which involves expanding support prototype vectors and concatenating them with query features to conduct conditional segmentation. However, such framework potentially focuses more on query features while may neglect the correlation between support and query features. In this paper, we propose a novel self-supervised few shot medical image segmentation network with Cross-Reference Transformer, which addresses the lack of interaction between the support image and the query image. We first enhance the correlation features between the support set image and the query image using a bidirectional cross-attention module. Then, we employ a cross-reference mechanism to mine and enhance the similar parts of support features and query features in high-dimensional channels. Experimental results show that the proposed model achieves good results on both CT dataset and MRI dataset., Comment: 6 pages,4 figures

Graph convolutional network (GCN) has been successfully applied to capture global non-consecutive and long-distance semantic information for text classification. However, while GCN-based methods have shown promising results in offline evaluations, they commonly follow a seen-token-seen-document paradigm by constructing a fixed document-token graph and cannot make inferences on new documents. It is a challenge to deploy them in online systems to infer steaming text data. In this work, we present a continual GCN model (ContGCN) to generalize inferences from observed documents to unobserved documents. Concretely, we propose a new all-token-any-document paradigm to dynamically update the document-token graph in every batch during both the training and testing phases of an online system. Moreover, we design an occurrence memory module and a self-supervised contrastive learning objective to update ContGCN in a label-free manner. A 3-month A/B test on Huawei public opinion analysis system shows ContGCN achieves 8.86% performance gain compared with state-of-the-art methods. Offline experiments on five public datasets also show ContGCN can improve inference quality. The source code will be released at https://github.com/Jyonn/ContGCN., Comment: 9 pages, 4 figures, AAAI 2023 accepted paper

Due to the complex behavior arising from non-uniqueness, symmetry, and bifurcations in the solution space, solving inverse problems of nonlinear differential equations (DEs) with multiple solutions is a challenging task. To address this, we propose homotopy physics-informed neural networks (HomPINNs), a novel framework that leverages homotopy continuation and neural networks (NNs) to solve inverse problems. The proposed framework begins with the use of NNs to simultaneously approximate unlabeled observations across diverse solutions while adhering to DE constraints. Through homotopy continuation, the proposed method solves the inverse problem by tracing the observations and identifying multiple solutions. The experiments involve testing the performance of the proposed method on one-dimensional DEs and applying it to solve a two-dimensional Gray-Scott simulation. Our findings demonstrate that the proposed method is scalable and adaptable, providing an effective solution for solving DEs with multiple solutions and unknown parameters. Moreover, it has significant potential for various applications in scientific computing, such as modeling complex systems and solving inverse problems in physics, chemistry, biology, etc., Comment: 20 pages, 15 figures, 7 tables

We investigate the $2\,^3\!S_1$--$2\,^3\!P_J$ ($J = 0, 1, 2$) transitions in $\rm^6Li^+$ using the optical Ramsey technique and achieve the most precise values of the hyperfine splittings of the $2\,^3\!S_1$ and $2\,^3\!P_J$ states, with smallest uncertainty of about 10~kHz. The present results reduce the uncertainties of previous experiments by a factor of 5 for the $2\,^3\!S_1$ state and a factor of 50 for the $2\,^3\!P_J$ states, and are in better agreement with theoretical values. Combining our measured hyperfine intervals of the $2\,^3\!S_1$ state with the latest quantum electrodynamic (QED) calculations, the improved Zemach radius of the $\rm^6Li$ nucleus is determined to be 2.44(2)~fm, with the uncertainty entirely due to the uncalculated QED effects of order $m\alpha^7$. The result is in sharp disagreement with the value 3.71(16) fm determined from simple models of the nuclear charge and magnetization distribution. We call for a more definitive nuclear physics value of the $\rm^6Li$ Zemach radius., Comment: 6 pages, 6 figures

We present a compact, long-term nearly continuous operation of a room-temperature Ca$^+$ optical clock setup towards a transportable clock, achieving an overall systematic uncertainty of $4.8\times 10^{-18}$ and an uptime rate of 97.8% over an 8-day period. The active liquid-cooling scheme is adopted, combined with the precise temperature measurement with 13 temperature sensors both inside and outside the vacuum chamber to ensure the accurate evaluation of the thermal environment for the optical clock. The environmental temperature uncertainty is evaluated as 293.31(0.4) K, corresponding to a blackbody radiation (BBR) frequency shift uncertainty of $4.6\times 10^{-18}$, which is reduced more than two times compared to our previous work. Through the frequency comparison between the room temperature Ca$^+$ optical clock and a cryogenic Ca$^+$ optical clock, the overall uncertainty of the clock comparison is $7.5\times 10^{-18}$, including a statistic uncertainty of $4.9\times 10^{-18}$ and a systematic uncertainty of $5.7\times 10^{-18}$. This work provides a set of feasible implementations for high-precision transportable ion optical clocks., Comment: 11 pages, 4 figures

Highly charged ions (HCIs) have drawn significant interest in quantum metrology and in search for new physics. Among these, Ni12+ is considered as one of the most promising candidates for the next generation of HCI optical clocks, due to its two E1-forbidden transitions M1 and E2, which occur in the visible spectral range. In this work, we used the Shanghai-Wuhan Electron Beam Ion Trap to perform a high-precision measurement of the M1 transition wavelength. Our approach involved an improved calibration scheme for the spectra, utilizing auxiliary Ar+ lines for calibration and correction. Our final measured result of the M1 transition wavelength demonstrates a five-fold improvement in accuracy compared to our previous findings, reaching the sub-picometer level accuracy. In combination with our rigorous atomic-structure calculations to capture the electron correlations and relativistic effects, the quantum electrodynamic (QED) corrections were extracted. Moreover, comparing with an estimate of the one-electron QED contributions by using the GRASP2018 package, we found that the present experimental accuracy is high enough for testing the higher-order QED corrections for such a complex system with four electrons in the p subshell., Comment: 15 pages, 5 figures

We constructed a transportable 40Ca+ optical clock (with an estimated minimum systematic shift uncertainty of 1.3*10^(-17) and a stability of 5*10^(-15)/sqrt{tau} ) that can operate outside the laboratory. We transported it from the Innovation Academy for Precision Measurement Science and Technology, Chinese Academy of Sciences, Wuhan to the National Institute of Metrology, Beijing. The absolute frequency of the 729 nm clock transition was measured for up to 35 days by tracing its frequency to the second of International System of Units. Some improvements were implemented in the measurement process, such as the increased effective up-time of 91.3 % of the 40Ca+ optical clock over a 35-day-period, the reduced statistical uncertainty of the comparison between the optical clock and hydrogen maser, and the use of longer measurement times to reduce the uncertainty of the frequency traceability link. The absolute frequency measurement of the 40Ca+ optical clock yielded a value of 411042129776400.26 (13) Hz with an uncertainty of 3.2*10^(-16), which is reduced by a factor of 1.7 compared with our previous results. As a result of the increase in the operating rate of the optical clock, the accuracy of 35 days of absolute frequency measurement can be comparable to the best results of different institutions in the world based on different optical frequency measurements., Comment: 15 pages, 5 figures

A thermal component is suggested to be the physical composition of the ejecta of several bright gamma-ray bursts (GRBs). Such a thermal component is discovered in the time-integrated spectra of several short GRBs as well as long GRBs. In this work, we present a comprehensive analysis of ten very short GRBs detected by Fermi Gamma-Ray Burst Monitor to search for the thermal component. We found that both the resultant low-energy spectral index and the peak energy in each GRB imply a common hard spectral feature, which is in favor of the main classification of the short/hard versus long/soft dichotomy in the GRB duration. We also found moderate evidence for the detection of thermal component in eight GRBs. Although such a thermal component contributes a small proportion of the global prompt gamma-ray emission, the modified thermal-radiation mechanism could enhance the proportion significantly, such as in subphotospheric dissipation., Comment: 14 pages, 6 figures, published in the Universe journal