Search

Vision-enabled language models (VLMs) are now used to build autonomous multimodal agents capable of taking actions in real environments. In this paper, we show that multimodal agents raise new safety risks, even though attacking agents is more challenging than prior attacks due to limited access to and knowledge about the environment. Our attacks use adversarial text strings to guide gradient-based perturbation over one trigger image in the environment: (1) our captioner attack attacks white-box captioners if they are used to process images into captions as additional inputs to the VLM; (2) our CLIP attack attacks a set of CLIP models jointly, which can transfer to proprietary VLMs. To evaluate the attacks, we curated VisualWebArena-Adv, a set of adversarial tasks based on VisualWebArena, an environment for web-based multimodal agent tasks. Within an L-infinity norm of $16/256$ on a single image, the captioner attack can make a captioner-augmented GPT-4V agent execute the adversarial goals with a 75% success rate. When we remove the captioner or use GPT-4V to generate its own captions, the CLIP attack can achieve success rates of 21% and 43%, respectively. Experiments on agents based on other VLMs, such as Gemini-1.5, Claude-3, and GPT-4o, show interesting differences in their robustness. Further analysis reveals several key factors contributing to the attack's success, and we also discuss the implications for defenses as well. Project page: https://chenwu.io/attack-agent Code and data: https://github.com/ChenWu98/agent-attack, Comment: 19 pages

Foundation models (FMs) are revolutionizing the analysis and understanding of remote sensing (RS) scenes, including aerial RGB, multispectral, and SAR images. However, hyperspectral images (HSIs), which are rich in spectral information, have not seen much application of FMs, with existing methods often restricted to specific tasks and lacking generality. To fill this gap, we introduce HyperSIGMA, a vision transformer-based foundation model for HSI interpretation, scalable to over a billion parameters. To tackle the spectral and spatial redundancy challenges in HSIs, we introduce a novel sparse sampling attention (SSA) mechanism, which effectively promotes the learning of diverse contextual features and serves as the basic block of HyperSIGMA. HyperSIGMA integrates spatial and spectral features using a specially designed spectral enhancement module. In addition, we construct a large-scale hyperspectral dataset, HyperGlobal-450K, for pre-training, which contains about 450K hyperspectral images, significantly surpassing existing datasets in scale. Extensive experiments on various high-level and low-level HSI tasks demonstrate HyperSIGMA's versatility and superior representational capability compared to current state-of-the-art methods. Moreover, HyperSIGMA shows significant advantages in scalability, robustness, cross-modal transferring capability, and real-world applicability., Comment: The code and models will be released at https://github.com/WHU-Sigma/HyperSIGMA

Backdoor attacks present significant threats to Large Language Models (LLMs), particularly with the rise of third-party services that offer API integration and prompt engineering. Untrustworthy third parties can plant backdoors into LLMs and pose risks to users by embedding malicious instructions into user queries. The backdoor-compromised LLM will generate malicious output when and input is embedded with a specific trigger predetermined by an attacker. Traditional defense strategies, which primarily involve model parameter fine-tuning and gradient calculation, are inadequate for LLMs due to their extensive computational and clean data requirements. In this paper, we propose a novel solution, Chain-of-Scrutiny (CoS), to address these challenges. Backdoor attacks fundamentally create a shortcut from the trigger to the target output, thus lack reasoning support. Accordingly, CoS guides the LLMs to generate detailed reasoning steps for the input, then scrutinizes the reasoning process to ensure consistency with the final answer. Any inconsistency may indicate an attack. CoS only requires black-box access to LLM, offering a practical defense, particularly for API-accessible LLMs. It is user-friendly, enabling users to conduct the defense themselves. Driven by natural language, the entire defense process is transparent to users. We validate the effectiveness of CoS through extensive experiments across various tasks and LLMs. Additionally, experiments results shows CoS proves more beneficial for more powerful LLMs.

Despite significant progress has been made in image deraining, existing approaches are mostly carried out on low-resolution images. The effectiveness of these methods on high-resolution images is still unknown, especially for ultra-high-definition (UHD) images, given the continuous advancement of imaging devices. In this paper, we focus on the task of UHD image deraining, and contribute the first large-scale UHD image deraining dataset, 4K-Rain13k, that contains 13,000 image pairs at 4K resolution. Based on this dataset, we conduct a benchmark study on existing methods for processing UHD images. Furthermore, we develop an effective and efficient vision MLP-based architecture (UDR-Mixer) to better solve this task. Specifically, our method contains two building components: a spatial feature rearrangement layer that captures long-range information of UHD images, and a frequency feature modulation layer that facilitates high-quality UHD image reconstruction. Extensive experimental results demonstrate that our method performs favorably against the state-of-the-art approaches while maintaining a lower model complexity. The code and dataset will be available at https://github.com/cschenxiang/UDR-Mixer.

Due to the potential application of regulating droplet shape by external fields in microfluidic technology and micro devices, it becomes increasingly important to understand the shape formation of a droplet in the presence of an electric field. How to understand and determine such a deformable boundary shape at equilibrium has been a long-term physical and mathematical challenge. Here, based on the theoretical model we propose, and combining the finite element method and the gradient descent algorithm, we successfully obtain the droplet shape by considering the contributions made by electrostatic energy, surface tension energy, and gravitational potential energy. We also carry out scaling analyses and obtain an empirical critical disruption condition with a universal scaling exponent 1/2 for the contact angle in terms of normalized volume. The master curve fits both the experimental and the numerical results very well.

Today's analog/mixed-signal (AMS) integrated circuit (IC) designs demand substantial manual intervention. The advent of multimodal large language models (MLLMs) has unveiled significant potential across various fields, suggesting their applicability in streamlining large-scale AMS IC design as well. A bottleneck in employing MLLMs for automatic AMS circuit generation is the absence of a comprehensive dataset delineating the schematic-netlist relationship. We therefore design an automatic technique for converting schematics into netlists, and create dataset AMSNet, encompassing transistor-level schematics and corresponding SPICE format netlists. With a growing size, AMSNet can significantly facilitate exploration of MLLM applications in AMS circuit design. We have made an initial set of netlists public, and will make both our netlist generation tool and the full dataset available upon publishing of this paper.

A high-precision feature extraction model is crucial for change detection (CD). In the past, many deep learning-based supervised CD methods learned to recognize change feature patterns from a large number of labelled bi-temporal images, whereas labelling bi-temporal remote sensing images is very expensive and often time-consuming; therefore, we propose a coarse-to-fine semi-supervised CD method based on consistency regularization (C2F-SemiCD), which includes a coarse-to-fine CD network with a multiscale attention mechanism (C2FNet) and a semi-supervised update method. Among them, the C2FNet network gradually completes the extraction of change features from coarse-grained to fine-grained through multiscale feature fusion, channel attention mechanism, spatial attention mechanism, global context module, feature refine module, initial aggregation module, and final aggregation module. The semi-supervised update method uses the mean teacher method. The parameters of the student model are updated to the parameters of the teacher Model by using the exponential moving average (EMA) method. Through extensive experiments on three datasets and meticulous ablation studies, including crossover experiments across datasets, we verify the significant effectiveness and efficiency of the proposed C2F-SemiCD method. The code will be open at: https://github.com/ChengxiHAN/C2F-SemiCDand-C2FNet.

The rapid advancement of automated artificial intelligence algorithms and remote sensing instruments has benefited change detection (CD) tasks. However, there is still a lot of space to study for precise detection, especially the edge integrity and internal holes phenomenon of change features. In order to solve these problems, we design the Change Guiding Network (CGNet), to tackle the insufficient expression problem of change features in the conventional U-Net structure adopted in previous methods, which causes inaccurate edge detection and internal holes. Change maps from deep features with rich semantic information are generated and used as prior information to guide multi-scale feature fusion, which can improve the expression ability of change features. Meanwhile, we propose a self-attention module named Change Guide Module (CGM), which can effectively capture the long-distance dependency among pixels and effectively overcome the problem of the insufficient receptive field of traditional convolutional neural networks. On four major CD datasets, we verify the usefulness and efficiency of the CGNet, and a large number of experiments and ablation studies demonstrate the effectiveness of CGNet. We're going to open-source our code at https://github.com/ChengxiHAN/CGNet-CD.

Benefiting from the developments in deep learning technology, deep-learning-based algorithms employing automatic feature extraction have achieved remarkable performance on the change detection (CD) task. However, the performance of existing deep-learning-based CD methods is hindered by the imbalance between changed and unchanged pixels. To tackle this problem, a progressive foreground-balanced sampling strategy on the basis of not adding change information is proposed in this article to help the model accurately learn the features of the changed pixels during the early training process and thereby improve detection performance.Furthermore, we design a discriminative Siamese network, hierarchical attention network (HANet), which can integrate multiscale features and refine detailed features. The main part of HANet is the HAN module, which is a lightweight and effective self-attention mechanism. Extensive experiments and ablation studies on two CDdatasets with extremely unbalanced labels validate the effectiveness and efficiency of the proposed method.

Change detection, as a research hotspot in the field of remote sensing, has witnessed continuous development and progress. However, the discrimination of boundary details remains a significant bottleneck due to the complexity of surrounding elements between change areas and backgrounds. Discriminating the boundaries of large change areas results in misalignment, while connecting boundaries occurs for small change targets. To address the above issues, a novel network based on the localization-then-refinement strategy is proposed in this paper, namely LRNet. LRNet consists of two stages: localization and refinement. In the localization stage, a three-branch encoder simultaneously extracts original image features and their differential features for interactive localization of the position of each change area. To minimize information loss during feature extraction, learnable optimal pooling (LOP) is proposed to replace the widely used max-pooling. Additionally, this process is trainable and contributes to the overall optimization of the network. To effectively interact features from different branches and accurately locate change areas of various sizes, change alignment attention (C2A) and hierarchical change alignment module (HCA) are proposed. In the refinement stage, the localization results from the localization stage are corrected by constraining the change areas and change edges through the edge-area alignment module (E2A). Subsequently, the decoder, combined with the difference features strengthened by C2A in the localization phase, refines change areas of different sizes, ultimately achieving accurate boundary discrimination of change areas. The proposed LRNet outperforms 13 other state-of-the-art methods in terms of comprehensive evaluation metrics and provides the most precise boundary discrimination results on the LEVIR-CD and WHU-CD datasets., Comment: 18 pages, 11 figures

This paper explores the generation of primordial black holes (PBHs) and scalar-induced gravitational waves (SIGWs) from the inflation potential with a tiny Lorentz function bump. We choose the Starobinsky model as basic potential, which satisfies the condition of observational constraints of the CMB at a large scale. We find that the tiny Lorentz function bump can enhance the primordial curvature power spectrum to $\mathcal{O}(10^{-2})$ at a small scale, leading to the formation of PBHs with sufficient abundance. Furthermore, we discover that the abundance of PBHs with mass $10^{-12}M_\odot$ is approximately $1$, which can make up almost all dark matter. In addition, the SIGWs generated by our models can be tested by the Square Kilometre Array (SKA), Pulsar Timing Arrays (PTA), TianQin, Taiji, Laser Interferometer Space Antenna (LISA), and DECIGO. It should be emphasized that, compared with the tiny Gaussian bump case, the Lorentz function bump case can easily produce a sufficient abundance of PBHs with a wider mass range, and accompanying a broader frequency range of SIGWs, which predicts the advantage of the Lorentz function bump for basic potential with big e-folding number $N$ around $55\sim60$., Comment: 13 pages, 10 figures

Capillary-driven flow of fluids occurs frequently in nature and has a wide range of technological applications in the fields of industry, agriculture, medicine, biotechnology, and microfluidics. By using the Onsager variational principle, we propose a model to systematically study the capillary imbibition in titled tubes, and find different laws of time-dependent capillary invasion length for liquid-liquid displacement system other than Lucas-Washburn type under different circumstances. The good agreement between our model and experimental results shows that the imbibition dynamics in a capillary tube with a prefilled liquid slug can be well captured by the dynamic equation derived in this paper. Our results bear important implications for macroscopic descriptions of multiphase flows in microfluidic systems and porous media.

The chi_c0(3915) was observed firstly by the Belle experiment in the omega J/psi invariant mass spectrum in the process B -> K omegaJ/psi, and then confirmed by the Babar experiment. The two experiments reported the resonant parameters of this particle in the both processes gammagamma -> omegaJ/psi and B -> K omega J/psi by assuming chi_c0(3915) as a S-wave Breit-Wigner resonance. We perform a global fit to the distributions of invariant mass of omega J/psi measured by the Belle and Babar experiments, and incorporate the measurement by the LHCb experiment additionally to extract the mass and width of chi_c0(3915). We obtain M = 3921.0 +/- 0.9 MeV/c2 and Gamma = 17.9 +/- 2.4 MeV which are consistent with the values on PDG within one standard deviation but with improved precision., Comment: 4 pages, 5 figures

Text-to-image diffusion models have achieved remarkable performance in image synthesis, while the text interface does not always provide fine-grained control over certain image factors. For instance, changing a single token in the text can have unintended effects on the image. This paper shows a simple modification of classifier-free guidance can help disentangle image factors in text-to-image models. The key idea of our method, Contrastive Guidance, is to characterize an intended factor with two prompts that differ in minimal tokens: the positive prompt describes the image to be synthesized, and the baseline prompt serves as a "baseline" that disentangles other factors. Contrastive Guidance is a general method we illustrate whose benefits in three scenarios: (1) to guide domain-specific diffusion models trained on an object class, (2) to gain continuous, rig-like controls for text-to-image generation, and (3) to improve the performance of zero-shot image editors.

Currently, Transformer is the most popular architecture for image dehazing, but due to its large computational complexity, its ability to handle long-range dependency is limited on resource-constrained devices. To tackle this challenge, we introduce the U-shaped Vision Mamba (UVM-Net), an efficient single-image dehazing network. Inspired by the State Space Sequence Models (SSMs), a new deep sequence model known for its power to handle long sequences, we design a Bi-SSM block that integrates the local feature extraction ability of the convolutional layer with the ability of the SSM to capture long-range dependencies. Extensive experimental results demonstrate the effectiveness of our method. Our method provides a more highly efficient idea of long-range dependency modeling for image dehazing as well as other image restoration tasks. The URL of the code is \url{https://github.com/zzr-idam/UVM-Net}. Our method takes only \textbf{0.009} seconds to infer a $325 \times 325$ resolution image (100FPS) without I/O handling time.

Recently, large models (Segment Anything model) came on the scene to provide a new baseline for polyp segmentation tasks. This demonstrates that large models with a sufficient image level prior can achieve promising performance on a given task. In this paper, we unfold a new perspective on polyp segmentation modeling by leveraging the Depth Anything Model (DAM) to provide depth prior to polyp segmentation models. Specifically, the input polyp image is first passed through a frozen DAM to generate a depth map. The depth map and the input polyp images are then concatenated and fed into a convolutional neural network with multiscale to generate segmented images. Extensive experimental results demonstrate the effectiveness of our method, and in addition, we observe that our method still performs well on images of polyps with noise. The URL of our code is \url{https://github.com/zzr-idam/Polyp-DAM}.

With the continuous advancement of imaging devices, the prevalence of Ultra-High-Definition (UHD) images is rising. Although many image restoration methods have achieved promising results, they are not directly applicable to UHD images on devices with limited computational resources due to the inherently high computational complexity of UHD images. In this paper, we focus on the task of low-light image enhancement (LLIE) and propose a novel LLIE method called MixNet, which is designed explicitly for UHD images. To capture the long-range dependency of features without introducing excessive computational complexity, we present the Global Feature Modulation Layer (GFML). GFML associates features from different views by permuting the feature maps, enabling efficient modeling of long-range dependency. In addition, we also design the Local Feature Modulation Layer (LFML) and Feed-forward Layer (FFL) to capture local features and transform features into a compact representation. This way, our MixNet achieves effective LLIE with few model parameters and low computational complexity. We conducted extensive experiments on both synthetic and real-world datasets, and the comprehensive results demonstrate that our proposed method surpasses the performance of current state-of-the-art methods. The code will be available at \url{https://github.com/zzr-idam/MixNet}.

Federated Learning (FL) addresses critical issues in machine learning related to data privacy and security, yet suffering from data insufficiency and imbalance under certain circumstances. The emergence of foundation models (FMs) offers potential solutions to the limitations of existing FL frameworks, e.g., by generating synthetic data for model initialization. However, due to the inherent safety concerns of FMs, integrating FMs into FL could introduce new risks, which remains largely unexplored. To address this gap, we conduct the first investigation on the vulnerability of FM integrated FL (FM-FL) under adversarial threats. Based on a unified framework of FM-FL, we introduce a novel attack strategy that exploits safety issues of FM to compromise FL client models. Through extensive experiments with well-known models and benchmark datasets in both image and text domains, we reveal the high susceptibility of the FM-FL to this new threat under various FL configurations. Furthermore, we find that existing FL defense strategies offer limited protection against this novel attack approach. This research highlights the critical need for enhanced security measures in FL in the era of FMs., Comment: Chen Wu and Xi Li are equal contribution. The corresponding author is Jiaqi Wang

Different types of phase transition from hadron to quark at high density near zero temperature may occur in the inner core of hybrid stars. We investigate the impacts of phase transition types and quark models on properties of hybrid star and quark cores. The quark-meson coupling (QMC) model is used to describe hadronic matter, and the MIT bag model as well as the Nambu-Jona-Lasinio (NJL) model are employed to describe quark matter for comparison. From the mass-radius curves obtained by using equations of state (EOS), we find that EOSs of hadron matter have a decisive influence on the maximum mass of a hybrid star in the first-order phase transition case, while quark matter EOSs have more impacts on results in the crossover transition case. It is also found in the present work that the thermodynamic correction arising from an interpolation scheme considerably stiffens the EOSs. Therefore the crossover type phase transition generally leads to hybrid stars with higher masses. In particular, by using the QMC model and the NJL model to construct crossover EOSs with thermodynamic correction, we discover that the maximum masses of hybrid stars can meet the recent observational constraint on mass from PSR J0952-0607, i.e., $2.35\pm0.17M_\odot$., Comment: 13pages, 10figures, 5 tables, revised version published in Physical Review D

Due to the potential applications in biomedical engineering, it becomes more and more important to understand the process of engulfment and internalization of nanoparticles (NPs) by cell membranes. Despite the fact that the interaction between cell membranes and passive particles has been widely studied, the interaction between cell membranes and self-propelled nonspherical NPs remains to be elucidated. Here we present a theoretical model to systematically investigate the influence of the active force, aspect ratio of NPs, particle size and membrane properties (adhesion energy density and membrane tension) on the cellular uptake of a nonspherical nanoparticle. It is found that the active force generated by an NP can trigger a type of first-order wrapping transition from a small partial wrapping state to a large one. In addition, the phase diagram in the force-aspect ratio (particle size, adhesion energy density and membrane tension) space displays more complex behaviors compared with that for the passive wrapping mediated merely by adhesion. These results may provide a useful guidance to the study of activity-driven cellular entry of active particles into cells.

Face inpainting is important in various applications, such as photo restoration, image editing, and virtual reality. Despite the significant advances in face generative models, ensuring that a person's unique facial identity is maintained during the inpainting process is still an elusive goal. Current state-of-the-art techniques, exemplified by MyStyle, necessitate resource-intensive fine-tuning and a substantial number of images for each new identity. Furthermore, existing methods often fall short in accommodating user-specified semantic attributes, such as beard or expression. To improve inpainting results, and reduce the computational complexity during inference, this paper proposes the use of Parallel Visual Attention (PVA) in conjunction with diffusion models. Specifically, we insert parallel attention matrices to each cross-attention module in the denoising network, which attends to features extracted from reference images by an identity encoder. We train the added attention modules and identity encoder on CelebAHQ-IDI, a dataset proposed for identity-preserving face inpainting. Experiments demonstrate that PVA attains unparalleled identity resemblance in both face inpainting and face inpainting with language guidance tasks, in comparison to various benchmarks, including MyStyle, Paint by Example, and Custom Diffusion. Our findings reveal that PVA ensures good identity preservation while offering effective language-controllability. Additionally, in contrast to Custom Diffusion, PVA requires just 40 fine-tuning steps for each new identity, which translates to a significant speed increase of over 20 times.

By designing a multi-channel millimeter Hall measurement configuration, we realize the carrier-density (locally) controllable measurement on the transport property in 2H MoS$_{2}$. We observe a linearly increased Hall conductivity and exponentially decreased resistivity as the increase of dc current. The intrinsically large band gap does not exhibit too much effect on our measurement, as far as the magnetic field is above the critical value, which is $B=6$ T for 2H-MoS$_{2}$. Instead, the edge effect which emerge as a result of one-dimensional channels. This is different from the Corbino geometry which is widely applied on semiconductors, where the edges are absent. At room temperature, we observe that the emergent quantized quantum Hall plateaus are at the same value for both the two measurements, which implies that the quantized conductivity does not depends on the non-Hermitian interactions, but the number of partially filled Landau levels, and this is in consistent with the previous theoretical works\cite{Siddiki}. At low-temperature limit, the Hall plateaus are destroyed due to the filtered contribution from the electrons above fermi energy, and in this case, the two measuremens exhibits stronger distinction, where we observe stronger fluctuations (of voltage, conductivity, and resistivity) at the currents between where there are Hall plateaus at higher temperature.

Federated learning (FL) represents a novel paradigm to machine learning, addressing critical issues related to data privacy and security, yet suffering from data insufficiency and imbalance. The emergence of foundation models (FMs) provides a promising solution to the problems with FL. For instance, FMs could serve as teacher models or good starting points for FL. However, the integration of FM in FL presents a new challenge, exposing the FL systems to potential threats. This paper investigates the robustness of FL incorporating FMs by assessing their susceptibility to backdoor attacks. Contrary to classic backdoor attacks against FL, the proposed attack (1) does not require the attacker fully involved in the FL process; (2) poses a significant risk in practical FL scenarios; (3) is able to evade existing robust FL frameworks/ FL backdoor defenses; (4) underscores the researches on the robustness of FL systems integrated with FMs. The effectiveness of the proposed attack is demonstrated by extensive experiments with various well-known models and benchmark datasets encompassing both text and image classification domains., Comment: This paper has been accepted by FL@FM-NeurIPS 23 (International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023). The corresponding author is Jiaqi Wang (jqwang@psu.edu)

Starting from a Hermitian operator with two distinct eigenvalues, we construct a non-Hermitian bipartite system in Gaussian orthogonal ensemble according to random matrix theory, where we introduce the off-diagonal fluctuations through random eigenkets and realizing the bipartite configuration consisting of two $D\times D$ subsystems (with $D$ the Hilbert space dimension). As required by the global thermalization (chaos), one of the two subsystems is full ranked, while the other is rank deficient. For the latter subsystem, there is a block with non-defective degeneracies containing the non-linear symmetries, as well as the accumulation effect of the linear map in adjacent eigenvectors. The maximally mixed state made by the eigenvectors of this special region exhibit not thermal ensmeble behavior (neither canonical or Gibbs), and exhibit similar character with the corresponding reduced density, which can be verified through the Loschmitch echo and variance of the imaginary spectrum. This non-defective degeneracy region partly meets the Lemma in 10.1103/PhysRevLett.122.220603 and theorem in 10.1103/PhysRevLett.120.150603. The coexistence of strong entanglement and initial state fidelity in this region make it possible to achieve a maximally mixed density which, however, not be a thermal canonical ensemble (with complete insensitivity to the environmental energy or temperature). Outside this region, the collection of eigenstates (reduced density) always exhibit restriction on the corresponding Hilbert space dimension, and thus suppress the thermaliation. There are abundant physics for those densities in Hermitian and non-Hermitian bases, where we investigate seperately in this work.

Change detection (CD) is a critical task in studying the dynamics of ecosystems and human activities using multi-temporal remote sensing images. While deep learning has shown promising results in CD tasks, it requires a large number of labeled and paired multi-temporal images to achieve high performance. Pairing and annotating large-scale multi-temporal remote sensing images is both expensive and time-consuming. To make deep learning-based CD techniques more practical and cost-effective, we propose an unsupervised single-temporal CD framework based on intra- and inter-image patch exchange (I3PE). The I3PE framework allows for training deep change detectors on unpaired and unlabeled single-temporal remote sensing images that are readily available in real-world applications. The I3PE framework comprises four steps: 1) intra-image patch exchange method is based on an object-based image analysis method and adaptive clustering algorithm, which generates pseudo-bi-temporal image pairs and corresponding change labels from single-temporal images by exchanging patches within the image; 2) inter-image patch exchange method can generate more types of land-cover changes by exchanging patches between images; 3) a simulation pipeline consisting of several image enhancement methods is proposed to simulate the radiometric difference between pre- and post-event images caused by different imaging conditions in real situations; 4) self-supervised learning based on pseudo-labels is applied to further improve the performance of the change detectors in both unsupervised and semi-supervised cases. Extensive experiments on two large-scale datasets demonstrate that I3PE outperforms representative unsupervised approaches and achieves F1 value improvements of 10.65% and 6.99% to the SOTA method. Moreover, I3PE can improve the performance of the ... (see the original article for full abstract)

This paper investigates the metric of previously proposed regular black holes, calculates their effective potentials, and plots the curves of the effective potentials. By determining the conserved quantities, the dynamical equations for particles and photons near the black hole are derived. The analysis encompasses timelike and null geodesics in different spacetimes, including bound geodesics, unstable circular geodesics, stable circular geodesics, and escape geodesics. The findings are presented through figures and tables. Furthermore, the bound geodesics of the four regular black hole spacetimes are analyzed, examining the average distance of particle orbits from the center of the event horizon, the precession behavior of the perihelion, and the probability of particles appearing inside the outer event horizon during motion. Based on these analyses, a general formula is proposed, which yields the existing metrics when specific parameter values are chosen. The impact of parameter variations on the effective potential and geodesics is then computed using this new formula., Comment: 23 pages, 13 figures

Designing reward functions is a longstanding challenge in reinforcement learning (RL); it requires specialized knowledge or domain data, leading to high costs for development. To address this, we introduce Text2Reward, a data-free framework that automates the generation and shaping of dense reward functions based on large language models (LLMs). Given a goal described in natural language, Text2Reward generates shaped dense reward functions as an executable program grounded in a compact representation of the environment. Unlike inverse RL and recent work that uses LLMs to write sparse reward codes or unshaped dense rewards with a constant function across timesteps, Text2Reward produces interpretable, free-form dense reward codes that cover a wide range of tasks, utilize existing packages, and allow iterative refinement with human feedback. We evaluate Text2Reward on two robotic manipulation benchmarks (ManiSkill2, MetaWorld) and two locomotion environments of MuJoCo. On 13 of the 17 manipulation tasks, policies trained with generated reward codes achieve similar or better task success rates and convergence speed than expert-written reward codes. For locomotion tasks, our method learns six novel locomotion behaviors with a success rate exceeding 94%. Furthermore, we show that the policies trained in the simulator with our method can be deployed in the real world. Finally, Text2Reward further improves the policies by refining their reward functions with human feedback. Video results are available at https://text-to-reward.github.io/ ., Comment: ICLR 2024 camera ready, 37 pages, 12 figures

Text-to-image generative models often reflect the biases of the training data, leading to unequal representations of underrepresented groups. This study investigates inclusive text-to-image generative models that generate images based on human-written prompts and ensure the resulting images are uniformly distributed across attributes of interest. Unfortunately, directly expressing the desired attributes in the prompt often leads to sub-optimal results due to linguistic ambiguity or model misrepresentation. Hence, this paper proposes a drastically different approach that adheres to the maxim that "a picture is worth a thousand words". We show that, for some attributes, images can represent concepts more expressively than text. For instance, categories of skin tones are typically hard to specify by text but can be easily represented by example images. Building upon these insights, we propose a novel approach, ITI-GEN, that leverages readily available reference images for Inclusive Text-to-Image GENeration. The key idea is learning a set of prompt embeddings to generate images that can effectively represent all desired attribute categories. More importantly, ITI-GEN requires no model fine-tuning, making it computationally efficient to augment existing text-to-image models. Extensive experiments demonstrate that ITI-GEN largely improves over state-of-the-art models to generate inclusive images from a prompt. Project page: https://czhang0528.github.io/iti-gen., Comment: Accepted to ICCV 2023 (Oral Presentation)

Change detection (CD) is a fundamental and important task for monitoring the land surface dynamics in the earth observation field. Existing deep learning-based CD methods typically extract bi-temporal image features using a weight-sharing Siamese encoder network and identify change regions using a decoder network. These CD methods, however, still perform far from satisfactorily as we observe that 1) deep encoder layers focus on irrelevant background regions and 2) the models' confidence in the change regions is inconsistent at different decoder stages. The first problem is because deep encoder layers cannot effectively learn from imbalanced change categories using the sole output supervision, while the second problem is attributed to the lack of explicit semantic consistency preservation. To address these issues, we design a novel similarity-aware attention flow network (SAAN). SAAN incorporates a similarity-guided attention flow module with deeply supervised similarity optimization to achieve effective change detection. Specifically, we counter the first issue by explicitly guiding deep encoder layers to discover semantic relations from bi-temporal input images using deeply supervised similarity optimization. The extracted features are optimized to be semantically similar in the unchanged regions and dissimilar in the changing regions. The second drawback can be alleviated by the proposed similarity-guided attention flow module, which incorporates similarity-guided attention modules and attention flow mechanisms to guide the model to focus on discriminative channels and regions. We evaluated the effectiveness and generalization ability of the proposed method by conducting experiments on a wide range of CD tasks. The experimental results demonstrate that our method achieves excellent performance on several CD tasks, with discriminative features and semantic consistency preserved., Comment: 15 pages,13 figures

Remote sensing image change detection aims to identify the differences between images acquired at different times in the same area. It is widely used in land management, environmental monitoring, disaster assessment and other fields. Currently, most change detection methods are based on Siamese network structure or early fusion structure. Siamese structure focuses on extracting object features at different times but lacks attention to change information, which leads to false alarms and missed detections. Early fusion (EF) structure focuses on extracting features after the fusion of images of different phases but ignores the significance of object features at different times for detecting change details, making it difficult to accurately discern the edges of changed objects. To address these issues and obtain more accurate results, we propose a novel network, Triplet UNet(T-UNet), based on a three-branch encoder, which is capable to simultaneously extract the object features and the change features between the pre- and post-time-phase images through triplet encoder. To effectively interact and fuse the features extracted from the three branches of triplet encoder, we propose a multi-branch spatial-spectral cross-attention module (MBSSCA). In the decoder stage, we introduce the channel attention mechanism (CAM) and spatial attention mechanism (SAM) to fully mine and integrate detailed textures information at the shallow layer and semantic localization information at the deep layer., Comment: 21 pages, 11 figures, 6 tables

We present a novel interpretation of the thermodynamics of perfect fluid dark matter (PFDM) black hole based on Misner-Sharp energy, and then investigate its evaporation behavior. We find that the ratio between dark sector initial density and black hole horizon radius significantly influences black hole evaporation behaviors. We demonstrate that the presence of the dark sector can significantly extend the lifetime of a black hole which is similar to the Reissner-Nordstrom case. Our work reformulates the thermodynamics of PFDM black holes and points out the existence of long-lived black holes in the presence of the dark sector., Comment: 14 pages, 2 figures, comments are welcomed. v2: typos corrected

In this work, we study the properties of neutron stars using the linear Relativistic Mean-Field (RMF) theory and consider multiple degrees of freedom inside neutron stars, including hyperons and $\Delta$ resonances. We investigate different coupling parameters $x_{\sigma \Delta}$ between $\Delta$ resonances and nucleons and compare the differences between neutron stars with and without strange mesons $\sigma^*$ and $\phi$. These effects include particle number distributions, equations of state (EOS), mass-radius relations, and tidal deformabilities. To overcome the "hyperon puzzle," we employ the $\sigma-cut$ scheme to obtain neutron stars with masses up to $2M_{\odot}$. We find that strange mesons appear at around 3$\rho_0$ and reduce the critical density of baryons in the high-density region. With increasing coupling parameter $x_{\sigma \Delta}$, the $\Delta$ resonances suppress hyperons, leading to a shift of the critical density towards lower values. The early appearance of $\Delta$ resonances may play a crucial role in the stability of neutron stars. Strange mesons soften the EOS slightly, while $\Delta$ resonances predominantly soften the EOS in the low-density region. By calculating tidal deformabilities and comparing with astronomical observation GW170817, we find that the inclusion of $\Delta$ resonances decreases the radius of neutron stars., Comment: 10 pages, 9 figures

Buildings are the basic carrier of social production and human life; roads are the links that interconnect social networks. Building and road information has important application value in the frontier fields of regional coordinated development, disaster prevention, auto-driving, etc. Mapping buildings and roads from very high-resolution (VHR) remote sensing images have become a hot research topic. However, the existing methods often ignore the strong spatial correlation between roads and buildings and extract them in isolation. To fully utilize the complementary advantages between buildings and roads, we propose a building-road collaborative extraction method based on multi-task and cross-scale feature interaction to improve the accuracy of both tasks in a complementary way. A multi-task interaction module is proposed to interact information across tasks and preserve the unique information of each task, which tackle the seesaw phenomenon in multitask learning. By considering the variation in appearance and structure between buildings and roads, a cross-scale interaction module is designed to automatically learn the optimal reception field for different tasks. Compared with many existing methods that train each task individually, the proposed collaborative extraction method can utilize the complementary advantages between buildings and roads by the proposed inter-task and inter-scale feature interactions, and automatically select the optimal reception field for different tasks. Experiments on a wide range of urban and rural scenarios show that the proposed algorithm can achieve building-road extraction with outstanding performance and efficiency., Comment: IEEE Transactions on Geoscience and Remote Sensing

Change detection (CD) is an important yet challenging task in the Earth observation field for monitoring Earth surface dynamics. The advent of deep learning techniques has recently propelled automatic CD into a technological revolution. Nevertheless, deep learning-based CD methods are still plagued by two primary issues: 1) insufficient temporal relationship modeling and 2) pseudo-change misclassification. To address these issues, we complement the strong temporal modeling ability of metric learning with the prominent fitting ability of segmentation and propose a deep change feature learning (DeepCL) framework for robust and explainable CD. Firstly, we designed a hard sample-aware contrastive loss, which reweights the importance of hard and simple samples. This loss allows for explicit modeling of the temporal correlation between bi-temporal remote sensing images. Furthermore, the modeled temporal relations are utilized as knowledge prior to guide the segmentation process for detecting change regions. The DeepCL framework is thoroughly evaluated both theoretically and experimentally, demonstrating its superior feature discriminability, resilience against pseudo changes, and adaptability to a variety of CD algorithms. Extensive comparative experiments substantiate the quantitative and qualitative superiority of DeepCL over state-of-the-art CD approaches., Comment: 12 pages,7 figures, submitted to IEEE Transactions on Image Processing

The efficacy of building footprint segmentation from remotely sensed images has been hindered by model transfer effectiveness. Many existing building segmentation methods were developed upon the encoder-decoder architecture of U-Net, in which the encoder is finetuned from the newly developed backbone networks that are pre-trained on ImageNet. However, the heavy computational burden of the existing decoder designs hampers the successful transfer of these modern encoder networks to remote sensing tasks. Even the widely-adopted deep supervision strategy fails to mitigate these challenges due to its invalid loss in hybrid regions where foreground and background pixels are intermixed. In this paper, we conduct a comprehensive evaluation of existing decoder network designs for building footprint segmentation and propose an efficient framework denoted as BFSeg to enhance learning efficiency and effectiveness. Specifically, a densely-connected coarse-to-fine feature fusion decoder network that facilitates easy and fast feature fusion across scales is proposed. Moreover, considering the invalidity of hybrid regions in the down-sampled ground truth during the deep supervision process, we present a lenient deep supervision and distillation strategy that enables the network to learn proper knowledge from deep supervision. Building upon these advancements, we have developed a new family of building segmentation networks, which consistently surpass prior works with outstanding performance and efficiency across a wide range of newly developed encoder networks.

Weakly-supervised change detection (WSCD) aims to detect pixel-level changes with only image-level annotations. Owing to its label efficiency, WSCD is drawing increasing attention recently. However, current WSCD methods often encounter the challenge of change missing and fabricating, i.e., the inconsistency between image-level annotations and pixel-level predictions. Specifically, change missing refer to the situation that the WSCD model fails to predict any changed pixels, even though the image-level label indicates changed, and vice versa for change fabricating. To address this challenge, in this work, we leverage global-scale and local-scale priors in WSCD and propose two components: a Dilated Prior (DP) decoder and a Label Gated (LG) constraint. The DP decoder decodes samples with the changed image-level label, skips samples with the unchanged label, and replaces them with an all-unchanged pixel-level label. The LG constraint is derived from the correspondence between changed representations and image-level labels, penalizing the model when it mispredicts the change status. Additionally, we develop TransWCD, a simple yet powerful transformer-based model, showcasing the potential of weakly-supervised learning in change detection. By integrating the DP decoder and LG constraint into TransWCD, we form TransWCD-DL. Our proposed TransWCD and TransWCD-DL achieve significant +6.33% and +9.55% F1 score improvements over the state-of-the-art methods on the WHU-CD dataset, respectively. Some performance metrics even exceed several fully-supervised change detection (FSCD) competitors. Code will be available at https://github.com/zhenghuizhao/TransWCD.

The electronic band structure is an intrinsic property of solid-state materials that is intimately connected to the crystalline arrangement of atoms. Moir\'e crystals, which emerge in twisted stacks of atomic layers, feature a band structure that can be continuously tuned by changing the twist angle between adjacent layers. This class of artificial materials blends the discrete nature of the moir\'e superlattice with intrinsic symmetries of the constituent materials, providing a versatile platform for investigation of correlated phenomena whose origins are rooted in the geometry of the superlattice, from insulating states at "magic angles" to flat bands in quasicrystals. Here we present a route to mechanically tune the twist angle of individual atomic layers with a precision of a fraction of a degree inside a scanning probe microscope, which enables continuous control of the electronic band structure in-situ. Using nanostructured rotor devices, we achieve the collective rotation of a single layer of atoms with minimal deformation of the crystalline lattice. In twisted bilayer graphene, we demonstrate nanoscale control of the moir\'e superlattice period via external rotations, as revealed using piezoresponse force microscopy. We also extend this methodology to create twistable boron nitride devices, which could enable dynamic control of the domain structure of moir\'e ferroelectrics. This approach provides a route for real-time manipulation of moir\'e materials, allowing for systematic exploration of the phase diagrams at multiple twist angles in a single device.

In a generic random system, the coexistence of extended and localized states can be evidenced by the subextensive width of energy distribution of a physical initial state in, for example, the quantum quenches which involving the local Hamiltonian. The robust thermalization is also evidenced in terms of the microscopic canonical ensemble average in the thermalization limit\cite{Shiraishi}, which satisfies the weak eigenstate thermalization hypothesis (ETH). In this article, we study the method of two-particle self-consistency for a system without condensation, i.e., without the inaccessible localizations that violating the ETH. We provide another pespective that considering the local conservation of an nonintegrable system the stubborn correlations between the three kinds of decompositions for the four-point functions, which can be regarded as the elements in a product of the self-energy and Green function matrices (i.e., the two-particle correlations in Kadanoff and Baym notation\cite{Vilk Y M}).

Oscillation modes of compact stars, in general, can serve as a fingerprint in determining the equation of state (EOS) of dense matter. In this study, we examine the impact of symmetry energy slope ($L$) on the oscillation frequencies of neutron stars (NSs) with nucleon-nucleon short range correlation (SRC) and admixed dark matter (DM) for the first time within the relativistic mean-field theory. By adjusting the $L$, we revise the EOS and coupling parameters in light of the SRC and DM effects, and construct the new sets. The results reveal that NSs containing SRC and DM inside are more likely to satisfy the observational constraints, and we find that smaller $L$ exhibits larger fundamental non-radial and radial frequencies, and that the effect on Large Separation (LG) is also mainly concentrated in the low-mass region. Moreover, we update the linear relationship between the non-radial frequency and mean density, and we further give empirical relations between non-radial and radial frequencies and tidal deformability at different $L$ for 1.4$M_{\odot}$ and 2$M_{\odot}$. These findings will enable us to more effectively confine the NS EOSs, in turn, also provide a strategy to place constraints on the $L$., Comment: 18 pages, 13 figures

Microwave impedance microscopy (MIM) is a near-field imaging technique that has been used to visualize the local conductivity of materials with nanoscale resolution across the GHz regime. In recent years, MIM has shown great promise for the investigation of topological states of matter, correlated electronic states and emergent phenomena in quantum materials. To explore these low-energy phenomena, many of which are only detectable in the milliKelvin regime, we have developed a novel low-temperature MIM incorporated into a dilution refrigerator. This setup which consists of a tuning-fork-based atomic force microscope with microwave reflectometry capabilities, is capable of reaching temperatures down to 70 mK during imaging and magnetic fields up to 9 T. To test the performance of this microscope, we demonstrate microwave imaging of the conductivity contrast between graphite and silicon dioxide at cryogenic temperatures and discuss the resolution and noise observed in these results. We extend this methodology to visualize edge conduction in Dirac semimetal cadmium arsenide in the quantum Hall regime, Comment: This article may be downloaded for personal use only. Any other use requires prior permission of the author and AIP Publishing. This article appeared in Review of Scientific Instruments and may be found at doi.org/10.1063/5.0159548

Photoplethysmogram (PPG) signals are easily contaminated by motion artifacts in real-world settings, despite their widespread use in Internet-of-Things (IoT) based wearable and smart health devices for cardiovascular health monitoring. This study proposed a lightweight deep neural network, called Tiny-PPG, for accurate and real-time PPG artifact segmentation on IoT edge devices. The model was trained and tested on a public dataset, PPG DaLiA, which featured complex artifacts with diverse lengths and morphologies during various daily activities of 15 subjects using a watch-type device (Empatica E4). The model structure, training method and loss function were specifically designed to balance detection accuracy and speed for real-time PPG artifact detection in resource-constrained embedded devices. To optimize the model size and capability in multi-scale feature representation, the model employed depth-wise separable convolution and atrous spatial pyramid pooling modules, respectively. Additionally, the contrastive loss was also utilized to further optimize the feature embeddings. With additional model pruning, Tiny-PPG achieved state-of-the-art detection accuracy of 87.4% while only having 19,726 model parameters (0.15 megabytes), and was successfully deployed on an STM32 embedded system for real-time PPG artifact detection. Therefore, this study provides an effective solution for resource-constraint IoT smart health devices in PPG artifact detection.

We investigate the eigenstate thermalization in terms of a Hermitian operator and the complex eigenkets that follows Gaussian ensemble distribution. With the non-Hermitian open bipartite system, there are, however, some global restrictions such that the elements share some of the properties of Gaussian orthogonal ensemble in diagonal and off-diagonal perspective. Such global restrictions enforce that one of the subsystem contains a nullspace with non-defective degeneracies (the primary subsystem), and the another full-ranked subsystem (the secondary subsystem). For the primary subsystem, the mixed densities in Hermitian and non-Hermitian basis exhibits global fluctuation and unidirectional (non-Hermitian skin effect) fluctuation, respectively. The former is due to the global restrictions of the whole system which plays teh role of environmental disorder, while the latter is due to the nonlocal symmetries which is allowed in the restricted Hilbert space. We also investigate the integrablity-chaos transition with independent perturbations in terms of the Berry autocorrelation in semicalssical limit, where there is a phase space spanned by the momentum-like projection and the range of local wave function.

Wearing face mask is an effective measure to reduce the risk of COVID-19 infections and control its transmission, thus its usage survey is important for better policy decision to mitigate the epidemic spread. Current existing worldwide surveys are mostly self-reported, whose accuracies are hard to guaranteed, and may exaggerate the percentage of face mask wearing. Therefore, we collected busking videos with a large amount on YouTube from December 2019 to December 2020, mainly from South Korea, and reported an objective investigation of face mask use in the crowds outdoor. It is found that the face mask wearing rate has an obvious positive correlation with effective reproductive number (Rt) in the South Korea, which indicates that the people in South Korea kept sensitive to the COVID-19 epidemic. The face mask wearing rate in South Korea is higher than some other countries, and two rate droppings in June and September also corresponds to the temporary remission in 2020. This study shows significant potentials to utilize public big video data to make an accurate worldwide survey of face mask use with the support of deep learning technology.