Search

Dense colored point clouds enhance visual perception and are of significant value in various robotic applications. However, existing learning-based point cloud upsampling methods are constrained by computational resources and batch processing strategies, which often require subdividing point clouds into smaller patches, leading to distortions that degrade perceptual quality. To address this challenge, we propose a novel 2D-3D hybrid colored point cloud upsampling framework (GaussianPU) based on 3D Gaussian Splatting (3DGS) for robotic perception. This approach leverages 3DGS to bridge 3D point clouds with their 2D rendered images in robot vision systems. A dual scale rendered image restoration network transforms sparse point cloud renderings into dense representations, which are then input into 3DGS along with precise robot camera poses and interpolated sparse point clouds to reconstruct dense 3D point clouds. We have made a series of enhancements to the vanilla 3DGS, enabling precise control over the number of points and significantly boosting the quality of the upsampled point cloud for robotic scene understanding. Our framework supports processing entire point clouds on a single consumer-grade GPU, such as the NVIDIA GeForce RTX 3090, eliminating the need for segmentation and thus producing high-quality, dense colored point clouds with millions of points for robot navigation and manipulation tasks. Extensive experimental results on generating million-level point cloud data validate the effectiveness of our method, substantially improving the quality of colored point clouds and demonstrating significant potential for applications involving large-scale point clouds in autonomous robotics and human-robot interaction scenarios., Comment: 7 pages, 5 figures

Large Language Models (LLMs) have garnered widespread attention due to their remarkable performance across various tasks. However, to mitigate the issue of hallucinations, LLMs often incorporate retrieval-augmented pipeline to provide them with rich external knowledge and context. Nevertheless, challenges stem from inaccurate and coarse-grained context retrieved from the retriever. Supplying irrelevant context to the LLMs can result in poorer responses, increased inference latency, and higher costs. This paper introduces a method called Instruction-Aware Contextual Compression, which filters out less informative content, thereby accelerating and enhancing the use of LLMs. The experimental results demonstrate that Instruction-Aware Contextual Compression notably reduces memory consumption and minimizes generation latency while maintaining performance levels comparable to those achieved with the use of the full context. Specifically, we achieved a 50% reduction in context-related costs, resulting in a 5% reduction in inference memory usage and a 2.2-fold increase in inference speed, with only a minor drop of 0.047 in Rouge-1. These findings suggest that our method strikes an effective balance between efficiency and performance., Comment: 20 pages

In this work, we investigate the resonance production in $\Lambda^+_c \to \gamma \pi^+ \Lambda$ decay through the triangle singularity (TS) mechanism, within an effective Lagrangian approach. We find that the appropriate loop decay process could develop a triangle singularity in the invariant mass $M_{\pi\Lambda}$ around $1.35$ GeV, with the shape depending on the quantum numbers of $\Sigma^*$ states that couple to the final $\pi\Lambda$ system. Especially, the $\Sigma(1380)1/2^-$ state newly predicted in the pentaquark model, if exists, significantly enhances the TS contribution and sharpens the TS peak due to the $S$-wave $\Sigma^*\pi\Sigma$ vertex in the loop. Therefore, the existence of the $\Sigma(1380)1/2^-$ state can be examined by measuring the TS signal in $\Lambda^+_c \to \gamma \pi^+ \Lambda$. Considering also possible tree-level diagrams, we additionally obtain the branching ratio ${\rm Br}(\Lambda^+_c \to \gamma \pi^+ \Lambda)\simeq 1.5\times 10^{-5}$. We suggest the investigation of this reaction by future BESIII, LHCb, and Belle experiments., Comment: 8 pages, 6 figures

A quantum network provides an infrastructure connecting quantum devices with revolutionary computing, sensing, and communication capabilities. As the best-known application of a quantum network, quantum key distribution (QKD) shares secure keys guaranteed by the laws of quantum mechanics. A quantum satellite constellation offers a solution to facilitate the quantum network on a global scale. The Micius satellite has verified the feasibility of satellite quantum communications, however, scaling up quantum satellite constellations is challenging, requiring small lightweight satellites, portable ground stations and real-time secure key exchange. Here we tackle these challenges and report the development of a quantum microsatellite capable of performing space-to-ground QKD using portable ground stations. The quantum microsatellite features a payload weighing approximately 23 kg, while the portable ground station weighs about 100 kg. These weights represent reductions by more than an order and two orders of magnitude, respectively, compared to the Micius satellite. Additionally, we multiplex bidirectional satellite-ground optical communication with quantum communication, enabling key distillation and secure communication in real-time. Using the microsatellite and the portable ground stations, we demonstrate satellite-based QKD with multiple ground stations and achieve the sharing of up to 0.59 million bits of secure keys during a single satellite pass. The compact quantum payload can be readily assembled on existing space stations or small satellites, paving the way for a satellite-constellation-based quantum and classical network for widespread real-life applications., Comment: 40 pages, 8 figures

Meta-learning has been extensively applied in the domains of few-shot learning and fast adaptation, achieving remarkable performance. While Meta-learning methods like Model-Agnostic Meta-Learning (MAML) and its variants provide a good set of initial parameters for the model, the model still tends to learn shortcut features, which leads to poor generalization. In this paper, we propose the formal conception of "learn to learn more precisely", which aims to make the model learn precise target knowledge from data and reduce the effect of noisy knowledge, such as background and noise. To achieve this target, we proposed a simple and effective meta-learning framework named Meta Self-Distillation(MSD) to maximize the consistency of learned knowledge, enhancing the models' ability to learn precise target knowledge. In the inner loop, MSD uses different augmented views of the same support data to update the model respectively. Then in the outer loop, MSD utilizes the same query data to optimize the consistency of learned knowledge, enhancing the model's ability to learn more precisely. Our experiment demonstrates that MSD exhibits remarkable performance in few-shot classification tasks in both standard and augmented scenarios, effectively boosting the accuracy and consistency of knowledge learned by the model., Comment: 10pages,4 figures, meta learning

Quantum nonlocality based on state discrimination describes the global property of the set of orthogonal states and has a wide range of applications in quantum cryptographic protocols. Strongest nonlocality is the strongest form of quantum nonlocality recently presented in multipartite quantum systems: a set of orthogonal multipartite quantum states is strongest nonlocal if the only orthogonality-preserving local measurements on the subsystems in every bipartition are trivial. In this work, we found a construction of strongest nonlocal sets in $\mathbb{C}^{d_{1}}\otimes \mathbb{C}^{d_{2}}\otimes \mathbb{C}^{d_{3}}$ $(2\leq d_{1}\leq d_{2}\leq d_{3})$ of size $d_2d_3+1$ without stopper states. Then we obtain the strongest nonlocal sets in four-partite systems with $d^3+1$ orthogonal states in $\mathbb{C}^d\otimes \mathbb{C}^{d}\otimes \mathbb{C}^{d}\otimes \mathbb{C}^{d}$ $(d\geq2)$ and $d_{2}d_{3}d_{4}+1$ orthogonal states in $\mathbb{C}^{d_{1}}\otimes \mathbb{C}^{d_{2}}\otimes \mathbb{C}^{d_{3}}\otimes \mathbb{C}^{d_{4}}$ $(2\leq d_{1}\leq d_{2}\leq d_{3}\leq d_{4})$. Surprisingly, the number of the elements in all above constructions perfectly reaches the recent conjectured lower bound and reduces the size of the strongest nonlocal set in $\mathbb{C}^{d}\otimes \mathbb{C}^{d}\otimes \mathbb{C}^{d}\otimes \mathbb{C}^{d}$ of [\href{https://doi.org/10.1103/PhysRevA.108.062407}{Phys. Rev. A \textbf{108}, 062407 (2023)}] by $d-2$. In particular, the general optimal construction of the strongest nonlocal set in four-partite system is completely solved for the first time, which further highlights the theory of quantum nonlocality from the perspective of state discrimination., Comment: 14 pages, 3 figures

Two-dimensional materials with Janus structure host novel physical properties due to their inversional symmetry breaking. However, it remains elusive to synthesize Janus monolayer crystals with tailored long-range magnetic orders. Here, we have developed a general method to fabricate uniform Janus CrTeSe monolayers by selective selenization of preformed CrTe2 monolayers with molecular beam epitaxy. The uniform Janus structure of CrTeSe with high crystal quality is confirmed by high-resolution scanning transmission electron microscopy. Spin-polarized scanning tunneling microscopy/spectroscopy measurements unveil that the Janus CrTeSe undergoes a charge density wave (CDW) transition and a robust antiferromagnetic order. The magnetic anisotropy of CrTeSe is drastically altered compared to monolayer CrTe2 by the breaking symmetries induced from the Janus structure and the CDW transition, as is substantiated with first principles calculations. Our research achieves the construction of large-area Janus structures, and artificially tailors the electronic and magnetic properties of Janus systems at the two-dimensional limit., Comment: 19 pages, 4 figures

Structured Sentiment Analysis (SSA) was cast as a problem of bi-lexical dependency graph parsing by prior studies. Multiple formulations have been proposed to construct the graph, which share several intrinsic drawbacks: (1) The internal structures of spans are neglected, thus only the boundary tokens of spans are used for relation prediction and span recognition, thus hindering the model's expressiveness; (2) Long spans occupy a significant proportion in the SSA datasets, which further exacerbates the problem of internal structure neglect. In this paper, we treat the SSA task as a dependency parsing task on partially-observed dependency trees, regarding flat spans without determined tree annotations as latent subtrees to consider internal structures of spans. We propose a two-stage parsing method and leverage TreeCRFs with a novel constrained inside algorithm to model latent structures explicitly, which also takes advantages of joint scoring graph arcs and headed spans for global optimization and inference. Results of extensive experiments on five benchmark datasets reveal that our method performs significantly better than all previous bi-lexical methods, achieving new state-of-the-art.

Affective computing stands at the forefront of artificial intelligence (AI), seeking to imbue machines with the ability to comprehend and respond to human emotions. Central to this field is emotion recognition, which endeavors to identify and interpret human emotional states from different modalities, such as speech, facial images, text, and physiological signals. In recent years, important progress has been made in generative models, including Autoencoder, Generative Adversarial Network, Diffusion Model, and Large Language Model. These models, with their powerful data generation capabilities, emerge as pivotal tools in advancing emotion recognition. However, up to now, there remains a paucity of systematic efforts that review generative technology for emotion recognition. This survey aims to bridge the gaps in the existing literature by conducting a comprehensive analysis of over 320 research papers until June 2024. Specifically, this survey will firstly introduce the mathematical principles of different generative models and the commonly used datasets. Subsequently, through a taxonomy, it will provide an in-depth analysis of how generative techniques address emotion recognition based on different modalities in several aspects, including data augmentation, feature extraction, semi-supervised learning, cross-domain, etc. Finally, the review will outline future research directions, emphasizing the potential of generative models to advance the field of emotion recognition and enhance the emotional intelligence of AI systems., Comment: Under Review

Three types of cycles are identified in the quantum jump trajectories of the Scovil--Schulz-DuBois (SSDB) machine: an R-cycle as refrigeration, an H-cycle as a heat engine, and an N-cycle in which the machine is neutral. The statistics of these cycles are investigated via a semi-Markov process method. We find that in the large time limit, whether the machine operates as a heat engine or refrigerator depends on a balance between the numbers of R-cycles and H-cycles per unit time. Further increasing the hot bath temperature above a certain threshold does not increase the machine's power output. The cause is that, in this situation, the N-cycle has a greater probability than the H-cycle and R-cycle. Although the SSDB machine operates by randomly switching among these three cycles, at the level of a single quantum jump trajectory, its heat engine efficiency and the refrigerator's coefficient of performance remain constant., Comment: 3 figures

The detailed study of the strong gravitational lensing of a Kerr black hole within Quantum Einstein Gravity (QEG) is performed. We calculate the photon sphere, the deflection angle of light, and observables on the equatorial plane under the strong deflection limit in a vacuum. The presence of quantum effects reduces the radius of the photon sphere, the magnification, the position of relativistic images, and the time delays on the same side of the lens. However, it increases the strong deflection angle, the separations, and the time delays on the opposite side of the lens. By modeling M87* and Sgr A* as the Kerr black hole within QEG, we find that the time delays are more significant in M87*, while other observables are more pronounced in Sgr A*. Furthermore, we consider the influence of plasma on the gravitational lensing effect. Plasma causes an additional deflection of light, increasing the magnification, images position and the time delays, but decreasing the separations. More importantly, we calculate the time delays under the strong deflection limit in the presence of plasma, and they increase with higher plasma concentrations. Our research may help to evaluate the observational imprints left by such quantum effects in the propagation of light and the impact of plasma around black holes on gravitational lensing., Comment: 15 pages, 15 figures

Clinical notes contain unstructured representations of patient histories, including the relationships between medical problems and prescription drugs. To investigate the relationship between cancer drugs and their associated symptom burden, we extract structured, semantic representations of medical problem and drug information from the clinical narratives of oncology notes. We present Clinical Concept Annotations for Cancer Events and Relations (CACER), a novel corpus with fine-grained annotations for over 48,000 medical problems and drug events and 10,000 drug-problem and problem-problem relations. Leveraging CACER, we develop and evaluate transformer-based information extraction (IE) models such as BERT, Flan-T5, Llama3, and GPT-4 using fine-tuning and in-context learning (ICL). In event extraction, the fine-tuned BERT and Llama3 models achieved the highest performance at 88.2-88.0 F1, which is comparable to the inter-annotator agreement (IAA) of 88.4 F1. In relation extraction, the fine-tuned BERT, Flan-T5, and Llama3 achieved the highest performance at 61.8-65.3 F1. GPT-4 with ICL achieved the worst performance across both tasks. The fine-tuned models significantly outperformed GPT-4 in ICL, highlighting the importance of annotated training data and model optimization. Furthermore, the BERT models performed similarly to Llama3. For our task, LLMs offer no performance advantage over the smaller BERT models. The results emphasize the need for annotated training data to optimize models. Multiple fine-tuned transformer models achieved performance comparable to IAA for several extraction tasks., Comment: This is a pre-copy-editing, author-produced PDF of an article accepted for publication in JAMIA following peer review. The definitive publisher-authenticated version is available online at https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocae231/7748302

We consider the derivative of the characteristic polynomial of $N \times N$ Haar distributed unitary matrices. In the limit $N \to \infty$, we give a formula for general non-integer moments of the derivative for values of the spectral variable inside the unit disc. The formula we obtain is given in terms of a confluent hypergeometric function. We give an extension of this result to joint moments taken at different points, again valid for non-integer moment orders. For integer moments, our results simplify and are expressed as a single Laguerre polynomial. Using these moment formulae, we derive the mean counting function of zeros of the derivative in the limit $N \to \infty$. This motivated us to consider related questions for the derivative of the Riemann zeta function. Assuming the Lindel\"of hypothesis, we show that integer moments away from but approaching the critical line agree with our random matrix results. Within random matrix theory, we obtain an explicit expression for the integer moments, for finite matrix size, as a sum over partitions. We use this to obtain asymptotic formulae in a regime where the spectral variable approaches or is on the unit circle., Comment: 45 pages

Audio-driven talking face generation aims to synthesize video with lip movements synchronized to input audio. However, current generative techniques face challenges in preserving intricate regional textures (skin, teeth). To address the aforementioned challenges, we propose a novel framework called SegTalker to decouple lip movements and image textures by introducing segmentation as intermediate representation. Specifically, given the mask of image employed by a parsing network, we first leverage the speech to drive the mask and generate talking segmentation. Then we disentangle semantic regions of image into style codes using a mask-guided encoder. Ultimately, we inject the previously generated talking segmentation and style codes into a mask-guided StyleGAN to synthesize video frame. In this way, most of textures are fully preserved. Moreover, our approach can inherently achieve background separation and facilitate mask-guided facial local editing. In particular, by editing the mask and swapping the region textures from a given reference image (e.g. hair, lip, eyebrows), our approach enables facial editing seamlessly when generating talking face video. Experiments demonstrate that our proposed approach can effectively preserve texture details and generate temporally consistent video while remaining competitive in lip synchronization. Quantitative and qualitative results on the HDTF and MEAD datasets illustrate the superior performance of our method over existing methods., Comment: 10 pages, 7 figures, 3 tables

Since the first instances of online education, where courses were uploaded to accessible and shared online platforms, this form of scaling the dissemination of human knowledge to reach a broader audience has sparked extensive discussion and widespread adoption. Recognizing that personalized learning still holds significant potential for improvement, new AI technologies have been continuously integrated into this learning format, resulting in a variety of educational AI applications such as educational recommendation and intelligent tutoring. The emergence of intelligence in large language models (LLMs) has allowed for these educational enhancements to be built upon a unified foundational model, enabling deeper integration. In this context, we propose MAIC (Massive AI-empowered Course), a new form of online education that leverages LLM-driven multi-agent systems to construct an AI-augmented classroom, balancing scalability with adaptivity. Beyond exploring the conceptual framework and technical innovations, we conduct preliminary experiments at Tsinghua University, one of China's leading universities. Drawing from over 100,000 learning records of more than 500 students, we obtain a series of valuable observations and initial analyses. This project will continue to evolve, ultimately aiming to establish a comprehensive open platform that supports and unifies research, technology, and applications in exploring the possibilities of online education in the era of large model AI. We envision this platform as a collaborative hub, bringing together educators, researchers, and innovators to collectively explore the future of AI-driven online education.

Multimodel Large Language Models(MLLMs) have achieved promising OCR-free Document Understanding performance by increasing the supported resolution of document images. However, this comes at the cost of generating thousands of visual tokens for a single document image, leading to excessive GPU memory and slower inference times, particularly in multi-page document comprehension. In this work, to address these challenges, we propose a High-resolution DocCompressor module to compress each high-resolution document image into 324 tokens, guided by low-resolution global visual features. With this compression module, to strengthen multi-page document comprehension ability and balance both token efficiency and question-answering performance, we develop the DocOwl2 under a three-stage training framework: Single-image Pretraining, Multi-image Continue-pretraining, and Multi-task Finetuning. DocOwl2 sets a new state-of-the-art across multi-page document understanding benchmarks and reduces first token latency by more than 50%, demonstrating advanced capabilities in multi-page questioning answering, explanation with evidence pages, and cross-page structure understanding. Additionally, compared to single-image MLLMs trained on similar data, our DocOwl2 achieves comparable single-page understanding performance with less than 20% of the visual tokens. Our codes, models, and data are publicly available at https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2., Comment: 15 pages, 7 figures

This work proposes FireRedTTS, a foundation text-to-speech framework, to meet the growing demands for personalized and diverse generative speech applications. The framework comprises three parts: data processing, foundation system, and downstream applications. First, we comprehensively present our data processing pipeline, which transforms massive raw audio into a large-scale high-quality TTS dataset with rich annotations and a wide coverage of content, speaking style, and timbre. Then, we propose a language-model-based foundation TTS system. The speech signal is compressed into discrete semantic tokens via a semantic-aware speech tokenizer, and can be generated by a language model from the prompt text and audio. Then, a two-stage waveform generator is proposed to decode them to the high-fidelity waveform. We present two applications of this system: voice cloning for dubbing and human-like speech generation for chatbots. The experimental results demonstrate the solid in-context learning capability of FireRedTTS, which can stably synthesize high-quality speech consistent with the prompt text and audio. For dubbing, FireRedTTS can clone target voices in a zero-shot way for the UGC scenario and adapt to studio-level expressive voice characters in the PUGC scenario via few-shot fine-tuning with 1-hour recording. Moreover, FireRedTTS achieves controllable human-like speech generation in a casual style with paralinguistic behaviors and emotions via instruction tuning, to better serve spoken chatbots.

In the present study, by adopting the advantage of ultrasonic techniques, we developed a Multichannel Pulsed Ultrasonic Doppler Velocimetry (MPUDV) to measure the 2D2C velocity fields of liquid metal flow. Due to the specially designed Ultrasonic host and post-processing scheme, the MPUDV system can reach a high spatiotemporal resolution of 50 Hz and 3 mm. The flow loop contains a cavity test section to ensure a classical recirculating flow was built to validate the accuracy of MPUDV in velocity field measurement. In the initial phase of the study, water with tracer particles was selected as the working liquid to ensure the velocity field measurements by the well-developed Particle Image Velocimetry (PIV). A comparison of the data obtained from the PIV and MPUDV methods revealed less than 3 differences in the 2D2C velocity field between the two techniques during simultaneous measurements of the same flow field. This finding strongly demonstrates the reliability of the MPUDV method developed in this paper. Moreover, the ternary alloy GaInSn was selected as the working liquid in the flow loop to validate the efficacy of the MPUDV in measuring 2D-2C velocity fields. A series of tests were conducted in the cavity at varying Reynolds numbers, ranging from 9103 to 24123. The measurements demonstrated that the MPUDV could accurately measure the flow structures characterized by a central primary circulation eddy and two secondary eddies in the opaque liquid metal. Furthermore, it was found that the vortex center of the primary circulating eddy and the size of the secondary eddies undergo significant alterations with varying Reynolds numbers, indicating the influence of inertial force on the flow characteristics in the recirculating flow. It is therefore demonstrated that the current MPUDV methodology is applicable for measuring a 2D2C velocity field in opaque liquid metal flows., Comment: 23 pages, 12 figures

In recent years, kagome materials have attracted significant attention due to their rich emergent phenomena arising from the quantum interplay of geometry, topology, spin, and correlations. However, in the search for kagome materials, it has been found that bulk compounds with electronic properties related to the kagome lattice are relatively scarce, primarily due to the hybridization of kagome layers with adjacent layers. Therefore, researchers have shown increasing interest in the construction of two-dimensional (2D) kagome materials, aiming to achieve clean kagome bands near the Fermi level in monolayer or few-layer systems. Substantial advancements have already been made in this area. In this review, we summarize the current progress in the construction and development of 2D kagome lattices. We begin by introducing the geometric and electronic structures of the kagome lattice and its variants. This is followed by a discussion on the experimental realizations and electronic structure characterizations of 2D kagome materials. Finally, we provide an outlook on the future development of 2D kagome lattices., Comment: 58 pages, 12 figures

Logs play a critical role in providing essential information for system monitoring and troubleshooting. Recently, with the success of pre-trained language models (PLMs) and large language models (LLMs) in natural language processing (NLP), smaller PLMs (such as BERT) and LLMs (like ChatGPT) have become the current mainstream approaches for log analysis. While LLMs possess rich knowledge, their high computational costs and unstable performance make LLMs impractical for analyzing logs directly. In contrast, smaller PLMs can be fine-tuned for specific tasks even with limited computational resources, making them more practical. However, these smaller PLMs face challenges in understanding logs comprehensively due to their limited expert knowledge. To better utilize the knowledge embedded within LLMs for log understanding, this paper introduces a novel knowledge enhancement framework, called LUK, which acquires expert knowledge from LLMs to empower log understanding on a smaller PLM. Specifically, we design a multi-expert collaboration framework based on LLMs consisting of different roles to acquire expert knowledge. In addition, we propose two novel pre-training tasks to enhance the log pre-training with expert knowledge. LUK achieves state-of-the-art results on different log analysis tasks and extensive experiments demonstrate expert knowledge from LLMs can be utilized more effectively to understand logs., Comment: Under review

Effective planning is essential for the success of any task, from organizing a vacation to routing autonomous vehicles and developing corporate strategies. It involves setting goals, formulating plans, and allocating resources to achieve them. LLMs are particularly well-suited for automated planning due to their strong capabilities in commonsense reasoning. They can deduce a sequence of actions needed to achieve a goal from a given state and identify an effective course of action. However, it is frequently observed that plans generated through direct prompting often fail upon execution. Our survey aims to highlight the existing challenges in planning with language models, focusing on key areas such as embodied environments, optimal scheduling, competitive and cooperative games, task decomposition, reasoning, and planning. Through this study, we explore how LLMs transform AI planning and provide unique insights into the future of LM-assisted planning.

Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. Website at https://rekep-robot.github.io.

On-device computing, or edge computing, is becoming increasingly important for remote sensing, particularly in applications like deep network-based perception on on-orbit satellites and unmanned aerial vehicles (UAVs). In these scenarios, two brain-like capabilities are crucial for remote sensing models: (1) high energy efficiency, allowing the model to operate on edge devices with limited computing resources, and (2) online adaptation, enabling the model to quickly adapt to environmental variations, weather changes, and sensor drift. This work addresses these needs by proposing an online adaptation framework based on spiking neural networks (SNNs) for remote sensing. Starting with a pretrained SNN model, we design an efficient, unsupervised online adaptation algorithm, which adopts an approximation of the BPTT algorithm and only involves forward-in-time computation that significantly reduces the computational complexity of SNN adaptation learning. Besides, we propose an adaptive activation scaling scheme to boost online SNN adaptation performance, particularly in low time-steps. Furthermore, for the more challenging remote sensing detection task, we propose a confidence-based instance weighting scheme, which substantially improves adaptation performance in the detection task. To our knowledge, this work is the first to address the online adaptation of SNNs. Extensive experiments on seven benchmark datasets across classification, segmentation, and detection tasks demonstrate that our proposed method significantly outperforms existing domain adaptation and domain generalization approaches under varying weather conditions. The proposed method enables energy-efficient and fast online adaptation on edge devices, and has much potential in applications such as remote perception on on-orbit satellites and UAV.

The visible-light camera, which is capable of environment perception and navigation assistance, has emerged as an essential imaging sensor for marine surface vessels in intelligent waterborne transportation systems (IWTS). However, the visual imaging quality inevitably suffers from several kinds of degradations (e.g., limited visibility, low contrast, color distortion, etc.) under complex weather conditions (e.g., haze, rain, and low-lightness). The degraded visual information will accordingly result in inaccurate environment perception and delayed operations for navigational risk. To promote the navigational safety of vessels, many computational methods have been presented to perform visual quality enhancement under poor weather conditions. However, most of these methods are essentially specific-purpose implementation strategies, only available for one specific weather type. To overcome this limitation, we propose to develop a general-purpose multi-scene visibility enhancement method, i.e., edge reparameterization- and attention-guided neural network (ERANet), to adaptively restore the degraded images captured under different weather conditions. In particular, our ERANet simultaneously exploits the channel attention, spatial attention, and reparameterization technology to enhance the visual quality while maintaining low computational cost. Extensive experiments conducted on standard and IWTS-related datasets have demonstrated that our ERANet could outperform several representative visibility enhancement methods in terms of both imaging quality and computational efficiency. The superior performance of IWTS-related object detection and scene segmentation could also be steadily obtained after ERANet-based visibility enhancement under complex weather conditions., Comment: 15 pages, 13 figures

The recent progress made in large language models (LLMs) has brought tremendous application prospects to the world. The growing model size demands LLM training on multiple GPUs, while data parallelism is the most popular distributed training strategy due to its simplicity, efficiency, and scalability. Current systems adopt the model-sharded data parallelism to enable memory-efficient training, however, existing model-sharded data-parallel systems fail to efficiently utilize GPU on a commodity GPU cluster with 100 Gbps (or 200 Gbps) inter-GPU bandwidth due to 1) severe interference between collective operation and GPU computation and 2) heavy CPU optimizer overhead. Recent works propose in-network aggregation (INA) to relieve the network bandwidth pressure in data-parallel training, but they are incompatible with model sharding due to the network design. To this end, we propose LuWu, a novel in-network optimizer that enables efficient model-in-network data-parallel training of a 100B-scale model on distributed GPUs. Such new data-parallel paradigm keeps a similar communication pattern as model-sharded data parallelism but with a centralized in-network optimizer execution. The key idea is to offload the entire optimizer states and parameters from GPU workers onto an in-network optimizer node and to offload the entire collective communication from GPU-implemented NCCL to SmartNIC-SmartSwitch co-optimization. The experimental results show that LuWu outperforms the state-of-the-art training system by 3.98x when training on a 175B model on an 8-worker cluster.

Flying through body-size narrow gaps in the environment is one of the most challenging moments for an underactuated multirotor. We explore a purely data-driven method to master this flight skill in simulation, where a neural network directly maps pixels and proprioception to continuous low-level control commands. This learned policy enables whole-body control through gaps with different geometries demanding sharp attitude changes (e.g., near-vertical roll angle). The policy is achieved by successive model-free reinforcement learning (RL) and online observation space distillation. The RL policy receives (virtual) point clouds of the gaps' edges for scalable simulation and is then distilled into the high-dimensional pixel space. However, this flight skill is fundamentally expensive to learn by exploring due to restricted feasible solution space. We propose to reset the agent as states on the trajectories by a model-based trajectory optimizer to alleviate this problem. The presented training pipeline is compared with baseline methods, and ablation studies are conducted to identify the key ingredients of our method. The immediate next step is to scale up the variation of gap sizes and geometries in anticipation of emergent policies and demonstrate the sim-to-real transformation., Comment: 9 pages, 8 figures, 2 tables

The imitation of cursive handwriting is mainly limited to generating handwritten words or lines. Multiple synthetic outputs must be stitched together to create paragraphs or whole pages, whereby consistency and layout information are lost. To close this gap, we propose a method for imitating handwriting at the paragraph level that also works for unseen writing styles. Therefore, we introduce a modified latent diffusion model that enriches the encoder-decoder mechanism with specialized loss functions that explicitly preserve the style and content. We enhance the attention mechanism of the diffusion model with adaptive 2D positional encoding and the conditioning mechanism to work with two modalities simultaneously: a style image and the target text. This significantly improves the realism of the generated handwriting. Our approach sets a new benchmark in our comprehensive evaluation. It outperforms all existing imitation methods at both line and paragraph levels, considering combined style and content preservation.

Human pose estimation (HPE) is crucial for various applications. However, deploying HPE algorithms in surveillance contexts raises significant privacy concerns due to the potential leakage of sensitive personal information (SPI) such as facial features, and ethnicity. Existing privacy-enhancing methods often compromise either privacy or performance, or they require costly additional modalities. We propose a novel privacy-enhancing system that generates privacy-enhanced portraits while maintaining high HPE performance. Our key innovations include the reversible recovery of SPI for authorized personnel and the preservation of contextual information. By jointly optimizing a privacy-enhancing module, a privacy recovery module, and a pose estimator, our system ensures robust privacy protection, efficient SPI recovery, and high-performance HPE. Experimental results demonstrate the system's robust performance in privacy enhancement, SPI recovery, and HPE.

This paper explores a simple extension of diffusion-based rectified flow Transformers for text-to-music generation, termed as FluxMusic. Generally, along with design in advanced Flux\footnote{https://github.com/black-forest-labs/flux} model, we transfers it into a latent VAE space of mel-spectrum. It involves first applying a sequence of independent attention to the double text-music stream, followed by a stacked single music stream for denoised patch prediction. We employ multiple pre-trained text encoders to sufficiently capture caption semantic information as well as inference flexibility. In between, coarse textual information, in conjunction with time step embeddings, is utilized in a modulation mechanism, while fine-grained textual details are concatenated with the music patch sequence as inputs. Through an in-depth study, we demonstrate that rectified flow training with an optimized architecture significantly outperforms established diffusion methods for the text-to-music task, as evidenced by various automatic metrics and human preference evaluations. Our experimental data, code, and model weights are made publicly available at: \url{https://github.com/feizc/FluxMusic}.

Cosine similarity between two documents can be computed using token embeddings formed by Large Language Models (LLMs) such as GPT-4, and used to categorize those documents across a range of uses. However, these similarities are ultimately dependent on the corpora used to train these LLMs, and may not reflect subjective similarity of individuals or how their biases and constraints impact similarity metrics. This lack of cognitively-aware personalization of similarity metrics can be particularly problematic in educational and recommendation settings where there is a limited number of individual judgements of category or preference, and biases can be particularly relevant. To address this, we rely on an integration of an Instance-Based Learning (IBL) cognitive model with LLM embeddings to develop the Instance-Based Individualized Similarity (IBIS) metric. This similarity metric is beneficial in that it takes into account individual biases and constraints in a manner that is grounded in the cognitive mechanisms of decision making. To evaluate the IBIS metric, we also introduce a dataset of human categorizations of emails as being either dangerous (phishing) or safe (ham). This dataset is used to demonstrate the benefits of leveraging a cognitive model to measure the subjective similarity of human participants in an educational setting., Comment: 7 Figures, 1 table

Unsigned distance fields (UDFs) allow for the representation of models with complex topologies, but extracting accurate zero level sets from these fields poses significant challenges, particularly in preserving topological accuracy and capturing fine geometric details. To overcome these issues, we introduce DCUDF2, an enhancement over DCUDF--the current state-of-the-art method--for extracting zero level sets from UDFs. Our approach utilizes an accuracy-aware loss function, enhanced with self-adaptive weights, to improve geometric quality significantly. We also propose a topology correction strategy that reduces the dependence on hyper-parameter, increasing the robustness of our method. Furthermore, we develop new operations leveraging self-adaptive weights to boost runtime efficiency. Extensive experiments on surface extraction across diverse datasets demonstrate that DCUDF2 outperforms DCUDF and existing methods in both geometric fidelity and topological accuracy. We will make the source code publicly available.

Randomized neural network (RNN) methods have been proposed for solving various partial differential equations (PDEs), demonstrating high accuracy and efficiency. However, initializing the fixed parameters remains a challenging issue. Additionally, RNNs often struggle to solve PDEs with sharp or discontinuous solutions. In this paper, we propose a novel approach called Adaptive Growing Randomized Neural Network (AG-RNN) to address these challenges. First, we establish a parameter initialization strategy based on frequency information to construct the initial RNN. After obtaining a numerical solution from this initial network, we use the residual as an error indicator. Based on the error indicator, we introduce growth strategies that expand the neural network, making it wider and deeper to improve the accuracy of the numerical solution. A key feature of AG-RNN is its adaptive strategy for determining the weights and biases of newly added neurons, enabling the network to expand in both width and depth without requiring additional training. Instead, all weights and biases are generated constructively, significantly enhancing the network's approximation capabilities compared to conventional randomized neural network methods. In addition, a domain splitting strategy is introduced to handle the case of discontinuous solutions. Extensive numerical experiments are conducted to demonstrate the efficiency and accuracy of this innovative approach.

"Benefit Game: Alien Seaweed Swarms" combines artificial life art and interactive game with installation to explore the impact of human activity on fragile seaweed ecosystems. The project aims to promote ecological consciousness by creating a balance in digital seaweed ecologies. Inspired by the real species "Laminaria saccharina", the author employs Procedural Content Generation via Machine Learning technology to generate variations of virtual seaweeds and symbiotic fungi. The audience can explore the consequences of human activities through gameplay and observe the ecosystem's feedback on the benefits and risks of seaweed aquaculture. This Benefit Game offers dynamic and real-time responsive artificial seaweed ecosystems for an interactive experience that enhances ecological consciousness., Comment: Paper accepted at ISEA 24, The 29th International Symposium on Electronic Art, Brisbane, Australia, 21-29 June 2024

Topological flat bands (TFBs) provide a promising platform to investigate intriguing fractionalization phenomena, such as the fractional Chern insulators (FCIs). Most of TFB models are established in two-dimensional Euclidean lattices with zero curvature. In this work, we systematically explore TFBs in a class of two-dimensional non-Euclidean lattices with constant negative curvature, {\emph i.e.,} the hyperbolic analogs of the kagome lattice. Based on the Abelian hyperbolic band theory, TFBs have been respectively found in the heptagon-kagome, the octagon-kagome, the nonagon-kagome and the decagon-kagome lattices by introducing staggered magnetic fluxes and the next nearest-neighbor hoppings. The flatness ratios of all hyperbolic TFB models are more than 15, which suggests that the hyperbolic FCIs can be realized in these TFB models. We further demonstrate the existence of a $\nu=1/2$ FCI state with open boundary conditions when hard-core bosons fill into these hyperbolic TFB models., Comment: 7 pages, 5 figures, comments are welcome

We solve the problem of how to classify the first-order vertex-algebraic deformations for any grading-restricted vertex algebra $V$ that is freely generated by homogeneous elements of positive weights. We approach by computing the second cohomology $H^2_{1/2}(V, V)$ constructed by Yi-Zhi Huang. We start with the cocycle on two generators and show that its cohomology class is completely determined by its singular part. To extend the cocycle to any pair of elements in $V$, we take a generating function approach, formulate the cocycle equation, and show that all the complementary solutions are coboundaries. Then we use a very general procedure to construct a particular solution. The procedure applies to vertex algebras that are not freely generated. As a by-product, we show that $H^2_{1/2}(V, V) = H^2_\infty(V, V)$. Using these results, we explicitly determine the first-order deformations of the universal Virasoro VOA $Vir_c$, universal affine VOA $V^l(\mathfrak{g})$, Heisenberg VOA $V^l(\mathfrak{h})$, and the universal Zamolodchikov VOA $W_3^c$., Comment: 64 pages. Comments are welcomed

Automating suturing during robotically-assisted surgery reduces the burden on the operating surgeon, enabling them to focus on making higher-level decisions rather than fatiguing themselves in the numerous intricacies of a surgical procedure. Accurate suture thread reconstruction and grasping are vital prerequisites for suturing, particularly for avoiding entanglement with surgical tools and performing complex thread manipulation. However, such methods must be robust to heavy perceptual degradation resulting from heavy noise and thread feature sparsity from endoscopic images. We develop a reconstruction algorithm that utilizes quadratic programming optimization to fit smooth splines to thread observations, satisfying reliability bounds estimated from measured observation noise. Additionally, we craft a grasping policy that generates gripper trajectories that maximize the probability of a successful grasp. Our full image-to-grasp pipeline is rigorously evaluated with over 400 grasping trials, exhibiting state-of-the-art accuracy. We show that this strategy can be applied to the various techniques in autonomous suture needle manipulation to achieve autonomous surgery in a generalizable way., Comment: 8 pages, 6 figures, Submitted to RAL

In this chapter, we summarize recent progress on the properties of accretion disks when the accretion rate exceeds the so-called Eddington limit based on multi-dimensional radiation magnetohydrodynamic simulations. We first summarize the classical models that are used to describe the accretion disks in the super-Eddington regime with an emphasis on the key uncertainties in these models. Then we show that radiation-driven outflows are ubiquitously found by numerical simulations of super-Eddington accretion disks. Some key physical processes on energy transport inside the disk are also identified by numerical simulations. Radiative and mechanical output as a function of mass accretion rates, black hole mass, spin, and magnetic field topology are summarized. Applications of super-Eddington accretion disks to different astrophysical systems, particularly tidal disruption events, are also discussed., Comment: 22 pages, 4 figures, invited review chapter for the book "New Frontiers in GRMHD Simulations"

In this work, we calculate correlators of long strings on AdS$_3\times$S$^3\times$T$^4$ with pure NS-NS flux. We first construct physical vertex operators that correspond to long strings. Due to the GSO projection, they depend on the parity of the spectral flow parameter $w$. For a given $w$, we construct the physical operators that have the lowest space-time weights in both the NS and R sector. Then, we calculate three point correlators for each possible type of parities of spectral flows. We find that the recursion relations of correlators in the bosonic SL$(2,\mathbb{R})$ WZW model can be understood from the equivalence of these superstring correlators with different picture choices. Furthermore, after carefully mapping the vertex operators to appropriate operators in the dual CFT, we find that once the fermionic contributions together with the picture changing effects are correctly taken into account, some mathematical identities of covering maps lead to the matching of the correlators of the two sides. We check this explicitly at the leading order in the conformal perturbation computation and conjecture that this remains correct to all orders., Comment: 57 pages, 4 tables; v2: typos corrected

The diffusion model has emerged as a powerful tool for generating atomic structures for materials science. This work calls attention to the deficiency of current particle-based diffusion models, which represent atoms as a point cloud, in generating even the simplest ordered crystalline structures. The problem is attributed to particles being trapped in local minima during the score-driven simulated annealing of the diffusion process, similar to the physical process of force-driven simulated annealing. We develop a solution, the grand canonical diffusion model, which adopts an alternative voxel-based representation with continuous rather than fixed number of particles. The method is applied towards generation of several common crystalline phases as well as the technologically important and challenging problem of grain boundary structures.

k-uniform mixed states are a significant class of states characterized by all k-party reduced states being maximally mixed. Novel methodologies are constructed for constructing k-uniform mixed states with the highest possible purity. By using the orthogonal partition of orthogonal arrays, a series of new $k$-uniform mixed states is derived. Consequently, an infinite number of higher-dimensional k-uniform mixed states, including those with highest purity, can be generated.

Let $X$ be a normal projective variety of dimension $d$ over an algebraically closed field and $f$ an automorphism of $X$. Suppose that the pullback $f^*|_{\mathsf{N}^1(X)_\mathbf{R}}$ of $f$ on the real N\'eron--Severi space $\mathsf{N}^1(X)_\mathbf{R}$ is unipotent and denote the index of the eigenvalue $1$ by $k+1$. We prove an upper bound for the polynomial volume growth $\mathrm{plov}(f)$ of $f$ as follows: \[ \mathrm{plov}(f) \le (k/2 + 1)d. \] This inequality is optimal in certain cases. Furthermore, we show that $k\le 2(d-1)$, extending a result of Dinh--Lin--Oguiso--Zhang for compact K\"ahler manifolds to arbitrary characteristic. Combining these two inequalities together, we obtain an optimal inequality that \[ \mathrm{plov}(f) \le d^2, \] which affirmatively answers questions of Cantat--Paris-Romaskevich and Lin--Oguiso--Zhang., Comment: 26 pages, comments are welcome

In this paper, we propose a novel symbiotic sensing and communication (SSAC) framework, comprising a base station (BS) and a passive sensing node. In particular, the BS transmits communication waveform to serve vehicle users (VUEs), while the sensing node is employed to execute sensing tasks based on the echoes in a bistatic manner, thereby avoiding the issue of self-interference. Besides the weak target of interest, the sensing node tracks VUEs and shares sensing results with BS to facilitate sensing-assisted beamforming. By considering both fully digital arrays and hybrid analog-digital (HAD) arrays, we investigate the beamforming design in the SSAC system. We first derive the Cramer-Rao lower bound (CRLB) of the two-dimensional angles of arrival estimation as the sensing metric. Next, we formulate an achievable sum rate maximization problem under the CRLB constraint, where the channel state information is reconstructed based on the sensing results. Then, we propose two penalty dual decomposition (PDD)-based alternating algorithms for fully digital and HAD arrays, respectively. Simulation results demonstrate that the proposed algorithms can achieve an outstanding data rate with effective localization capability for both VUEs and the weak target. In particular, the HAD beamforming design exhibits remarkable performance gain compared to conventional schemes, especially with fewer radio frequency chains., Comment: 16 pages, 11 figures, submitted to IEEE journals for possible publication

When applying integrated sensing and communications (ISAC) in future mobile networks, many sensing tasks have low latency requirements, preferably being implemented at terminals. However, terminals often have limited computing capabilities and energy supply. In this paper, we investigate the effectiveness of leveraging the advanced computing capabilities of mobile edge computing (MEC) servers and the cloud server to address the sensing tasks of ISAC terminals. Specifically, we propose a novel three-tier integrated sensing, communication, and computing (ISCC) framework composed of one cloud server, multiple MEC servers, and multiple terminals, where the terminals can optionally offload sensing data to the MEC server or the cloud server. The offload message is sent via the ISAC waveform, whose echo is used for sensing. We jointly optimize the computation offloading and beamforming strategies to minimize the average execution latency while satisfying sensing requirements. In particular, we propose a low-complexity distributed algorithm to solve the problem. Firstly, we use the alternating direction method of multipliers (ADMM) and derive the closed-form solution for offloading decision variables. Subsequently, we convert the beamforming optimization sub-problem into a weighted minimum mean-square error (WMMSE) problem and propose a fractional programming based algorithm. Numerical results demonstrate that the proposed ISCC framework and distributed algorithm significantly reduce the execution latency and the energy consumption of sensing tasks at a lower computational complexity compared to existing schemes., Comment: 15 pages, 12 figures, submitted to IEEE journals for possible publication

The exotic neutral gauge boson is a powerful candidate for the new physics beyond the standard model. As a promising model, the left-right symmetric model has been proposed to explain the neutrino mass, dark matter, and matter-antimatter asymmetry, etc., in which exotic gauge bosons $Z^\prime, W^{\prime \pm}$ have been put forward as well as other new right-handed particles. We investigate the $\mu^+ \mu^- \to q\bar{q} $ and $ \mu^+ \mu^- \to l^+ l^- $ processes involving the $Z^\prime$ boson as an intermediate particle. The coupling strength, decay width and mass are the key parameters on the production and decay processes of the $Z^\prime$ boson. The results indicate that the angular distributions of final particles are sensitive to the couplings of $Z^\prime$ to the other fermions. Asymmetries defined from the angular distributions are ideal quantities to demonstrate the discrepancy between the standard model process and the processes with $Z^\prime$ participated and they are also appropriate observables to discriminate the couplings of $Z^\prime$ to other particles. Compared with the current results at the Large Hadron Collider (LHC), the future muon collider has a great potential to explore the new parameter space with $Z^\prime$ boson., Comment: 22 pages, 10 figures

Reading text from images (either natural scenes or documents) has been a long-standing research topic for decades, due to the high technical challenge and wide application range. Previously, individual specialist models are developed to tackle the sub-tasks of text reading (e.g., scene text recognition, handwritten text recognition and mathematical expression recognition). However, such specialist models usually cannot effectively generalize across different sub-tasks. Recently, generalist models (such as GPT-4V), trained on tremendous data in a unified way, have shown enormous potential in reading text in various scenarios, but with the drawbacks of limited accuracy and low efficiency. In this work, we propose Platypus, a generalized specialist model for text reading. Specifically, Platypus combines the best of both worlds: being able to recognize text of various forms with a single unified architecture, while achieving excellent accuracy and high efficiency. To better exploit the advantage of Platypus, we also construct a text reading dataset (called Worms), the images of which are curated from previous datasets and partially re-labeled. Experiments on standard benchmarks demonstrate the effectiveness and superiority of the proposed Platypus model. Model and data will be made publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/OCR/Platypus., Comment: Accepted by ECCV2024