Search

Unlike traditional mesh-based approximations of differential operators, machine learning methods, which exploit the automatic differentiation of neural networks, have attracted increasing attention for their potential to mitigate stability issues encountered in the numerical simulation of hyperbolic conservation laws. However, solutions to hyperbolic problems are often piecewise smooth, rendering the differential form invalid along discontinuity interfaces and limiting the effectiveness of standard learning approaches. In this work, we propose lift-and-embed learning methods for solving scalar hyperbolic equations with discontinuous solutions, which consist of (i) embedding the Rankine-Hugoniot jump condition within a higher-dimensional space through the inclusion of an augmented variable in the solution ansatz; (ii) utilizing physics-informed neural networks to manage the increased dimensionality and to address both linear and quasi-linear problems within a unified learning framework; and (iii) projecting the trained network solution back onto the original lower-dimensional plane to obtain the approximate solution. Besides, the location of discontinuity can be parametrized as extra model parameters and inferred concurrently with the training of network solution. With collocation points sampled on piecewise surfaces rather than distributed over the entire lifted space, we conduct numerical experiments on various benchmark problems to demonstrate the capability of our methods in resolving discontinuous solutions without spurious numerical smearing and oscillations.

The massive semi-detached binary with a long-term decreasing orbital period may involve a rapid mass-transfer phase in Case A, and thus they are good astrophysical laboratories for investigating the evolution of massive binary stars. In this work, by using the long-term observational light curves from the OGLE project and other data in the low-metallicity LMC, four semi-detached massive binaries with long-term decreases in the orbital periods are detected from 165 EB-type close binaries. It is found that the more massive component in S07798 is filling its Roche lobe where the period decrease is caused by mass transfer from the primary to the secondary. However, the other three (S03065, S12631, S16873) are semi-detached binaries with a lobe-filling secondary where the mass transfer between the components should cause the period to increase if the angular momentum is conservative. The long-term period decreases in these three systems may be caused by the angular momentum loss. Additionally, the orbital periods of three systems (S03065, S07798, S16873) are detected to show cyclic variation with periods shorter than 11 years, which can be plausibly explained by the presence of close-in third bodies in these massive binaries. Based on all of these results, it is suggested that the detected four semi-detached binaries almost have multiplicity. The companion stars are crucial for the origin and evolution of these massive close binaries.

When presented with one or a few photos of a previously unseen object, humans can instantly recognize it in different scenes. Although the human brain mechanism behind this phenomenon is still not fully understood, this work introduces a novel technical realization of this task. It consists of two phases: (1) generating a Similarity Density Map (SDM) by convolving the scene image with the given object image patch(es) so that the highlight areas in the SDM indicate the possible locations; (2) obtaining the object occupied areas in the scene through a Region Alignment Network (RAN). The RAN is constructed on a backbone of Deep Siamese Network (DSN), and different from the traditional DSNs, it aims to obtain the object accurate regions by regressing the location and area differences between the ground truths and the predicted ones indicated by the highlight areas in SDM. By pre-learning from labels annotated in traditional datasets, the SDM-RAN can detect previously unknown objects without fine-tuning. Experiments were conducted on the MS COCO, PASCAL VOC datasets. The results indicate that the proposed method outperforms state-of-the-art methods on the same task.

Prior methods propose to offset the escalating costs of modern foundation models by dropping specific parts of their contexts with hand-designed rules, while attempting to preserve their original performance. We overcome this trade-off with Neural Attention Memory Models (NAMMs), introducing a learned network for memory management that improves both the performance and efficiency of transformers. We evolve NAMMs atop pre-trained transformers to provide different latent contexts focusing on the most relevant information for individual layers and attention heads. NAMMs are universally applicable to any model using self-attention as they condition exclusively on the values in the produced attention matrices. Learning NAMMs on a small set of problems, we achieve substantial performance improvements across multiple long-context benchmarks while cutting the model's input contexts up to a fraction of the original sizes. We show the generality of our conditioning enables zero-shot transfer of NAMMs trained only on language to entirely new transformer architectures even across input modalities, with their benefits carrying over to vision and reinforcement learning., Comment: 29 pages, 14 figures. Preprint, under submission. Source code is available at https://github.com/SakanaAI/evo-memory

We show that under Space Curve Shortening flow any closed immersed curve in $\mathbb R^n$ whose projection onto $\mathbb{R}^2\times\{\vec{0}\}$ is convex remains smooth until it shrinks to a point. Throughout its evolution, the projection of the curve onto $\mathbb{R}^2\times\{\vec{0}\}$ remains convex. As an application, we show that any closed immersed curve in $\mathbb R^n$ can be perturbed to an immersed curve in $\mathbb R^{n+2}$ whose evolution by Space Curve Shortening shrinks to a point., Comment: 33 pages, 5 figures. Added affiliation, no other changes

Neural operators, which learn mappings between the function spaces, have been applied to solve boundary value problems in various ways, including learning mappings from the space of the forcing terms to the space of the solutions with the substantial requirements of data pairs. In this work, we present a data-free neural operator integrated with physics, which learns the Green kernel directly. Our method proceeds in three steps: 1. The governing equations for the Green's function are reformulated into an interface problem, where the delta Dirac function is removed; 2. The interface problem is embedded in a lifted space of higher-dimension to handle the jump in the derivative, but still solved on a two-dimensional surface without additional sampling cost; 3. Deep neural networks are employed to address the curse of dimensionality caused by this lifting operation. The approximate Green's function obtained through our approach is then used to construct preconditioners for the linear systems allowed by its mathematical properties. Furthermore, the spectral bias of it revealed through both theoretical analysis and numerical validation contrasts with the smoothing effects of traditional iterative solvers, which motivates us to propose a hybrid iterative method that combines these two solvers. Numerical experiments demonstrate the effectiveness of our approximate Green's function in accelerating iterative methods, proving fast convergence for solving indefinite problems even involving discontinuous coefficients.

Point cloud is a critical 3D representation with many emerging applications. Because of the point sparsity and irregularity, high-quality rendering of point clouds is challenging and often requires complex computations to recover the continuous surface representation. On the other hand, to avoid visual discomfort, the motion-to-photon latency has to be very short, under 10 ms. Existing rendering solutions lack in either quality or speed. To tackle these challenges, we present a framework that unlocks interactive, free-viewing and high-fidelity point cloud rendering. We train a generic neural network to estimate 3D elliptical Gaussians from arbitrary point clouds and use differentiable surface splatting to render smooth texture and surface normal for arbitrary views. Our approach does not require per-scene optimization, and enable real-time rendering of dynamic point cloud. Experimental results demonstrate the proposed solution enjoys superior visual quality and speed, as well as generalizability to different scene content and robustness to compression artifacts. The code is available at https://github.com/huzi96/gaussian-pcloud-render ., Comment: Published at CVPR 2024 Workshop on AIS: Vision, Graphics and AI for Streaming (https://ai4streaming-workshop.github.io/)

Large multimodal models have demonstrated impressive problem-solving abilities in vision and language tasks, and have the potential to encode extensive world knowledge. However, it remains an open challenge for these models to perceive, reason, plan, and act in realistic environments. In this work, we introduce Can-Do, a benchmark dataset designed to evaluate embodied planning abilities through more diverse and complex scenarios than previous datasets. Our dataset includes 400 multimodal samples, each consisting of natural language user instructions, visual images depicting the environment, state changes, and corresponding action plans. The data encompasses diverse aspects of commonsense knowledge, physical understanding, and safety awareness. Our fine-grained analysis reveals that state-of-the-art models, including GPT-4V, face bottlenecks in visual perception, comprehension, and reasoning abilities. To address these challenges, we propose NeuroGround, a neurosymbolic framework that first grounds the plan generation in the perceived environment states and then leverages symbolic planning engines to augment the model-generated plans. Experimental results demonstrate the effectiveness of our framework compared to strong baselines. Our code and dataset are available at https://embodied-planning.github.io.

The IW And-type phenomenon in cataclysmic variables presents a significant challenge to the accretion disk instability model. Using photometric data from the All-Sky Automated Survey for Supernovae, the Zwicky Transient Facility, and the Transiting Exoplanet Survey Satellite, we identify Karachurin 12 as a new non-eclipsing IW And-type object with a cycle period of 35.69(3) days. We also report for the first time that Karachurin 12 is a negative superhump (NSH) system featuring a precessing tilted disks, with precession, orbital, and NSH periods of 4.9588(2) days, 0.3168895(13) days, and 0.2979861(8) days, respectively. Our analysis, using dips as index and NSHs as probe, reveals diverse cycle patterns in Karachurin 12, with NSH amplitude varying throughout the cycle. These findings offer new insights for studying tilted disks and the IW And-type phenomenon. The mass-transfer burst model has difficulty explaining the observed variations in NSH amplitude, especially given the uncertainty surrounding the origin of the mass transfer burst. Meanwhile, the tilted thermally unstable disk model indicates a possible connection to the IW And-type phenomenon, but it also struggles to account for the detailed variations in Karachurin 12. Therefore, a wider range of factors must be considered to fully understand the complex changes in Karachurin 12., Comment: 20 pages, 9 figures and 3 tables, accepted for publication in The Astrophysical Journal

Whisper and other large-scale automatic speech recognition models have made significant progress in performance. However, their performance on many low-resource languages, such as Kazakh, is not satisfactory. It is worth researching how to utilize low-cost data to improve the performance of Whisper on under-represented languages. In this study, we utilized easily accessible unpaired speech and text data and combined the language model GPT with Whisper on Kazakh. We implemented end of transcript (EOT) judgment modification and hallucination penalty to improve the performance of speech recognition. Further, we employed the decoding average token log probability as a criterion to select samples from unlabeled speech data and used pseudo-labeled data to fine-tune the model to further improve its performance. Ultimately, we achieved more than 10\% absolute WER reduction in multiple experiments, and the whole process has the potential to be generalized to other under-represented languages., Comment: Accepted by INTERSPEECH 2024;Minor typo correction

Heartbeat stars (HBSs) with tidally excited oscillations (TEOs) are ideal astrophysical laboratories for studying the internal properties of the systems. In this paper, five new HBSs exhibiting TEOs are discovered using TESS photometric data. The orbital parameters are derived using a corrected version of Kumar et al.'s model based on the Markov Chain Monte Carlo (MCMC) method. The TEOs in these objects are examined, and their pulsation phases and modes are identified. The pulsation phases of the TEOs in TIC 266809405, TIC 266894805, and TIC 412881444 are consistent with the dominant $l=2$, $m=0$, or $\pm2$ spherical harmonic. For TIC 11619404, although the TEO phase is close to the $m=+2$ mode, the $m = 0$ mode cannot be excluded because of the low inclination in this system. The TEO phase in TIC 447927324 shows a large deviation ($>2\sigma$) from the adiabatic expectations, suggesting that it is expected to be a traveling wave rather than a standing wave. In addition, these TEOs occur at relatively low orbital harmonics, and we cautiously suggest that this may be an observational bias. These objects are valuable sources for studying the structure and evolution of eccentricity orbit binaries and extending the TESS HBS catalog with TEOs., Comment: 9pages, 5figures

Heartbeat stars (HBSs) are ideal astrophysical laboratories to study the formation and evolution of binary stars in eccentric orbits and the internal structural changes of their components under strong tidal action. We discover 23 new HBSs based on TESS photometric data. The orbital parameters, including orbital period, eccentricity, orbital inclination, argument of periastron, and epoch of periastron passage of these HBSs are derived by using a corrected version of Kumar et al.'s model based on the Markov Chain Monte Carlo (MCMC) method. The preliminary results show that these HBSs have orbital periods in the range from 2.7 to 20 days and eccentricities in the range from 0.08 to 0.70. The eccentricity-period relation of these objects shows a positive correlation between eccentricity and period, and also shows the existence of orbital circularization. The Hertzsprung-Russell diagram shows that the HBSs are not all located in a particular area. The distribution of the derived parameters suggests a selection bias within the TESS survey towards massive HBSs with shorter orbital periods, higher temperatures and luminosities. These objects are a very useful source to study the structure and evolution of eccentricity orbit binaries and to extend the TESS HBS catalog., Comment: 10 pages, 7 figures

The data circulation is a complex scenario involving a large number of participants and different types of requirements, which not only has to comply with the laws and regulations, but also faces multiple challenges in technical and business areas. In order to systematically and comprehensively address these issues, it is essential to have a comprehensive and profound understanding of 'data circulation'. The traditional analysis method tends to proceed based on the traditional circulation model of commodities, that is, tangible objects, which has some defects and shortcomings, and tends to be a formalized approach, which is faced numerous challenges in practice. This paper analyzes the circulation of data with a philosophical approach, obtains the new explication of data and executing entity, and provides a new definition of the concepts of data utilization and data key stakeholders (objects). At the same time, it puts forward the idea of ``data alienation'', and constructs a new interpretive framework of ``data circulation''. Based on the framework of this interpretation, it is clearly proposed that ``data alienation'' is the core of ``data circulation'', benefit distribution is the driving force, and legal compliance is the foundation, and further discussed the three modes of ``data circulation''. It further discusses the three modes of ``data circulation''. It is pointed out that ``data circulation'' is different from traditional ``commodity circulation''. To achieve ``data circulation'',a comprehensive information infrastructure needs to be established. from a theoretical point of view, it lays a solid foundation for the development of ``data circulation''.

Rocket recycling is a crucial pursuit in aerospace technology, aimed at reducing costs and environmental impact in space exploration. The primary focus centers on rocket landing control, involving the guidance of a nonlinear underactuated rocket with limited fuel in real-time. This challenging task prompts the application of reinforcement learning (RL), yet goal-oriented nature of the problem poses difficulties for standard RL algorithms due to the absence of intermediate reward signals. This paper, for the first time, significantly elevates the success rate of rocket landing control from 8% with a baseline controller to 97% on a high-fidelity rocket model using RL. Our approach, called Random Annealing Jump Start (RAJS), is tailored for real-world goal-oriented problems by leveraging prior feedback controllers as guide policy to facilitate environmental exploration and policy learning in RL. In each episode, the guide policy navigates the environment for the guide horizon, followed by the exploration policy taking charge to complete remaining steps. This jump-start strategy prunes exploration space, rendering the problem more tractable to RL algorithms. The guide horizon is sampled from a uniform distribution, with its upper bound annealing to zero based on performance metrics, mitigating distribution shift and mismatch issues in existing methods. Additional enhancements, including cascading jump start, refined reward and terminal condition, and action smoothness regulation, further improve policy performance and practical applicability. The proposed method is validated through extensive evaluation and Hardware-in-the-Loop testing, affirming the effectiveness, real-time feasibility, and smoothness of the proposed controller., Comment: IROS 2024 Oral

The concept of an intelligent augmented reality (AR) assistant has significant, wide-ranging applications, with potential uses in medicine, military, and mechanics domains. Such an assistant must be able to perceive the environment and actions, reason about the environment state in relation to a given task, and seamlessly interact with the task performer. These interactions typically involve an AR headset equipped with sensors which capture video, audio, and haptic feedback. Previous works have sought to facilitate the development of intelligent AR assistants by visualizing these sensor data streams in conjunction with the assistant's perception and reasoning model outputs. However, existing visual analytics systems do not focus on user modeling or include biometric data, and are only capable of visualizing a single task session for a single performer at a time. Moreover, they typically assume a task involves linear progression from one step to the next. We propose a visual analytics system that allows users to compare performance during multiple task sessions, focusing on non-linear tasks where different step sequences can lead to success. In particular, we design visualizations for understanding user behavior through functional near-infrared spectroscopy (fNIRS) data as a proxy for perception, attention, and memory as well as corresponding motion data (acceleration, angular velocity, and gaze). We distill these insights into embedding representations that allow users to easily select groups of sessions with similar behaviors. We provide two case studies that demonstrate how to use these visualizations to gain insights about task performance using data collected during helicopter copilot training tasks. Finally, we evaluate our approach by conducting an in-depth examination of a think-aloud experiment with five domain experts., Comment: 11 pages, 6 figures. This is the author's version of the article that has been accepted for publication in IEEE Transactions on Visualization and Computer Graphics (TVCG)

Intelligence is key to advancing integrated circuit (IC) fabrication. Recent breakthroughs in Large Multimodal Models (LMMs) have unlocked unparalleled abilities in understanding images and text, fostering intelligent fabrication. Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication large multimodal model for wafer defect knowledge query. FabGPT manifests expertise in conducting defect detection in Scanning Electron Microscope (SEM) images, performing root cause analysis, and providing expert question-answering (Q&A) on fabrication processes. FabGPT matches enhanced multimodal features to automatically detect minute defects under complex wafer backgrounds and reduce the subjectivity of manual threshold settings. Besides, the proposed modulation module and interactive corpus training strategy embed wafer defect knowledge into the pre-trained model, effectively balancing Q&A queries related to defect knowledge and original knowledge and mitigating the modality bias issues. Experiments on in-house fab data (SEM-WaD) show that our FabGPT achieves significant performance improvement in wafer defect detection and knowledge querying.

Despite their nearly universal adoption for large language models, the internal workings of transformers are not well understood. We aim to better understand the impact of removing or reorganizing information throughout the layers of a pretrained transformer. Such an understanding could both yield better usage of existing models as well as to make architectural improvements to produce new variants. We present a series of empirical studies on frozen models that show that the lower and final layers of pretrained transformers differ from middle layers, but that middle layers have a surprising amount of uniformity. We further show that some classes of problems have robustness to skipping layers, running the layers in an order different from how they were trained, or running the layers in parallel. Our observations suggest that even frozen pretrained models may gracefully trade accuracy for latency by skipping layers or running layers in parallel., Comment: 12 pages total, including references and appendices

Synthesizing realistic 3D indoor scenes is a challenging task that traditionally relies on manual arrangement and annotation by expert designers. Recent advances in autoregressive models have automated this process, but they often lack semantic understanding of the relationships and hierarchies present in real-world scenes, yielding limited performance. In this paper, we propose Forest2Seq, a framework that formulates indoor scene synthesis as an order-aware sequential learning problem. Forest2Seq organizes the inherently unordered collection of scene objects into structured, ordered hierarchical scene trees and forests. By employing a clustering-based algorithm and a breadth-first traversal, Forest2Seq derives meaningful orderings and utilizes a transformer to generate realistic 3D scenes autoregressively. Experimental results on standard benchmarks demonstrate Forest2Seq's superiority in synthesizing more realistic scenes compared to top-performing baselines, with significant improvements in FID and KL scores. Our additional experiments for downstream tasks and ablation studies also confirm the importance of incorporating order as a prior in 3D scene generation., Comment: ECCV 2024

Tilted disk precession exists in different objects. Negative superhumps (NSHs) in cataclysmic variable stars (CVs) are believed to arise from the interaction between the reverse precession of a tilted disk and the streams from the secondary star.Utilizing TESS photometry, we present a comprehensive investigation into the tilted disk precession and NSHs in the dwarf nova (DN) HS 2325+8205, employing eclipse minima, eclipse depths, NSH frequencies, and NSH amplitudes and the correlation between them as the windows. We identified NSHs with a period of 0.185671(17) days in HS 2325+8205. The NSH frequency exhibits variability with a period of 3.943(9) days, akin to the tilted disk precession period validated in novae-like stars (NLs, SDSS J0812) and intermediate polars (IPs, TV Col).The O-C of eclipse minima were similarly found to vary cyclically in period 4.135(5) days, characterized by a faster rise than fall. Furthermore, the NSH amplitude exhibits complex and diverse variations, which may be linked to changes in the disk radius, mass transfer rate, and the apparent area of the hot spot. For the first time in DNe, we observe bi-periodic variations in eclipse depth (P1= 4.131(4) d and P2= 2.065(2) d ~ Pprec/2), resembling those seen in IPs, suggesting that variations with P2 are not attributable to an accretion curtain, as previously suspected. Moreover, NSH amplitude and eclipse depth decrease with increasing NSH frequency, while NSH amplitude correlates positively with eclipse depth.These complex variations observed across multiple observational windows provide substantial evidence for understanding of tilted disk precession and NSHs., Comment: 19 pages, 10 figures and 3 tables; accepted for publication in ApJ

Low-light videos often exhibit spatiotemporal incoherent noise, compromising visibility and performance in computer vision applications. One significant challenge in enhancing such content using deep learning is the scarcity of training data. This paper introduces a novel low-light video dataset, consisting of 40 scenes with various motion scenarios under two distinct low-lighting conditions, incorporating genuine noise and temporal artifacts. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly and refine it via an image-based approach for pixel-wise frame alignment across different light levels. We provide benchmarks based on four different technologies: convolutional neural networks, transformers, diffusion models, and state space models (mamba). Our experimental results demonstrate the significance of fully registered video pairs for low-light video enhancement (LLVE) and the comprehensive evaluation shows that the models trained with our dataset outperform those trained with the existing datasets. Our dataset and links to benchmarks are publicly available at https://doi.org/10.21227/mzny-8c77., Comment: arXiv admin note: text overlap with arXiv:2402.01970

Neural image representations have recently emerged as a promising technique for storing, streaming, and rendering visual data. Coupled with learning-based workflows, these novel representations have demonstrated remarkable visual fidelity and memory efficiency. However, existing neural image representations often rely on explicit uniform data structures without content adaptivity or computation-intensive implicit models, limiting their adoption in real-time graphics applications. Inspired by recent advances in radiance field rendering, we propose Image-GS, a content-adaptive image representation. Using anisotropic 2D Gaussians as the basis, Image-GS shows high memory efficiency, supports fast random access, and offers a natural level of detail stack. Leveraging a tailored differentiable renderer, Image-GS fits a target image by adaptively allocating and progressively optimizing a set of 2D Gaussians. The generalizable efficiency and fidelity of Image-GS are validated against several recent neural image representations and industry-standard texture compressors on a diverse set of images. Notably, its memory and computation requirements solely depend on and linearly scale with the number of 2D Gaussians, providing flexible controls over the trade-off between visual fidelity and run-time efficiency. We hope this research offers insights for developing new applications that require adaptive quality and resource control, such as machine perception, asset streaming, and content generation.

The merger of compact binary stars produces short gamma-ray bursts (sGRBs), involving channels such as neutron star - neutron star (BNS) and neutron star - black hole (NS-BH). The association between sGRB 170817A and gravitational wave GW 170817 provides reliable evidence for the BNS channel. The spatial distribution and merger rate differ between BNS mergers and NS-BH mergers. Some speculations suggest that sGRBs with extended emission (EE) may represent another distinct population. We compared the offset distributions of these two types of samples and found that they follow the same distribution. Utilizing non-parametric methods, we investigated the origin of these burst types in terms of their formation rate. We examined the luminosity function and formation rate of sGRBs without any assuming. The luminosity function can be described as $\psi(L_{0}) \propto L_{0}^{-0.09 \pm 0.01}$ for $L_{0} < L_0^b$ ($\psi(L_{0}) \propto L_{0}^{-0.57 \pm 0.02}$ for $L_{0} > L_0^b$) for standard sGRBs and $\psi(L_{0}) \propto L_{0}^{-0.11 \pm 0.004}$ for $L_{0} < L_0^b$ ($\psi(L_{0}) \propto L_{0}^{-0.61 \pm 0.01}$ for $L_{0} > L_0^b$) for sGRBs with EE. The formation rate is characterized as $\rho(z) \propto (1 + z)^{-4.21 \pm 0.22}$ for $z < 0.8$ and $\rho(z) \propto (1 + z)^{-0.22 \pm 0.74}$ for $0.8 < z < 3$ for standard sGRBs, while for sGRBs with EE, it is $\rho(z) \propto (1 + z)^{-4.30 \pm 0.13}$ for $z < 0.8$ and $\rho(z) \propto (1 + z)^{-0.33 \pm 0.66}$ for $0.8 < z < 3$. Based on these findings, we suggest that there is no significant difference in the progenitor stars of sGRBs with and without EE, considering the spatial offset and formation rate perspectives., Comment: 16 pages, 11 figures and 2 tables

Accurately digitizing physical objects is central to many applications, including virtual/augmented reality, industrial design, and e-commerce. Prior research has demonstrated efficient and faithful reconstruction of objects' geometric shapes and visual appearances, which suffice for digitally representing rigid objects. In comparison, physical properties, such as elasticity and pressure, are also indispensable to the behavioral fidelity of digitized deformable objects. However, existing approaches to acquiring these quantities either rely on invasive specimen collection or expensive/bulky laboratory setups, making them inapplicable to consumer-level usage. To fill this gap, we propose a wearable and non-invasive computing framework that allows users to conveniently estimate the material elasticity and internal pressure of deformable objects through finger touches. This is achieved by modeling their local surfaces as pressurized elastic shells and analytically deriving the two physical properties from finger-induced wrinkling patterns. Together with photogrammetry-reconstructed geometry and textures, the two estimated physical properties enable us to faithfully replicate the motion and deformation behaviors of several deformable objects. For the pressure estimation, our model achieves a relative error of 3.5%. In the interaction experiments, the virtual-physical deformation discrepancy measures less than 10.1%. Generalization to objects of irregular shape further demonstrates the potential of our approach in practical applications. We envision this work to provide insights for and motivate research toward democratizing the ubiquitous and pervasive digitization of our physical surroundings in daily, industrial, and scientific scenarios., Comment: 10 pages, 6 figures

In recent years, the increasing demand for dynamic 3D assets in design and gaming applications has given rise to powerful generative pipelines capable of synthesizing high-quality 4D objects. Previous methods generally rely on score distillation sampling (SDS) algorithm to infer the unseen views and motion of 4D objects, thus leading to unsatisfactory results with defects like over-saturation and Janus problem. Therefore, inspired by recent progress of video diffusion models, we propose to optimize a 4D representation by explicitly generating multi-view videos from one input image. However, it is far from trivial to handle practical challenges faced by such a pipeline, including dramatic temporal inconsistency, inter-frame geometry and texture diversity, and semantic defects brought by video generation results. To address these issues, we propose DG4D, a novel multi-stage framework that generates high-quality and consistent 4D assets without score distillation. Specifically, collaborative techniques and solutions are developed, including an attention injection strategy to synthesize temporal-consistent multi-view videos, a robust and efficient dynamic reconstruction method based on Gaussian Splatting, and a refinement stage with diffusion prior for semantic restoration. The qualitative results and user preference study demonstrate that our framework outperforms the baselines in generation quality by a considerable margin. Code will be released at \url{https://github.com/jasongzy/EG4D}.

We reported a cyclic variation of $O-C$ diagram with a semi-amplitude of 0.0033 days and a period of 1.05 years for the pulsating eclipsing binary HZ Dra. The cyclic variation can be explained by the light travel-time effect via the presence of a close-in third body orbiting around HZ Dra in an elliptical orbit with a maximum semi-major axis of 0.92 au. Based on the W-D code, the contribution of the third light to the total system is determined to be 29 $\%$, which is in agreement with the estimated value. Our light curve modelling indicates an evolving hot and cool spot on the surface of the primary and secondary components, respectively. Their positions are roughly symmetrical to the inner Lagrangian point L1, which could be used to explain the variation in the O$^{'}$Connell effect. Our frequency analysis detects 1 radial p-mode, 7 non-radial p-modes and 1 non-radial g-mode. In addition, a total of 6 multiplets are identified, spaced by the orbital frequency, which can be explained as a tidally split mode caused by the equilibrium tides of the close binary system with a circular orbit. These pulsating features suggest that the primary of HZ Dra is a $\delta$ Scuti star, pulsating in both p- and g-mode and influenced by tidal forces., Comment: 15 pages, 11 figures. The first version is Under Review in ApJ

The rising popularity of multimodal large language models (MLLMs) has sparked a significant increase in research dedicated to evaluating these models. However, current evaluation studies predominantly concentrate on the ability of models to comprehend and reason within a unimodal (vision-only) context, overlooking critical performance evaluations in complex multimodal reasoning tasks that integrate both visual and text contexts. Furthermore, tasks that demand reasoning across multiple modalities pose greater challenges and require a deep understanding of multimodal contexts. In this paper, we introduce a comprehensive assessment framework named MM-InstructEval, which integrates a diverse array of metrics to provide an extensive evaluation of the performance of various models and instructions across a broad range of multimodal reasoning tasks with vision-text contexts. MM-InstructEval enhances the research on the performance of MLLMs in complex multimodal reasoning tasks, facilitating a more thorough and holistic zero-shot evaluation of MLLMs. We firstly utilize the "Best Performance" metric to determine the upper performance limit of each model across various datasets. The "Mean Relative Gain" metric provides an analysis of the overall performance across different models and instructions, while the "Stability" metric evaluates their sensitivity to variations. Historically, the research has focused on evaluating models independently or solely assessing instructions, overlooking the interplay between models and instructions. To address this gap, we introduce the "Adaptability" metric, designed to quantify the degree of adaptability between models and instructions. Evaluations are conducted on 31 models (23 MLLMs) across 16 multimodal datasets, covering 6 tasks, with 10 distinct instructions. The extensive analysis enables us to derive novel insights., Comment: Under review, the new version of MM-BigBench: arXiv:2310.09036

We present the eclipsing Heartbeat Star KIC 9704906 with tidally excited oscillations (TEOs) and gamma Doradus/delta Scuti hybrid pulsations. The derived parameters show that it has an orbital period of $P$=8.7529108(1) days, a high eccentricity of $e$=0.467(3), and a high inclination of $i$=78$^{\circ}$.81(6). The mass ratio $q$=0.981(5), the relative radii (radius divided by semi-major axis) $r_1$=0.0639(2), and $r_2$=0.0715(4) indicate that the secondary component has a less mass and a larger radius, and may have evolved off the main sequence. The eight derived TEO candidates, $n$ = 3, 4, 5, 6, 7, 12, 40, and 44 harmonics, are consistent with or close to the dominant spherical harmonic $l=2$, $m=0$, or $\pm2$, assuming that the spin and orbital axes are aligned, and the pulsations are adiabatic and standing waves. We also identify ten independent frequency candidates, but one of them, $\mathit{f_{7}}$, is more like a modulation of a quasi-periodic signal and the orbits. According to the g-mode frequencies, we find that the rotation period of one component is 11.52(29) days. Although the masses and radii cannot be further constrained due to the lack of sufficient high-precision spectra, the fascinating phenomena in the Fourier spectra are evident and valuable in this system.

Spiking Neural Networks (SNNs) aim to bridge the gap between neuroscience and machine learning by emulating the structure of the human nervous system. However, like convolutional neural networks, SNNs are vulnerable to adversarial attacks. To tackle the challenge, we propose a biologically inspired methodology to enhance the robustness of SNNs, drawing insights from the visual masking effect and filtering theory. First, an end-to-end SNN-based image purification model is proposed to defend against adversarial attacks, including a noise extraction network and a non-blind denoising network. The former network extracts noise features from noisy images, while the latter component employs a residual U-Net structure to reconstruct high-quality noisy images and generate clean images. Simultaneously, a multi-level firing SNN based on Squeeze-and-Excitation Network is introduced to improve the robustness of the classifier. Crucially, the proposed image purification network serves as a pre-processing module, avoiding modifications to classifiers. Unlike adversarial training, our method is highly flexible and can be seamlessly integrated with other defense strategies. Experimental results on various datasets demonstrate that the proposed methodology outperforms state-of-the-art baselines in terms of defense effectiveness, training time, and resource consumption., Comment: 8 pages, 5 figures, ECAI2024 under review

Pretrained language models underpin several AI applications, but their high computational cost for training limits accessibility. Initiatives such as BLOOM and StarCoder aim to democratize access to pretrained models for collaborative community development. However, such existing models face challenges: limited multilingual capabilities, continual pretraining causing catastrophic forgetting, whereas pretraining from scratch is computationally expensive, and compliance with AI safety and development laws. This paper presents Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. Continually pretrained from StarCoderPlus on 435 billion additional tokens, Aurora-M surpasses 2 trillion tokens in total training token count. It is the first open-source multilingual model fine-tuned on human-reviewed safety instructions, thus aligning its development not only with conventional red-teaming considerations, but also with the specific concerns articulated in the Biden-Harris Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Aurora-M is rigorously evaluated across various tasks and languages, demonstrating robustness against catastrophic forgetting and outperforming alternatives in multilingual settings, particularly in safety evaluations. To promote responsible open-source LLM development, Aurora-M and its variants are released at https://huggingface.co/collections/aurora-m/aurora-m-models-65fdfdff62471e09812f5407 ., Comment: Preprint

3-dB couplers, which are commonly used in photonic integrated circuits for on-chip information processing, precision measurement, and quantum computing, face challenges in achieving robust performance due to their limited 3-dB bandwidths and sensitivity to fabrication errors. To address this, we introduce topological physics to nanophotonics, developing a framework for topological 3-dB couplers. These couplers exhibit broad working wavelength range and robustness against fabrication dimensional errors. By leveraging valley-Hall topology and mirror symmetry, the photonic-crystal-slab couplers achieve ideal 3-dB splitting characterized by a wavelength-insensitive scattering matrix. Tolerance analysis confirms the superiority on broad bandwidth of 48 nm and robust splitting against dimensional errors of 20 nm. We further propose a topological interferometer for on-chip distance measurement, which also exhibits robustness against dimensional errors. This extension of topological principles to the fields of interferometers, may open up new possibilities for constructing robust wavelength division multiplexing, temperature-drift-insensitive sensing, and optical coherence tomography applications., Comment: 20 pages, 4 figures

In the pursuit of advancing holographic display technology, we face a unique yet persistent roadblock: the inflexibility of learned holography in adapting to various hardware configurations. This is due to the variances in the complex optical components and system settings in existing holographic displays. Although the emerging learned approaches have enabled rapid and high-quality hologram generation, any alteration in display hardware still requires a retraining of the model. Our work introduces a configurable learned model that interactively computes 3D holograms from RGB-only 2D images for a variety of holographic displays. The model can be conditioned to predefined hardware parameters of existing holographic displays such as working wavelengths, pixel pitch, propagation distance, and peak brightness without having to retrain. In addition, our model accommodates various hologram types, including conventional single-color and emerging multi-color holograms that simultaneously use multiple color primaries in holographic displays. Notably, we enabled our hologram computations to rely on identifying the correlation between depth estimation and 3D hologram synthesis tasks within the learning domain for the first time in the literature. We employ knowledge distillation via a student-teacher learning strategy to streamline our model for interactive performance. Achieving up to a 2x speed improvement compared to state-of-the-art models while consistently generating high-quality 3D holograms with different hardware configurations., Comment: 14 pages, 5 figures

We present a novel application of evolutionary algorithms to automate the creation of powerful foundation models. While model merging has emerged as a promising approach for LLM development due to its cost-effectiveness, it currently relies on human intuition and domain knowledge, limiting its potential. Here, we propose an evolutionary approach that overcomes this limitation by automatically discovering effective combinations of diverse open-source models, harnessing their collective intelligence without requiring extensive additional training data or compute. Our approach operates in both parameter space and data flow space, allowing for optimization beyond just the weights of the individual models. This approach even facilitates cross-domain merging, generating models like a Japanese LLM with Math reasoning capabilities. Surprisingly, our Japanese Math LLM achieved state-of-the-art performance on a variety of established Japanese LLM benchmarks, even surpassing models with significantly more parameters, despite not being explicitly trained for such tasks. Furthermore, a culturally-aware Japanese VLM generated through our approach demonstrates its effectiveness in describing Japanese culture-specific content, outperforming previous Japanese VLMs. This work not only contributes new state-of-the-art models back to the open-source community, but also introduces a new paradigm for automated model composition, paving the way for exploring alternative, efficient approaches to foundation model development.

Diffusion models offer unprecedented image generation capabilities given just a text prompt. While emerging control mechanisms have enabled users to specify the desired spatial arrangements of the generated content, they cannot predict or control where viewers will pay more attention due to the complexity of human vision. Recognizing the critical necessity of attention-controllable image generation in practical applications, we present a saliency-guided framework to incorporate the data priors of human visual attention into the generation process. Given a desired viewer attention distribution, our control module conditions a diffusion model to generate images that attract viewers' attention toward desired areas. To assess the efficacy of our approach, we performed an eye-tracked user study and a large-scale model-based saliency analysis. The results evidence that both the cross-user eye gaze distributions and the saliency model predictions align with the desired attention distributions. Lastly, we outline several applications, including interactive design of saliency guidance, attention suppression in unwanted regions, and adaptive generation for varied display/viewing conditions.

Quantization is a crucial technique for deploying deep learning models on resource-constrained devices, such as embedded FPGAs. Prior efforts mostly focus on quantizing matrix multiplications, leaving other layers like BatchNorm or shortcuts in floating-point form, even though fixed-point arithmetic is more efficient on FPGAs. A common practice is to fine-tune a pre-trained model to fixed-point for FPGA deployment, but potentially degrading accuracy. This work presents QFX, a novel trainable fixed-point quantization approach that automatically learns the binary-point position during model training. Additionally, we introduce a multiplier-free quantization strategy within QFX to minimize DSP usage. QFX is implemented as a PyTorch-based library that efficiently emulates fixed-point arithmetic, supported by FPGA HLS, in a differentiable manner during backpropagation. With minimal effort, models trained with QFX can readily be deployed through HLS, producing the same numerical results as their software counterparts. Our evaluation shows that compared to post-training quantization, QFX can quantize models trained with element-wise layers quantized to fewer bits and achieve higher accuracy on both CIFAR-10 and ImageNet datasets. We further demonstrate the efficacy of multiplier-free quantization using a state-of-the-art binarized neural network accelerator designed for an embedded FPGA (AMD Xilinx Ultra96 v2). We plan to release QFX in open-source format.

Document-level Relation Triplet Extraction (DocRTE) is a fundamental task in information systems that aims to simultaneously extract entities with semantic relations from a document. Existing methods heavily rely on a substantial amount of fully labeled data. However, collecting and annotating data for newly emerging relations is time-consuming and labor-intensive. Recent advanced Large Language Models (LLMs), such as ChatGPT and LLaMA, exhibit impressive long-text generation capabilities, inspiring us to explore an alternative approach for obtaining auto-labeled documents with new relations. In this paper, we propose a Zero-shot Document-level Relation Triplet Extraction (ZeroDocRTE) framework, which generates labeled data by retrieval and denoising knowledge from LLMs, called GenRDK. Specifically, we propose a chain-of-retrieval prompt to guide ChatGPT to generate labeled long-text data step by step. To improve the quality of synthetic data, we propose a denoising strategy based on the consistency of cross-document knowledge. Leveraging our denoised synthetic data, we proceed to fine-tune the LLaMA2-13B-Chat for extracting document-level relation triplets. We perform experiments for both zero-shot document-level relation and triplet extraction on two public datasets. The experimental results illustrate that our GenRDK framework outperforms strong baselines., Comment: Accepted by WWW 2024

Only bosonic molecular species have been directly laser cooled to date, primarily due to an abundance of bosonic isotopes in nature and to their simpler hyperfine structure. Fermionic molecules provide new opportunities for ultracold chemistry, quantum simulation, and precision measurements. Here we report direct laser cooling of a fermionic molecular isotopologue, calcium monodeuteride (CaD). With a nuclear spin I = 1, only 5 hyperfine states need to be addressed for rotational closure in optical cycling. These hyperfine states are unresolved for typical experimental linewidths. We present a method for efficiently producing alkaline-earth metal hydrides and deuterides. We demonstrate rotational closure and show magnetically assisted Sisyphus cooling in one dimension for a beam of CaD molecules. Our results indicate that the experimental complexity for laser cooling CaD is similar to that of calcium monohydride (CaH). Laser cooling of CaD is a promising first step for production of ultracold and trapped atomic deuterium., Comment: 8 pages, 5 figures

Tidally excited oscillations (TEOs) in Heartbeat Stars (HBSs) are an essential probe of the internal properties of the systems, but their potential has yet to be fully exploited. Based on the orbital parameters of TEO candidates from our previous works, we identify the pulsation phases and amplitudes of TEOs in fourteen Kepler HBSs. Most pulsation phases of most systems can be explained by the dominant being $l=2$, $m=0$, or $\pm2$ spherical harmonic, assuming that the spin and orbital axes are aligned, and the pulsations are adiabatic and standing waves. The largest deviation ($>6\sigma$) occurs in KIC 8459354, which can be explained by the spin-orbit misalignment, and KIC 5877364 has a similar scenario. For KIC 11122789, almost half of the harmonics show large deviations; we cautiously suggest that these harmonics may not be considered TEO candidates. A similar scenario also exists in KIC 6290740. This phases and mode identification approach can also be used inversely to verify the TEO candidates derived by the Fourier analysis. Furthermore, the harmonics with large deviations ($>2\sigma$) in KIC 4377638, KIC 5090937, and KIC 11403032 can be expected to be travelling waves rather than standing waves. In addition, we also suggest that the apsidal motion could cause large deviations in TEO phases from theoretical values., Comment: published on MNRAS