Search

The salient multimodal capabilities and interactive experience of GPT-4o highlight its critical role in practical applications, yet it lacks a high-performing open-source counterpart. In this paper, we introduce Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM) adept at concurrently processing and analyzing modalities of image, video, audio, and text, while delivering an advanced multimodal interactive experience and strong performance. We propose an effective multimodal training schema starting with 7B model and proceeding through two stages of multimodal alignment and multitask fine-tuning across audio, image, video, and text modal. This approach equips the language model with the ability to handle visual and audio data effectively. Demonstrating strong performance across various omni-modal and multimodal benchmarks, we aim for this contribution to serve as a competitive baseline for the open-source community in advancing multimodal understanding and real-time interaction.

The opening of the gravitational wave window has significantly enhanced our capacity to explore the universe's most extreme and dynamic sector. In the mHz frequency range, a diverse range of compact objects, from the most massive black holes at the farthest reaches of the Universe to the lightest white dwarfs in our cosmic backyard, generate a complex and dynamic symphony of gravitational wave signals. Once recorded by gravitational wave detectors, these unique fingerprints have the potential to decipher the birth and growth of cosmic structures over a wide range of scales, from stellar binaries and stellar clusters to galaxies and large-scale structures. The TianQin space-borne gravitational wave mission is scheduled for launch in the 2030s, with an operational lifespan of five years. It will facilitate pivotal insights into the history of our universe. This document presents a concise overview of the detectable sources of TianQin, outlining their characteristics, the challenges they present, and the expected impact of the TianQin observatory on our understanding of them., Comment: TianQin Gravitational Wave Whitepaper, 72 pages, 30 figures

In the milli-Hertz frequency band, stochastic gravitational-wave background can be composed of both astronomical and cosmological sources, both can be anisotropic. Numerically depicting these anisotropies can be critical in revealing the underlying properties of their origins. For the first time, we perform a theoretical analysis of the constraining ability of TianQin on multiple moments of the stochastic background. First, we find that with a one-year operation, for a background with a signal-to-noise ratio of 16, TianQin can recover the multiple moments up to $l=4$. We also identified a unique feature of the stochastic background sky map, which is the mirror symmetry along the fixed orbital plane of TianQin. Thirdly, we explain the difference in anisotropy recovering ability between TianQin and LISA, by employing the criteria of the singularity of the covariance matrix (which is the condition number). Finally, we find that since the different data channel combinations correspond to different singularities, certain combinations might have an advantage in stochastic background map-making. We believe that the findings of this work can provide an important reference to future stochastic background analysis pipelines. It can also serve as a guideline for designing better gravitational-wave detectors aiming to decipher anisotropies in the stochastic background., Comment: 12 pages, 6 figures

In recent years, Large Language Models (LLMs) have demonstrated significant improvements across a variety of tasks, one of which is the long-context capability. The key to improving long-context performance lies in effective data organization and management strategies that integrate data from multiple domains and optimize the context window during training. Through extensive experimental analysis, we identified three key challenges in designing effective data management strategies that enable the model to achieve long-context capability without sacrificing performance in other tasks: (1) a shortage of long documents across multiple domains, (2) effective construction of context windows, and (3) efficient organization of large-scale datasets. To address these challenges, we introduce DataSculpt, a novel data management framework designed for long-context training. We first formulate the organization of training data as a multi-objective combinatorial optimization problem, focusing on attributes including relevance, homogeneity, integrity, and efficiency. Specifically, our approach utilizes a coarse-to-fine methodology to optimize training data organization both efficiently and effectively. We begin by clustering the data based on semantic similarity (coarse), followed by a multi-objective greedy search within each cluster to score and concatenate documents into various context windows (fine). Our comprehensive evaluations demonstrate that DataSculpt significantly enhances long-context training performance, resulting in improvements of 18.09% in retrieval augmentation, 21.23% in summarization, 21.27% in reading comprehension, and a 3.81% increase in code completion, while also maintaining overall model proficiency with a 4.88% improvement.

Space-borne detectors, including TianQin and Laser Interferometry Space Antenna (LISA), are tasked with the simultaneous observation of Galactic foreground, astrophysical and cosmological stochastic gravitational-wave backgrounds (SGWBs). For the first time, we employ a space-borne detector network to identify the multi-component SGWB. Specifically, we develop a tailored likelihood for cross-correlation detection with such networks. Combined with the likelihood, we use the simulated datasets of the TianQin + LISA network to conduct model selection and parameter estimation. Our results indicate that, after 4 years of operation, the network could detect a single SGWB from either astrophysical or cosmological origins, with an energy density $\Omega_{\rm ast/cos}$ (10 mHz) on the order of $10^{-12}$, despite the presence of a Galactic foreground. Furthermore, to distinguish the cosmological background from both a Galactic foreground and the expected astrophysical background, the energy density $\Omega_{\rm cos}$ should reach around $2\times 10^{-11}$., Comment: 17 pages, 9 figures

The general capabilities of Large Language Models (LLM) highly rely on the composition and selection on extensive pretraining datasets, treated as commercial secrets by several institutions. To mitigate this issue, we open-source the details of a universally applicable data processing pipeline and validate its effectiveness and potential by introducing a competitive LLM baseline. Specifically, the data processing pipeline consists of broad collection to scale up and reweighting to improve quality. We then pretrain a 7B model BaichuanSEED with 3T tokens processed by our pipeline without any deliberate downstream task-related optimization, followed by an easy but effective supervised fine-tuning stage. BaichuanSEED demonstrates consistency and predictability throughout training and achieves comparable performance on comprehensive benchmarks with several commercial advanced large language models, such as Qwen1.5 and Llama3. We also conduct several heuristic experiments to discuss the potential for further optimization of downstream tasks, such as mathematics and coding., Comment: 19 pages, 6 figures

Relativistic charged particle beam can be used as destructive beam weapons in space for debris removal tasks. The trajectories of charged particles are affected by both electric and magnetic forces in the Earth's magnetic field. In this paper, we firstly analyzed the correlation parameters of the charged particle beam as a weapon when it propagated in the geomagnetic field. Then the models were constructed based on COMSOL Multiphysics and the IGRF model was adopted in the simulation. The gyro-radius and the related uncertainty were analyzed by simulation of the charged particle transport in the geomagnetic field at different altitudes. The charged beam spot radius divergency was also simulated. The magnetic field pinch effect can be found and can limit the beam spreading., Comment: 10 pages, 7 figures

With the advancement of audio generation, generative models can produce highly realistic audios. However, the proliferation of deepfake general audio can pose negative consequences. Therefore, we propose a new task, deepfake general audio detection, which aims to identify whether audio content is manipulated and to locate deepfake regions. Leveraging an automated manipulation pipeline, a dataset named FakeSound for deepfake general audio detection is proposed, and samples can be viewed on website https://FakeSoundData.github.io. The average binary accuracy of humans on all test sets is consistently below 0.6, which indicates the difficulty humans face in discerning deepfake audio and affirms the efficacy of the FakeSound dataset. A deepfake detection model utilizing a general audio pre-trained model is proposed as a benchmark system. Experimental results demonstrate that the performance of the proposed model surpasses the state-of-the-art in deepfake speech detection and human testers., Comment: Accepted by INTERSPEECH 2024

Unsupervised (a.k.a. Self-supervised) representation learning (URL) has emerged as a new paradigm for time series analysis, because it has the ability to learn generalizable time series representation beneficial for many downstream tasks without using labels that are usually difficult to obtain. Considering that existing approaches have limitations in the design of the representation encoder and the learning objective, we have proposed Contrastive Shapelet Learning (CSL), the first URL method that learns the general-purpose shapelet-based representation through unsupervised contrastive learning, and shown its superior performance in several analysis tasks, such as time series classification, clustering, and anomaly detection. In this paper, we develop TimeCSL, an end-to-end system that makes full use of the general and interpretable shapelets learned by CSL to achieve explorable time series analysis in a unified pipeline. We introduce the system components and demonstrate how users interact with TimeCSL to solve different analysis tasks in the unified pipeline, and gain insight into their time series by exploring the learned shapelets and representation.

Weak-signal limit is often used in estimating stochastic gravitational-wave background (SGWB) intensities. This approximation fails and the signal-to-noise ratio (SNR) can be much weaker when background signals are loud compared to the detector noise. In this work, we highlight this limitation for the SGWB detection using space-borne detector networks. For the TianQin + LISA network, the SNR estimated under the weak-signal limit might be off by as large as an order of magnitude. Contour plots of SNR over the parameter spaces are also presented to indicate regions susceptible to this discrepancy. Our results suggest that DA and DB type extragalactic double white dwarfs may yield an SGWB with SNR surpassing 100 after 1 year of operation in the weak-signal-limit scenario, with a redshift-independent merger rate of about $500\,\,{\rm Mpc^{-3}\,Myr^{-1}}$. In fact, this value falls significantly below the necessary threshold. Similar influences arise for first-order phase transitions, yet pinning down parameter regions remains formidable due to model uncertainties., Comment: 11 pages, 7 figures

Plasmon, a collective mode of electronic excitation in semiconductor materials, provides a unique avenue for the detection of light dark matter (DM). In this work, we first generalize the theoretical formalism of plasmon to treat the relativistic DM particles. We demonstrate that the plasmon resonance effect for sub-MeV DM can be significantly enhanced, particularly in scenarios of boosted DM with a light mediator. Utilizing the first data from SENSEI at SNOLAB, we derive a new strong limit on the sub-MeV DM-electron scattering cross section, which is more stringent than those from liquid detectors., Comment: 6 pages(double column)+ 3 pages , 4 figures

The Primakoff mechanism is one of the primary channels for the production of solar axion. In canonical estimation of the Primakoff photon-axion conversion rate, the recoil effect is neglected and a static structure factor is adopted. By use of the linear response theory, we provide a dynamic description of the solar Primakoff process. It is found that the collective electrons overtake ions as the dominant factor, in contrast to the static screening picture where ions contribute more to the photon-axion conversion. Nonetheless, the resulting axion flux is only 1-2% lower than the standard estimate based on the static structure factor., Comment: matches the published version

In recent times, the field of unsupervised representation learning (URL) for time series data has garnered significant interest due to its remarkable adaptability across diverse downstream applications. Unsupervised learning goals differ from downstream tasks, making it tricky to ensure downstream task utility by focusing only on temporal feature characterization. Researchers have proposed multiple transformations to extract discriminative patterns implied in informative time series, trying to fill the gap. Despite the introduction of a variety of feature engineering techniques, e.g. spectral domain, wavelet transformed features, features in image form and symbolic features etc. the utilization of intricate feature fusion methods and dependence on heterogeneous features during inference hampers the scalability of the solutions. To address this, our study introduces an innovative approach that focuses on aligning and binding time series representations encoded from different modalities, inspired by spectral graph theory, thereby guiding the neural encoder to uncover latent pattern associations among these multi-modal features. In contrast to conventional methods that fuse features from multiple modalities, our proposed approach simplifies the neural architecture by retaining a single time series encoder, consequently leading to preserved scalability. We further demonstrate and prove mechanisms for the encoder to maintain better inductive bias. In our experimental evaluation, we validated the proposed method on a diverse set of time series datasets from various domains. Our approach outperforms existing state-of-the-art URL methods across diverse downstream tasks.

Space-based gravitational wave detectors such as TianQin, LISA, and TaiJi have the potential to outperform themselves through joint observation. To achieve this, it is desirable to practice joint data analysis in advance on simulated data that encodes the intrinsic correlation among the signals found in different detectors that operate simultaneously. In this paper, we introduce \texttt{GWSpace}, a package that can simulate the joint detection data from TianQin, LISA, and TaiJi. The software is not a groundbreaking work that starts from scratch. Rather, we use as many open-source resources as possible, tailoring them to the needs of simulating the multi-mission science data and putting everything into a ready-to-go and easy-to-use package. We shall describe the main components, the construction, and a few examples of application of the package. A common coordinate system, namely the Solar System Barycenter (SSB) coordinate system, is utilized to calculate spacecraft orbits for all three missions. The paper also provides a brief derivation of the detection process and outlines the general waveform of sources detectable by these detectors., Comment: 24 pages, 13 figures, GWSpace will be uploaded at https://github.com/TianQinSYSU/GWSpace

Despite advancements of end-to-end (E2E) models in speech recognition, named entity recognition (NER) is still challenging but critical for semantic understanding. Previous studies mainly focus on various rule-based or attention-based contextual biasing algorithms. However, their performance might be sensitive to the biasing weight or degraded by excessive attention to the named entity list, along with a risk of false triggering. Inspired by the success of the class-based language model (LM) in NER in conventional hybrid systems and the effective decoupling of acoustic and linguistic information in the factorized neural Transducer (FNT), we propose C-FNT, a novel E2E model that incorporates class-based LMs into FNT. In C-FNT, the LM score of named entities can be associated with the name class instead of its surface form. The experimental results show that our proposed C-FNT significantly reduces error in named entities without hurting performance in general word recognition., Comment: Accepted in INTERSPEECH 2024

A chiplet is an integrated circuit that encompasses a well-defined subset of an overall system's functionality. In contrast to traditional monolithic system-on-chips (SoCs), chiplet-based architecture can reduce costs and increase reusability, representing a promising avenue for continuing Moore's Law. Despite the advantages of multi-chiplet architectures, floorplan design in a chiplet-based architecture has received limited attention. Conflicts between cost and performance necessitate a trade-off in chiplet floorplan design since additional latency introduced by advanced packaging can decrease performance. Consequently, balancing power, performance, cost, area, and reliability is of paramount importance. To address this challenge, we propose Floorplet, a framework comprising simulation tools for performance reporting and comprehensive models for cost and reliability optimization. Our framework employs the open-source Gem5 simulator to establish the relationship between performance and floorplan for the first time, guiding the floorplan optimization of multi-chiplet architecture. The experimental results show that our framework decreases inter-chiplet communication costs by 24.81%., Comment: accepted by TCAD, 12 pages, 10 figures

TianQin and LISA are space-based laser interferometer gravitational wave (GW) detectors planned to be launched in the mid-2030s. Both detectors will detect low-frequency GWs around $10^{-2}\,{\rm Hz}$, however, TianQin is more sensitive to frequencies above this common sweet-spot while LISA is more sensitive to frequencies below $10^{-2}\,{\rm Hz}$. Therefore, TianQin and LISA will be able to detect the same sources but with different accuracy depending on the source and its parameters. We consider some of the most important astrophysical sources -- massive black hole binaries, stellar-mass black hole binaries, double white dwarfs, extreme mass ratio inspirals, light and heavy intermediate mass ratio inspirals, as well as the stochastic gravitational background of astrophysical origin -- that TianQin and LISA will be able to detect. For each of these sources, we analyze how far they can be detected (detection distance) and how well their parameters can be measured (detection accuracy) using a Fisher Matrix analysis. We compare the results obtained by the three detection scenarios (TianQin alone, LISA alone, and joint detection by LISA and TianQin) highlighting the gains from joint detection as well as the contribution of TianQin and LISA to a combined study of astrophysical sources. In particular, we consider the different orientations, lifetimes, and duty cycles of the two detectors to explore how they can give a more complete picture when working together., Comment: 49 pages, 63 figures

In recent years, the importance of the electronic in-medium effect in the sub-GeV dark matter (DM) direct detection has been recognized and a coherent formulation of the DM-electron scattering based the linear response theory has been well established in the literature. In this paper, we apply the formulation to the scattering between DM particles and solar medium, and it is found that the dynamic structure factor inherently incorporate the particle-particle scattering and in-medium effect. Using this tool and taking a benchmark model as an example, we demonstrate how the in-medium effect affect the scattering of DM particles in the Sun, in both the heavy and light mediator limit. Formulae derived in this work lay the foundation for accurately calculating the spectra of solar-accelerated DM particles, which is of particular importance for the detection of DM particles via plasmon in semiconductor targets., Comment: 16+2 pages, 9 pictures, matches the published version

Single gravitational-wave detectors face inherent limitations in detecting the anisotropy of the stochastic background. In this work, we explore the sensitivity to anisotropic backgrounds with a network of space-borne detectors. We find that the separation between detectors plays an important role in determining the sensitivity. For the first time, we observe as large as three orders of magnitude enhancement in detection sensitivity for the multipoles with $l=5$ and 6, compared to coinciding detectors. Coordinating and optimizing the separation between two space-borne detectors can significantly enhance the network's sensitivity to the multipole components of the stochastic background. For the TianQin + LISA network, benefiting from detector separation, it is possible to achieve sensitivity levels of 2-3 orders of magnitude better than using TianQin or LISA detector alone. These findings pave the way to uncover the underlying physics of anisotropy through gravitational-wave detections., Comment: 19 pages, 13 figures

Recently, end-to-end (E2E) automatic speech recognition (ASR) models have made great strides and exhibit excellent performance in general speech recognition. However, there remain several challenging scenarios that E2E models are not competent in, such as code-switching and named entity recognition (NER). Data augmentation is a common and effective practice for these two scenarios. However, the current data augmentation methods mainly rely on audio splicing and text-to-speech (TTS) models, which might result in discontinuous, unrealistic, and less diversified speech. To mitigate these potential issues, we propose a novel data augmentation method by applying the text-based speech editing model. The augmented speech from speech editing systems is more coherent and diversified, also more akin to real speech. The experimental results on code-switching and NER tasks show that our proposed method can significantly outperform the audio splicing and neural TTS based data augmentation systems., Comment: Accepted by Interspeech 2023

The utilization of discrete speech tokens, divided into semantic tokens and acoustic tokens, has been proven superior to traditional acoustic feature mel-spectrograms in terms of naturalness and robustness for text-to-speech (TTS) synthesis. Recent popular models, such as VALL-E and SPEAR-TTS, allow zero-shot speaker adaptation through auto-regressive (AR) continuation of acoustic tokens extracted from a short speech prompt. However, these AR models are restricted to generate speech only in a left-to-right direction, making them unsuitable for speech editing where both preceding and following contexts are provided. Furthermore, these models rely on acoustic tokens, which have audio quality limitations imposed by the performance of audio codec models. In this study, we propose a unified context-aware TTS framework called UniCATS, which is capable of both speech continuation and editing. UniCATS comprises two components, an acoustic model CTX-txt2vec and a vocoder CTX-vec2wav. CTX-txt2vec employs contextual VQ-diffusion to predict semantic tokens from the input text, enabling it to incorporate the semantic context and maintain seamless concatenation with the surrounding context. Following that, CTX-vec2wav utilizes contextual vocoding to convert these semantic tokens into waveforms, taking into consideration the acoustic context. Our experimental results demonstrate that CTX-vec2wav outperforms HifiGAN and AudioLM in terms of speech resynthesis from semantic tokens. Moreover, we show that UniCATS achieves state-of-the-art performance in both speech continuation and editing., Comment: Accepted to AAAI 2024

Recent studies have shown great promise in unsupervised representation learning (URL) for multivariate time series, because URL has the capability in learning generalizable representation for many downstream tasks without using inaccessible labels. However, existing approaches usually adopt the models originally designed for other domains (e.g., computer vision) to encode the time series data and {rely on strong assumptions to design learning objectives, which limits their ability to perform well}. To deal with these problems, we propose a novel URL framework for multivariate time series by learning time-series-specific shapelet-based representation through a popular contrasting learning paradigm. To the best of our knowledge, this is the first work that explores the shapelet-based embedding in the unsupervised general-purpose representation learning. A unified shapelet-based encoder and a novel learning objective with multi-grained contrasting and multi-scale alignment are particularly designed to achieve our goal, and a data augmentation library is employed to improve the generalization. We conduct extensive experiments using tens of real-world datasets to assess the representation quality on many downstream tasks, including classification, clustering, and anomaly detection. The results demonstrate the superiority of our method against not only URL competitors, but also techniques specially designed for downstream tasks. Our code has been made publicly available at https://github.com/real2fish/CSL., Comment: Accepted by VLDB 2024, 14 pages

Machine learning has emerged as a powerful tool for time series analysis. Existing methods are usually customized for different analysis tasks and face challenges in tackling practical problems such as partial labeling and domain shift. To improve the performance and address the practical problems universally, we develop UniTS, a novel framework that incorporates self-supervised representation learning (or pre-training). The components of UniTS are designed using sklearn-like APIs to allow flexible extensions. We demonstrate how users can easily perform an analysis task using the user-friendly GUIs, and show the superior performance of UniTS over the traditional task-specific methods without self-supervised pre-training on five mainstream tasks and two practical settings., Comment: Accepted by SIGMOD 24

The method of time delay interferometry (TDI) is proposed to cancel the laser noise in space-borne gravitational-wave detectors. Among all different TDI combinations, the most commonly used ones are the orthogonal channels A, E and T, where A and E are signal-sensitive and T is signal-insensitive. Meanwhile, for the detection of stochastic gravitational-wave background, one needs to introduce the overlap reduction function to characterize the correlation between channels. For the calculation of overlap reduction function, it is often convenient to work in the low-frequency approximation, and assuming the equal-arm Michelson channels. However, if one wishes to work on the overlap reduction function of $\rm A/E$ channels, then the low-frequency approximation fails. We derive the exact form of overlap reduction function for $\rm A/E$ channels. Based on the overlap reduction function, we calculate the sensitivity curves of TianQin, TianQin I+II and TianQin + LISA. We conclude that the detection sensitivity calculated with $\rm A/E$ channels is mostly consistent with that obtained from the equal-arm Michelson channels., Comment: 18 pages,10 figures