Search

We present the scintillation velocity measurements of FRB~20201124A from the FAST observations, which reveal an annual variation. This annual variation is further supported by changes detected in the scintillation arc as observed from the secondary spectrum. We attribute the annual velocity variation to the presence of a moderately anisotropic scattering screen located at a distance of 0.4$\pm$0.1~kpc from Earth. Our results prove that the scintillation of this FRB is mainly caused by material close to Earth on a Galactic scale. However, scintillation observations of other FRBs may expose their surrounding environment or uncover possible orbital motion if scintillation is caused by materials in their host galaxy., Comment: 4 figures, 1 tables, accepted for publication in ApJL

We study both stream-based and pool-based active learning with neural network approximations. A recent line of works proposed bandit-based approaches that transformed active learning into a bandit problem, achieving both theoretical and empirical success. However, the performance and computational costs of these methods may be susceptible to the number of classes, denoted as $K$, due to this transformation. Therefore, this paper seeks to answer the question: "How can we mitigate the adverse impacts of $K$ while retaining the advantages of principled exploration and provable performance guarantees in active learning?" To tackle this challenge, we propose two algorithms based on the newly designed exploitation and exploration neural networks for stream-based and pool-based active learning. Subsequently, we provide theoretical performance guarantees for both algorithms in a non-parametric setting, demonstrating a slower error-growth rate concerning $K$ for the proposed approaches. We use extensive experiments to evaluate the proposed algorithms, which consistently outperform state-of-the-art baselines., Comment: Published on ICLR 2024, 40 Pages

Recent advancements in text-to-image models have significantly enhanced image generation capabilities, yet a notable gap of open-source models persists in bilingual or Chinese language support. To address this need, we present Taiyi-Diffusion-XL, a new Chinese and English bilingual text-to-image model which is developed by extending the capabilities of CLIP and Stable-Diffusion-XL through a process of bilingual continuous pre-training. This approach includes the efficient expansion of vocabulary by integrating the most frequently used Chinese characters into CLIP's tokenizer and embedding layers, coupled with an absolute position encoding expansion. Additionally, we enrich text prompts by large vision-language model, leading to better images captions and possess higher visual quality. These enhancements are subsequently applied to downstream text-to-image models. Our empirical results indicate that the developed CLIP model excels in bilingual image-text retrieval.Furthermore, the bilingual image generation capabilities of Taiyi-Diffusion-XL surpass previous models. This research leads to the development and open-sourcing of the Taiyi-Diffusion-XL model, representing a notable advancement in the field of image generation, particularly for Chinese language applications. This contribution is a step forward in addressing the need for more diverse language support in multimodal research. The model and demonstration are made publicly available at \href{https://huggingface.co/IDEA-CCNL/Taiyi-Stable-Diffusion-XL-3.5B/}, fostering further research and collaboration in this domain., Comment: Taiyi-Diffusion-XL Tech Report

With the open-sourcing of text-to-image models (T2I) such as stable diffusion (SD) and stable diffusion XL (SD-XL), there is an influx of models fine-tuned in specific domains based on the open-source SD model, such as in anime, character portraits, etc. However, there are few specialized models in certain domains, such as interior design, which is attributed to the complex textual descriptions and detailed visual elements inherent in design, alongside the necessity for adaptable resolution. Therefore, text-to-image models for interior design are required to have outstanding prompt-following capabilities, as well as iterative collaboration with design professionals to achieve the desired outcome. In this paper, we collect and optimize text-image data in the design field and continue training in both English and Chinese on the basis of the open-source CLIP model. We also proposed a fine-tuning strategy with curriculum learning and reinforcement learning from CLIP feedback to enhance the prompt-following capabilities of our approach so as to improve the quality of image generation. The experimental results on the collected dataset demonstrate the effectiveness of the proposed approach, which achieves impressive results and outperforms strong baselines.

Various large language models (LLMs) have been proposed in recent years, including closed- and open-source ones, continually setting new records on multiple benchmarks. However, the development of LLMs still faces several issues, such as high cost of training models from scratch, and continual pre-training leading to catastrophic forgetting, etc. Although many such issues are addressed along the line of research on LLMs, an important yet practical limitation is that many studies overly pursue enlarging model sizes without comprehensively analyzing and optimizing the use of pre-training data in their learning process, as well as appropriate organization and leveraging of such data in training LLMs under cost-effective settings. In this work, we propose Ziya2, a model with 13 billion parameters adopting LLaMA2 as the foundation model, and further pre-trained on 700 billion tokens, where we focus on pre-training techniques and use data-centric optimization to enhance the learning process of Ziya2 on different stages. We define three data attributes and firstly establish data-centric scaling laws to illustrate how different data impacts LLMs. Experiments show that Ziya2 significantly outperforms other models in multiple benchmarks especially with promising results compared to representative open-source ones. Ziya2 (Base) is released at https://huggingface.co/IDEA-CCNL/Ziya2-13B-Base and https://modelscope.cn/models/Fengshenbang/Ziya2-13B-Base/summary.

Annual variations of interstellar scintillation can be modelled to constrain parameters of the ionized interstellar medium. If a pulsar is in a binary system, then investigating the orbital parameters is possible through analysis of the orbital variation of scintillation. In observations carried out from 2011 January to 2020 August by the European Pulsar Timing Array radio telescopes, PSRs~J0613$-$0200 and J0636+5128 show strong annual variations in their scintillation velocity, while the former additionally exhibits an orbital fluctuation. Bayesian theory and Markov-chain-Monte-Carlo methods are used to interpret these periodic variations. We assume a thin and anisotropic scattering screen model, and discuss the mildly and extremely anisotropic scattering cases. PSR~J0613$-$0200 is best described by mildly anisotropic scattering, while PSR~J0636+5128 exhibits extremely anisotropic scattering. We measure the distance, velocity and degree of anisotropy of the scattering screen for our two pulsars, finding that scattering screen distances from Earth for PSRs~J0613$-$0200 and J0636+5128 are 316$^{+28}_{-20}$\,pc and 262$^{+96}_{-38}$\,pc, respectively. The positions of these scattering screens are coincident with the shell of the Local Bubble towards both pulsars. These associations add to the growing evidence of the Local Bubble shell as a dominant region of scattering along many sightlines., Comment: Accepted by SCIENCE CHINA Physics, Mechanics & Astronomy ( SCPMA)

Interstellar scattering (ISS) of radio pulsar emission can be used as a probe of the ionised interstellar medium (IISM) and causes corruptions in pulsar timing experiments. Two types of ISS phenomena (intensity scintillation and pulse broadening) are caused by electron density fluctuations on small scales (< 0.01 AU). Theory predicts that these are related, and both have been widely employed to study the properties of the IISM. Larger scales ($\sim$1-100\,AU) cause measurable changes in dispersion and these can be correlated with ISS observations to estimate the fluctuation spectrum over a very wide scale range. IISM measurements can often be modeled by a homogeneous power-law spatial spectrum of electron density with the Kolmogorov ($-11/3$) spectral exponent. Here we aim to test the validity of using the Kolmogorov exponent with PSR~J0826+2637. We do so using observations of intensity scintillation, pulse broadening and dispersion variations across a wide fractional bandwidth (20 -- 180\,MHz). We present that the frequency dependence of the intensity scintillation in the high frequency band matches the expectations of a Kolmogorov spectral exponent but the pulse broadening in the low frequency band does not change as rapidly as predicted with this assumption. We show that this behavior is due to an inhomogeneity in the scattering region, specifically that the scattering is dominated by a region of transverse size $\sim$40\,AU. The power spectrum of the electron density, however, maintains the Kolmogorov spectral exponent from spatial scales of 5$\times10^{-6}$\,AU to $\sim$100\,AU., Comment: Accepted for publication in MNRAS, typo fixed

We propose a new paradigm for zero-shot learners that is format agnostic, i.e., it is compatible with any format and applicable to a list of language tasks, such as text classification, commonsense reasoning, coreference resolution, and sentiment analysis. Zero-shot learning aims to train a model on a given task such that it can address new learning tasks without any additional training. Our approach converts zero-shot learning into multiple-choice tasks, avoiding problems in commonly used large-scale generative models such as FLAN. It not only adds generalization ability to models but also significantly reduces the number of parameters. Our method shares the merits of efficient training and deployment. Our approach shows state-of-the-art performance on several benchmarks and produces satisfactory results on tasks such as natural language inference and text classification. Our model achieves this success with only 235M parameters, which is substantially smaller than state-of-the-art models with billions of parameters. The code and pre-trained models are available at https://github.com/IDEA-CCNL/Fengshenbang-LM ., Comment: EMNLP 2022

Nowadays, foundation models become one of fundamental infrastructures in artificial intelligence, paving ways to the general intelligence. However, the reality presents two urgent challenges: existing foundation models are dominated by the English-language community; users are often given limited resources and thus cannot always use foundation models. To support the development of the Chinese-language community, we introduce an open-source project, called Fengshenbang, which leads by the research center for Cognitive Computing and Natural Language (CCNL). Our project has comprehensive capabilities, including large pre-trained models, user-friendly APIs, benchmarks, datasets, and others. We wrap all these in three sub-projects: the Fengshenbang Model, the Fengshen Framework, and the Fengshen Benchmark. An open-source roadmap, Fengshenbang, aims to re-evaluate the open-source community of Chinese pre-trained large-scale models, prompting the development of the entire Chinese large-scale model community. We also want to build a user-centered open-source ecosystem to allow individuals to access the desired models to match their computing resources. Furthermore, we invite companies, colleges, and research institutions to collaborate with us to build the large-scale open-source model-based ecosystem. We hope that this project will be the foundation of Chinese cognitive intelligence., Comment: Added the Chinese version and is now a bilingual paper

With the increasing application of machine learning in high-stake decision-making problems, potential algorithmic bias towards people from certain social groups poses negative impacts on individuals and our society at large. In the real-world scenario, many such problems involve positive and unlabeled data such as medical diagnosis, criminal risk assessment and recommender systems. For instance, in medical diagnosis, only the diagnosed diseases will be recorded (positive) while others will not (unlabeled). Despite the large amount of existing work on fairness-aware machine learning in the (semi-)supervised and unsupervised settings, the fairness issue is largely under-explored in the aforementioned Positive and Unlabeled Learning (PUL) context, where it is usually more severe. In this paper, to alleviate this tension, we propose a fairness-aware PUL method named FairPUL. In particular, for binary classification over individuals from two populations, we aim to achieve similar true positive rates and false positive rates in both populations as our fairness metric. Based on the analysis of the optimal fair classifier for PUL, we design a model-agnostic post-processing framework, leveraging both the positive examples and unlabeled ones. Our framework is proven to be statistically consistent in terms of both the classification error and the fairness metric. Experiments on the synthetic and real-world data sets demonstrate that our framework outperforms state-of-the-art in both PUL and fair classification.

Interstellar scintillation analysis of pulsars allows us to probe the small-scale distribution and inhomogeneities of the ionized interstellar medium. Our priority is to present the data set and the basic measurements of scintillation parameters of pulsars employing long-term scintillation observations carried out from 2011 January to 2020 August by the European Pulsar Timing Array radio telescopes in the 21-cm and 11-cm bands. Additionally, we aim to identify future possible lines of study using this long-term scintillation dataset. We present the long-term time series of $\nu_{\rm d}$ and $\tau_{\rm d}$ for 13 pulsars. Sanity-checks and comparisons indicate that the scintillation parameters of our work and previously published works are mostly consistent. For two pulsars, PSRs~J1857+0943 and J1939+2134, we were able to obtain measurements of the $\nu_{\rm d}$ at both bands, which allows us to derive the time series of frequency scaling indices with a mean and a standard deviation of 2.82$\pm$1.95 and 3.18$\pm$0.60, respectively. We found some interesting features which will be studied in more detail in subsequent papers in this series: (i) in the time series of PSR~J1939+2134, where the scintillation bandwidth sharply increases or decreases associated with a sharp change of dispersion measure; (ii) PSR~J0613$-$0200 and PSR~J0636+5126 show a strong annual variation in the time series of the $\tau_{\rm d}$; (iii) PSR~J1939+2134 shows a weak anti-correlation between scintillation timescale and dispersion in WSRT data., Comment: 18 pages, 6 figures, 4 tables, accepted for publication in A&A

Context. Interstellar scintillation (ISS) of pulsar emission can be used both as a probe of the ionised interstellar medium (IISM) and cause corruptions in pulsar timing experiments. Of particular interest are so-called scintillation arcs which can be used to measure time-variable interstellar scattering delays directly, potentially allowing high-precision improvements to timing precision. Aims. The primary aim of this study is to carry out the first sizeable and self-consistent census of diffractive pulsar scintillation and scintillation-arc detectability at low frequencies, as a primer for larger-scale IISM studies and pulsar-timing related propagation studies with the LOw-Frequency ARray (LOFAR) High Band Antennae (HBA). Results. In this initial set of 31 sources, 15 allow full determination of the scintillation properties; nine of these show detectable scintillation arcs at 120-180 MHz. Eight of the observed sources show unresolved scintillation; and the final eight don't display diffractive scintillation. Some correlation between scintillation detectability and pulsar brightness and dispersion measure is apparent, although no clear cut-off values can be determined. Our measurements across a large fractional bandwidth allow a meaningful test of the frequency scaling of scintillation parameters, uncorrupted by influences from refractive scintillation variations. Conclusions. Our results indicate the powerful advantage and great potential of ISS studies at low frequencies and the complex dependence of scintillation detectability on parameters like pulsar brightness and interstellar dispersion. This work provides the first installment of a larger-scale census and longer-term monitoring of interstellar scintillation effects at low frequencies., Comment: 16 pages, 6 figures, accepted for publication in A&A

Most fair machine learning methods either highly rely on the sensitive information of the training samples or require a large modification on the target models, which hinders their practical application. To address this issue, we propose a two-stage training algorithm named FAIRIF. It minimizes the loss over the reweighted data set (second stage) where the sample weights are computed to balance the model performance across different demographic groups (first stage). FAIRIF can be applied on a wide range of models trained by stochastic gradient descent without changing the model, while only requiring group annotations on a small validation set to compute sample weights. Theoretically, we show that, in the classification setting, three notions of disparity among different groups can be mitigated by training with the weights. Experiments on synthetic data sets demonstrate that FAIRIF yields models with better fairness-utility trade-offs against various types of bias; and on real-world data sets, we show the effectiveness and scalability of FAIRIF. Moreover, as evidenced by the experiments with pretrained models, FAIRIF is able to alleviate the unfairness issue of pretrained models without hurting their performance.

Active learning theories and methods have been extensively studied in classical statistical learning settings. However, deep active learning, i.e., active learning with deep learning models, is usually based on empirical criteria without solid theoretical justification, thus suffering from heavy doubts when some of those fail to provide benefits in real applications. In this paper, by exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven deep active learning method (dynamicAL) which selects samples to maximize training dynamics. In particular, we prove that the convergence speed of training and the generalization performance are positively correlated under the ultra-wide condition and show that maximizing the training dynamics leads to better generalization performance. Furthermore, to scale up to large deep neural networks and data sets, we introduce two relaxations for the subset selection problem and reduce the time complexity from polynomial to constant. Empirical results show that dynamicAL not only outperforms the other baselines consistently but also scales well on large deep learning models. We hope our work would inspire more attempts on bridging the theoretical findings of deep networks and practical impacts of deep active learning in real applications., Comment: Accepted by NeurIPS 2022

The general aim of the recommender system is to provide personalized suggestions to users, which is opposed to suggesting popular items. However, the normal training paradigm, i.e., fitting a recommender model to recover the user behavior data with pointwise or pairwise loss, makes the model biased towards popular items. This results in the terrible Matthew effect, making popular items be more frequently recommended and become even more popular. Existing work addresses this issue with Inverse Propensity Weighting (IPW), which decreases the impact of popular items on the training and increases the impact of long-tail items. Although theoretically sound, IPW methods are highly sensitive to the weighting strategy, which is notoriously difficult to tune. In this work, we explore the popularity bias issue from a novel and fundamental perspective -- cause-effect. We identify that popularity bias lies in the direct effect from the item node to the ranking score, such that an item's intrinsic property is the cause of mistakenly assigning it a higher ranking score. To eliminate popularity bias, it is essential to answer the counterfactual question that what the ranking score would be if the model only uses item property. To this end, we formulate a causal graph to describe the important cause-effect relations in the recommendation process. During training, we perform multi-task learning to achieve the contribution of each cause; during testing, we perform counterfactual inference to remove the effect of item popularity. Remarkably, our solution amends the learning process of recommendation which is agnostic to a wide range of models -- it can be easily implemented in existing methods. We demonstrate it on Matrix Factorization (MF) and LightGCN [20]. Experiments on five real-world datasets demonstrate the effectiveness of our method., Comment: To Appear in SIGKDD 2021

With the observations from \textit{Rossi X-ray Timing Explorer}, we search and study the X-ray bursts of accreting millisecond X-ray pulsar SAX~J1748.9-2021 during its 2010 outburst. We find 13 X-ray bursts, including 12 standard type-\uppercase\expandafter{\romannumeral1} X-ray bursts and an irregular X-ray burst which lacks cooling tail. During the outburst, the persistent emission occurred at $\sim$(1-5)$\%\rm {\dot{M}_{Edd}}$. We use a combination model of a blackbody (BB), a powerlaw, and a line component to fit the persistent emission spectra. Another BB is added into the combination model to account for the emission of the X-ray bursts due to the thermonuclear burning on the surface of the neutron star. Finally, we modify the combination model with a multiplicative factor $f_{\rm a}$, plus a BB to fit the spectra during the X-ray bursts. It is found that the $f_{\rm a}$ is inversely correlated with the burst flux in some cases. Our analysis suggests that the ignition depth of the irregular X-ray burst is obviously smaller than those of the type-\uppercase\expandafter{\romannumeral1} X-ray bursts. We argue that the detected type-\uppercase\expandafter{\romannumeral1} X-ray bursts originate from helium-rich or pure-helium environment, while the irregular X-ray burst originates from the thermonuclear flash in a shallow ocean., Comment: 12 pages, 7 figures, Accepted for publication in Astrophysics and Space Science