Search

We introduce synax, a novel library for automatically differentiable simulation of Galactic synchrotron emission. Built on the JAX framework, synax leverages JAX's capabilities, including batch acceleration, just-in-time compilation, and hardware-specific optimizations (CPU, GPU, TPU). Crucially, synax uses JAX's automatic differentiation (AD) mechanism, enabling precise computation of derivatives with respect to any model parameters. This feature facilitates powerful inference algorithms, such as Hamiltonian Monte Carlo (HMC) and gradient-based optimization, which enables inference over models that would otherwise be computationally prohibitive. In its initial release, synax supports synchrotron intensity and polarization calculations down to GHz frequencies, alongside several models of the Galactic magnetic field (GMF), cosmic ray (CR) spectra, and thermal electron density fields. We demonstrate the transformative potential of AD for tasks involving full posterior inference using gradient-based techniques or Maximum Likelihood Estimation (MLE) optimization. Notably, we show that GPU acceleration brings a twenty-fold enhancement in efficiency, while HMC achieves a two-fold improvement over standard random walk Metropolis-Hastings (RWMH) when performing inference over a four-parameter test model. HMC still works on a more complex, 16-parameter model while RWMH fails to converge. Additionally, we showcase the application of synax in optimizing the GMF based on the Haslam 408 MHz map, achieving residuals with a standard deviation below 1 K., Comment: Submitted to ApJS, 11 pages, 8 figures, comments welcome. Code at https://github.com/dkn16/Synax

The tumultuous effects of ultraviolet photons that source cosmic reionization, the subsequent compression and shock-heating of low-density regions, and the modulation of baryons in shallow potential wells induced by the passage of ionization fronts, collectively introduce perturbations to the evolution of the intergalactic medium in the post-reionization era. These enduring fluctuations persist deep into the post-reionization era, casting a challenge upon precision cosmology endeavors targeting tracers in this cosmic era. Simultaneously, these relics from reionization also present a unique opportunity to glean insights into the astrophysics that govern the epoch of reionization. In this work, we propose a first study of the cross-correlation of \lya forest and 21 cm intensity mapping, accounting for the repercussions of inhomogeneous reionization in the post-reionization era. We investigate the ability of SKA $\times$ DESI-like, SKA $\times$ MUST-like, and PUMA $\times$ MUST-like instrumental setups to achieve a high signal-to-noise ratio (SNR) in the redshift range $3.5 \leq z \leq 4$. Moreover, we assess how alterations in integration time, survey area, and reionization scenarios impact the SNR. Furthermore, we forecast the cross-correlation's potential to constrain cosmological parameters under varying assumptions: considering or disregarding reionization relics, marginalizing over reionization astrophysics, and assuming perfect knowledge of reionization. Notably, our findings underscore the remarkable capability of a futuristic PUMA $\times$ MUST-like setup, with a modest 100-hour integration time over a 100 sq. deg. survey, to constrain the ionization efficiency error to $\sigma_\zeta = 3.42 $., Comment: Updated to match accepted version for publication

Gaussian processes (GPs) have been extensively utilized as nonparametric models for component separation in 21 cm data analyses. This exploits the distinct spectral behavior of the cosmological and foreground signals, which are modeled through the GP covariance kernel. Previous approaches have employed a global GP kernel along all lines of sight (LoS). In this work, we study Bayesian approaches that allow for spatial variations in foreground kernel parameters, testing them against simulated HI intensity mapping observations. We consider a no-pooling (NP) model, which treats each LoS independently by fitting for separate covariance kernels, and a hierarchical Gaussian Process (HGP) model that allows for variation in kernel parameters between different LoS, regularized through a global hyperprior. We find that accounting for spatial variations in the GP kernel parameters results in a significant improvement in cosmological signal recovery, achieving up to a 30% reduction in the standard deviation of the residual distribution and improved model predictive performance. Allowing for spatial variations in GP kernel parameters also improves the recovery of the HI power spectra and wavelet scattering transform coefficients. Whilst the NP model achieves superior recovery as measured by the residual distribution, it demands extensive computational resources, faces formidable convergence challenges, and is prone to overfitting. Conversely, the HGP model strikes a balance between the accuracy and robustness of the signal recovery. Further improvements to the HGP model will require more physically motivated modeling of foreground spatial variations., Comment: 20 pages, 11 figures, 2 tables. Submitted to ApJ, comments welcome

We present a simulation-based framework to forecast the HI power spectrum on non-linear scales ($k\gtrsim 1\ {\rm Mpc^{-1}}$), as measured by interferometer arrays like MeerKAT in the low-redshift ($z\leq 1.0$) universe. Building on a galaxy-based HI mock catalog, we meticulously consider various factors, including the emission line profiles of HI discs and some observational settings, and explore their impacts on the HI power spectrum. While it is relatively insensitive to the profile shape of HI emission line at these scales, we identify a strong correlation with the profile width, that is, the Full Width at Half Maxima (FWHM, also known as $W_{\rm 50}$ in observations) in this work. By modeling the width function of $W_{50}$ as a function of $v_{\rm max}$, we assign each HI source a emission line profile and find that the resulting HI power spectrum is comparatively close to results from particles in the IllustrisTNG hydrodynamical simulation. After implementing $k$-space cuts matching the MeerKAT data, our prediction replicates the trend of the measurements obtained by MeerKAT at $z\approx 0.44$, though with a significantly lower amplitude. Utilizing a Monte Carlo Markov Chain sampling method, we constrain the parameter $A_{W_{\rm 50}}$ in the $W_{\rm 50}$ models and $\Omega_{\rm HI}$ with the MeerKAT measurements and find that a strong degeneracy exists between these two parameters.

The Minkowski Functionals (MFs), a set of topological summary statistics, have emerged as a powerful tool for extracting non-Gaussian information. We investigate the prospect of constraining the reionization parameters using the MFs of the 21 cm brightness temperature field from the epoch of reionization (EoR). Realistic effects, including thermal noise, synthesized beam, and foreground avoidance, are applied to the mock observations from the radio interferometric array experiments such as the Hydrogen Epoch of Reionization Array (HERA) and the Square Kilometre Array (SKA). We demonstrate that the MFs of the 21 cm signal measured with SKA-Low can be used to distinguish different reionization models, whereas the MF measurement with a HERA-like array cannot be made accurately enough. We further forecast the accuracies with which the MF measurements can place constraints on reionization parameters, using the standard MCMC analysis for parameter inference based on forward modeling. We find that for SKA-Low observation, MFs provide unbiased estimations of the reionization parameters with accuracies comparable to the power spectrum (PS) analysis. Furthermore, joint constraints using both MFs and PS can improve the constraint accuracies by up to $30\%$ compared to those with the PS alone. Nevertheless, the constraint accuracies can be degraded if the EoR window is shrunk with strong foreground avoidance. Our analysis demonstrates the promise of MFs as a set of summary statistics that extract complementary information from the 21 cm EoR field to the two-point statistics, which suggests a strong motivation for incorporating the MFs into the data analysis of future 21 cm observations., Comment: 20 pages, 10 figures, match the version that accpeted by ApJ

Optic nerve head (ONH) detection has been a crucial area of study in ophthalmology for years. However, the significant discrepancy between fundus image datasets, each generated using a single type of fundus camera, poses challenges to the generalizability of ONH detection approaches developed based on semantic segmentation networks. Despite the numerous recent advancements in general-purpose semantic segmentation methods using convolutional neural networks (CNNs) and Transformers, there is currently a lack of benchmarks for these state-of-the-art (SoTA) networks specifically trained for ONH detection. Therefore, in this article, we make contributions from three key aspects: network design, the publication of a dataset, and the establishment of a comprehensive benchmark. Our newly developed ONH detection network, referred to as ODFormer, is based upon the Swin Transformer architecture and incorporates two novel components: a multi-scale context aggregator and a lightweight bidirectional feature recalibrator. Our published large-scale dataset, known as TongjiU-DROD, provides multi-resolution fundus images for each participant, captured using two distinct types of cameras. Our established benchmark involves three datasets: DRIONS-DB, DRISHTI-GS1, and TongjiU-DROD, created by researchers from different countries and containing fundus images captured from participants of diverse races and ages. Extensive experimental results demonstrate that our proposed ODFormer outperforms other state-of-the-art (SoTA) networks in terms of performance and generalizability. Our dataset and source code are publicly available at mias.group/ODFormer.

The information regarding how the intergalactic medium is reionized by astrophysical sources is contained in the tomographic three-dimensional 21 cm images from the epoch of reionization. In Zhao et al. (2022a) ("Paper I"), we demonstrated for the first time that density estimation likelihood-free inference (DELFI) can be applied efficiently to perform a Bayesian inference of the reionization parameters from the 21 cm images. Nevertheless, the 3D image data needs to be compressed into informative summaries as the input of DELFI by, e.g., a trained 3D convolutional neural network (CNN) as in Paper I (DELFI-3D CNN). Here in this paper, we introduce an alternative data compressor, the solid harmonic wavelet scattering transform (WST), which has a similar, yet fixed (i.e. no training), architecture to CNN, but we show that this approach (i.e. solid harmonic WST with DELFI) outperforms earlier analyses based on 3D 21 cm images using DELFI-3D CNN in terms of credible regions of parameters. Realistic effects, including thermal noise and residual foreground after removal, are also applied to the mock observations from the Square Kilometre Array (SKA). We show that under the same inference strategy using DELFI, the 21 cm image analysis with solid harmonic WST outperforms the 21 cm power spectrum analysis. This research serves as a proof of concept, demonstrating the potential to harness the strengths of WST and simulation-based inference to derive insights from future 21 cm light-cone image data., Comment: 21 pages, 11 figures, 7 tables. Accepted for publication in ApJ. Comments welcome

X-ray photons can penetrate deep into the intergalactic medium (IGM), leading to preheating of the IGM prior to cosmic reionization. X-ray preheating wipes out some of the small-scale structures that would otherwise be present prior to the passage of an ionization front. Accurate modeling of the small-scale structure is vital to the post-reionization IGM since the small-scale structure is ultimately the dominant source of long-lasting relics from hydrogen reionization. However, the precise impact of X-ray preheating in the fossils from hydrogen reionization is highly uncertain. In this work, we explore and establish for the first time, the long-lasting impact of X-ray preheating in the post-reionization IGM via hydrodynamic simulations with high-mass resolution. We find that the addition of X-ray preheating astrophysics leads to an overall lesser impact of the effect of inhomogeneous reionization in the Lyman-$\alpha$ forest -- depending on specific X-ray prescription -- at low redshifts ($z \sim 2$) with respect to a model with no X-ray preheating. However, at high redshifts ($z \sim 4$), our results indicate a strengthening of the relics of reionization in the Lyman-$\alpha$ forest because the IGM becomes more transparent compared to the scenario with no preheating. Thus, the absence of X-ray preheating in Lyman-$\alpha$ modeling can lead to a biased inference of cosmological parameters. Nevertheless, optimistically, the inclusion of X-ray preheating emerges as a promising novel avenue to probe the astrophysics of cosmic dawn., Comment: Updated to match accepted version for publication, 19 pages, 10 figures

Generative adversarial networks (GANs) are frequently utilized in astronomy to construct an emulator of numerical simulations. Nevertheless, training GANs can prove to be a precarious task, as they are prone to instability and often lead to mode collapse problems. Conversely, the diffusion model also has the ability to generate high-quality data without adversarial training. It has shown superiority over GANs with regard to several natural image datasets. In this study, we undertake a quantitative comparison between the denoising diffusion probabilistic model (DDPM) and StyleGAN2 (one of the most robust types of GANs) via a set of robust summary statistics from scattering transform. In particular, we utilize both models to generate the images of 21 cm brightness temperature mapping, as a case study, conditionally based on astrophysical parameters that govern the process of cosmic reionization. Using our new Fr\'echet Scattering Distance (FSD) as the evaluation metric to quantitatively compare the sample distribution between generative models and simulations, we demonstrate that DDPM outperforms StyleGAN2 on varied sizes of training sets. Through Fisher forecasts, we demonstrate that on our datasets, StyleGAN2 exhibits mode collapses in varied ways, while DDPM yields a more robust generation. We also explore the role of classifier-free guidance in DDPM and show the preference for a non-zero guidance scale only when the training data is limited. Our findings indicate that the diffusion model presents a promising alternative to GANs in the generation of accurate images. These images can subsequently provide reliable parameter constraints, particularly in the realm of astrophysics., Comment: 14 pages, 10 figures, 1 table. Accepted for publication in MNRAS. Comments welcome

The Square Kilometre Array (SKA) will have the sensitivity to take the 3D light-cones of the 21 cm signal from the epoch of reionization. This signal, however, is highly non-Gaussian and can not be fully interpreted by the traditional statistic using power spectrum. In this work, we introduce the 3D ScatterNet that combines the normalizing flows with solid harmonic wavelet scattering transform, a 3D CNN featurizer with inductive bias, to perform implicit likelihood inference (ILI) from 21 cm light-cones. We show that 3D ScatterNet outperforms the ILI with a fine-tuned 3D CNN in the literature. It also reaches better performance than ILI with the power spectrum for varied light-cone effects and varied signal contaminations., Comment: 9 pages, 4 figures, 2 tables. Accepted to ICML 2023 Machine Learning for Astrophysics workshop. Comments and suggestions are welcome

The ability to compress observational data and accurately estimate physical parameters relies heavily on informative summary statistics. In this paper, we introduce the use of mutual information (MI) as a means of evaluating the quality of summary statistics in inference tasks. MI can assess the sufficiency of summaries, and provide a quantitative basis for comparison. We propose to estimate MI using the Barber-Agakov lower bound and normalizing flow based variational distributions. To demonstrate the effectiveness of our method, we compare three different summary statistics (namely the power spectrum, bispectrum, and scattering transform) in the context of inferring reionization parameters from mock images of 21~cm observations with Square Kilometre Array. We find that this approach is able to correctly assess the informativeness of different summary statistics and allows us to select the optimal set of statistics for inference tasks., Comment: Accepted at the ICML 2023 Workshop on Machine Learning for Astrophysics, comments welcome

Large-scale numerical simulations ($\gtrsim 500\rm{Mpc}$) of cosmic reionization are required to match the large survey volume of the upcoming Square Kilometre Array (SKA). We present a multi-fidelity emulation technique for generating large-scale lightcone images of cosmic reionization. We first train generative adversarial networks (GAN) on small-scale simulations and transfer that knowledge to large-scale simulations with hundreds of training images. Our method achieves high accuracy in generating lightcone images, as measured by various statistics with mostly percentage errors. This approach saves computational resources by 90% compared to conventional training methods. Our technique enables efficient and accurate emulation of large-scale images of the Universe., Comment: Accepted at the ICML 2023 Workshop on Machine Learning for Astrophysics

The task of repository-level code completion is to continue writing the unfinished code based on a broader context of the repository. While for automated code completion tools, it is difficult to utilize the useful information scattered in different files. We propose RepoCoder, a simple, generic, and effective framework to address the challenge. It streamlines the repository-level code completion process by incorporating a similarity-based retriever and a pre-trained code language model in an iterative retrieval-generation pipeline. RepoCoder makes effective utilization of repository-level information for code completion and has the ability to generate code at various levels of granularity. Moreover, we propose a new benchmark RepoEval, which consists of the latest and high-quality real-world repositories covering line, API invocation, and function body completion scenarios. Experimental results indicate that RepoCoder significantly improves the In-File completion baseline by over 10% in all settings and consistently outperforms the vanilla retrieval-augmented code completion approach. Furthermore, we validate the effectiveness of RepoCoder through comprehensive analysis, providing valuable insights for future research. Our source code and benchmark are publicly available: https://github.com/microsoft/CodeT/tree/main/RepoCoder, Comment: accepted by EMNLP 2023 main conference

We systematically study how three large language models with code capabilities - CodeT5, Codex, and ChatGPT - generalize to out-of-domain data. We consider two fundamental applications - code summarization, and code generation. We split data into domains following its natural boundaries - by an organization, by a project, and by a module within the software project. We establish that samples from each new domain present all the models with a significant challenge of distribution shift. We study how established methods adapt models to better generalize to new domains. Our experiments show that while multitask learning alone is a reasonable baseline, combining it with few-shot finetuning on examples retrieved from training data can achieve very strong performance. Moreover, this solution can outperform direct finetuning for very low-data scenarios. Finally, we consider variations of this approach to create a more broadly applicable method to adapt to multiple domains at once. We find that for code generation, a model adapted to multiple domains simultaneously performs on par with those adapted to a single domain

Pre-trained language models have achieved promising success in code retrieval tasks, where a natural language documentation query is given to find the most relevant existing code snippet. However, existing models focus only on optimizing the documentation code pairs by embedding them into latent space, without the association of external knowledge. In this paper, we propose a generation-augmented query expansion framework. Inspired by the human retrieval process - sketching an answer before searching, in this work, we utilize the powerful code generation model to benefit the code retrieval task. Specifically, we demonstrate that rather than merely retrieving the target code snippet according to the documentation query, it would be helpful to augment the documentation query with its generation counterpart - generated code snippets from the code generation model. To the best of our knowledge, this is the first attempt that leverages the code generation model to enhance the code retrieval task. We achieve new state-of-the-art results on the CodeSearchNet benchmark and surpass the baselines significantly.

Fine-tuning large language models for different tasks can be costly and inefficient, and even methods that reduce the number of tuned parameters still require full gradient-based optimization. We propose HyperTuning, a novel approach to model adaptation that uses a hypermodel to generate task-specific parameters for a fixed downstream model. We demonstrate a simple setup for hypertuning with HyperT5, a T5-based hypermodel that produces soft prefixes or LoRA parameters for a frozen T5 model from few-shot examples. We train HyperT5 in two stages: first, hyperpretraining with a modified conditional language modeling objective that trains a hypermodel to generate parameters; second, multi-task fine-tuning (MTF) on a large number of diverse language tasks. We evaluate HyperT5 on P3, MetaICL and Super-NaturalInstructions datasets, and show that it can effectively generate parameters for unseen tasks. Moreover, we show that using hypermodel-generated parameters as initializations for further parameter-efficient fine-tuning improves performance. HyperTuning can thus be a flexible and efficient way to leverage large language models for diverse downstream applications.