Search

We introduce the Python program THALAS (TensorFlow Hydrodynamics Analysis for Lyman-Alpha Simulations), which maps baryon fields (baryon density, temperature, and velocity) to Ly$\alpha$ optical depth fields in both real space and redshift space. Unlike previous Ly$\alpha$ codes, THALAS is fully differentiable, enabling a wide variety of potential applications for general analysis of hydrodynamical simulations and cosmological inference. To demonstrate THALAS's capabilities, we apply it to the Ly$\alpha$ forest inversion problem: given a Ly$\alpha$ optical depth field, we reconstruct the corresponding real-space dark matter density field. Such applications are relevant to both cosmological and three-dimensional tomographic analyses of Lyman Alpha forest data., Comment: 19 pages, 6 figures

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements., Comment: accepted by SC '24

Cosmological information is usually extracted from the Lyman-$\alpha$ forest correlations using only either large-scale information interpreted through linear theory or using small-scale information interpreted by means of expensive hydrodynamical simulations. A complete cosmological interpretation of the 3D correlations at all measurable scales is challenged by the need of more realistic models including the complex growth of non-linear small scales that can only be studied within large hydrodynamical simulations. Past work were often limited by the trade off between the simulated cosmological volume and the resolution of the low-density intergalactic medium from which the Lyman-$\alpha$ signal originates. We conduct a suite of hydrodynamical simulations of the intergalactic medium, including one of the largest Lyman-$\alpha$ simulations ever performed in terms of volume (640 $h^{-1}\mathrm{Mpc}$), alongside simulations in smaller volumes with resolutions up to 25 $h^{-1}\mathrm{kpc}$. We compare the 3D Lyman-$\alpha$ power spectra predicted by those simulations to different non-linear models. The inferred Lyman-$\alpha$ bias and RSD parameters, $b_\alpha$ and $\beta_\alpha$ are in remarkable agreement with those measured in SDSS and DESI data. We find that, contrary to intuition, the convergence of large-scale modes of the 3D Lyman-$\alpha$ power spectra, which determines $\beta_\alpha$, is primarily influenced by the resolution of the simulation box through mode coupling, rather than the box size itself. Finally, we study the BAO signal encoded in the 3D Lyman-$\alpha$ power spectra. For the first time with a hydrodynamical simulation, we clearly detect the BAO signal, however we only marginally detect its damping, associated with the non-linear growth of the structures., Comment: 17 pages, 12 figures

We present a novel maximum a posteriori estimator to jointly estimate band-powers and the covariance of the three-dimensional power spectrum (P3D) of Lyman-alpha forest flux fluctuations, called MAPLE. Our Wiener-filter based algorithm reconstructs a window-deconvolved P3D in the presence of complex survey geometries typical for Lyman-alpha surveys that are sparsely sampled transverse to and densely sampled along the line-of-sight. We demonstrate our method on idealized Gaussian random fields with two selection functions: (i) a sparse sampling of 30 background sources per square degree designed to emulate the currently observing the Dark Energy Spectroscopic Instrument (DESI); (ii) a dense sampling of 900 background sources per square degree emulating the upcoming Prime Focus Spectrograph Galaxy Evolution Survey. Our proof-of-principle shows promise, especially since the algorithm can be extended to marginalize jointly over nuisance parameters and contaminants, i.e.offsets introduced by continuum fitting. Our code is implemented in JAX and is publicly available on GitHub., Comment: 6 pages, 4 figures

The epoch of hydrogen reionization is complete by $z=5$, but its progression at higher redshifts is uncertain. Measurements of Ly$\alpha$ forest opacity show large scatter at $z<6$, suggestive of spatial fluctuations in neutral fraction ($x_\mathrm{HI}$), temperature, or ionizing background, either individually or in combination. However, these effects are degenerate, necessitating modeling these physics in tandem in order to properly interpret the observations. We begin this process by developing a framework for modeling the reionization history and associated temperature fluctuations, with the intention of incorporating ionizing background fluctuations at a later time. To do this, we generate several reionization histories using semi-numerical code AMBER, selecting histories with volume-weighted neutral fractions that adhere to the observed CMB optical depth and dark pixel fractions. Implementing these histories in the \texttt{Nyx} cosmological hydrodynamics code, we examine the evolution of gas within the simulation, and the associated metrics of the Ly$\alpha$ forest opacity. We find that the pressure smoothing scale within the IGM is strongly correlated with the adiabatic index of the temperature-density relation. We find that while models with 20,000 K photoheating at reionization are better able to reproduce the shape of the observed $z=5$ 1D flux power spectrum than those with 10,000 K, they fail to match the highest wavenumbers. The simulated autocorrelation function and optical depth distributions are systematically low and narrow, respectively, compared to the observed values, but are in better agreement when the reionization history is longer in duration, more symmetric in its distribution of reionization redshifts, or if there are remaining neutral regions at $z<6$. The systematically low variance likely requires the addition of a fluctuating UVB., Comment: To be submitted to MNRAS. Comments welcome

We perform the first measurement of the thermal and ionization state of the intergalactic medium (IGM) across 0.9 < z < 1.5 using 301 \lya absorption lines fitted from 12 HST STIS quasar spectra, with a total pathlength of \Delta z=2.1. We employ the machine-learning-based inference method that uses joint b-N distributions obtained from \lyaf decomposition. Our results show that the HI photoionization rates, \Gamma, are in good agreement with the recent UV background synthesis models, with \log (\Gamma/s^{-1})={-11.79}^{0.18}_{-0.15}, -11.98}^{0.09}_{-0.09}, and {-12.32}^{0.10}_{-0.12} at z=1.4, 1.2, and 1 respectively. We obtain the IGM temperature at the mean density, T_0, and the adiabatic index, \gamma, as [\log (T_0/K), \gamma]= [{4.13}^{+0.12}_{-0.10}, {1.34}^{+0.10}_{-0.15}], [{3.79}^{+0.11}_{-0.11}, {1.70}^{+0.09}_{-0.09}] and [{4.12}^{+0.15}_{-0.25}, {1.34}^{+0.21}_{-0.26}] at z=1.4, 1.2 and 1 respectively. Our measurements of T_0 at z=1.4 and 1.2 are consistent with the expected trend from z<3 temperature measurements as well as theoretical expectations that, in the absence of any non-standard heating, the IGM should cool down after HeII reionization. Whereas, our T_0 measurements at z=1 show unexpectedly high IGM temperature. However, because of the relatively large uncertainty in these measurements of the order of \Delta T_0~5000 K, mostly emanating from the limited redshift path length of available data in these bins, we can not definitively conclude whether the IGM cools down at z<1.5. Lastly, we generate a mock dataset to test the constraining power of future measurement with larger datasets. The results demonstrate that, with redshift pathlength \Delta z \sim 2 for each redshift bin, three times the current dataset, we can constrain the T_0 of IGM within 1500K. Such precision would be sufficient to conclusively constrain the history of IGM thermal evolution at z < 1.5.

Small-scale correlations measured in the Lyman-$\alpha$ (Ly$\alpha$) forest encode information about the intergalactic medium and the primordial matter power spectrum. In this article, we present and implement a simple method to measure the 3-dimensional power spectrum, $P_{\rm 3D}$, of the Ly$\alpha$ forest at wavenumbers $k$ corresponding to small, $\sim$ Mpc scales. In order to estimate $P_{\rm 3D}$ from sparsely and unevenly distributed data samples, we rely on averaging 1-dimensional Fourier Transforms, as previously carried out to estimate the 1-dimensional power spectrum of the Ly$\alpha$ forest, $P_{\rm 1D}$. This methodology exhibits a very low computational cost. We confirm the validity of this approach through its application to Nyx cosmological hydrodynamical simulations. Subsequently, we apply our method to the eBOSS DR16 Ly$\alpha$ forest sample, providing as a proof of principle, a first $P_{\rm 3D}$ measurement averaged over two redshift bins $z=2.2$ and $z=2.4$. This work highlights the potential for forthcoming $P_{\rm 3D}$ measurements, from upcoming large spectroscopic surveys, to untangle degeneracies in the cosmological interpretation of $P_{\rm 1D}$., Comment: 28 pages, 9 figures; matches published version

The auto-correlation function of the Lyman-$\alpha$ (Ly$\alpha$) forest flux from high-z quasars can statistically probe all scales of the intergalactic medium (IGM) just after the epoch of reionization. The thermal state of the IGM, which is determined by the physics of reionization, sets the amount of small-scale power seen in the \lya forest. To study the sensitivity of the auto-correlation function to the thermal state of the IGM, we compute the auto-correlation function from cosmological hydrodynamical simulations with semi-numerical models of the thermal state of the IGM. We create mock data sets of 20 quasars to forecast constraints on $T_0$ and $\gamma$, which characterize a tight temperature-density relation in the IGM, at $5.4 \leq z \leq 6$. At $z = 5.4$ we find that an ideal data set constrains $T_0$ to 29\% and $\gamma$ to 9\%. In addition, we investigate four realistic reionization scenarios that combine temperature and ultra-violet background (UVB) fluctuations at $z = 5.8$. We find that, when using mock data generated from a model that includes temperature and UVB fluctuations, we can rule out a model with no temperature or UVB fluctuations at $>1\sigma$ level 50.5\% of the time., Comment: 18 pages, 13 figures, comments welcome. arXiv admin note: text overlap with arXiv:2208.09013

Dense gas in minihalos with masses of $10^6-10^8~M_\odot$ can shield themselves from reionization for $\sim100$ Myr after being exposed to the UV background. These self-shielded systems, often unresolved in cosmological simulations, can introduce strong absorption in quasar spectra. This paper is the first systematic study on the impact of these systems on the Ly$\alpha$ forest. We first derive the HI column density profile of photoevaporating minihalos by conducting 1D radiation-hydrodynamics simulations. We utilize these results to estimate the Ly$\alpha$ opacity from minihalos in a large-scale simulation that cannot resolve self-shielding. When the ionization rate of the background radiation is $0.03\times10^{-12}~{\rm s}^{-1}$, as expected near the end of reionization at $z\sim5.5$, we find that the incidence rate of damped Ly$\alpha$ absorbers increases by a factor of $\sim2-4$ compared to at $z=4.5$. The Ly$\alpha$ flux is, on average, suppressed by $\sim 3\%$ of its mean due to minihalos. The absorption features enhance the 1D power spectrum up to $\sim5\%$ at $k\sim0.1~h~{\rm Mpc}^{-1}~({\rm or}~10^{-3}~{\rm km}^{-1}~{\rm s})$, which is comparable to the enhancement caused by inhomogeneous reionization. The flux is particularly suppressed in the vicinity of large halos along the line-of-sight direction at separations of up to $10~h^{-1}~{\rm Mpc}$ at $r_\perp\lesssim2~h^{-1}~{\rm Mpc}$. However, these effects become much smaller for higher ionizing rates ($\gtrsim0.3\times10^{-12}~{\rm s}^{-1}$) expected in the post-reionization Universe. Our findings highlight the need to consider minihalo absorption when interpreting the Ly$\alpha$ forest at $z\gtrsim5.5$. Moreover, the sensitivity of these quantities to the ionizing background intensity can be exploited to constrain the intensity itself., Comment: 16 pages; 11 figures; Accepted for publication in ApJ

Recently, the Lyman-$\alpha$ (Ly$\alpha$) forest flux auto-correlation function has been shown to be sensitive to the mean free path of hydrogen-ionizing photons, $\lambda_{\text{mfp}}$, for simulations at $z \geq 5.4$. Measuring $\lambda_{\text{mfp}}$ at these redshifts will give vital information on the ending of reionization. Here we present the first observational measurements of the Ly$\alpha$ forest flux auto-correlation functions in ten redshift bins from $5.1 \leq z \leq 6.0$. We use a sample of 35 quasar sightlines at $z > 5.7$ from the extended XQR-30 data set, this data has signal-to-noise ratios of $> 20$ per spectral pixel. We carefully account for systematic errors in continuum reconstruction, instrumentation, and contamination by damped Ly$\alpha$ systems. With these measurements, we introduce software tools to generate auto-correlation function measurements from any simulation. For an initial comparison, we show our auto-correlation measurements with simulation models for recently measured $\lambda_{\text{mfp}}$ values and find good agreements. Further work in modeling and understanding the covariance matrices of the data is necessary to get robust measurements of $\lambda_{\text{mfp}}$ from this data., Comment: 17 pages, 13 figures, submitted to MNRAS

At $z \lesssim 1$, shock heating caused by large-scale velocity flows and possibly violent feedback from galaxy formation, converts a significant fraction of the cool gas ($T\sim 10^4$ K) in the intergalactic medium (IGM) into warm-hot phase (WHIM) with $T >10^5$K, resulting in a significant deviation from the previously tight power-law IGM temperature-density relationship, $T=T_0 (\rho / {\bar{\rho}})^{\gamma -1}$. This study explores the impact of the WHIM on measurements of the low-$z$ IGM thermal state, $[T_0,\gamma]$, based on the $b$-$N_{H I}$ distribution of the Lyman-$\alpha$ forest. Exploiting a machine learning-enabled simulation-based inference method trained on Nyx hydrodynamical simulations, we demonstrate that [$T_0$, $\gamma$] can still be reliably measured from the $b$-$N_{H I}$ distribution at $z=0.1$, notwithstanding the substantial WHIM in the IGM. To investigate the effects of different feedback, we apply this inference methodology to mock spectra derived from the IllustrisTNG and Illustris simulations at $z=0.1$. The results suggest that the underlying $[T_0,\gamma]$ of both simulations can be recovered with biases as low as $|\Delta \log(T_0/\text{K})| \lesssim 0.05$ dex, $|\Delta \gamma | \lesssim 0.1$, smaller than the precision of a typical measurement. Given the large differences in the volume-weighted WHIM fractions between the three simulations (Illustris 38\%, IllustrisTNG 10\%, Nyx 4\%) we conclude that the $b$-$N_{H I}$ distribution is not sensitive to the WHIM under realistic conditions. Finally, we investigate the physical properties of the detectable Lyman-$\alpha$ absorbers, and discover that although their $T$ and $\Delta$ distributions remain mostly unaffected by feedback, they are correlated with the photoionization rate used in the simulation., Comment: 23 pages, 24 figures

Hydrodynamical cosmological simulations are a powerful tool for accurately predicting the properties of the intergalactic medium (IGM) and for producing mock skies that can be compared against observational data. However, the need to resolve density fluctuation in the IGM puts a stringent requirement on the resolution of such simulations which in turn limits the volumes which can be modelled, even on most powerful supercomputers. In this work, we present a novel modeling method which combines physics-driven simulations with data-driven generative neural networks to produce outputs that are qualitatively and statistically close to the outputs of hydrodynamical simulations employing 8 times higher resolution. We show that the Ly-$\alpha$ flux field, as well as the underlying hydrodynamic fields, have greatly improved statistical fidelity over a low-resolution simulation. Importantly, the design of our neural network allows for sampling multiple realizations from a given input, enabling us to quantify the model uncertainty. Using test data, we demonstrate that this model uncertainty correlates well with the true error of the Ly-$\alpha$ flux prediction. Ultimately, our approach allows for training on small simulation volumes and applying it to much larger ones, opening the door to producing accurate Ly-$\alpha$ mock skies in volumes of Hubble size, as will be probed with DESI and future spectroscopic sky surveys., Comment: 13 pages, 11 figures

As supercomputers advance towards exascale capabilities, computational intensity increases significantly, and the volume of data requiring storage and transmission experiences exponential growth. Adaptive Mesh Refinement (AMR) has emerged as an effective solution to address these two challenges. Concurrently, error-bounded lossy compression is recognized as one of the most efficient approaches to tackle the latter issue. Despite their respective advantages, few attempts have been made to investigate how AMR and error-bounded lossy compression can function together. To this end, this study presents a novel in-situ lossy compression framework that employs the HDF5 filter to improve both I/O costs and boost compression quality for AMR applications. We implement our solution into the AMReX framework and evaluate on two real-world AMR applications, Nyx and WarpX, on the Summit supercomputer. Experiments with 4096 CPU cores demonstrate that AMRIC improves the compression ratio by up to 81X and the I/O performance by up to 39X over AMReX's original compression solution., Comment: 12 pages, 18 figures, 3 tables, accepted by ACM/IEEE SC '23

Super-Resolution (SR) techniques aim to enhance data resolution, enabling the retrieval of finer details, and improving the overall quality and fidelity of the data representation. There is growing interest in applying SR methods to complex spatiotemporal systems within the Scientific Machine Learning (SciML) community, with the hope of accelerating numerical simulations and/or improving forecasts in weather, climate, and related areas. However, the lack of standardized benchmark datasets for comparing and validating SR methods hinders progress and adoption in SciML. To address this, we introduce SuperBench, the first benchmark dataset featuring high-resolution datasets (up to $2048\times2048$ dimensions), including data from fluid flows, cosmology, and weather. Here, we focus on validating spatial SR performance from data-centric and physics-preserved perspectives, as well as assessing robustness to data degradation tasks. While deep learning-based SR methods (developed in the computer vision community) excel on certain tasks, despite relatively limited prior physics information, we identify limitations of these methods in accurately capturing intricate fine-scale features and preserving fundamental physical properties and constraints in scientific data. These shortcomings highlight the importance and subtlety of incorporating domain knowledge into ML models. We anticipate that SuperBench will significantly advance SR methods for scientific tasks.

We present and validate the catalog of Lyman-$\alpha$ forest fluctuations for 3D analyses using the Early Data Release (EDR) from the Dark Energy Spectroscopic Instrument (DESI) survey. We used 88,511 quasars collected from DESI Survey Validation (SV) data and the first two months of the main survey (M2). We present several improvements to the method used to extract the Lyman-$\alpha$ absorption fluctuations performed in previous analyses from the Sloan Digital Sky Survey (SDSS). In particular, we modify the weighting scheme and show that it can improve the precision of the correlation function measurement by more than 20%. This catalog can be downloaded from https://data.desi.lbl.gov/public/edr/vac/edr/lya/fuji/v0.3 and it will be used in the near future for the first DESI measurements of the 3D correlations in the Lyman-$\alpha$ forest.

We present the one-dimensional Lyman-$\alpha$ forest power spectrum measurement using the first data provided by the Dark Energy Spectroscopic Instrument (DESI). The data sample comprises $26,330$ quasar spectra, at redshift $z > 2.1$, contained in the DESI Early Data Release and the first two months of the main survey. We employ a Fast Fourier Transform (FFT) estimator and compare the resulting power spectrum to an alternative likelihood-based method in a companion paper. We investigate methodological and instrumental contaminants associated to the new DESI instrument, applying techniques similar to previous Sloan Digital Sky Survey (SDSS) measurements. We use synthetic data based on log-normal approximation to validate and correct our measurement. We compare our resulting power spectrum with previous SDSS and high-resolution measurements. With relatively small number statistics, we successfully perform the FFT measurement, which is already competitive in terms of the scale range. At the end of the DESI survey, we expect a five times larger Lyman-$\alpha$ forest sample than SDSS, providing an unprecedented precise one-dimensional power spectrum measurement., Comment: 23 pages, 23 figures, Journal version

The Ly$\alpha$ forest (LAF) at $z>5$ probes the thermal and reionization history of the intergalactic medium (IGM) and the nature of dark matter, but its interpretation requires comparison to cosmological hydrodynamical simulations. At high-$z$, convergence of these simulations is more exacting since transmission is dominated by underdense voids that are challenging to resolve. With evidence mounting for a late end to reionization, small structures down to the sub-kpc level may survive to later times than conventionally thought due to the reduced time for pressure smoothing to impact the gas, further tightening simulation resolution requirements. We perform a suite of simulations using the Eulerian cosmological hydrodynamics code Nyx, spanning domain sizes of 1.25-10 $h^{-1}$ Mpc and 5-80 $h^{-1}$ kpc cells, and explore the interaction of these variables with the timing of reionization on the properties of the matter distribution and the simulated LAF at $z=5.5$. In observable Ly$\alpha$ power, convergence within 10% is achieved for $k< 0.1$ s/km, but larger $k$ shows deviation of up to 20 percent. While a later reionization retains more small structure in the density field, because of the greater thermal broadening there is little difference in the convergence of LAF power between early ($z=9$) and later ($z=6$) reionizations. We conclude that at $z\sim5.5$, resolutions of 10 kpc are necessary for convergence of LAF power at $k<0.1$ s/km, while higher-$k$ modes require higher resolution, and that the timing of reionization does not significantly impact convergence given realistic photoheating., Comment: Submitted to MNRAS. Comments welcome

We present a new approach to measure the power-law temperature density relationship $T=T_0 (\rho / \bar{\rho})^{\gamma -1}$ and the UV background photoionization rate $\Gamma_{\rm HI}$ of the IGM based on the Voigt profile decomposition of the Ly$\alpha$ forest into a set of discrete absorption lines with Doppler parameter $b$ and the neutral hydrogen column density $N_{\rm HI}$. Previous work demonstrated that the shape of the $b$-$N_{\rm HI}$ distribution is sensitive to the IGM thermal parameters $T_0$ and $\gamma$, whereas our new inference algorithm also takes into account the normalization of the distribution, i.e. the line-density d$N$/d$z$, and we demonstrate that precise constraints can also be obtained on $\Gamma_{\rm HI}$. We use density-estimation likelihood-free inference (DELFI) to emulate the dependence of the $b$-$N_{\rm HI}$ distribution on IGM parameters trained on an ensemble of 624 Nyx hydrodynamical simulations at $z = 0.1$, which we combine with a Gaussian process emulator of the normalization. To demonstrate the efficacy of this approach, we generate hundreds of realizations of realistic mock HST/COS datasets, each comprising 34 quasar sightlines, and forward model the noise and resolution to match the real data. We use this large ensemble of mocks to extensively test our inference and empirically demonstrate that our posterior distributions are robust. Our analysis shows that by applying our new approach to existing Ly$\alpha$ forest spectra at $z\simeq 0.1$, one can measure the thermal and ionization state of the IGM with very high precision ($\sigma_{\log T_0} \sim 0.08$ dex, $\sigma_\gamma \sim 0.06$, and $\sigma_{\log \Gamma_{\rm HI}} \sim 0.07$ dex)., Comment: 20 pages, 17 figures, accepted for publication in MNRAS

We compare two state-of-the-art numerical codes to study the overall accuracy in modeling the intergalactic medium and reproducing Lyman-$\alpha$ forest observables for DESI and high-resolution data sets. The codes employ different approaches to solving both gravity and modeling the gas hydrodynamics. The first code, Nyx, solves the Poisson equation using the Particle-Mesh (PM) method and the Euler equations using a finite volume method. The second code, \CRKHACC, uses a Tree-PM method to solve for gravity, and an improved Lagrangian smoothed particle hydrodynamics (SPH) technique, where fluid elements are modeled with particles, to treat the intergalactic gas. We compare the convergence behavior of the codes in flux statistics as well as the degree to which the codes agree in the converged limit. We find good agreement overall with differences being less than observational uncertainties, and a particularly notable $\lesssim$1\% agreement in the 1D flux power spectrum. This agreement was achieved by applying a tessellation methodology for reconstructing the density in \CRKHACC instead of using an SPH kernel as is standard practice. We show that use of the SPH kernel can lead to significant and unnecessary biases in flux statistics; this is especially prominent at high redshifts, $z \sim 5$, as the Lyman-$\alpha$ forest mostly comes from lower-density regions which are intrinsically poorly sampled by SPH particles., Comment: submitted to MNRAS

Lossy compression is one of the most efficient solutions to reduce storage overhead and improve I/O performance for HPC applications. However, existing parallel I/O libraries cannot fully utilize lossy compression to accelerate parallel write due to the lack of deep understanding on compression-write performance. To this end, we propose to deeply integrate predictive lossy compression with HDF5 to significantly improve the parallel-write performance. Specifically, we propose analytical models to predict the time of compression and parallel write before the actual compression to enable compression-write overlapping. We also introduce an extra space in the process to handle possible data overflows resulting from prediction uncertainty in compression ratios. Moreover, we propose an optimization to reorder the compression tasks to increase the overlapping efficiency. Experiments with up to 4,096 cores from Summit show that our solution improves the write performance by up to 4.5X and 2.9X over the non-compression and lossy compression solutions, respectively, with only 1.5% storage overhead (compared to original data) on two real-world HPC applications., Comment: 13 pages, 18 figures, accepted by ACM/IEEE SC'22

Despite efforts over several decades, direct-detection experiments have not yet led to the discovery of the dark matter (DM) particle. This has led to increasing interest in alternatives to the Lambda CDM (LCDM) paradigm and alternative DM scenarios (including fuzzy DM, warm DM, self-interacting DM, etc.). In many of these scenarios, DM particles cannot be detected directly and constraints on their properties can ONLY be arrived at using astrophysical observations. The Dark Energy Spectroscopic Instrument (DESI) is currently one of the most powerful instruments for wide-field surveys. The synergy of DESI with ESA's Gaia satellite and future observing facilities will yield datasets of unprecedented size and coverage that will enable constraints on DM over a wide range of physical and mass scales and across redshifts. DESI will obtain spectra of the Lyman-alpha forest out to z~5 by detecting about 1 million QSO spectra that will put constraints on clustering of the low-density intergalactic gas and DM halos at high redshift. DESI will obtain radial velocities of 10 million stars in the Milky Way (MW) and Local Group satellites enabling us to constrain their global DM distributions, as well as the DM distribution on smaller scales. The paradigm of cosmological structure formation has been extensively tested with simulations. However, the majority of simulations to date have focused on collisionless CDM. Simulations with alternatives to CDM have recently been gaining ground but are still in their infancy. While there are numerous publicly available large-box and zoom-in simulations in the LCDM framework, there are no comparable publicly available WDM, SIDM, FDM simulations. DOE support for a public simulation suite will enable a more cohesive community effort to compare observations from DESI (and other surveys) with numerical predictions and will greatly impact DM science., Comment: Contributed white paper to Snowmass 2021, CF03; minor revisions

The non-linear process of cosmic structure formation produces gravitationally bound overdensities of dark matter known as halos. The abundances, density profiles, ellipticities, and spins of these halos can be tied to the underlying fundamental particle physics that governs dark matter at microscopic scales. Thus, macroscopic measurements of dark matter halos offer a unique opportunity to determine the underlying properties of dark matter across the vast landscape of dark matter theories. This white paper summarizes the ongoing rapid development of theoretical and experimental methods, as well as new opportunities, to use dark matter halo measurements as a pillar of dark matter physics., Comment: White paper submitted to the Proceedings of the US Community Study on the Future of Particle Physics (Snowmass 2021). 88 pages, 9 figures. Comments welcome

Powerful new observational facilities will come online over the next decade, enabling a number of discovery opportunities in the "Cosmic Frontier", which targets understanding of the physics of the early universe, dark matter and dark energy, and cosmological probes of fundamental physics, such as neutrino masses and modifications of Einstein gravity. Synergies between different experiments will be leveraged to present new classes of cosmic probes as well as to minimize systematic biases present in individual surveys. Success of this observational program requires actively pairing it with a well-matched state-of-the-art simulation and modeling effort. Next-generation cosmological modeling will increasingly focus on physically rich simulations able to model outputs of sky surveys spanning multiple wavebands. These simulations will have unprecedented resolution, volume coverage, and must deliver guaranteed high-fidelity results for individual surveys as well as for the cross-correlations across different surveys. The needed advances are as follows: (1) Development of scientifically rich and broadly-scoped simulations, which capture the relevant physics and correlations between probes (2) Accurate translation of simulation results into realistic image or spectral data to be directly compared with observations (3) Improved emulators and/or data-driven methods serving as surrogates for expensive simulations, constructed from a finite set of full-physics simulations (4) Detailed and transparent verification and validation programs for both simulations and analysis tools. (Abridged), Comment: Submitted to the Proceedings of the US Community Study on the Future of Particle Physics (Snowmass 2021)

We present the use of self-supervised learning to explore and exploit large unlabeled datasets. Focusing on 42 million galaxy images from the latest data release of the Dark Energy Spectroscopic Instrument (DESI) Legacy Imaging Surveys, we first train a self-supervised model to distill low-dimensional representations that are robust to symmetries, uncertainties, and noise in each image. We then use the representations to construct and publicly release an interactive semantic similarity search tool. We demonstrate how our tool can be used to rapidly discover rare objects given only a single example, increase the speed of crowd-sourcing campaigns, and construct and improve training sets for supervised applications. While we focus on images from sky surveys, the technique is straightforward to apply to any scientific dataset of any dimensionality. The similarity search web app can be found at https://github.com/georgestein/galaxy_search, Comment: 5 pages, 2 figures. The similarity search web app can be found at https://github.com/georgestein/galaxy_search. Accepted to the Fourth Workshop on Machine Learning and the Physical Sciences (NeurIPS 2021). ArXiv admin note: text overlap with arXiv:2110.00023

The thermal state of the intergalactic medium (IGM) contains vital information about the epoch of reionization, one of the most transformative yet poorly understood periods in the young universe. This thermal state is encoded in the small-scale structure of Lyman-$\alpha$ (Ly$\alpha$) absorption in quasar spectra. The 1D flux power spectrum measures the average small-scale structure along quasar sightlines. At high redshifts, where the opacity is large, averaging mixes high signal-to-noise ratio transmission spikes with noisy absorption troughs. Wavelet amplitudes are an alternate statistic that maintains spatial information while quantifying fluctuations at the same spatial frequencies as the power spectrum, giving them the potential to more sensitively measure the small-scale structure. Previous Ly$\alpha$ forest studies using wavelet amplitude probability density functions (PDFs) used limited spatial frequencies and neglected strong correlations between PDF bins and across wavelets scales, resulting in sub-optimal and unreliable parameter inference. Here we present a novel method for performing statistical inference using wavelet amplitude PDFs that spans the full range of spatial frequencies probed by the power spectrum and that fully accounts for these correlations. We applied this procedure to realistic mock data drawn from a simple thermal model parameterized by the temperature at mean density, $T_0$, and find that wavelets deliver 1$\sigma$ constraints on $T_0$ that are on average 7% more sensitive at $z=5$ (12% at $z=6$) than those from the power spectrum. We consider the possibility of combing wavelet PDFs with the power, but find that this does not lead to improved sensitivity., Comment: Accepted for publication in MNRAS, 22 Pages, 16 Figures

We employ self-supervised representation learning to distill information from 76 million galaxy images from the Dark Energy Spectroscopic Instrument Legacy Imaging Surveys' Data Release 9. Targeting the identification of new strong gravitational lens candidates, we first create a rapid similarity search tool to discover new strong lenses given only a single labelled example. We then show how training a simple linear classifier on the self-supervised representations, requiring only a few minutes on a CPU, can automatically classify strong lenses with great efficiency. We present 1192 new strong lens candidates that we identified through a brief visual identification campaign, and release an interactive web-based similarity search tool and the top network predictions to facilitate crowd-sourcing rapid discovery of additional strong gravitational lenses and other rare objects: https://github.com/georgestein/ssl-legacysurvey., Comment: 24 Pages, 15 figures, published in ApJ, data at github.com/georgestein/ssl-legacysurvey

Generating large volume hydrodynamical simulations for cosmological observables is a computationally demanding task necessary for next generation observations. In this work, we construct a novel fully convolutional variational auto-encoder (VAE) to synthesize hydrodynamic fields conditioned on dark matter fields from N-body simulations. After training the model on a single hydrodynamical simulation, we are able to probabilistically map new dark matter only simulations to corresponding full hydrodynamical outputs. By sampling over the latent space of our VAE, we can generate posterior samples and study the variance of the mapping. We find that our reconstructed field provides an accurate representation of the target hydrodynamical fields as well as a reasonable variance estimates. This approach has promise for the rapid generation of mocks as well as for implementation in a full Bayesian inverse model of observed data., Comment: 13 pages, 11 figures

Full-physics cosmological simulations are powerful tools for studying the formation and evolution of structure in the universe but require extreme computational resources. Here, we train a convolutional neural network to use a cheaper N-body-only simulation to reconstruct the baryon hydrodynamic variables (density, temperature, and velocity) on scales relevant to the Lyman-$\alpha$ (Ly$\alpha$) forest, using data from Nyx simulations. We show that our method enables rapid estimation of these fields at a resolution of $\sim$20kpc, and captures the statistics of the Ly$\alpha$ forest with much greater accuracy than existing approximations. Because our model is fully-convolutional, we can train on smaller simulation boxes and deploy on much larger ones, enabling substantial computational savings. Furthermore, as our method produces an approximation for the hydrodynamic fields instead of Ly$\alpha$ flux directly, it is not limited to a particular choice of ionizing background or mean transmitted flux., Comment: 10 pages, 6 figures

We use a contrastive self-supervised learning framework to estimate distances to galaxies from their photometric images. We incorporate data augmentations from computer vision as well as an application-specific augmentation accounting for galactic dust. We find that the resulting visual representations of galaxy images are semantically useful and allow for fast similarity searches, and can be successfully fine-tuned for the task of redshift estimation. We show that (1) pretraining on a large corpus of unlabeled data followed by fine-tuning on some labels can attain the accuracy of a fully-supervised model which requires 2-4x more labeled data, and (2) that by fine-tuning our self-supervised representations using all available data labels in the Main Galaxy Sample of the Sloan Digital Sky Survey (SDSS), we outperform the state-of-the-art supervised learning method.

Sky surveys are the largest data generators in astronomy, making automated tools for extracting meaningful scientific information an absolute necessity. We show that, without the need for labels, self-supervised learning recovers representations of sky survey images that are semantically useful for a variety of scientific tasks. These representations can be directly used as features, or fine-tuned, to outperform supervised methods trained only on labeled data. We apply a contrastive learning framework on multi-band galaxy photometry from the Sloan Digital Sky Survey (SDSS) to learn image representations. We then use them for galaxy morphology classification, and fine-tune them for photometric redshift estimation, using labels from the Galaxy Zoo 2 dataset and SDSS spectroscopy. In both downstream tasks, using the same learned representations, we outperform the supervised state-of-the-art results, and we show that our approach can achieve the accuracy of supervised models while using 2-4 times fewer labels for training., Comment: The codes, trained models, and data can be found at https://portal.nersc.gov/project/dasrepo/self-supervised-learning-sdss

Measurements of the Ly$\alpha$ forest based on large numbers of quasar spectra from sky surveys such as SDSS/eBOSS accurately probe the distribution of matter on small scales and thus provide important constraints on several ingredients of the cosmological model. A main summary statistic derived from those measurements is the one-dimensional power spectrum, P1D, of the Ly$\alpha$ absorption. However, model predictions for P1D rely on expensive hydrodynamical simulations of the intergalactic medium, which was the limiting factor in previous analyses. Datasets from upcoming surveys such as DESI will push observational accuracy near the 1%-level and probe even smaller scales. This observational push mandate seven more accurate simulations as well as more careful exploration of parameter space. In this work we evaluate the robustness and accuracy of simulations and the statistical framework used to constrain cosmological parameters. We present a comparison between the grid-based simulation code Nyx and SPH-based code Gadget in the context ofP1D. In addition, we perform resolution and box-size convergence tests using Nyx code. We use a Gaussian process emulation scheme to reduce the number of simulations required for exploration of parameter space without sacrificing the model accuracy. We demonstrate the ability to produce unbiased parameter constraints in an end-to-end inference test using mock eBOSS- and DESI-like data, and we advocate for the usage of adaptive sampling schemes as opposed to using a fixed Latin hypercube design., Comment: 36 pages, 17 figures, accepted for publication in JCAP

The Tri-Agency Cosmological Simulations (TACS) Task Force was formed when Program Managers from the Department of Energy (DOE), the National Aeronautics and Space Administration (NASA), and the National Science Foundation (NSF) expressed an interest in receiving input into the cosmological simulations landscape related to the upcoming DOE/NSF Vera Rubin Observatory (Rubin), NASA/ESA's Euclid, and NASA's Wide Field Infrared Survey Telescope (WFIRST). The Co-Chairs of TACS, Katrin Heitmann and Alina Kiessling, invited community scientists from the USA and Europe who are each subject matter experts and are also members of one or more of the surveys to contribute. The following report represents the input from TACS that was delivered to the Agencies in December 2018., Comment: 36 pages, 3 figures. Delivered to NASA, NSF, and DOE in Dec 2018

Cosmological probes pose an inverse problem where the measurement result is obtained through observations, and the objective is to infer values of model parameters which characterize the underlying physical system -- our Universe. Modern cosmological probes increasingly rely on measurements of the small-scale structure, and the only way to accurately model physical behavior on those scales, roughly 65 Mpc/h or smaller, is via expensive numerical simulations. In this paper, we provide a detailed description of a novel statistical framework for obtaining accurate parameter constraints by combining observations with a very limited number of cosmological simulations. The proposed framework utilizes multi-output Gaussian process emulators that are adaptively constructed using Bayesian optimization methods. We compare several approaches for constructing multi-output emulators that enable us to take possible inter-output correlations into account while maintaining the efficiency needed for inference. Using Lyman alpha forest flux power spectrum, we demonstrate that our adaptive approach requires considerably fewer --- by a factor of a few in Lyman alpha P(k) case considered here --- simulations compared to the emulation based on Latin hypercube sampling, and that the method is more robust in reconstructing parameters and their Bayesian credible intervals., Comment: 23 pages, 17 figures, submitted to ApJ

The expansion of the Universe is understood to have accelerated during two epochs: in its very first moments during a period of Inflation and much more recently, at $z < 1$, when Dark Energy is hypothesized to drive cosmic acceleration. The undiscovered mechanisms behind these two epochs represent some of the most important open problems in fundamental physics. The large cosmological volume at $2 < z < 5$, together with the ability to efficiently target high-$z$ galaxies with known techniques, enables large gains in the study of Inflation and Dark Energy. A future spectroscopic survey can test the Gaussianity of the initial conditions up to a factor of ~50 better than our current bounds, crossing the crucial theoretical threshold of $\sigma(f_{NL}^{\rm local})$ of order unity that separates single field and multi-field models. Simultaneously, it can measure the fraction of Dark Energy at the percent level up to $z = 5$, thus serving as an unprecedented test of the standard model and opening up a tremendous discovery space., Comment: Science white paper submitted to the Astro2020 Decadal Survey

The intense radiation emitted by luminous quasars dramatically alters the ionization state of their surrounding IGM. This so-called proximity effect extends out to tens of Mpc, and manifests as large coherent regions of enhanced Lyman-$\alpha$ (Ly$\alpha$) forest transmission in absorption spectra of background sightlines. Here we present a novel method based on Ly$\alpha$ forest tomography, which is capable of mapping these quasar `light echoes' in three dimensions. Using a dense grid (10-100) of faint ($m_r\approx24.7\,\mathrm{mag}$) background galaxies as absorption probes, one can measure the ionization state of the IGM in the vicinity of a foreground quasar, yielding detailed information about the quasar's radiative history and emission geometry. An end-to-end analysis - combining cosmological hydrodynamical simulations post-processed with a quasar emission model, realistic estimates of galaxy number densities, and instrument + telescope throughput - is conducted to explore the feasibility of detecting quasar light echoes. We present a new fully Bayesian statistical method that allows one to reconstruct quasar light echoes from thousands of individual low S/N transmission measurements. Armed with this machinery, we undertake an exhaustive parameter study and show that light echoes can be convincingly detected for luminous ($M_{1450} < -27.5\,\mathrm{mag}$ corresponding to $m_{1450} < 18.4\,\mathrm{mag}$ at $z\simeq 3.6$) quasars at redshifts $3 5$ is sufficient, requiring three hour integrations using existing instruments on 8m class telescopes., Comment: 22 pages, 21 figures

We present new measurements of the flux power-spectrum P(k) of the $z<0.5$ HI Lyman-$\alpha$ forest spanning scales k ~ 0.001-0.1 s/km. These results were derived from 65 far ultraviolet quasar spectra (resolution R~18000) observed with the Cosmic Origin Spectrograph (COS) on board the Hubble Space Telescope. The analysis required careful masking of all contaminating, coincident absorption from HI and metal-line transitions of the Galactic interstellar medium and intervening absorbers as well as proper treatment of the complex COS line-spread function. From the P(k) measurements, we estimate the HI photoionization rate ($\Gamma_{\rm HI}$) in the z<0.5 intergalactic medium. Our results confirm most of the previous $\Gamma_{\rm HI}$ estimates. We conclude that previous concerns of a photon underproduction crisis are now resolved by demonstrating that the measured $\Gamma_{\rm HI}$ can be accounted for by ultraviolet emission from quasars alone. In a companion paper, we will present constraints on the thermal state of the $z<0.5$ intergalactic medium from the P(k) measurements presented here., Comment: 15 pages, 8 figures, Accepted for publication in MNRAS

We determine the thermal evolution of the intergalactic medium (IGM) over $3\, \mathrm{Gyr}$ of cosmic time $1.8

During reionization, neutral hydrogen in the intergalactic medium (IGM) imprints a damping wing absorption feature on the spectrum of high-redshift quasars. A detection of this signature provides compelling evidence for a significantly neutral Universe, and enables measurements of the hydrogen neutral fraction $x_{\rm HI}(z)$ at that epoch. Obtaining reliable quantitative constraints from this technique, however, is challenging due to stochasticity induced by the patchy inside-out topology of reionization, degeneracies with quasar lifetime, and the unknown unabsorbed quasar spectrum close to rest-frame Ly$\alpha$. We combine a large-volume semi-numerical simulation of reionization topology with 1D radiative transfer through high-resolution hydrodynamical simulations of the high-redshift Universe to construct models of quasar transmission spectra during reionization. Our state-of-the-art approach captures the distribution of damping wing strengths in biased quasar halos that should have reionized earlier, as well as the erosion of neutral gas in the quasar environment caused by its own ionizing radiation. Combining this detailed model with our new technique for predicting the quasar continuum and its associated uncertainty, we introduce a Bayesian statistical method to jointly constrain the neutral fraction of the Universe and the quasar lifetime from individual quasar spectra. We apply this methodology to the spectra of the two highest redshift quasars known, ULAS J1120+0641 and ULAS J1342+0928, and measured volume-averaged neutral fractions $\langle x_{\rm HI} \rangle(z=7.09)=0.48^{+0.26}_{-0.26}$ and $\langle x_{\rm HI} \rangle(z=7.54)=0.60^{+0.20}_{-0.23}$ (posterior medians and 68% credible intervals) when marginalized over quasar lifetimes of $10^3 \leq t_{\rm q} \leq 10^8$ years., Comment: 18 pages, 13 figures. Submitted to ApJ

The HeII transverse proximity effect - enhanced HeII Ly{\alpha} transmission in a background sightline caused by the ionizing radiation of a foreground quasar - offers a unique opportunity to probe the emission properties of quasars, in particular the emission geometry (obscuration, beaming) and the quasar lifetime. Building on the foreground quasar survey published in Schmidt+2017, we present a detailed model of the HeII transverse proximity effect, specifically designed to include light travel time effects, finite quasar ages, and quasar obscuration. We post-process outputs from a cosmological hydrodynamical simulation with a fluctuating HeII UV background model, plus the added effect of the radiation from a single bright foreground quasar. We vary the age $t_\mathrm{age}$ and obscured sky fractions $\Omega_\mathrm{obsc}$ of the foreground quasar, and explore the resulting effect on the HeII transverse proximity effect signal. Fluctuations in IGM density and the UV background, as well as the unknown orientation of the foreground quasar, result in a large variance of the HeII Ly{\alpha} transmission along the background sightline. We develop a fully Bayesian statistical formalism to compare far UV HeII Ly{\alpha} transmission spectra of the background quasars to our models, and extract joint constraints on $t_\mathrm{age}$ and $\Omega_\mathrm{obsc}$ for the six Schmidt+2017 foreground quasars with the highest implied HeII photoionization rates. Our analysis suggests a bimodal distribution of quasar emission properties, whereby one foreground quasar, associated with a strong HeII transmission spike, is relatively old $(22\,\mathrm{Myr})$ and unobscured $\Omega_\mathrm{obsc}<35\%$, whereas three others are either younger than $(10\,\mathrm{Myr})$ or highly obscured $(\Omega_\mathrm{obsc}>70\%)$., Comment: 19 pages, 6 figures, submitted to ApJ

We present the most distant detection of cosmic voids ($z \sim 2.3$) and the first detection of three-dimensional voids in the Lyman-$\alpha$ forest. We used a 3D tomographic map of the absorption with effective comoving spatial resolution of $2.5\,h^{-1}\mathrm{Mpc}$ and volume of $3.15\times 10^5\,h^{-3}\mathrm{Mpc}^3$, which was reconstructed from moderate-resolution Keck-I/LRIS spectra of 240 background Lyman-break galaxies and quasars in a $0.16\,\mathrm{deg}^2$ footprint in the COSMOS field. Voids were detected using a spherical overdensity finder calibrated from hydrodynamical simulations of the intergalactic medium. This allows us to identify voids in the IGM corresponding to voids in the underlying matter density field, yielding a consistent volume fraction of voids in both data (19.5%) and simulations (18.2%). We fit excursion set models to the void radius function and compare the radially-averaged stacked profiles of large voids ($r > 5$ $h^{-1}$ Mpc) to stacked voids in mock observations and the simulated density field. Comparing with 432 coeval galaxies with spectroscopic redshifts in the same volume as the tomographic map, we find that the tomography-identified voids are underdense in galaxies by 5.95$\sigma$ compared to random cells., Comment: 19 pages, 10 figures, accepted by ApJ; includes revisions from referee report