Search

In-context learning (ICL) adapts LLMs by providing demonstrations without fine-tuning the model parameters; however, it does not differentiate between demonstrations and quadratically increases the complexity of Transformer LLMs, exhausting the memory. As a solution, we propose Mixtures of In-Context Learners (MoICL), a novel approach to treat subsets of demonstrations as experts and learn a weighting function to merge their output distributions based on a training set. In our experiments, we show performance improvements on 5 out of 7 classification datasets compared to a set of strong baselines (up to +13\% compared to ICL and LENS). Moreover, we enhance the Pareto frontier of ICL by reducing the inference time needed to achieve the same performance with fewer demonstrations. Finally, MoICL is more robust to out-of-domain (up to +11\%), imbalanced (up to +49\%), or noisy demonstrations (up to +38\%) or can filter these out from datasets. Overall, MoICL is a more expressive approach to learning from demonstrations without exhausting the context window or memory.

The SRG/eROSITA All-Sky Surveys (eRASSs) combine the advantages of complete sky coverage and the energy resolution provided by the charge couple device and offer the most holistic and detailed view of the diffuse soft X-ray background (SXRB) to date. The first eRASS (eRASS1) was completed at solar minimum, when solar wind charge exchange emission was minimal, providing the clearest view of the SXRB. We aim to extract spatial and spectral information from each constituent of the SXRB in the western Galactic hemisphere, focusing on the local hot bubble (LHB). We extracted and analysed eRASS1 spectra from almost all directions in the western Galactic hemisphere by dividing the sky into equal signal-to-noise bins. We fitted all bins with fixed spectral templates of known background constituents. We find the temperature of the LHB exhibits a north-south dichotomy at high latitudes ($|b|>30^{\circ}$), with the south being hotter, with a mean temperature at $kT=121.8\pm0.6\,$eV and the north at $kT=100.8\pm0.5\,$eV. At low latitudes, the LHB temperature increases towards the Galactic plane, especially towards the inner Galaxy. The LHB emission measure (${\rm EM_{LHB}}$) enhances approximately towards the Galactic poles. The ${\rm EM_{LHB}}$ map shows clear anti-correlation with the local dust column density. In particular, we found tunnels of dust cavities filled with hot plasma, potentially forming a wider network of hot interstellar medium. We also constructed a three-dimensional LHB model from ${\rm EM_{LHB}}$, assuming constant density. The average thermal pressure of the LHB is $P_{\rm thermal}/k=10100^{+1200}_{-1500}\,{\rm cm^{-3}\,K}$, a lower value than typical supernova remnants and wind-blown bubbles. This could be an indication of the LHB being open towards high Galactic latitudes., Comment: 31 pages, 37 figures, published in A&A

(Abridged) Sgr A$^\star$ is the electromagnetic counterpart of the accreting supermassive black hole in the Galactic center. Its emission is variable in the near-infrared (NIR) and X-ray wavelengths on short timescales. The physical origin of NIR and X-ray flares is still under debate. We introduce a model for the production of NIR and X-ray flares from an active region in Sgr A$^\star$, where particle acceleration takes place intermittently. In contrast to other radiation models for Sgr A$^\star$ flares, the particle acceleration is not assumed to be instantaneous. We studied the evolution of the particle distribution and the emitted electromagnetic radiation from the flaring region by numerically solving the kinetic equations for electrons and photons. Our calculations took the finite duration of particle acceleration, radiative energy losses, and physical escape from the flaring region into account. To gain better insight into the relation of the model parameters, we complemented our numerical study with analytical calculations. Flares are produced when the acceleration episode has a finite duration. The rising part in the light curve of a flare is related to the particle acceleration timescale, while the decay is controlled by the cooling or escape timescale of particles. Bright X-ray flares, such as the one observed in 2014, have $\gamma$-ray counterparts that might be detected by the Cherenkov Telescope Array Observatory. Our model for NIR and X-ray flares favors an interpretation of diffusive nonresonant particle acceleration in magnetized turbulence. If direct acceleration by the reconnection electric field in macroscopic current sheets causes the energization of particles during flares in Sgr A$^\star$, then models considering the injection of preaccelerated particles into a blob where particles cool and/or escape would be appropriate to describe the flare., Comment: 13 pages, 1 appendix, accepted for publication in A&A

Reinforcement Learning from Human Feedback aligns the outputs of Large Language Models with human values and preferences. Central to this process is the reward model (RM), which translates human feedback into training signals for optimising LLM behaviour. However, RMs can develop biases by exploiting spurious correlations in their training data, such as favouring outputs based on length or style rather than true quality. These biases can lead to incorrect output rankings, sub-optimal model evaluations, and the amplification of undesirable behaviours in LLMs alignment. This paper addresses the challenge of correcting such biases without additional data and training, introducing the concept of Post-hoc Reward Calibration. We first propose an intuitive approach to estimate the bias term and, thus, remove it to approximate the underlying true reward. We then extend the approach to a more general and robust form with the Locally Weighted Regression. Focusing on the prevalent length bias, we validate our proposed approaches across three experimental settings, demonstrating consistent improvements: (1) a 3.11 average performance gain across 33 reward models on the RewardBench dataset; (2) enhanced alignment of RM rankings with GPT-4 evaluations and human preferences based on the AlpacaEval benchmark; and (3) improved Length-Controlled win rate of the RLHF process in multiple LLM--RM combinations. Our method is computationally efficient and generalisable to other types of bias and RMs, offering a scalable and robust solution for mitigating biases in LLM alignment. Our code and results are available at https://github.com/ZeroYuHuang/Reward-Calibration., Comment: Preprint

Do speakers of different languages talk differently about what they see? Behavioural and cognitive studies report cultural effects on perception; however, these are mostly limited in scope and hard to replicate. In this work, we conduct the first large-scale empirical study of cross-lingual variation in image descriptions. Using a multimodal dataset with 31 languages and images from diverse locations, we develop a method to accurately identify entities mentioned in captions and present in the images, then measure how they vary across languages. Our analysis reveals that pairs of languages that are geographically or genetically closer tend to mention the same entities more frequently. We also identify entity categories whose saliency is universally high (such as animate beings), low (clothing accessories) or displaying high variance across languages (landscape). In a case study, we measure the differences in a specific language pair (e.g., Japanese mentions clothing far more frequently than English). Furthermore, our method corroborates previous small-scale studies, including 1) Rosch et al. (1976)'s theory of basic-level categories, demonstrating a preference for entities that are neither too generic nor too specific, and 2) Miyamoto et al. (2006)'s hypothesis that environments afford patterns of perception, such as entity counts. Overall, our work reveals the presence of both universal and culture-specific patterns in entity mentions.

The hot phase of the circumgalactic medium (CGM) allows us to probe the inflow and outflow of gas within a galaxy, which is responsible for dictating the evolution of the galaxy. Studying the hot CGM sheds light on a better understanding of gas physics, which is crucial to inform and constrain simulation models. With the recent advances in observational measurements probing the hot CGM in X-rays and tSZ, we have a new avenue for widening our knowledge of gas physics and feedback by exploiting the information from current/future observations. In this paper, we use the TNG300 hydrodynamical simulations to build a fully self-consistent forward model for the hot CGM. We construct a lightcone and generate mock X-ray observations. We quantify the projection effects, namely the locally correlated large-scale structure in X-rays and the effect due to satellite galaxies misclassified as centrals which affects the measured hot CGM galactocentric profiles in stacking experiments. We present an analytical model that describes the intrinsic X-ray surface brightness profile across the stellar and halo mass bins. The increasing stellar mass bins result in decreasing values of $\beta$, the exponent quantifying the slope of the intrinsic galactocentric profiles. We carry forward the current state-of-the-art by also showing the impact of the locally correlated environment on the measured X-ray surface brightness profiles. We also present, for the first time, the effect of misclassified centrals in stacking experiments for three stellar mass bins: $10^{10.5-11}\ M_\odot$, $10^{11-11.2}\ M_\odot$, and $10^{11.2-11.5}\ M_\odot$. We find that the contaminating effect of the misclassified centrals on the stacked profiles increases when the stellar mass decreases., Comment: 14 pages, 10 figures, Submitted to A&A, comments welcome

Quasi-periodic eruptions (QPEs) are an extreme X-ray variability phenomenon associated with low-mass supermassive black holes. First discovered in the nucleus of the galaxy GSN 069, they have been so far securely detected in five other galaxies, including RX J1301.9+2747. When detected, the out-of-QPE emission (quiescence) is consistent with the high-energy tail of thermal emission from an accretion disk. We present the X-ray and radio properties of RX J1301.9+2747, both in quiescence and during QPEs. We analyse X-ray data taken during five XMM-Newton observations between 2000 and 2022. The last three observations were taken in coordination with radio observations with the Karl G. Jansky Very Large Array. We also make use of EXOSAT, ROSAT, and Chandra archival observations taken between 1983 and 2009. XMM-Newton detected 34 QPEs of which 8 have significantly lower amplitudes than the others. No correlated radio/X-ray variability was observed during QPEs. In terms of timing properties, the QPEs in RX J1301.9+2747 do not exhibit the striking regularity observed in the discovery source GSN 069. In fact there is no clear repetition pattern between QPEs: the average time separation between their peaks is about four hours, but it can be as short as one, and as long as six hours. The QPE spectral properties of RX J1301.9+2747 as a function of energy are however very similar to those of GSN 069 and of other QPE sources. The quiescent emission of RX J1301.9+2747 is more complex than that of GSN 069, as it requires a soft X-ray excess-like component in addition to the thermal emission from the accretion disk. Its long-term X-ray quiescent flux variations are of low-amplitude and not strictly monotonic, with a general decay over $\sim 22$ years. We discuss our observational results in terms of some of the ideas and models that have been proposed so far for the physical origin of QPEs., Comment: Accepted for publication in Astronomy & Astrophysics. 22 pages and 17 figures in the main text; 11 pages, 5 tables, and 4 figures in the Appendix. Abstract shortened to comply with arXiv requirements

Large-scale magnetic fields are observed off the midplanes of disk galaxies, indicating that they harbour magnetised halos. These halos are crucial to studies of galaxy evolution, galactic-scale outflows, and feedback from star formation activity. Identifying the magnetised halo of the Milky Way is challenging because of the potential contamination from foreground emission arising in local spiral arms. Additionally, it is unclear how our magnetic halo is influenced by recently revealed large-scale structures such as the X-ray emitting eROSITA Bubbles, which, according to previous simulations, might be transient structures powered by the Galactic Center or the Galaxy's star-forming ring. Here we report the identification of several kpc-scale magnetised structures based on their polarized radio emission and their gamma-ray counterparts, which can be interpreted as the radiation of relativistic electrons. These non-thermal structures extend far above and below the Galactic plane and are spatially coincident with the thermal X-ray emission from the eROSITA Bubbles. The morphological consistency of these structures suggests a common origin, which can be sustained by Galactic outflows driven by the active star-forming regions located at 3-5 kpc from the Galactic Centre. These results reveal how X-ray-emitting and magnetised halos of spiral galaxies can be related to intense star formation activities and suggest that the X-shaped coherent magnetic structures observed in their halos can stem from galaxy outflows., Comment: Initially submitted on March 2nd, 2024

We aim at characterizing the population of low-luminosity X-ray sources in the Galactic plane by studying their X-ray spectra and periodic signals in the light curves. We are performing an X-ray survey of the Galactic disk using XMM-Newton, and the source XMMU J173029.8-330920 was serendipitously discovered in our campaign. We performed a follow-up observation of the source using our pre-approved NuSTAR target of opportunity time. We used various phenomenological models in xspec for the X-ray spectral modeling. We also computed the Lomb-Scargle periodogram to search for X-ray periodicity. A Monte Carlo method was used to simulate 1000 artificial light curves to estimate the significance of the detected period. We also searched for X-ray, optical, and infrared counterparts of the source in various catalogs. The spectral modeling indicates the presence of an intervening cloud with $N_{\rm H}\sim(1.5-2.3)\times10^{23}\ \rm cm^{-2}$ that partially absorbs the incoming X-ray photons. The X-ray spectra are best fit by a model representing emission from a collisionally ionized diffuse gas with plasma temperature $kT=26^{+11}_{-5}$ keV. Furthermore, an Fe $K_{\alpha}$ line at $6.47^{+0.13}_{-0.06}$ keV was detected with an equivalent width of the line of $312\pm104$ eV. We discovered a coherent pulsation with a period of $521.7\pm0.8$ s. The 3-10 keV pulsed fraction of the source is around $\sim$50-60\%. The hard X-ray emission with plasma temperature $kT=26^{+11}_{-5}$ keV, iron $K_{\alpha}$ emission at 6.4 keV and a periodic behavior of $521.7\pm0.8$ s suggest XMMU J173029.8-33092 to be an intermediate polar. We estimated the mass of the central white dwarf to be $0.94-1.4\ M_{\odot}$ by assuming a distance to the source of $\sim1.4-5$ kpc., Comment: 7 pages, 4 figures, accepted for publication in A&A

Quasi-periodic eruptions (QPEs) are repeated X-ray flares from galactic nuclei. Despite some diversity in the recurrence and amplitude of eruptions, their striking regularity has motivated theorists to associate QPEs with orbital systems. Among the known QPE sources, eRO-QPE2 has shown the most regular flare timing and luminosity since its discovery. We report here on its long-term evolution over $\sim3.3\,$yr from discovery and find that: i) the average QPE recurrence time per epoch has decreased over time, albeit not at a uniform rate; ii) the distinct alternation between consecutive long and short recurrence times found at discovery has not been significant since; iii) the spectral properties, namely flux and temperature of both eruptions and quiescence components, have remained remarkably consistent within uncertainties. We attempted to interpret these results as orbital period and eccentricity decay coupled with orbital and disk precession. However, since gaps between observations are too long, we are not able to distinguish between an evolution dominated by just a decreasing trend, or by large modulations (e.g. due to the precession frequencies at play). In the former case, the observed period decrease is roughly consistent with that of a star losing orbital energy due to hydrodynamic gas drag from disk collisions, although the related eccentricity decay is too fast and additional modulations have to contribute too. In the latter case, no conclusive remarks are possible on the orbital evolution and the nature of the orbiter due to the many effects at play. However, these two cases come with distinctive predictions for future X-ray data: in the former, we expect all future observations to show a shorter recurrence time than the latest epoch, while in the latter we expect some future observations to be found with a larger recurrence, hence an apparent temporary period increase., Comment: Version updated after re-submission to A&A with minor revisions. Appendix B in v1 was moved to the main text

A variety of scientific fields like proteomics and spintronics have created a new demand for on-chip devices capable of sensing parameters localized within a few tens of micrometers. Nano and microelectromechanical systems (NEMS/MEMS) are extensively employed for monitoring parameters that exert uniform forces over hundreds of micrometers or more, such as acceleration, pressure, and magnetic fields. However, they can show significantly degraded sensing performance when targeting more localized parameters, like the mass of a single cell. To address this challenge, we present a new MEMS device that leverages the destructive interference of two topological radiofrequency (RF) counter-propagating wave modes along a piezoelectric Aluminum Scandium Nitride (AlScN) Su-Schrieffer-Heeger (SSH) interface. The reported MEMS device opens up opportunities for further purposes, including achieving more stable frequency sources for communication and timing applications.

Multilingual Large Language Models (LLMs) achieve remarkable levels of zero-shot cross-lingual transfer performance. We speculate that this is predicated on their ability to align languages without explicit supervision from parallel sentences. While representations of translationally equivalent sentences in different languages are known to be similar after convergence, however, it remains unclear how such cross-lingual alignment emerges during pre-training of LLMs. Our study leverages intrinsic probing techniques, which identify which subsets of neurons encode linguistic features, to correlate the degree of cross-lingual neuron overlap with the zero-shot cross-lingual transfer performance for a given model. In particular, we rely on checkpoints of BLOOM, a multilingual autoregressive LLM, across different training steps and model scales. We observe a high correlation between neuron overlap and downstream performance, which supports our hypothesis on the conditions leading to effective cross-lingual transfer. Interestingly, we also detect a degradation of both implicit alignment and multilingual abilities in certain phases of the pre-training process, providing new insights into the multilingual pretraining dynamics., Comment: Accepted to Findings of the Association for Computational Linguistics: ACL 2024

The properties of the warm-hot intergalactic medium (WHIM) in cosmic filaments are among the least quantified units in modern astrophysics. The Spectrum Roentgen Gamma/eROSITA All Sky Survey ((SRG/eRASS) provides a unique opportunity to study the X-ray emission of the WHIM. We applied both imaging and spectroscopic stacking techniques to the data of the first four eRASS scans to inspect the X-ray emissions from 7817 cosmic filaments identified from Sloan Digital Sky Survey (SDSS) optical galaxy samples. We obtained a $9\sigma$ significant detection of the total X-ray signal from filaments in the 0.3--1.2~keV band. Here, we introduce a novel method to estimate the contamination fraction from unmasked X-ray halos, active galactic nuclei, and X-ray binaries associated with filament galaxies. We found an approximately 40\% contamination fraction for these unmasked sources, suggesting that the remaining 60\% of the signal could be coming from the WHIM and a $5.4\sigma$ detection significance of the WHIM. Moreover, we modeled the temperature and baryon density contrast of the detected WHIM by fitting the stacked spectrum and surface brightness profile. The best-fit temperature $\log(T/\mathrm{K})=6.84\pm0.07$, obtained by using a single temperature model, is marginally higher than in the simulation results. This could be due to the fitting of a single temperature model on a multi-temperature spectrum. Assuming a 0.2 solar abundance, the best-fit baryon density contrast $\log\Delta_\mathrm{b}=1.88\pm0.18$ is in general agreement with the X-ray emitting phases in the IllustrisTNG simulation. This result suggests that the broadband X-ray emission traces the high end of the temperature and density values that characterize the entire WHIM population., Comment: 15+8 pages, 11+7 figures, accepted for publication in A&A

For the Laplacian of an $n$-Riemannian manifold $X$, the Weyl law states that the $k$-th eigenvalue is asymptotically proportional to $(k/V)^{2/n}$, where $V$ is the volume of $X$. We show that this result can be derived via physical considerations by demanding that the gravitational potential for a compactification on $X$ behaves in the expected $(4+n)$-dimensional way at short distances. In simple product compactifications, when particle motion on $X$ is ergodic, for large $k$ the eigenfunctions oscillate around a constant, and the argument is relatively straightforward. The Weyl law thus allows to reconstruct the four-dimensional Planck mass from the asymptotics of the masses of the spin 2 Kaluza--Klein modes. For warped compactifications, a puzzle appears: the Weyl law still depends on the ordinary volume $V$, while the Planck mass famously depends on a weighted volume obtained as an integral of the warping function. We resolve this tension by arguing that in the ergodic case the eigenfunctions oscillate now around a power of the warping function rather than around a constant, a property that we call weighted quantum ergodicity. This has implications for the problem of gravity localization, which we discuss. We show that for spaces with D$p$-brane singularities the spectrum is discrete only for $p =6,7,8$, and for these cases we rigorously prove the Weyl law by applying modern techniques from RCD theory., Comment: 35 pages, 1 figure

The growing number of parameter-efficient adaptations of a base large language model (LLM) calls for studying whether we can reuse such trained adapters to improve performance for new tasks. We study how to best build a library of adapters given multi-task data and devise techniques for both zero-shot and supervised task generalization through routing in such library. We benchmark existing approaches to build this library and introduce model-based clustering, MBC, a method that groups tasks based on the similarity of their adapter parameters, indirectly optimizing for transfer across the multi-task dataset. To re-use the library, we present a novel zero-shot routing mechanism, Arrow, which enables dynamic selection of the most relevant adapters for new inputs without the need for retraining. We experiment with several LLMs, such as Phi-2 and Mistral, on a wide array of held-out tasks, verifying that MBC-based adapters and Arrow routing lead to superior generalization to new tasks. We make steps towards creating modular, adaptable LLMs that can match or outperform traditional joint training.

Large language models (LLMs) often exhibit undesirable behaviours, such as generating untruthful or biased content. Editing their internal representations has been shown to be effective in mitigating such behaviours on top of the existing alignment methods. We propose a novel inference-time editing method, namely spectral editing of activations (SEA), to project the input representations into directions with maximal covariance with the positive demonstrations (e.g., truthful) while minimising covariance with the negative demonstrations (e.g., hallucinated). We also extend our method to non-linear editing using feature functions. We run extensive experiments on benchmarks concerning truthfulness and bias with six open-source LLMs of different sizes and model families. The results demonstrate the superiority of SEA in effectiveness, generalisation to similar tasks, as well as computation and data efficiency. We also show that SEA editing only has a limited negative impact on other model capabilities., Comment: 24 pages, NeurIPS 2024

Language models (LMs) are bound to their tokenizer, which maps raw text to a sequence of vocabulary items (tokens). This restricts their flexibility: for example, LMs trained primarily on English may still perform well in other natural and programming languages, but have vastly decreased efficiency due to their English-centric tokenizer. To mitigate this, we should be able to swap the original LM tokenizer with an arbitrary one, on the fly, without degrading performance. Hence, in this work we define a new problem: Zero-Shot Tokenizer Transfer (ZeTT). The challenge at the core of ZeTT is finding embeddings for the tokens in the vocabulary of the new tokenizer. Since prior heuristics for initializing embeddings often perform at chance level in a ZeTT setting, we propose a new solution: we train a hypernetwork taking a tokenizer as input and predicting the corresponding embeddings. We empirically demonstrate that the hypernetwork generalizes to new tokenizers both with encoder (e.g., XLM-R) and decoder LLMs (e.g., Mistral-7B). Our method comes close to the original models' performance in cross-lingual and coding tasks while markedly reducing the length of the tokenized sequence. We also find that the remaining gap can be quickly closed by continued training on less than 1B tokens. Finally, we show that a ZeTT hypernetwork trained for a base (L)LM can also be applied to fine-tuned variants without extra training. Overall, our results make substantial strides toward detaching LMs from their tokenizer.

There are two contradictory views of the eROSITA bubbles: either a 10 kpc-scale pair of giant bubbles blown by the Galactic center (GC), or a 100 pc-scale local structure coincidentally located in the direction of GC. A key element of this controversy is the distance to the bubbles. Based on the 3D dust distribution in the Galactic plane, we found three isolated, distant (500-800 pc) clouds at intermediate Galactic latitudes. Their projected morphologies perfectly match the X-ray shadows on the defining features of the north eROSITA bubble, i.e., the North Polar Spur (NPS) and the Lotus Petal Cloud (LPC), indicating that both the NPS and LPC are distant with a distance lower limit of nearly 1kpc. In the X-ray dark region between the NPS and LPC, we found a few polarized radio arcs and attributed them to the bubble's shock front. These arcs match up perfectly with the outer border of the NPS and LPC and provide a way to define the bubble's border. The border defined in this way can be well described by the line-of-sight tangent of a 3D skewed cup model rooted in the GC. We conclude that, instead of being two independent, distant features, NPS and LPC compose a single, giant bubble, which, therefore, is most plausibly a 10-kpc scale bubble rooted at the GC., Comment: Accepted for publication in ApJL. 11 pages, 5 figures

The eROSITA will deliver an unprecedented volume of X-ray survey observations, 20-30 times more sensitive than ROSAT in the soft band (0.5-2 keV) and for the first time imaging in the hard band (2-10 keV) including galaxy clusters and groups along with obscured and unobscured AGNs. This calls for a powerful theoretical effort to control the systematics and biases that may affect the data analysis. We investigate the detection technique and selection effects in the galaxy group and AGN populations of a mock eROSITA survey at the depth of eRASS:4. We create a $30\times 30$ deg$^{2}$ mock observation based on the cosmological hydrodynamical simulation Magneticum Pathfinder within z=0-0.2. We combine a physical background extracted from the real eFEDS background analysis with realistic simulations of X-ray emission for the hot gas, AGNs and X-ray binaries. We apply a detection procedure equivalent to the reduction done on eRASS data and evaluate the completeness and contamination to reconstruct the luminosity functions of the extended and point sources in the catalogue. We assess the completeness of extended detections as a function of the input X-ray flux and halo. We achieve full recovery of the brightest (most massive) clusters and AGNs. However, a significant fraction of galaxy groups remains undetected. Examining the gas properties between the detected and undetected galaxy groups at fixed halo mass, we observe that the detected population exhibits, on average, higher X-ray brightness compared to the undetected ones. Moreover, we find that X-ray luminosity primarily correlates with the hot gas fraction, rather than temperature or metallicity. Our simulation suggests the presence of a systematic selection effect in current surveys, resulting in X-ray survey catalogues predominantly composed of the lowest-entropy, gas-richest, and highest surface brightness halos on galaxy group scales., Comment: 18 pages, 14 figures, submitted to A&A

State-of-the-art neurosymbolic learning systems use probabilistic reasoning to guide neural networks towards predictions that conform to logical constraints over symbols. Many such systems assume that the probabilities of the considered symbols are conditionally independent given the input to simplify learning and reasoning. We study and criticise this assumption, highlighting how it can hinder optimisation and prevent uncertainty quantification. We prove that loss functions bias conditionally independent neural networks to become overconfident in their predictions. As a result, they are unable to represent uncertainty over multiple valid options. Furthermore, we prove that these loss functions are difficult to optimise: they are non-convex, and their minima are usually highly disconnected. Our theoretical analysis gives the foundation for replacing the conditional independence assumption and designing more expressive neurosymbolic probabilistic models., Comment: Accepted at ICML 2024

Graphs are ubiquitous in various fields, and deep learning methods have been successful applied in graph classification tasks. However, building large and diverse graph datasets for training can be expensive. While augmentation techniques exist for structured data like images or numerical data, the augmentation of graph data remains challenging. This is primarily due to the complex and non-Euclidean nature of graph data. In this paper, it has been proposed a novel augmentation strategy for graphs that operates in a non-Euclidean space. This approach leverages graphon estimation, which models the generative mechanism of networks sequences. Computational results demonstrate the effectiveness of the proposed augmentation framework in improving the performance of graph classification models. Additionally, using a non-Euclidean distance, specifically the Gromow-Wasserstein distance, results in better approximations of the graphon. This framework also provides a means to validate different graphon estimation approaches, particularly in real-world scenarios where the true graphon is unknown., Comment: 6 pages, 3 figures

Multi-wavelength extragalactic nuclear transients, particularly those detectable as multi-messengers, are among the primary drivers for the next-generation observatories. X-ray quasi-periodic eruptions (QPEs) are the most recent and perhaps most peculiar addition to this group. Here, we report a first estimate of the volumetric rate of QPEs based on the first four discoveries with the eROSITA X-ray telescope onboard the Spectrum Roentgen Gamma observatory. Under the assumption, supported by a suite of simulated light curves, that these four sources sample the intrinsic population somewhat homogeneously, we correct for their detection efficiency and compute a QPE abundance of $\mathscr{R}_{{\rm vol}} = 0.60_{-0.43}^{+4.73} \times 10^{-6}\,$Mpc$^{-3}$ above an intrinsic average $\log L_{\rm 0.5-2.0\,keV}^{\rm peak} > 41.7$. Since the exact lifetime of QPEs ($\tau_{\rm life}$) is currently not better defined than between a few years or few decades, we convert this to a formation rate of $\mathscr{R}_{\rm vol}/\tau_{\rm life}\approx 0.6 \times 10^{-7} (\tau_{\rm life}/10\,\mathrm{y})^{-1}\,$Mpc$^{-3}\,$year$^{-1}$. As a comparison, this value is a factor $\sim10\,\tau_{\rm life}$ times smaller than the formation rate of tidal disruption events. The origin of QPEs is still debated, although lately most models suggest that they are the electromagnetic counterpart of extreme mass ratio inspirals (EMRIs). In this scenario, the QPE rate would thus be the first-ever constraint (i.e. a lower limit) to the EMRI rate from observations alone. Future discoveries of QPEs and advances in their theoretical modeling will consolidate or rule out their use for constraining the number of EMRIs detectable by the LISA mission., Comment: Accepted for publication in A&A. Comments welcome

For many years, it has been claimed that the Galactic ridge X-ray emission at the Galactic Center (GC) is truly diffuse in nature. However, with the advancement of modern X-ray satellites, it has been found that most of the diffuse emission is actually comprised of thousands of previously unresolved X-ray point sources. Further, many studies suggest that a vast majority of these X-ray point sources are magnetic cataclysmic variables (mCVs) and active binaries. One unambiguous way to identify these mCVs and other sources is by detecting their X-ray periodicity. Therefore, we systematically searched for periodic X-ray sources in the inner Galactic disk, including the GC region. We have used data from our ongoing XMM-Newton Heritage survey of the inner Galactic disk ($350^{\circ}\lesssim l\lesssim+7^{\circ}$ and $-1^{\circ}\lesssim b\lesssim +1^{\circ}$) plus the XMM-Newton archival observations of the GC. We computed the Lomb-Scargle periodogram of the light curves for the periodicity search. We fitted the energy spectra of the sources using a simple power-law model plus three Gaussians at 6.4, 6.7, and 6.9 keV for the iron $K$ emission complex. We detected periodicity in 26 sources. For 14 of them, this is the first discovery of periodicity. For the other 12 sources, we found periods similar to those already known, indicating no significant period evolution. We also searched for the Gaia counterparts of the periodic sources to estimate their distances using the Gaia parallax. We found a likely Gaia counterpart for seven sources. We have classified the sources into four categories based on the periodicity, hardness ratio, and the equivalent width of Fe $K$ line emission. Of the 14 sources where we detect the periodicity for the first time, four are likely to be intermediate polars, five are likely to be polars, two are neutron star X-ray binaries, and three are of unknown nature., Comment: 19 pages, 9 figures, accepted for publication in A&A

Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value representations for past tokens, whose size scales linearly with the input sequence length and batch size. As a solution, we propose Dynamic Memory Compression (DMC), a method for online key-value cache compression at inference time. Most importantly, the model learns to apply different compression ratios in different heads and layers. We retrofit pre-trained LLMs such as Llama 2 (7B, 13B and 70B) into DMC Transformers, achieving up to 7x throughput increase during auto-regressive inference on an NVIDIA H100 GPU. DMC is applied via continued pre-training on a negligible percentage of the original data without adding any extra parameters. DMC preserves the original downstream performance with up to 4x cache compression, outperforming up-trained grouped-query attention (GQA) and key-value eviction policies (H$_2$O, TOVA). GQA and DMC can be even combined to obtain compounded gains. Hence, DMC can serve as a drop-in replacement for KV caching in existing LLMs to fit longer contexts and larger batches within any given memory budget.

The detection of blue-shifted absorption lines likely associated with ionized Iron K-shell transitions in the X-ray spectra of many Active Galactic Nuclei (AGN) suggests the presence of a highly ionized gas outflowing with mildly relativistic velocities (0.03c-0.6c), named Ultra-Fast Outflow (UFO). Within the SUBWAYS project we characterized these winds starting from a sample of 22 radio-quiet quasars at 0.1 < z < 0.4, and compared the results with similar studies in the literature on samples of 42 local radio-quiet Seyfert galaxies and 14 high redshift radio-quiet quasars. The scope of our work is a statistical study of UFO parameters and incidence, considering key physical properties of the sources, e.g. supermassive black hole (SMBH) mass, bolometric luminosity, accretion rates and Spectral Energy Distribution, with the aim of gaining new insights into the UFO launching mechanisms. We find indications that highly luminous AGN with steeper X-ray/UV ratio, are more likely to host UFO. The presence of UFO is not significantly related to any other AGN property in our sample. These findings suggest that the UFO phenomenon may be transient. Focusing on AGN with UFO, other important results are: (1) faster UFO have larger ionization parameters and column densities; (2) X-ray radiation plays a more crucial role in driving highly ionized winds compared to UV; (3) the correlation between outflow velocity and luminosity is significantly flatter than what expected for radiatively driven winds; (4) more massive BH experience higher wind mass-losses, suppressing accretion of matter onto the BH; (5) the UFO launching radius is positively correlated with the Eddington ratio. Furthermore, our analysis suggest the involvement of multiple launching mechanisms, including radiation pressure and magneto-hydrodynamic processes, rather than pointing to a single, universally applicable mechanism., Comment: 52 pages, 154 figures and 7 tables. Accepted in Astronomy & Astrophysics, in press

Despite the success of existing instruction-tuned models, we find that they usually struggle to respond to queries with multiple instructions. This impairs their performance in complex problems whose solution consists of multiple intermediate tasks. Thus, we contend that part of the fine-tuning data mixture should be sequential--containing a chain of interrelated tasks. We first approach sequential instruction tuning from a task-driven perspective, manually creating interpretable intermediate tasks for multilingual and visual question answering: namely "translate then predict" and "caption then answer". Next, we automate this process by turning instructions in existing datasets (e.g., Alpaca and FlanCoT) into diverse and complex sequential instructions, making our method general-purpose. Models that underwent our sequential instruction tuning show improved results in coding, maths, and open-ended generation. Moreover, we put forward a new benchmark named SeqEval to evaluate a model's ability to follow all the instructions in a sequence, which further corroborates the benefits of our fine-tuning method. We hope that our endeavours will open new research avenues on instruction tuning for complex tasks., Comment: 21pages, 8 figures

Monochromatic gamma-ray signals constitute a potential smoking gun signature for annihilating or decaying dark matter particles that could relatively easily be distinguished from astrophysical or instrumental backgrounds. We provide an updated assessment of the sensitivity of the Cherenkov Telescope Array (CTA) to such signals, based on observations of the Galactic centre region as well as of selected dwarf spheroidal galaxies. We find that current limits and detection prospects for dark matter masses above 300 GeV will be significantly improved, by up to an order of magnitude in the multi-TeV range. This demonstrates that CTA will set a new standard for gamma-ray astronomy also in this respect, as the world's largest and most sensitive high-energy gamma-ray observatory, in particular due to its exquisite energy resolution at TeV energies and the adopted observational strategy focussing on regions with large dark matter densities. Throughout our analysis, we use up-to-date instrument response functions, and we thoroughly model the effect of instrumental systematic uncertainties in our statistical treatment. We further present results for other potential signatures with sharp spectral features, e.g.~box-shaped spectra, that would likewise very clearly point to a particle dark matter origin., Comment: 44 pages JCAP style (excluding author list and references), 19 figures; minor changes to match published version

We prove several functional and geometric inequalities only assuming the linearity and a quantitative $\mathrm{L}^\infty$-to-Lipschitz smoothing of the heat semigroup in metric-measure spaces. Our results comprise a Buser inequality, a lower bound on the size of the nodal set of a Laplacian eigenfunction, and different estimates involving the Wasserstein distance. The approach works in large variety settings, including Riemannian manifolds with a variable Kato-type lower bound on the Ricci curvature tensor, $\mathsf{RCD}(K,\infty)$ spaces, and some sub-Riemannian structures, such as Carnot groups, the Grushin plane and the $\mathbb{SU}(2)$ group., Comment: 23 pages

Quasi-Periodic Eruptions (QPEs) are luminous X-ray outbursts recurring on hour timescales, observed from the nuclei of a growing handful of nearby low-mass galaxies. Their physical origin is still debated, and usually modeled as (a) accretion disk instabilities or (b) interaction of a supermassive black hole (SMBH) with a lower mass companion in an extreme mass-ratio inspiral (EMRI). EMRI models can be tested with several predictions related to the short- and long-term behavior of QPEs. In this study, we report on the ongoing 3.5-year NICER and XMM-Newton monitoring campaign of eRO-QPE1, which is known to exhibit erratic QPEs that have been challenging for the simplest EMRI models to explain. We report 1) complex, non-monotonic evolution in the long-term trends of QPE energy output and inferred emitting area; 2) the disappearance of the QPEs (within NICER detectability) in October 2023, then reappearance by January 2024 at a luminosity $\sim$100x fainter (and temperature $\sim$3x cooler) than initial discovery; 3) radio non-detections with MeerKAT and VLA observations partly contemporaneous with our NICER campaign (though not during outbursts); and 4) the presence of a possible $\sim$6-day modulation of the QPE timing residuals, which aligns with the expected nodal precession timescale of the underlying accretion disk. Our results tentatively support EMRI-disk collision models powering the QPEs, and we demonstrate that the timing modulation of QPEs may be used to jointly constrain the SMBH spin and disk density profile., Comment: Accepted for publication in ApJ

[abridged] Hot plasma plays a crucial role in regulating the baryon cycle within the Milky Way, flowing from the energetic sources in the Galactic center and disc, to the corona and the halo. This hot plasma represents an important fraction of the Galactic baryons, plays a key role in galactic outflows and is an important ingredient in galaxy evolution models. Taking advantage of the Spectrum-Roentgen-Gamma (SRG))/eROSITA first all-sky survey (eRASS1), in this work, we aim to provide a panoramic view of the hot circumgalactic medium (CGM) of the Milky Way. Here we present the eROSITA eRASS1 half sky maps in narrow energy bands corresponding to the most prominent soft X-ray lines: OVII and OVIII, which allow us to constrain the distribution of the hot plasma within and surrounding the Milky Way. We corrected the maps by removing the expected contribution associated with the cosmic X-ray background, the time-variable solar wind charge exchange, and the local hot bubble. We applied corrections to mitigate the effect of absorption, therefore highlighting the emission from the CGM of the Milky Way. We use the line ratio of the oxygen lines as a proxy to constrain the temperature of the warm-hot CGM, and we define a pseudo-temperature $\mathcal{T}$ map. The map highlights how different regions are dominated by different thermal components. Towards the outer halo, the temperature distribution of the CGM on angular scales of 2-20 deg is consistent with being constant $\Delta \mathcal{T} / \langle \mathcal{T}\rangle \leq 4\%$, with a marginal detection of $\Delta \mathcal{T} / \langle \mathcal{T}\rangle = 2.7 \% \pm 0.2\%$ (statistical) $\pm 0.6\%$ (systematic) in the southern hemisphere. Instead, significant variations $\sim 12\%$ are observed on many tens of degrees scales when comparing the northern and southern hemispheres., Comment: 15 pages, 14 figures, 1 table, A&A in press

Understanding how the properties of galaxies relate to the properties of the hot circum-galactic medium (CGM) around them can constrain galaxy evolution models. We measured the X-ray luminosity of the hot CGM based on the surface brightness profiles of central galaxy samples measured from Spectrum Roentgen Gamma (SRG)/eROSITA all-sky survey data. We related the X-ray luminosity to the galaxies' stellar and halo mass, and we compared the observed relations to the self-similar model and intrinsic (i.e., not forward-modeled) output of the IllustrisTNG, EAGLE, and SIMBA simulations. The average hot CGM X-ray luminosity ($L_{\rm X,CGM}$) correlates with the galaxy's stellar mass ($M_*$). It increases from $(1.6 \pm 2.1)\times10^{39} \rm erg\,s^{-1}$ to $(3.4 \pm 0.3)\times10^{41} \rm erg\,s^{-1}$, when $\log(M_*)$ increases from 10.0 to 11.5. A power law describes the correlation as $\log(L_{\rm X,CGM})= (2.4\pm 0.1)\times \log(M_*)+(14.6\pm1.5)$. The hot CGM X-ray luminosity as a function of halo mass is measured within $\log(M_{\rm 500c})=11.3-13.7$, extending our knowledge of the scaling relation by more than two orders of magnitude. $L_{\rm X,CGM}$ increases with $M_{\rm 500c}$ from $(3.0 \pm 1.6)\times10^{39}\ \rm erg\,s^{-1}$ at $\log(M_{\rm 500c})=11.3$ to $(1.3 \pm 0.1)\times10^{42}\ \rm erg\,s^{-1}$ at $\log(M_{\rm 500c})=13.7$. The relation follows a power law of $\log(L_{\rm X,CGM})= (1.32\pm 0.05)\times \log(M_{\rm 500c})+(24.1\pm0.7)$. Our observations highlight the necessity of non-gravitational processes at the galaxy group scale while suggesting these processes are sub-dominant at the galaxy scale. We show that the outputs of current cosmological galaxy simulations generally align with the observational results uncovered here but with possibly important deviations in selected mass ranges., Comment: Abstract abridged for arXiv submission, accepted by A&A

The circumgalactic medium (CGM) provides the material needed for galaxy formation and influences galaxy evolution. The hot ($T>10^6K$) CGM is poorly detected around galaxies with stellar masses ($M_*$) lower than $3\times10^{11}M_\odot$ due to the low surface brightness. We used the X-ray data from the first four SRG/eROSITA All-Sky Surveys (eRASS:4). Based on the SDSS spectroscopic survey and halo-based group finder algorithm, we selected central galaxies with spectroscopic redshifts of $z_{\rm spec}<0.2$ and stellar masses of $10.0<\log(M_*/M_\odot)<11.5$ (85,222 galaxies) -- or halo masses of $11.5<\log(M_{\rm 200m}/M_\odot)<14.0$ (125,512 galaxies). By stacking the X-ray emission around galaxies, masking the detected X-ray point sources and carefully modeling the X-ray emission from the unresolved active galactic nuclei (AGN) and X-ray binaries (XRB), we obtain the X-ray emission from the hot CGM. We detected the X-ray emission around MW-mass and more massive central galaxies extending up to the virial radius ($R_{\rm vir}$). We used a $\beta$ model to describe the X-ray surface brightness profile and found $\beta =0.43^{+0.10}_{-0.06}\,(0.37^{+0.04}_{-0.02})$ for MW-mass (M31-mass) galaxies.We estimated the baryon budget of the hot CGM and obtained a value that is lower than the prediction of $\Lambda$CDM cosmology, indicating significant gas depletion in these halos. We extrapolated the hot CGM profile measured within $R_{\rm vir}$ to larger radii and found that within $\approx 3 R_{\rm vir}$, the baryon budget is close to the $\Lambda$CDM cosmology prediction. Our results set a firm footing for the presence of the hot CGM around such galaxies. These measurements constitute a new benchmark for galaxy evolution models and possible implementations of feedback processes therein., Comment: Abstract abridged for arXiv submission, accepted by A&A

A dozen of patches of polarized radio emission spanning tens of degrees in the form of coherent and stationary loops are observed at radio frequencies across the sky. Their origin is usually associated to nearby shocks, possibly arising from close supernovae explosions. The origin of the radio Loop XII remains so far unknown. We report an anti-correlation of the radio polarized emission of loop XII with a large patch of soft X-ray emission found with SRG/eROSITA in excess of the background surface brightness, in the same region. The soft X-ray seemingly coherent patch in excess of the background emission, which we dub as the Goat Horn complex, extends over a remarkable area of $\sim 1000$ deg$^2$ and includes an arc-shaped enhancement potentially tracing a cold front. An anti-correlation of the X-ray intensity with the temperature of the plasma responsible for the X-ray emission is also observed. The X-ray bright arc seems to anticipate the radio loop XII by some degrees on the sky. This behavior can be recast in terms of a correlation between X-ray surface brightness and radio depolarization. We explore and discuss different possible scenarios for the source of the diffuse emission in the Goat Horn complex: a large supernova remnant; an outflow from active star formation regions in nearby Galactic spiral arms; a hot atmosphere around the Large Magellanic Cloud. In order to probe these scenarios further, a more detailed characterization on the velocity of the hot gas is required., Comment: 12 pages, 14 figures

The Gemini-Monoceros X-ray enhancement is a rich field for studying diffuse X-ray emission and supernova remnants (SNRs). With the launch of eROSITA onboard the SRG platform in 2019, we are now able to fully study these sources. Many of the SNRs in the vicinity are suspected to be very old remnants, which are severely understudied in X-rays due to numerous observational challenges. In addition, identification of new faint large SNRs might help to solve the long-standing discrepancy of observed and expected number of Galactic SNRs. We performed a detailed X-ray spectral analysis of the entire diffuse structure and a detailed background analysis of the vicinity. We also made use of multi-wavelength data to better understand the morphology and to constrain the distances to the different sources. We estimated the plasma properties of the sources and calculated a grid of model SNRs to determine the individual SNR properties. Most of the diffuse plasma of the Monogem Ring SNR is well described by a single non-equilibrium ionization (NEI) component with an average temperature of $kT = 0.14\pm 0.03$ keV. We obtain an age of $\approx 1.2\cdot 10^5$ yr - consistent with PSR B0656+14 - for the Monogem Ring. In the south-east, we found evidence for a hotter second plasma component and a possible new SNR candidate at $\approx 300$ pc, with the new candidate having an age of $\approx 50,000$ yr. We were also able to improve on previous studies on the more distant Monoceros Loop and PKS 0646+06 SNRs. We obtained significantly higher temperatures than previous studies, and for PKS 0646+06 a much lower estimated age of the SNR. We also found a new SNR candidate G190.4+12.5 which most likely is located at $D > 1.5 $ kpc, expanding into a low density medium at a high distance from the Galactic plane, with an estimated age of $40,000-60,000$ yr., Comment: Accepted by A&A - 02/04/2024

Aims: During the Spectrum Roentgen Gamma (SRG)/ eROSITA all-sky surveys, X-ray sources close to the South Ecliptic Pole (SEP) are observed almost every 4 hours. We aim to identify the sources exhibiting the most significant long-term X-ray variability within 3 degrees of the SEP in the first three surveys, and investigate their properties. Methods: We determined the variability significance of all sources observed by eROSITA within 3 degrees of the SEP by using thresholds on the Bayesian excess variance (SCATT_LO) and the maximum amplitude deviation (AMPL_SIG). Sources exhibiting a variability significance above $3\sigma$ were subdivided into likely Galactic and extragalactic sources, by using spectral and photometric information of their optical counterparts. We quantified the X-ray normalised excess variances of all variable sources, and also calculated the periodograms of the brightest ones. Results: Out of more than $10^4$ X-ray sources detected by eROSITA within 3 degrees of the SEP, we identified 453 that exhibit significant X-ray variability. SCATT_LO is significantly more sensitive to detecting variable sources in this field, but AMPL_SIG helps provide a more complete variability sample. Of those variable sources, 168 were classified as likely extragalactic, and 235 as likely Galactic. The periodograms of most bright and variable extragalactic sources are approximately described by an aliased power law ($P\propto\nu^{-\alpha}$) with an index of $\alpha\approx 1$. We identified a potential tidal disruption event, and long-term transient sources. The stellar X-ray variability was predominantly caused by bright X-ray flares from coronally active stars., Comment: 22 pages, 29 figures

X-ray quasi-periodic eruptions (QPEs) are a novel addition to the group of extragalactic transients. In this work, we report the discovery of two further galaxies showing QPEs, eRO-QPE3 and eRO-QPE4, with the eROSITA X-ray telescope on board the Spectrum Roentgen Gamma observatory. Among the properties in common with those of known QPEs are: the thermal-like spectral shape in eruption (up to $kT\sim110-120$ eV) and quiescence ($kT\sim50-90$ eV) and its evolution during the eruptions (with a harder rise than decay); the lack of strong canonical signatures of active nuclei (from current optical, UV, infrared and radio data); and the low-mass nature of the host galaxies ($\log M_*\approx 9-10$) and their massive central black holes ($\log M_{\rm BH}\approx 5-7$). These discoveries also bring several new insights into the QPE population: i) eRO-QPE3 shows eruptions on top of a decaying quiescence flux, providing further evidence for a connection between QPEs and a preceding tidal disruption event; ii) eRO-QPE3 exhibits the longest recurrence times and faintest peak luminosity of QPEs, compared to the known QPE population, excluding a correlation between the two; iii) we find evidence, for the first time, of a transient component that is harder, albeit much fainter, than the thermal QPE spectrum in eRO-QPE4; and iv) eRO-QPE4 displays the appearance (or significant brightening) of the quiescence disk component after the detection of QPEs, supporting its short-lived nature against a preexisting active galactic nucleus. Overall, the newly discovered properties (e.g., recent origin and/or transient nature of the quiescent accretion disk; lack of correlation between eruption recurrence timescales and luminosity) are qualitatively consistent with recent models that identify QPEs as extreme mass-ratio inspirals., Comment: Accepted for publication in A&A

The eROSITA telescope array aboard the Spektrum Roentgen Gamma (SRG) satellite began surveying the sky in December 2019, with the aim of producing all-sky X-ray source lists and sky maps of an unprecedented depth. Here we present catalogues of both point-like and extended sources using the data acquired in the first six months of survey operations (eRASS1; completed June 2020) over the half sky whose proprietary data rights lie with the German eROSITA Consortium. We describe the observation process, the data analysis pipelines, and the characteristics of the X-ray sources. With nearly 930000 entries detected in the most sensitive 0.2-2.3 keV energy range, the eRASS1 main catalogue presented here increases the number of known X-ray sources in the published literature by more than 60%, and provides a comprehensive inventory of all classes of X-ray celestial objects, covering a wide range of physical processes. A smaller catalogue of 5466 sources detected in the less sensitive but harder 2.3-5 keV band is the result of the first true imaging survey of the entire sky above 2 keV. We show that the number counts of X-ray sources in eRASS1 are consistent with those derived over narrower fields by past X-ray surveys of a similar depth, and we explore the number counts variation as a function of the location in the sky. Adopting a uniform all-sky flux limit (at 50% completeness) of F_{0.5-2 keV} > 5 \times 10^{-14}$ erg\,s$^{-1}$\,cm$^{-2}$, we estimate that the eROSITA all-sky survey resolves into individual sources about 20% of the cosmic X-ray background in the 1-2 keV range. The catalogues presented here form part of the first data release (DR1) of the SRG/eROSITA all-sky survey. Beyond the X-ray catalogues, DR1 contains all detected and calibrated event files, source products (light curves and spectra), and all-sky maps. Illustrative examples of these are provided., Comment: 39 pages, 23 figures. Accepted for publication in A&A. Accompanying eROSITA-DE Data Release 1

Large Language Models (LLMs) are difficult to fully fine-tune (e.g., with instructions or human feedback) due to their sheer number of parameters. A family of parameter-efficient sparse fine-tuning methods have proven promising in terms of performance but their memory requirements increase proportionally to the size of the LLMs. In this work, we scale sparse fine-tuning to state-of-the-art LLMs like LLaMA 2 7B and 13B. We propose SpIEL, a novel sparse fine-tuning method which, for a desired density level, maintains an array of parameter indices and the deltas of these parameters relative to their pretrained values. It iterates over: (a) updating the active deltas, (b) pruning indices (based on the change of magnitude of their deltas) and (c) regrowth of indices. For regrowth, we explore two criteria based on either the accumulated gradients of a few candidate parameters or their approximate momenta estimated using the efficient SM3 optimizer. We experiment with instruction-tuning of LLMs on standard dataset mixtures, finding that SpIEL is often superior to popular parameter-efficient fine-tuning methods like LoRA (low-rank adaptation) in terms of performance and comparable in terms of run time. We additionally show that SpIEL is compatible with both quantization and efficient optimizers, to facilitate scaling to ever-larger model sizes. We release the code for SpIEL at https://github.com/AlanAnsell/peft and for the instruction-tuning experiments at https://github.com/ducdauge/sft-llm.

We obtain a vanishing result for solutions of the inequality $|\Delta u|\le q_1|u|+q_2|\nabla u|$ that decay to zero along a very general warped cylindrical end of a Riemannian manifold. The appropriate decay condition at infinity on $u$ is related to the behavior of the potential functions $q_1$ and $q_2$ and to the asymptotic geometry of the end. The main ingredient is a new Carleman estimate of independent interest. Geometric applications to conformal deformations and to minimal graphs are presented., Comment: Minor corrections following referee's suggestions. Final version, to appear on IMRN

Machine learning models typically focus on specific targets like creating classifiers, often based on known population feature distributions in a business context. However, models calculating individual features adapt over time to improve precision, introducing the concept of decoupling: shifting from point evaluation to data distribution. We use calibration strategies as strategy for decoupling machine learning (ML) classifiers from score-based actions within business logic frameworks. To evaluate these strategies, we perform a comparative analysis using a real-world business scenario and multiple ML models. Our findings highlight the trade-offs and performance implications of the approach, offering valuable insights for practitioners seeking to optimize their decoupling efforts. In particular, the Isotonic and Beta calibration methods stand out for scenarios in which there is shift between training and testing data.

Taking advantage of the structure of large datasets to pre-train Deep Learning models is a promising strategy to decrease the need for supervised data. Self-supervised learning methods, such as contrastive and its variation are a promising way towards obtaining better representations in many Deep Learning applications. Soundscape ecology is one application in which annotations are expensive and scarce, therefore deserving investigation to approximate methods that do not require annotations to those that rely on supervision. Our study involves the use of the methods Barlow Twins and VICReg to pre-train different models with the same small dataset with sound patterns of bird and anuran species. In a downstream task to classify those animal species, the models obtained results close to supervised ones, pre-trained in large generic datasets, and fine-tuned with the same task., Comment: 10 pages, 2 figures, 3 tables

By June of 2020, the extended ROentgen Survey with an Imaging Telescope Array (eROSITA) on board the Spectrum Roentgen Gamma observatory had completed its first of the planned eight X-ray all-sky survey (eRASS1). The large effective area of the X-ray telescope makes it ideal for a survey of the faint X-ray diffuse emission over half of the sky with an unprecedented energy resolution and position accuracy. In this work, we produce the X-ray diffuse emission maps of the eRASS1 data with a current calibration, covering the energy range from 0.2 to 8.0 keV. We validated these maps by comparison with X-ray background maps derived from the ROSAT All Sky Survey (RASS). We generated X-ray images with a pixel area of 9 arcmin$^2$ using the observations available to the German eROSITA consortium. The contribution of the particle background to the photons was subtracted from the final maps. We also subtracted all the point sources above a flux threshold dependent on the goal of the subtraction, exploiting the eRASS1 catalog that will soon be available. The accuracy of the eRASS1 maps is shown by a flux match to the RASS X-ray maps, obtained by converting the eROSITA rates into equivalent ROSAT count rates in the standard ROSAT energy bands R4, R5, R6, and R7, within 1.25$\sigma$. We find small residual deviations in the R4, R5, and R6 bands, where eROSITA tends to observe lower flux than ROSAT (~11%), while a better agreement is achieved in the R7 band (~1%). The eRASS maps exhibit lower noise levels than RASS maps at the same resolution above 0.3 keV. We report the average surface brightness and total flux of different large sky regions as a reference. The detection of faint emission from diffuse hot gas in the Milky Way is corroborated by the consistency of the eRASS1 and RASS maps shown in this paper and by their comparable flux dynamic range., Comment: A&A, in press. 27 pages; 26 figures; 3 tables

A prompt is a sequence of symbol or tokens, selected from a vocabulary according to some rule, which is prepended/concatenated to a textual query. A key problem is how to select the sequence of tokens: in this paper we formulate it as a combinatorial optimization problem. The high dimensionality of the token space com-pounded by the length of the prompt sequence requires a very efficient solution. In this paper we propose a Bayesian optimization method, executed in a continuous em-bedding of the combinatorial space. In this paper we focus on hard prompt tuning (HPT) which directly searches for discrete tokens to be added to the text input with-out requiring access to the large language model (LLM) and can be used also when LLM is available only as a black-box. This is critically important if LLMs are made available in the Model as a Service (MaaS) manner as in GPT-4. The current manu-script is focused on the optimization of discrete prompts for classification tasks. The discrete prompts give rise to difficult combinatorial optimization problem which easily become intractable given the dimension of the token space in realistic applications. The optimization method considered in this paper is Bayesian optimization (BO) which has become the dominant approach in black-box optimization for its sample efficiency along with its modular structure and versatility. In this paper we use BoTorch, a library for Bayesian optimization research built on top of pyTorch. Albeit preliminary and obtained using a 'vanilla' version of BO, the experiments on RoB-ERTa on six benchmarks, show a good performance across a variety of tasks and enable an analysis of the tradeoff between size of the search space, accuracy and wall clock time.