Search

DBSCAN has been widely used in density-based clustering algorithms. However, with the increasing demand for Multi-density clustering, previous traditional DSBCAN can not have good clustering results on Multi-density datasets. In order to address this problem, an adaptive Multi-density DBSCAN algorithm (AMD-DBSCAN) is proposed in this paper. An improved parameter adaptation method is proposed in AMD-DBSCAN to search for multiple parameter pairs (i.e., Eps and MinPts), which are the key parameters to determine the clustering results and performance, therefore allowing the model to be applied to Multi-density datasets. Moreover, only one hyperparameter is required for AMD-DBSCAN to avoid the complicated repetitive initialization operations. Furthermore, the variance of the number of neighbors (VNN) is proposed to measure the difference in density between each cluster. The experimental results show that our AMD-DBSCAN reduces execution time by an average of 75% due to lower algorithm complexity compared with the traditional adaptive algorithm. In addition, AMD-DBSCAN improves accuracy by 24.7% on average over the state-of-the-art design on Multi-density datasets of extremely variable density, while having no performance loss in Single-density scenarios. Our code and datasets are available at https://github.com/AlexandreWANG915/AMD-DBSCAN., Comment: Accepted at DSAA2022

Integrating free-text explanations to in-context learning of large language models (LLM) is shown to elicit strong reasoning capabilities along with reasonable explanations. In this paper, we consider the problem of leveraging the explanations generated by LLM to improve the training of small reasoners, which are more favorable in real-production deployment due to their low cost. We systematically explore three explanation generation approaches from LLM and utilize a multi-task learning framework to facilitate small models to acquire strong reasoning power together with explanation generation capabilities. Experiments on multiple reasoning tasks show that our method can consistently and significantly outperform finetuning baselines across different settings, and even perform better than finetuning/prompting a 60x larger GPT-3 (175B) model by up to 9.5% in accuracy. As a side benefit, human evaluation further shows that our method can generate high-quality explanations to justify its predictions, moving towards the goal of explainable AI.

The Principal Component Analysis (PCA) method and the Singular Value Decomposition (SVD) method are widely used for foreground subtraction in 21 cm intensity mapping experiments. We show their equivalence, and point out that the condition for completely clean separation of foregrounds and cosmic 21 cm signal using the PCA/SVD is unrealistic. We propose a PCA-based foreground subtraction method, dubbed "Singular Vector Projection (SVP)" method, which exploits a priori information of the left and/or right singular vectors of the foregrounds. We demonstrate with simulation tests that this new, semi-blind method can reduce the error of the recovered 21 cm signal by orders of magnitude, even if only the left and/or right singular vectors in the largest few modes are exploited. The SVP estimators provide a new, effective approach for 21 cm observations to remove foregrounds and uncover the physics in the cosmic 21 cm signal., Comment: Comments welcome

Atomic hydrogen (H I) gas, mostly residing in dark matter halos after cosmic reionization, is the fuel for star formation. Its relation with properties of host halo is the key to understand the cosmic H I distribution. In this work, we propose a flexible, empirical model of H I-halo relation. In this model, while the H I mass depends primarily on the mass of host halo, there is also secondary dependence on other halo properties. We apply our model to the observation data of the Arecibo Fast Legacy ALFA Survey (ALFALFA), and find it can successfully fit to the cosmic H I abundance ($\Omega_{\rm HI}$), average H I-halo mass relation $\langle M_{\rm HI}|M_{\rm h}\rangle$, and the H I clustering. The bestfit of the ALFALFA data rejects with high confidence level the model with no secondary halo dependence of H I mass and the model with secondary dependence on halo spin parameter ($\lambda$), and shows strong dependence on halo formation time ($a_{1/2}$) and halo concentration ($c_{\rm vir}$). In attempt to explain these findings from the perspective of hydrodynamical simulations, the IllustrisTNG simulation confirms the dependence of H I mass on secondary halo parameters. However, the IllustrisTNG results show strong dependence on $\lambda$ and weak dependence on $c_{\rm vir}$ and $a_{1/2}$, and also predict a much larger value of H I clustering on large scales than observations. This discrepancy between the simulation and observation calls for improvements in understanding the H I-halo relation from both theoretical and observational sides., Comment: 14 pages, 8 figures, 1 table. Submitted to ApJ. Comments welcome

It has been recently shown that the astrophysics of reionization can be extracted from the Ly$\alpha$ forest power spectrum by marginalizing the memory of reionization over cosmological information. This impact of cosmic reionization on the Ly$\alpha$ forest power spectrum can survive cosmological time scales because cosmic reionization, which is inhomogeneous, and subsequent shocks from denser regions can heat the gas in low-density regions to $\sim 3\times10^4$ K and compress it to mean-density. Current approach of marginalization over the memory of reionization, however, is not only model-dependent, based on the assumption of a specific reionization model, but also computationally expensive. Here we propose a simple analytical template for the impact of cosmic reionization, thereby treating it as a broadband systematic to be marginalized over for Bayesian inference of cosmological information from the Ly$\alpha$ forest in a model-independent manner. This template performs remarkably well with an error of $\leq 6 \%$ at large scales $k \approx 0.19$ Mpc$^{-1}$ where the effect of the memory of reionization is important, and reproduces the broadband effect of the memory of reionization in the Ly$\alpha$ forest correlation function, as well as the expected bias of cosmological parameters due to this systematic. The template can successfully recover the morphology of forecast errors in cosmological parameter space as expected when assuming a specific reionization model for marginalization purposes, with a slight overestimation of tens of per cent for the forecast errors on the cosmological parameters. We further propose a similar template for this systematic on the Ly$\alpha$ forest 1D power spectrum., Comment: Updated to match accepted version for publication, 14 pages, 10 figures

The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining. Code, pretraining data, and pretrained models are available at https://github.com/jzbjyb/OmniTab., Comment: NAACL 2022

Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses. In this paper, we propose a new dialog pre-training framework called DialogVED, which introduces continuous latent variables into the enhanced encoder-decoder pre-training framework to increase the relevance and diversity of responses. With the help of a large dialog corpus (Reddit), we pre-train the model using the following 4 tasks adopted in language models (LMs) and variational autoencoders (VAEs): 1) masked language model; 2) response generation; 3) bag-of-words prediction; and 4) KL divergence reduction. We also add additional parameters to model the turn structure in dialogs to improve the performance of the pre-trained model. We conduct experiments on PersonaChat, DailyDialog, and DSTC7-AVSD benchmarks for response generation. Experimental results show that our model achieves the new state-of-the-art results on all these datasets., Comment: 13 pages, 1 figures, 9 tables

Summarizing sales calls is a routine task performed manually by salespeople. We present a production system which combines generative models fine-tuned for customer-agent setting, with a human-in-the-loop user experience for an interactive summary curation process. We address challenging aspects of dialogue summarization task in a real-world setting including long input dialogues, content validation, lack of labeled data and quality evaluation. We show how GPT-3 can be leveraged as an offline data labeler to handle training data scarcity and accommodate privacy constraints in an industrial setting. Experiments show significant improvements by our models in tackling the summarization and content validation tasks on public datasets., Comment: To be published in NAACL 2022

The first measurements of the 21 cm brightness temperature power spectrum from the epoch of reionization will very likely be achieved in the near future by radio interferometric array experiments such as the Hydrogen Epoch of Reionization Array (HERA) and the Square Kilometre Array (SKA). Standard MCMC analyses use an explicit likelihood approximation to infer the reionization parameters from the 21 cm power spectrum. In this paper, we present a new Bayesian inference of the reionization parameters where the likelihood is implicitly defined through forward simulations using density estimation likelihood-free inference (DELFI). Realistic effects including thermal noise and foreground avoidance are also applied to the mock observations from the HERA and SKA. We demonstrate that this method recovers accurate posterior distributions for the reionization parameters, and outperforms the standard MCMC analysis in terms of the location and size of credible parameter regions. With the minutes-level processing time once the network is trained, this technique is a promising approach for the scientific interpretation of future 21 cm power spectrum observation data. Our code 21cmDELFI-PS is publicly available at this link., Comment: 15 pages, 8 figures, 4 tables. Accepted for publication in ApJ. Comments welcome

Knowledge-grounded dialogue systems are challenging to build due to the lack of training data and heterogeneous knowledge sources. Existing systems perform poorly on unseen topics due to limited topics covered in the training data. In addition, heterogeneous knowledge sources make it challenging for systems to generalize to other tasks because knowledge sources in different knowledge representations require different knowledge encoders. To address these challenges, we present PLUG, a language model that homogenizes different knowledge sources to a unified knowledge representation for knowledge-grounded dialogue generation tasks. PLUG is pre-trained on a dialogue generation task conditioned on a unified essential knowledge representation. It can generalize to different downstream knowledge-grounded dialogue generation tasks with a few training examples. The empirical evaluation on two benchmarks shows that our model generalizes well across different knowledge-grounded tasks. It can achieve comparable performance with state-of-the-art methods under a fully-supervised setting and significantly outperforms other methods in zero-shot and few-shot settings., Comment: Accepted to NAACL 2022

The impact of cosmic reionization on the Ly$\alpha$ forest power spectrum has recently been shown to be significant even at low redshifts ($z \sim 2$). This memory of reionization survives cosmological time scales because high-entropy mean-density gas is heated to $\sim 3\times10^4$ K by reionization, which is inhomogeneous, and subsequent shocks from denser regions. In the near future, the first measurements of the Ly$\alpha$ forest 3D power spectrum will be very likely achieved by upcoming observational efforts such as the Dark Energy Spectroscopic Instrument (DESI). In addition to abundant cosmological information, these observations have the potential to extract the astrophysics of reionization from the Ly$\alpha$ forest. We forecast, for the first time, the accuracy with which the measurements of Ly$\alpha$ forest 3D power spectrum can place constraints on the reionization parameters with DESI. Specifically, we demonstrate that the constraints on the ionization efficiency, $\zeta$, and the threshold mass for haloes that host ionizing sources, $m_{\rm turn}$, will have the $1\sigma$ error at the level of $\zeta = 25.0 \pm 11.6$ and $\log_{10} (m_{\rm turn}/{\rm M}_\odot) = 8.7^{+0.36}_{-0.70}$, respectively. The Ly$\alpha$ forest 3D power spectrum will thus provide an independent probe of reionization, probably even earlier in detection with DESI, with a sensitivity only slightly worse than the upcoming 21 cm power spectrum measurement with the Hydrogen Epoch of Reionization Array (HERA), i.e.\ $\sigma_{\rm DESI} / \sigma_{\rm HERA} \approx 1.5$ for $\zeta$ and $\sigma_{\rm DESI}/\sigma_{\rm HERA} \approx 2.0$ for $\log_{10}(m_{\rm turn} / $M$_\odot)$. Nevertheless, the Ly$\alpha$ forest constraint will be improved about three times tighter than the current constraint from reionization observations with high-z galaxy priors., Comment: Updated to match accepted version for publication. 19 pages, 12 figures

Tomographic three-dimensional 21 cm images from the epoch of reionization contain a wealth of information about the reionization of the intergalactic medium by astrophysical sources. Conventional power spectrum analysis cannot exploit the full information in the 21 cm data because the 21 cm signal is highly non-Gaussian due to reionization patchiness. We perform a Bayesian inference of the reionization parameters where the likelihood is implicitly defined through forward simulations using density estimation likelihood-free inference (DELFI). We adopt a trained 3D Convolutional Neural Network (CNN) to compress the 3D image data into informative summaries (DELFI-3D CNN). We show that this method recovers accurate posterior distributions for the reionization parameters. Our approach outperforms earlier analysis based on two-dimensional 21 cm images. In contrast, an MCMC analysis of the 3D lightcone-based 21 cm power spectrum alone and using a standard explicit likelihood approximation results in less accurate credible parameter regions than inferred by the DELFI-3D CNN, both in terms of the location and shape of the contours. Our proof-of-concept study implies that the DELFI-3D CNN can effectively exploit more information in the 3D 21 cm images than a 2D CNN or power spectrum analysis. This technique can be readily extended to include realistic effects and is therefore a promising approach for the scientific interpretation of future 21 cm observation data., Comment: 23 pages, 19 figures, 4 tables. Accepted for publication in ApJ. Comments welcome

Large pretrained generative models like GPT-3 often suffer from hallucinating non-existent or incorrect content, which undermines their potential merits in real applications. Existing work usually attempts to detect these hallucinations based on a corresponding oracle reference at a sentence or document level. However ground-truth references may not be readily available for many free-form text generation applications, and sentence- or document-level detection may fail to provide the fine-grained signals that would prevent fallacious content in real time. As a first step to addressing these issues, we propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes (HAllucination DEtection dataSet). To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations. To mitigate label imbalance during annotation, we utilize an iterative model-in-loop strategy. We conduct comprehensive data analyses and create multiple baseline models., Comment: Accepted by ACL2022 main conference

The paper estimates the need for additional attorney hours annually to address the legal needs of indigent clients throughout the United States in matters that comprise the so-called social determinants of health (SDoH). The result will inform stakeholders such as policy makers and private donors so they can allocate resources appropriately and design programs to close the do-called justice gap. As a pilot study, the scope of the project covers only a few major justice problems related to the social determinants of health (standard housing, evictions and foreclosures, guardianships for the incapacitated, and victims of domestic violence) because they significantly impact health outcomes and there are data available. Based on our calculations, we estimate that the total number of attorney hours to address only these five legal issues is over 34 million per year. To put that in perspective, the American Bar Association estimated that in 2018, the year from which much of our data comes, the total number of practicing attorneys in the United States was 1,338,678 (Weiss). Thus, to provide the needed hours, every single practicing attorney in the United States would need to contribute about 26 hours a year. While many lawyers do in fact contribute pro bono hours, they address the full range of legal needs that go well beyond just the five we studied., Comment: 23 pages

Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. But this comes with a significant computational cost, as the attention mechanism's complexity scales quadratically with sequence length. Efficient transformer variants have received increasing interest in recent works. Among them, a linear-complexity recurrent variant has proven well suited for autoregressive generation. It approximates the softmax attention with randomized or heuristic feature maps, but can be difficult to train and may yield suboptimal accuracy. This work aims to convert a pretrained transformer into its efficient recurrent counterpart, improving efficiency while maintaining accuracy. Specifically, we propose a swap-then-finetune procedure: in an off-the-shelf pretrained transformer, we replace the softmax attention with its linear-complexity recurrent alternative and then finetune. With a learned feature map, our approach provides an improved tradeoff between efficiency and accuracy over the standard transformer and other recurrent variants. We also show that the finetuning process has lower training cost relative to training these recurrent variants from scratch. As many models for natural language tasks are increasingly dependent on large-scale pretrained transformers, this work presents a viable approach to improving inference efficiency without repeating the expensive pretraining process., Comment: EMNLP 2021

Heating of neutral gas by energetic sources is crucial for the prediction of the 21 cm signal during the epoch of reionization (EoR). To investigate differences induced on statistics of the 21 cm signal by various source types, we use five radiative transfer simulations which have the same stellar UV emission model and varying combinations of more energetic sources, such as X-ray binaries (XRBs), accreting nuclear black holes (BHs) and hot interstellar medium emission (ISM). We find that the efficient heating from the ISM increases the average global 21~cm signal, while reducing its fluctuations and thus power spectrum. A clear impact is also observed in the bispectrum in terms of scale and timing of the transition between a positive and a negative value. The impact of XRBs is similar to that of the ISM, although it is delayed in time and reduced in intensity because of the less efficient heating. Due to the paucity of nuclear BHs, the behaviour of the 21~cm statistics in their presence is very similar to that of a case when only stars are considered, with the exception of the latest stages of reionization, when the effect of BHs is clearly visible. We find that differences between the source scenarios investigated here are larger than the instrumental noise of SKA1-low at $z \gtrsim 7-8$, suggesting that in the future it might be possible to constrain the spectral energy distribution of the sources contributing to the reionization process., Comment: 22pages,10 figures, apj accepted

The 21 cm linear polarization due to Thomson scattering off free electrons can probe the distribution of neutral hydrogen in the intergalactic medium during the epoch of reionization, complementary to the 21 cm temperature fluctuations. Previous study (Babich & Loeb 2005) estimated the strength of polarization with a toy model and claimed that it can be detected with 1-month observation of the Square Kilometre Array (SKA). Here we revisit this investigation with account of nonlinear terms due to inhomogeneous reionization, using seminumerical reionization simulations to provide the realistic estimation of the 21 cm TE and EE angular power spectra ($C^{\rm TE}_\ell$ and $C^{\rm EE}_\ell$). We find that (1) both power spectra are enhanced on sub-bubble scales but suppressed on super-bubble scales, compared with previous results; (2) $C^{\rm TE}_\ell$ displays a zero-crossing at $\ell<100$, and its angular scale is sensitive to the scale-dependence of H I bias on large scales; (3) the ratios of the power spectrum to its maximum value during reionization at a given $\ell$, i.e. $C^{\rm TE}_\ell / C^{\rm TE}_{\ell,{\rm max}} $ and $C^{\rm EE}_{\ell}/C^{\rm EE}_{\ell,{\rm max}}$, show robust correlations with the global ionized fraction. However, measurement of this signal will be very challenging not only because the overall strength is weaker than the sensitivity of SKA, but also because of the polarized foregrounds from diffuse synchrotron emission, and Faraday rotation which modifies the observed polarization. Nevertheless, the 21 cm linear polarization signal may still likely be detectable through other approaches, e.g. its cross-correlation with other probes., Comment: 21 pages, 11 figures, Published in ApJ

Small-scale density fluctuations can significantly affect reionization but are typically modelled quite crudely. Unresolved fluctuations in numerical simulations and analytical calculations are included using a gas clumping factor, typically assumed to be independent of the local environment. In Paper I, we presented an improved, local density-dependent model for the sub-grid gas clumping. Here we extend this using an empirical stochastic model based on the results from high-resolution numerical simulations which fully resolve all relevant fluctuations. Our model reproduces well both the mean density-clumping relation and its scatter. We applied our stochastic model, along with the mean clumping one and the Paper I deterministic model, to create a large-volume realisation of the clumping field, and used these in radiative transfer simulations of cosmic reionization. Our results show that the simplistic mean clumping model delays reionization compared to local density-dependent models, despite producing fewer recombinations overall. This is due to the very different spatial distribution of clumping, resulting in much higher photoionization rates in the latter cases. The mean clumping model produces smaller HII regions throughout most of the reionization, but those percolate faster at late times. It also causes a significant delay in the 21-cm fluctuations peak and yields lower non-Gaussianity and many fewer bright pixels in the PDF distribution. The stochastic density-dependent model shows relatively minor differences from the deterministic one, mostly concentrated around overlap, where it significantly suppresses the 21-cm fluctuations, and at the bright tail of the 21-cm PDFs, where it produces noticeably more bright pixels., Comment: 18 pages, 17 figures Minor change: enlarged the inset table in Fig. 1

Intensity mapping of the H I 21 cm line and the CO 2.61 mm line from the epoch of reionization has emerged as powerful, complementary, probes of the high-redshift Universe. However, both maps and their cross-correlation are dominated by foregrounds. We propose a new analysis by which the signal is unbiased by foregrounds, i.e. it can be measured without foreground mitigation. We construct the antisymmetric part of two-point cross-correlation between intensity maps of the H I 21 cm line and the CO 2.61 mm line, arising because the statistical fluctuations of two fields have different evolution in time. We show that the sign of this new signal can distinguish model-independently whether inside-out reionization happens during some interval of time. More importantly, within the framework of the excursion set model of reionization, we demonstrate that the slope of the dipole of H I-CO cross-power spectrum at large scales is linear to the rate of change of global neutral fraction of hydrogen in a manner independent of reionization parameters, until the slope levels out near the end of reionization, but this trend might possibly depend on the framework of reionization modelling. The H I-CO dipole may be a smoking-gun probe for the speed of reionization, or "standard speedometer" for cosmic reionization. Observations of this new signal will unveil the global reionization history from the midpoint to near the completion of reionization., Comment: Updated to match the accepted version for publication at the ApJ. 12 pages, 12 figures, 1 table. arXiv admin note: substantial text overlap with arXiv:2009.02765

Intensity mapping of the HI 21 cm line and the CO 2.61 mm line from the epoch of reionization has emerged as powerful, complementary, probes of the high-redshift Universe. However, both maps and their cross-correlation are dominated by foregrounds. We propose a new analysis by which the signal is unbiased by foregrounds, i.e. it can be measured without foreground mitigation. We construct the antisymmetric part of the HI-CO cross-correlation, arising because the statistical fluctuations of two fields have different evolution in time. We show that the sign of this new signal can distinguish model-independently whether inside-out reionization happens during some interval of time., Comment: arXiv:2009.02765 (now withdrawn) and arXiv:2009.02766 are now merged into a long paper that has now been accepted for publication at the ApJ. This published version is updated in arXiv:2009.02766

In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.

Conventional wisdom was that thermal relics from the epoch of reionization (EOR) would vanish swiftly. Recently, however, it was shown that these relics can survive to lower redshifts ($z \sim 2$) than previously thought, due to gas at mean density being heated to $T \sim 3 \times 10^4$ K by reionization, which is inhomogeneous, and shocks. Given the high sensitivities of upcoming Ly$\alpha$ forest surveys, this effect will be a novel broadband systematic for cosmological application. From the astrophysical point of view, however, the imprint of inhomogeneous reionization can shed light on the EOR and cosmic dawn. We utilize a hybrid method -- which includes two different simulation codes capable of handling the huge dynamical range -- to show the impact of patchy reionization on the Ly$\alpha$ forest and its dependence on different astrophysical scenarios. We found statistically significant deviations in the 1D Ly$\alpha$ power spectrum at $k = 0.14$ cMpc$^{-1}$ that range from $\sim 1\%$ at $z = 2$ up to almost $\sim 20\%$ at $z = 4$. The deviations in the 3D Ly$\alpha$ power spectrum, at the same wavenumber, are large and range from a few per cent at $z = 2$ up to $\sim 50\%$ at $z = 4$, although these deviations ignore the effect of He II reionization and AGN feedback at $z<4$. By exploiting different $k$-dependence of power spectrum among various astrophysical scenarios, the effect of patchy reionization on the Ly$\alpha$ forest power spectrum can open a new window into cosmic reionization and possibly cosmic dawn., Comment: Updated to match accepted version for publication. 13 pages, 10 figures

The bubble size distribution of ionized hydrogen regions probes the information about the morphology of \HII\ bubbles during the reionization. Conventionally, the \HII\ bubble size distribution can be derived from the tomographic imaging data of the redshifted 21~cm signal from the epoch of reionization, which, however, is observationally challenging even for the upcoming large radio interferometer arrays. Given that these interferometers promise to measure the 21~cm power spectrum accurately, we propose a new method, which is based on the artificial neural networks (ANN), to reconstruct the \HII\ bubble size distribution from the 21~cm power spectrum. We demonstrate that the reconstruction from the 21~cm power spectrum can be almost as accurate as directly measured from the imaging data with the fractional error $\lesssim 10\%$, even with thermal noise at the sensitivity level of the Square Kilometre Array. Nevertheless, the reconstruction implicitly exploits the modelling in reionization simulations, and hence the recovered \HII\ bubble size distribution is not an independent summary statistic from the power spectrum, and should be used only as the indicator for understanding \HII\ bubble morphology and its evolution., Comment: 17 pages, 14 figures. Published in RAA

Self-attention mechanisms have achieved great success on a variety of NLP tasks due to its flexibility of capturing dependency between arbitrary positions in a sequence. For problems such as query-based summarization (Qsumm) and knowledge graph reasoning where each input sequence is associated with an extra query, explicitly modeling such conditional contextual dependencies can lead to a more accurate solution, which however cannot be captured by existing self-attention mechanisms. In this paper, we propose \textit{conditional self-attention} (CSA), a neural network module designed for conditional dependency modeling. CSA works by adjusting the pairwise attention between input tokens in a self-attention module with the matching score of the inputs to the given query. Thereby, the contextual dependencies modeled by CSA will be highly relevant to the query. We further studied variants of CSA defined by different types of attention. Experiments on Debatepedia and HotpotQA benchmark datasets show CSA consistently outperforms vanilla Transformer and previous models for the Qsumm problem.