Search

Information retrieval across different languages is an increasingly important challenge in natural language processing. Recent approaches based on multilingual pre-trained language models have achieved remarkable success, yet they often optimize for either monolingual, cross-lingual, or multilingual retrieval performance at the expense of others. This paper proposes a novel hybrid batch training strategy to simultaneously improve zero-shot retrieval performance across monolingual, cross-lingual, and multilingual settings while mitigating language bias. The approach fine-tunes multilingual language models using a mix of monolingual and cross-lingual question-answer pair batches sampled based on dataset size. Experiments on XQuAD-R, MLQA-R, and MIRACL benchmark datasets show that the proposed method consistently achieves comparable or superior results in zero-shot retrieval across various languages and retrieval tasks compared to monolingual-only or cross-lingual-only training. Hybrid batch training also substantially reduces language bias in multilingual retrieval compared to monolingual training. These results demonstrate the effectiveness of the proposed approach for learning language-agnostic representations that enable strong zero-shot retrieval performance across diverse languages., Comment: 15 pages, 2 figures, 13 tables

In the real world, documents are organized in different formats and varied modalities. Traditional retrieval pipelines require tailored document parsing techniques and content extraction modules to prepare input for indexing. This process is tedious, prone to errors, and has information loss. To this end, we propose Document Screenshot Embedding} (DSE), a novel retrieval paradigm that regards document screenshots as a unified input format, which does not require any content extraction preprocess and preserves all the information in a document (e.g., text, image and layout). DSE leverages a large vision-language model to directly encode document screenshots into dense representations for retrieval. To evaluate our method, we first craft the dataset of Wiki-SS, a 1.3M Wikipedia web page screenshots as the corpus to answer the questions from the Natural Questions dataset. In such a text-intensive document retrieval setting, DSE shows competitive effectiveness compared to other text retrieval methods relying on parsing. For example, DSE outperforms BM25 by 17 points in top-1 retrieval accuracy. Additionally, in a mixed-modality task of slide retrieval, DSE significantly outperforms OCR text retrieval methods by over 15 points in nDCG@10. These experiments show that DSE is an effective document retrieval paradigm for diverse types of documents. Model checkpoints, code, and Wiki-SS collection will be released.

Alignment is a standard procedure to fine-tune pre-trained large language models (LLMs) to follow natural language instructions and serve as helpful AI assistants. We have observed, however, that the conventional alignment process fails to enhance the factual accuracy of LLMs, and often leads to the generation of more false facts (i.e. hallucination). In this paper, we study how to make the LLM alignment process more factual, by first identifying factors that lead to hallucination in both alignment steps:\ supervised fine-tuning (SFT) and reinforcement learning (RL). In particular, we find that training the LLM on new knowledge or unfamiliar texts can encourage hallucination. This makes SFT less factual as it trains on human labeled data that may be novel to the LLM. Furthermore, reward functions used in standard RL can also encourage hallucination, because it guides the LLM to provide more helpful responses on a diverse set of instructions, often preferring longer and more detailed responses. Based on these observations, we propose factuality-aware alignment, comprised of factuality-aware SFT and factuality-aware RL through direct preference optimization. Experiments show that our proposed factuality-aware alignment guides LLMs to output more factual responses while maintaining instruction-following capability.

Synthesized in the cores of stars and supernovae, most metals disperse over cosmic scales and are ultimately deposited well outside the gravitational potential of their host galaxies. Since their presence is well visible through their X-ray emission lines in the hot gas pervading galaxy clusters, measuring metal abundances in the intracluster medium (ICM) offers us a unique view of chemical enrichment of the Universe as a whole. Despite extraordinary progress in the field thanks to four decades of X-ray spectroscopy using CCD (and gratings) instruments, understanding the precise stellar origins of the bulk of metals, and when the latter were mixed on Mpc scales, requires an X-ray mission capable of spatial, non-dispersive high resolution spectroscopy covering at least the soft X-ray band over a large field of view. In this White Paper, we demonstrate how the Line Emission Mapper (LEM) probe mission concept will revolutionize our current picture of the ICM enrichment. Specifically, we show that LEM will be able to (i) spatially map the distribution of ten key chemical elements out to the virial radius of a nearby relaxed cluster and (ii) measure metal abundances in serendipitously discovered high-redshift protoclusters. Altogether, these key observables will allow us to constrain the chemical history of the largest gravitationally bound structures of the Universe. They will also solve key questions such as the universality of the initial mass function (IMF) and the initial metallicity of the stellar populations producing these metals, as well as the relative contribution of asymptotic giant branch (AGB) stars, core-collapse, and Type Ia supernovae to enrich the cosmic web over Mpc scales. Concrete observing strategies are also briefly discussed., Comment: 19 pages. White paper for a mission concept to be submitted for the 2023 NASA Astrophysics Probes opportunity

High-resolution spectroscopy in soft X-rays will open a new window to map multiphase gas in galaxy clusters and probe physics of the intracluster medium (ICM), including chemical enrichment histories, circulation of matter and energy during large-scale structure evolution, stellar and black hole feedback, halo virialization, and gas mixing processes. An eV-level spectral resolution, large field-of-view, and effective area are essential to separate cluster emissions from the Galactic foreground and efficiently map the cluster outskirts. Several mission concepts that meet these criteria have been proposed recently, e.g., LEM, HUBS, and SuperDIOS. This theoretical study explores what information on ICM physics could be recovered with such missions and the associated challenges. We emphasize the need for a comprehensive comparison between simulations and observations to interpret the high-resolution spectroscopic observations correctly. Using Line Emission Mapper (LEM) characteristics as an example, we demonstrate that it enables the use of soft X-ray emission lines (e.g., O VII/VIII and Fe-L complex) from the cluster outskirts to measure the thermodynamic, chemical, and kinematic properties of the gas up to $r_{200}$ and beyond. By generating mock observations with full backgrounds, analysing their images/spectra with observational approaches, and comparing the recovered characteristics with true ones from simulations, we develop six key science drivers for future missions, including the exploration of multiphase gas in galaxy clusters (e.g., temperature fluctuations, phase-space distributions), metallicity, ICM gas bulk motions and turbulence power spectra, ICM-cosmic filament interactions, and advances for cluster cosmology., Comment: 24 pages, 26 figures, submitted to MNRAS. Comments are welcome

This paper presents ConvRerank, a conversational passage re-ranker that employs a newly developed pseudo-labeling approach. Our proposed view-ensemble method enhances the quality of pseudo-labeled data, thus improving the fine-tuning of ConvRerank. Our experimental evaluation on benchmark datasets shows that combining ConvRerank with a conversational dense retriever in a cascaded manner achieves a good balance between effectiveness and efficiency. Compared to baseline methods, our cascaded pipeline demonstrates lower latency and higher top-ranking effectiveness. Furthermore, the in-depth analysis confirms the potential of our approach to improving the effectiveness of conversational search., Comment: SIGIR 2023

Various techniques have been developed in recent years to improve dense retrieval (DR), such as unsupervised contrastive learning and pseudo-query generation. Existing DRs, however, often suffer from effectiveness tradeoffs between supervised and zero-shot retrieval, which some argue was due to the limited model capacity. We contradict this hypothesis and show that a generalizable DR can be trained to achieve high accuracy in both supervised and zero-shot retrieval without increasing model size. In particular, we systematically examine the contrastive learning of DRs, under the framework of Data Augmentation (DA). Our study shows that common DA practices such as query augmentation with generative models and pseudo-relevance label creation using a cross-encoder, are often inefficient and sub-optimal. We hence propose a new DA approach with diverse queries and sources of supervision to progressively train a generalizable DR. As a result, DRAGON, our dense retriever trained with diverse augmentation, is the first BERT-base-sized DR to achieve state-of-the-art effectiveness in both supervised and zero-shot evaluations and even competes with models using more complex late interaction (ColBERTv2 and SPLADE++).

This paper introduces Sparsified Late Interaction for Multi-vector (SLIM) retrieval with inverted indexes. Multi-vector retrieval methods have demonstrated their effectiveness on various retrieval datasets, and among them, ColBERT is the most established method based on the late interaction of contextualized token embeddings of pre-trained language models. However, efficient ColBERT implementations require complex engineering and cannot take advantage of off-the-shelf search libraries, impeding their practical use. To address this issue, SLIM first maps each contextualized token vector to a sparse, high-dimensional lexical space before performing late interaction between these sparse token embeddings. We then introduce an efficient two-stage retrieval architecture that includes inverted index retrieval followed by a score refinement module to approximate the sparsified late interaction, which is fully compatible with off-the-shelf lexical search libraries such as Lucene. SLIM achieves competitive accuracy on MS MARCO Passages and BEIR compared to ColBERT while being much smaller and faster on CPUs. To our knowledge, we are the first to explore using sparse token representations for multi-vector retrieval. Source code and data are integrated into the Pyserini IR toolkit.

Multi-vector retrieval methods combine the merits of sparse (e.g. BM25) and dense (e.g. DPR) retrievers and have achieved state-of-the-art performance on various retrieval tasks. These methods, however, are orders of magnitude slower and need much more space to store their indices compared to their single-vector counterparts. In this paper, we unify different multi-vector retrieval models from a token routing viewpoint and propose conditional token interaction via dynamic lexical routing, namely CITADEL, for efficient and effective multi-vector retrieval. CITADEL learns to route different token vectors to the predicted lexical ``keys'' such that a query token vector only interacts with document token vectors routed to the same key. This design significantly reduces the computation cost while maintaining high accuracy. Notably, CITADEL achieves the same or slightly better performance than the previous state of the art, ColBERT-v2, on both in-domain (MS MARCO) and out-of-domain (BEIR) evaluations, while being nearly 40 times faster. Code and data are available at https://github.com/facebookresearch/dpr-scale.

The Line Emission Mapper (LEM) is an X-ray Probe for the 2030s that will answer the outstanding questions of the Universe's structure formation. It will also provide transformative new observing capabilities for every area of astrophysics, and to heliophysics and planetary physics as well. LEM's main goal is a comprehensive look at the physics of galaxy formation, including stellar and black-hole feedback and flows of baryonic matter into and out of galaxies. These processes are best studied in X-rays, and emission-line mapping is the pressing need in this area. LEM will use a large microcalorimeter array/IFU, covering a 30x30' field with 10" angular resolution, to map the soft X-ray line emission from objects that constitute galactic ecosystems. These include supernova remnants, star-forming regions, superbubbles, galactic outflows (such as the Fermi/eROSITA bubbles in the Milky Way and their analogs in other galaxies), the Circumgalactic Medium in the Milky Way and other galaxies, and the Intergalactic Medium at the outskirts and beyond the confines of galaxies and clusters. LEM's 1-2 eV spectral resolution in the 0.2-2 keV band will make it possible to disentangle the faintest emission lines in those objects from the bright Milky Way foreground, providing groundbreaking measurements of the physics of these plasmas, from temperatures, densities, chemical composition to gas dynamics. While LEM's main focus is on galaxy formation, it will provide transformative capability for all classes of astrophysical objects, from the Earth's magnetosphere, planets and comets to the interstellar medium and X-ray binaries in nearby galaxies, AGN, and cooling gas in galaxy clusters. In addition to pointed observations, LEM will perform a shallow all-sky survey that will dramatically expand the discovery space., Comment: 18 pages. White paper for a mission concept to be submitted for the 2023 NASA Astrophysics Probes opportunity. v2: All-sky survey figure expanded, references fixed. v3: Added energy resolution measurements for prototype detector array. v4: Author list and reference fixes

Quantifying the parameters and corresponding uncertainties of hundreds of strongly lensed quasar systems holds the key to resolving one of the most important scientific questions: the Hubble constant ($H_{0}$) tension. The commonly used Markov chain Monte Carlo (MCMC) method has been too time-consuming to achieve this goal, yet recent work has shown that convolution neural networks (CNNs) can be an alternative with seven orders of magnitude improvement in speed. With 31,200 simulated strongly lensed quasar images, we explore the usage of Vision Transformer (ViT) for simulated strong gravitational lensing for the first time. We show that ViT could reach competitive results compared with CNNs, and is specifically good at some lensing parameters, including the most important mass-related parameters such as the center of lens $\theta_{1}$ and $\theta_{2}$, the ellipticities $e_1$ and $e_2$, and the radial power-law slope $\gamma'$. With this promising preliminary result, we believe the ViT (or attention-based) network architecture can be an important tool for strong lensing science for the next generation of surveys. The open source of our code and data is in \url{https://github.com/kuanweih/strong_lensing_vit_resnet}., Comment: Accepted by ECCV 2022 AI for Space Workshop

Pre-trained language models have been successful in many knowledge-intensive NLP tasks. However, recent work has shown that models such as BERT are not ``structurally ready'' to aggregate textual information into a [CLS] vector for dense passage retrieval (DPR). This ``lack of readiness'' results from the gap between language model pre-training and DPR fine-tuning. Previous solutions call for computationally expensive techniques such as hard negative mining, cross-encoder distillation, and further pre-training to learn a robust DPR model. In this work, we instead propose to fully exploit knowledge in a pre-trained language model for DPR by aggregating the contextualized token embeddings into a dense vector, which we call agg*. By concatenating vectors from the [CLS] token and agg*, our Aggretriever model substantially improves the effectiveness of dense retrieval models on both in-domain and zero-shot evaluations without introducing substantial training overhead. Code is available at https://github.com/castorini/dhr, Comment: Published in Transactions of the Association for Computational Linguistics

Lexical and semantic matching capture different successful approaches to text retrieval and the fusion of their results has proven to be more effective and robust than either alone. Prior work performs hybrid retrieval by conducting lexical and semantic matching using different systems (e.g., Lucene and Faiss, respectively) and then fusing their model outputs. In contrast, our work integrates lexical representations with dense semantic representations by densifying high-dimensional lexical representations into what we call low-dimensional dense lexical representations (DLRs). Our experiments show that DLRs can effectively approximate the original lexical representations, preserving effectiveness while improving query latency. Furthermore, we can combine dense lexical and semantic representations to generate dense hybrid representations (DHRs) that are more flexible and yield faster retrieval compared to existing hybrid techniques. In addition, we explore it jointly training lexical and semantic representations in a single model and empirically show that the resulting DHRs are able to combine the advantages of the individual components. Our best DHR model is competitive with state-of-the-art single-vector and multi-vector dense retrievers in both in-domain and zero-shot evaluation settings. Furthermore, our model is both faster and requires smaller indexes, making our dense representation framework an attractive approach to text retrieval. Our code is available at https://github.com/castorini/dhr., Comment: Published in ACM Transactions on Information Systems

The total masses of galaxy clusters characterize many aspects of astrophysics and the underlying cosmology. It is crucial to obtain reliable and accurate mass estimates for numerous galaxy clusters over a wide range of redshifts and mass scales. We present a transfer-learning approach to estimate cluster masses using the ugriz-band images in the SDSS Data Release 12. The target masses are derived from X-ray or SZ measurements that are only available for a small subset of the clusters. We designed a semi-supervised deep learning model consisting of two convolutional neural networks. In the first network, a feature extractor is trained to classify the SDSS photometric bands. The second network takes the previously trained features as inputs to estimate their total masses. The training and testing processes in this work depend purely on real observational data. Our algorithm reaches a mean absolute error (MAE) of 0.232 dex on average and 0.214 dex for the best fold. The performance is comparable to that given by redMaPPer, 0.192 dex. We have further applied a joint integrated gradient and class activation mapping method to interpret such a two-step neural network. The performance of our algorithm is likely to improve as the size of training dataset increases. This proof-of-concept experiment demonstrates the potential of deep learning in maximizing the scientific return of the current and future large cluster surveys., Comment: Accepted for publication in MNRAS; 10 pages, 7 figures

Learned sparse and dense representations capture different successful approaches to text retrieval and the fusion of their results has proven to be more effective and robust. Prior work combines dense and sparse retrievers by fusing their model scores. As an alternative, this paper presents a simple approach to densifying sparse representations for text retrieval that does not involve any training. Our densified sparse representations (DSRs) are interpretable and can be easily combined with dense representations for end-to-end retrieval. We demonstrate that our approach can jointly learn sparse and dense representations within a single model and then combine them for dense retrieval. Experimental results suggest that combining our DSRs and dense representations yields a balanced tradeoff between effectiveness and efficiency.

This paper describes a compact and effective model for low-latency passage retrieval in conversational search based on learned dense representations. Prior to our work, the state-of-the-art approach uses a multi-stage pipeline comprising conversational query reformulation and information retrieval modules. Despite its effectiveness, such a pipeline often includes multiple neural models that require long inference times. In addition, independently optimizing each module ignores dependencies among them. To address these shortcomings, we propose to integrate conversational query reformulation directly into a dense retrieval model. To aid in this goal, we create a dataset with pseudo-relevance labels for conversational search to overcome the lack of training data and to explore different training strategies. We demonstrate that our model effectively rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. Finally, after observing that our model learns to adjust the $L_2$ norm of query token embeddings, we leverage this property for hybrid retrieval and to support error analysis., Comment: Published in EMNLP 2021

A vital step towards the widespread adoption of neural retrieval models is their resource efficiency throughout the training, indexing and query workflows. The neural IR community made great advancements in training effective dual-encoder dense retrieval (DR) models recently. A dense text retrieval model uses a single vector representation per query and passage to score a match, which enables low-latency first stage retrieval with a nearest neighbor search. Increasingly common, training approaches require enormous compute power, as they either conduct negative passage sampling out of a continuously updating refreshing index or require very large batch sizes for in-batch negative sampling. Instead of relying on more compute capability, we introduce an efficient topic-aware query and balanced margin sampling technique, called TAS-Balanced. We cluster queries once before training and sample queries out of a cluster per batch. We train our lightweight 6-layer DR model with a novel dual-teacher supervision that combines pairwise and in-batch negative teachers. Our method is trainable on a single consumer-grade GPU in under 48 hours (as opposed to a common configuration of 8x V100s). We show that our TAS-Balanced training method achieves state-of-the-art low-latency (64ms per query) results on two TREC Deep Learning Track query sets. Evaluated on NDCG@10, we outperform BM25 by 44%, a plainly trained DR by 19%, docT5query by 11%, and the previous best DR model by 5%. Additionally, TAS-Balanced produces the first dense retriever that outperforms every other method on recall at any cutoff on TREC-DL and allows more resource intensive re-ranking models to operate on fewer passages to improve results further., Comment: Accepted at SIGIR 2021 (Full Paper track)

We present the clustering analysis of photometric luminous red galaxies (LRGs) at a redshift range of $0.1\leq z \leq 1.05$ using $615,317$ photometric LRGs selected from the Hyper Suprime-Cam Subaru Strategic Program covering $\sim124$ deg$^{2}$. Our sample covers a broad range of stellar masses and photometric redshifts and enables a halo occupation distribution analysis to study the redshift and stellar-mass dependence of dark halo properties of LRGs. We find a tight correlation between the characteristic dark halo mass to host central LRGs, $M_{\min}$, and the number density of LRGs independently of redshifts, indicating that the formation of LRGs is associated with the global environment. The $M_{\min}$ of LRGs depends only weakly on the stellar mass $M_{\star}$ at $M_{\star} \lesssim 10^{10.75}h^{-2} M_{\odot}$ at $0.3

Pyserini is an easy-to-use Python toolkit that supports replicable IR research by providing effective first-stage retrieval in a multi-stage ranking architecture. Our toolkit is self-contained as a standard Python package and comes with queries, relevance judgments, pre-built indexes, and evaluation scripts for many commonly used IR test collections. We aim to support, out of the box, the entire research lifecycle of efforts aimed at improving ranking with modern neural approaches. In particular, Pyserini supports sparse retrieval (e.g., BM25 scoring using bag-of-words representations), dense retrieval (e.g., nearest-neighbor search on transformer-encoded representations), as well as hybrid retrieval that integrates both approaches. This paper provides an overview of toolkit features and presents empirical results that illustrate its effectiveness on two popular ranking tasks. We also describe how our group has built a culture of replicability through shared norms and tools that enable rigorous automated testing.

We study the angular correlation function of star-forming galaxies and properties of their host dark matter halos at z>1 using the Hyper-Suprime Cam (HSC) SSP survey. We use [OII] emitters identified using two narrow-band (NB) filters, NB816 and NB921, in the Deep/UltraDeep layers, which respectively cover large angular areas of 16.3 deg^2 and 16.9 deg^2. Our sample contains 8302 and 9578 [OII] emitters at z=1.19 (NB816) and z=1.47 (NB921), respectively. We detect a strong clustering signal over a wide angular range, 0.001 < \theta < 1 [deg], with the bias $b=1.61^{+0.13}_{-0.11}$ (z=1.19) and $b=2.09^{+0.17}_{-0.15}$ (z=1.47). We also find a clear deviation of the correlation from a simple power-law form. To interpret the measured clustering signal, we adopt a halo occupation distribution (HOD) model that is constructed to explain the spatial distribution of galaxies selected by a star formation rate. The observed correlation function and number density are simultaneously explained by the best-fitting HOD model. From the constrained HOD model, the average mass of halos hosting the [OII] emitters is derived to be $\log{M_{eff}/(h^{-1}M_\odot)}=12.70^{+0.09}_{-0.07}$ and $12.61^{+0.09}_{-0.05}$ at z=1.19 and 1.47, respectively, which will become halos with the present-day mass, $M\sim 1.5 \times 10^{13}h^{-1}M_\odot$. The satellite fraction of the [OII] emitter sample is found to be $f_{sat}\sim 0.15$. All these values are consistent with the previous studies of similar samples, but we obtain tighter constraints even in a larger parameter space due to the larger sample size from the HSC. The results obtained for host halos of [OII] emitters in this paper enable the construction of mock galaxy catalogs and the systematic forecast study of cosmological constraints from upcoming emission line galaxy surveys such as the Subaru PFS survey., Comment: 21 pages, 13 figures, 5 tables, accepted to PASJ

Most galaxies in the nearby Universe are gravitationally bound to a cluster or group of galaxies. Their optical contents, such as optical richness, are crucial for understanding the co-evolution of galaxies and large-scale structures in modern astronomy and cosmology. The determination of optical richness can be challenging. We propose a self-supervised approach for estimating optical richness from multi-band optical images. The method uses the data properties of the multi-band optical images for pre-training, which enables learning feature representations from a large but unlabeled dataset. We apply the proposed method to the Sloan Digital Sky Survey. The result shows our estimate of optical richness lowers the mean absolute error and intrinsic scatter by 11.84% and 20.78%, respectively, while reducing the need for labeled training data by up to 60%. We believe the proposed method will benefit astronomy and cosmology, where a large number of unlabeled multi-band images are available, but acquiring image labels is costly., Comment: Accepted to NeurIPS 2020 Workshop on Machine Learning and the Physical Sciences

We present an approach to ranking with dense representations that applies knowledge distillation to improve the recently proposed late-interaction ColBERT model. Specifically, we distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product, thus enabling single-step ANN search. Our key insight is that during distillation, tight coupling between the teacher model and the student model enables more flexible distillation strategies and yields better learned representations. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT, while only making modest sacrifices in terms of effectiveness. By combining our dense representations with sparse representations derived from document expansion, we are able to approach the effectiveness of a standard cross-encoder reranker using BERT that is orders of magnitude slower.

In this paper, we propose a two-stage ranking approach for recommending linear TV programs. The proposed approach first leverages user viewing patterns regarding time and TV channels to identify potential candidates for recommendation and then further leverages user preferences to rank these candidates given textual information about programs. To evaluate the method, we conduct empirical studies on a real-world TV dataset, the results of which demonstrate the superior performance of our model in terms of both recommendation accuracy and time efficiency., Comment: 8 pages

Conversational search plays a vital role in conversational information seeking. As queries in information seeking dialogues are ambiguous for traditional ad-hoc information retrieval (IR) systems due to the coreference and omission resolution problems inherent in natural language dialogue, resolving these ambiguities is crucial. In this paper, we tackle conversational passage retrieval (ConvPR), an important component of conversational search, by addressing query ambiguities with query reformulation integrated into a multi-stage ad-hoc IR system. Specifically, we propose two conversational query reformulation (CQR) methods: (1) term importance estimation and (2) neural query rewriting. For the former, we expand conversational queries using important terms extracted from the conversational context with frequency-based signals. For the latter, we reformulate conversational queries into natural, standalone, human-understandable queries with a pretrained sequence-tosequence model. Detailed analyses of the two CQR methods are provided quantitatively and qualitatively, explaining their advantages, disadvantages, and distinct behaviors. Moreover, to leverage the strengths of both CQR methods, we propose combining their output with reciprocal rank fusion, yielding state-of-the-art retrieval effectiveness, 30% improvement in terms of NDCG@3 compared to the best submission of TREC CAsT 2019., Comment: 28 pages. Accepted to ACM Transactions on Information Systems, Special Issue on Conversational Search and Recommendation. The first two authors contributed equally. Code: https://github.com/castorini/chatty-goose

This paper presents an empirical study of conversational question reformulation (CQR) with sequence-to-sequence architectures and pretrained language models (PLMs). We leverage PLMs to address the strong token-to-token independence assumption made in the common objective, maximum likelihood estimation, for the CQR task. In CQR benchmarks of task-oriented dialogue systems, we evaluate fine-tuned PLMs on the recently-introduced CANARD dataset as an in-domain task and validate the models using data from the TREC 2019 CAsT Track as an out-domain task. Examining a variety of architectures with different numbers of parameters, we demonstrate that the recent text-to-text transfer transformer (T5) achieves the best results both on CANARD and CAsT with fewer parameters, compared to similar transformer architectures.

We applied the T5 sequence-to-sequence model to tackle the AI2 WinoGrande Challenge by decomposing each example into two input text strings, each containing a hypothesis, and using the probabilities assigned to the "entailment" token as a score of the hypothesis. Our first (and only) submission to the official leaderboard yielded 0.7673 AUC on March 13, 2020, which is the best known result at this time and beats the previous state of the art by over five points.

The unprecedented depth and area surveyed by the Subaru Strategic Program with the Hyper Suprime-Cam (HSC-SSP) have enabled us to construct and publish the largest distant cluster sample out to $z\sim 1$ to date. In this exploratory study of cluster galaxy evolution from $z=1$ to $z=0.3$, we investigate the stellar mass assembly history of brightest cluster galaxies (BCGs), evolution of stellar mass and luminosity distributions, stellar mass surface density profile, as well as the population of radio galaxies. Our analysis is the first high redshift application of the top N richest cluster selection, which is shown to allow us to trace the cluster galaxy evolution faithfully. Over the 230 deg$^2$ area of the current HSC-SSP footprint, selecting the top 100 clusters in each of the 4 redshift bins allows us to observe the buildup of galaxy population in descendants of clusters whose $z\approx 1$ mass is about $2\times 10^{14}\,M_\odot$. Our stellar mass is derived from a machine-learning algorithm, which is found to be unbiased and accurate with respect to the COSMOS data. We find very mild stellar mass growth in BCGs (about 35% between $z=1$ and 0.3), and no evidence for evolution in both the total stellar mass-cluster mass correlation and the shape of the stellar mass surface density profile. We also present the first measurement of the radio luminosity distribution in clusters out to $z\sim 1$, and show hints of changes in the dominant accretion mode powering the cluster radio galaxies at $z\sim 0.8$., Comment: ApJ, in press (17 pages, 16 figures)

We present clustering properties from 579,492 Lyman break galaxies (LBGs) at z~4-6 over the 100 deg^2 sky (corresponding to a 1.4 Gpc^3 volume) identified in early data of the Hyper Suprime-Cam (HSC) Subaru strategic program survey. We derive angular correlation functions (ACFs) of the HSC LBGs with unprecedentedly high statistical accuracies at z~4-6, and compare them with the halo occupation distribution (HOD) models. We clearly identify significant ACF excesses in 10"<$\theta$<90", the transition scale between 1- and 2-halo terms, suggestive of the existence of the non-linear halo bias effect. Combining the HOD models and previous clustering measurements of faint LBGs at z~4-7, we investigate dark-matter halo mass (Mh) of the z~4-7 LBGs and its correlation with various physical properties including the star-formation rate (SFR), the stellar-to-halo mass ratio (SHMR), and the dark matter accretion rate (dotMh) over a wide-mass range of Mh/M$_\odot$=4x10^10-4x10^12. We find that the SHMR increases from z~4 to 7 by a factor of ~4 at Mh~1x10^11 M$_\odot$, while the SHMR shows no strong evolution in the similar redshift range at Mh~1x10^12 M$_\odot$. Interestingly, we identify a tight relation of SFR/dotMh-Mh showing no significant evolution beyond 0.15 dex in this wide-mass range over z~4-7. This weak evolution suggests that the SFR/dotMh-Mh relation is a fundamental relation in high-redshift galaxy formation whose star formation activities are regulated by the dark matter mass assembly. Assuming this fundamental relation, we calculate the cosmic SFR densities (SFRDs) over z=0-10 (a.k.a. Madau-Lilly plot). The cosmic SFRD evolution based on the fundamental relation agrees with the one obtained by observations, suggesting that the cosmic SFRD increase from z~10 to 4-2 (decrease from z~4-2 to 0) is mainly driven by the increase of the halo abundance (the decrease of the accretion rate)., Comment: 34 pages, 27 figures, PASJ in press

We present an optically-selected cluster catalog from the Hyper Suprime-Cam (HSC) Subaru Strategic Program. The HSC images are sufficiently deep to detect cluster member galaxies down to $M_*\sim 10^{10.2}M_\odot$ even at $z\sim 1$, allowing a reliable cluster detection at such high redshifts. We apply the CAMIRA algorithm to the HSC Wide S16A dataset covering $\sim 232$ deg$^2$ to construct a catalog of 1921 clusters at redshift $0.115$ that roughly corresponds to $M_{\rm 200m}\gtrsim 10^{14}h^{-1}M_\odot$. We confirm good cluster photometric redshift performance, with the bias and scatter in $\Delta z/(1+z)$ being better than 0.005 and 0.01 over most of the redshift range, respectively. We compare our cluster catalog with large X-ray cluster catalogs from XXL and XMM-LSS surveys and find good correlation between richness and X-ray properties. We also study the miscentering effect from the distribution of offsets between optical and X-ray cluster centers. We confirm the high ($>0.9$) completeness and purity for high mass clusters by analyzing mock galaxy catalogs., Comment: 18 pages, 18 figures, accepted for publication in PASJ; cluster catalogs are available at http://www.slac.stanford.edu/~oguri/cluster/