Search

Recently, various studies have been directed towards exploring dense passage retrieval techniques employing pre-trained language models, among which the masked auto-encoder (MAE) pre-training architecture has emerged as the most promising. The conventional MAE framework relies on leveraging the passage reconstruction of decoder to bolster the text representation ability of encoder, thereby enhancing the performance of resulting dense retrieval systems. Within the context of building the representation ability of the encoder through passage reconstruction of decoder, it is reasonable to postulate that a ``more demanding'' decoder will necessitate a corresponding increase in the encoder's ability. To this end, we propose a novel token importance aware masking strategy based on pointwise mutual information to intensify the challenge of the decoder. Importantly, our approach can be implemented in an unsupervised manner, without adding additional expenses to the pre-training phase. Our experiments verify that the proposed method is both effective and robust on large-scale supervised passage retrieval datasets and out-of-domain zero-shot retrieval benchmarks., Comment: Work in progress

Deep pre-trained language models (e,g. BERT) are effective at large-scale text retrieval task. Existing text retrieval systems with state-of-the-art performance usually adopt a retrieve-then-reranking architecture due to the high computational cost of pre-trained language models and the large corpus size. Under such a multi-stage architecture, previous studies mainly focused on optimizing single stage of the framework thus improving the overall retrieval performance. However, how to directly couple multi-stage features for optimization has not been well studied. In this paper, we design Hybrid List Aware Transformer Reranking (HLATR) as a subsequent reranking module to incorporate both retrieval and reranking stage features. HLATR is lightweight and can be easily parallelized with existing text retrieval systems so that the reranking process can be performed in a single yet efficient processing. Empirical experiments on two large-scale text retrieval datasets show that HLATR can efficiently improve the ranking performance of existing multi-stage text retrieval methods., Comment: Work in progress. HLAR part of the "AliceMind SLM + HLAR" method in MS MARCO Passage Ranking Submission

Additional file 1: Table S1. Demographic characteristics of HCWs included and excluded in this study. Table S2. Distribution of cytokines in three physical examinations. Table S3. Distribution of cytokines before discharge and in three physical examinations. Table S4. Levels of cytokines before discharge and in three physical examinations. Table S5. Distribution of lymphocyte subsets in three physical examinations. Table S6. Distribution of lymphocyte subsets before discharge and in three physical examinations. Table S7. Relative numbers of lymphocyte subsets before discharge and in three physical examinations. Table S8. Pearson correlations between indicators of cytokines and lymphocyte subsets (N=508). Figure S1. Number of HCWs with COVID-19 discharged from hospitals across time. Figure S2. Antibody titres against SARS-CoV-2 among 136 HCWs who had titres of antibodies both in first and third physical examinations. Figure S3. Levels of cytokines categorized by the time of physical examination in 114 HCWs who competed all three physical examinations. Figure S4. Relative numbers of lymphocyte subsets categorized by the time of physical examination in 114 HCWs who competed all three physical examinations

Additional file 2: Table S2. Primers used for PCR amplification of MYB genes.

Additional file 1: Table S1. Primers used for qRT-PCR analysis of tree peony and transgenic tobacco samples.

Additional file 5: Table S1. Statistics of assembly quality. Table S2. KEGG enrichment of the DEGs between cyanic and acyanic flowers. Table S3. Primers used for qRT-PCR.

Additional file 4: Table S4. Unigene annotation in public databases.