Search

Corpus retrieval is an important research direction in the field of corpora, which can help learners or applications search for relevant content and improve learning and application efficiency. Traditional retrieval methods have problems such as high development costs, complex structural design, and low data coverage. This article conducts research based on Lucene technology, leveraging its advantages such as inverted index structure, incremental indexing, object-oriented design, text analysis interfaces, and query engine services, to provide a complete solution for developing high-performance large-scale corpus full-text retrieval systems. Firstly, this article studies the foundation of Lucene technology, including system architecture, functional packages, and data flow analysis. Next, study probability retrieval models and directly model the relevance of user needs. Then, this article conducts system design and implementation, including system function design, inverted index design, and retrieval function implementation. Finally, this article conducts system testing and challenges, including building a testing environment, functional testing, and reliability testing. [ABSTRACT FROM AUTHOR]

This work introduces a companion reproducible paper with the aim of allowing the exact replication of the methods, experiments, and results discussed in a previous work Claude et al., (2016). In that parent paper, we proposed many and varied techniques for compressing indexes which exploit that highly repetitive collections are formed mostly of documents that are near-copies of others. More concretely, we describe a replication framework, called uiHRDC (universal indexes for Highly Repetitive Document Collections) , that allows our original experimental setup to be easily replicated using various document collections. The corresponding experimentation is carefully explained, providing precise details about the parameters that can be tuned for each indexing solution. Finally, note that we also provide uiHRDC as reproducibility package. • We summarize the original results and motivate the proposed experimental setup. • We explain the replication framework, including datasets, query patterns, source code and scripts. • We detail all configuration parameters for each solution, explaining the better configurations. • We host the framework at GitHub, and publish it through Mendeley Data. [ABSTRACT FROM AUTHOR]

Abstract While Information Retrieval (IR) systems have gained success in Web-style search engines in the past two decades, nonetheless, the DataBase (DB) paradigm remains prevalent in handling data in enterprise environments and digital libraries, and is gaining even more importance in the Semantic Web with the increasing need to handle partly structured (NoSQL) data. This paper describes SemIndex+ , a semantic-aware indexing and querying framework that allows semantic search, result selection, and result ranking of structured (relational DB-style), unstructured (IR-style), and partly structured (NoSQL) data. Various weighting functions and a parallelized search algorithm have been developed for that purpose and are presented here. We provide a general keyword query model allowing the user to choose the results' semantic coverage and expressiveness based on her needs. Different from alternative solutions involving query relaxation, query refinement, or query disambiguation, our approach incorporates semantics at the most basic data indexing level: providing more opportunities toward speedups and semantic coverage. An extensive experimental evaluation, comparing SemIndex+ with alternative methods, highlights our approach's flexibility and effectiveness, which in turn impact efficiency (requiring less or more time following the user specified index and query semantic coverages). Highlights • Search, select and rank unstructured, structured (relational) and partly structured (NoSQL) data. • Maps a textual data collection and a semantic knowledge base into a tightly-coupled semantic graph. • Involves users during semantic index creation, initial query formulation and query refinement. • Parallelized query processing, with a dedicated model for answer weighting and relevance scoring. • Comparative experiments with legacy methods highlight solution's flexibility & effectiveness. [ABSTRACT FROM AUTHOR]

Abstract In the past decade, there has been an increasing need for semantic-aware data search and indexing in textual (structured and NoSQL) databases, as full-text search systems became available to non-experts where users have no knowledge about the data being searched and often formulate query keywords which are different from those used by the authors in indexing relevant documents, thus producing noisy and sometimes irrelevant results. In this paper, we address the problem of semantic-aware querying and provide a general framework for modeling and processing semantic-based keyword queries in textual databases, i.e., considering the lexical and semantic similarities/disparities when matching user query and data index terms. To do so, we design and construct a semantic-aware inverted index structure called SemIndex , extending the standard inverted index by constructing a tightly coupled inverted index graph that combines two main resources: a semantic network and a standard inverted index on a collection of textual data. We then provide a general keyword query model with specially tailored query processing algorithms built on top of SemIndex , in order to produce semantic-aware results, allowing the user to choose the results' semantic coverage and expressiveness based on her needs. To investigate the practicality and effectiveness of SemIndex , we discuss its physical design within a standard commercial RDBMS allowing to create, store, and query its graph structure, thus enabling the system to easily scale up and handle large volumes of data. We have conducted a battery of experiments to test the performance of SemIndex , evaluating its construction time, storage size, query processing time, and result quality, in comparison with legacy inverted index. Results highlight both the effectiveness and scalability of our approach. [ABSTRACT FROM AUTHOR]

Highlights • Investigate K-means clustering for large collections of sparse vectors of high dimensionality. • Proposed utilizing the inverted list data-structure to improve run-time of K-means. • Heuristics proposed for initial centroid selection and centroid updates. • Proposed approach outperforms the run-time of K-means by up to 35x on a collection of 14M tweets. Abstract K-means, along with its several other variants, is the most widely used family of partitional clustering algorithms. Generally speaking, this family of algorithm starts by initializing a number of data points as cluster centres, and then iteratively refines these cluster centres based on the current partition of the dataset. Given a set of cluster centres, inducing the partition over the dataset involves finding the nearest (or most similar) cluster centre for each data point, which is an O (NK) operation, N and K being the number of data points and the number of clusters, respectively. In our proposed approach, we avoid the explicit computation of these distances for the case of sparse vectors, e.g. documents, by utilizing a fundamental operation, namely TOP (x), which gives a list of the top most similar vectors with respect to the vector x. A standard way to store sparse vectors and retrieve the top most similar ones given a query vector, is with the help of the inverted list data structure. In our proposed method, we use the TOP (x) function to first select cluster centres that are likely to be dissimilar to each other. Secondly, to obtain the partition during each iteration of K-means, we avoid the explicit computation of the pair-wise similarities between the centroid and the non-centroid vectors. Thirdly, we avoid recomputation of the cluster centroids by adopting a centrality based heuristic. We demonstrate the effectiveness of our proposed algorithm on TREC-2011 Microblog dataset, a large collection of about 14 M tweets. Our experiments demonstrate that our proposed method is about 35x faster and produces more effective clusters in comparison to the standard K-means algorithm. [ABSTRACT FROM AUTHOR]

Many applications need to compute similarity between two records, where each record is a set of elements. Supporting such similarity queries is critical to applications such as document search, data cleaning, error correction, plagiarism detection, and searching on molecule databases. In practice, when we consider billions of records each of which is composed of weighted elements, we can hardly accommodate all of the weighting information for evaluating existing in-memory query methods. It is challenging to efficiently drive filter-and-verify framework for a massive number of records with thorough consideration of token weights, especially in the context of using external storage. We address the practical problem of how to search similar records efficiently using weighted tokens. We accelerate existing methods especially for the IO-intensive environments. We present a novel inverted index , and show how to probe the segmented lengths to skip dissimilar records efficiently. By introducing a certain number of suffix tokens in the record entry on each inverted list, an in-place strategy is proposed to improve the pruning power significantly. We show how many in-place tokens are sufficient to eliminate the impact of verification, in both theoretical and practical aspects. The methods are extended for supporting different metrics. Based on large-scale datasets, the experiments with HDD and SSD show that our in-place structure offers 3X ∼ 9X better performance compared to existing indexes, and the pruning schemes improve its performance by a factor of 5X ∼ 27X. In total, the proposed method is 2X ∼ 20X more efficient than the length-based method at the cost of 2X space overhead. [ABSTRACT FROM AUTHOR]

Search in social network is continuously being expanded to enhance user experience. Besides basic textual retrieval, users can also search based on features such as spatial proximity, temporal freshness and/or social closeness. To efficiently process each advanced query type, customized indexing mechanisms have been developed. However, such mechanisms only perform well for the query types that they were designed for; moreover, they are not readily adaptable to support other query types. In this paper, we propose an interval-at-a-time (IAAT) framework as a first attempt to provide a one-size-fits-all solution to social media retrieval with spatial, temporal and social constraints. In addition, the algorithm relies on inverted index only, which makes it compatible with conventional search engines. The inverted lists are sorted by document id and the insertion is very fast because only append operation is involved. Experiments conducted on two large-scale Twitter datasets show that though IAAT is a unified strategy, it performs better than most of the state-of-the-art customized solutions in a variety of query types. [ABSTRACT FROM AUTHOR]

To realize ciphertext retrieval without decryption in the public key settings, Boneh et al. in 2004 proposed a cryptographic primitive named Public-key Encryption with Keyword Search (PEKS) and proposed the first PEKS scheme. Following Boneh et al.'s work, various PEKS schemes were proposed; however, most schemes are vulnerable to inside keyword guessing attacks (IKGA). Huang and Li proposed a new primitive named Public-key Authenticated Encryption with Keyword Search (PAEKS) in 2017, which resists the IKGA in the setting of only one test server. Their main idea to resist IKGA is to require the data sender to authenticate the ciphertext while encrypting a keyword. However, in the era of big data, as the number of encrypted files increases, huge storage overhead and heavy retrieval costs become problems in PAEKS. In this paper, we proposed a new PAEKS scheme that costs low storage space and supports fast search. We introduced the ciphertext deduplication into PAEKS to reduce storage space and leveraged the inverted index to accelerate the retrieval process. We simulated the proposed scheme and several related schemes with a desktop computer. The experiment results show that our proposed scheme is of lower storage overhead and higher retrieval efficiency compared with related schemes. [ABSTRACT FROM AUTHOR]

Indexing highly repetitive collections has become a relevant problem with the emergence of large repositories of versioned documents, among other applications. These collections may reach huge sizes, but are formed mostly of documents that are near-copies of others. Traditional techniques for indexing these collections fail to properly exploit their regularities in order to reduce space. We introduce new techniques for compressing inverted indexes that exploit this near-copy regularity. They are based on run-length, Lempel–Ziv, or grammar compression of the differential inverted lists, instead of the usual practice of gap-encoding them. We show that, in this highly repetitive setting, our compression methods significantly reduce the space obtained with classical techniques, at the price of moderate slowdowns. Moreover, our best methods are universal, that is, they do not need to know the versioning structure of the collection, nor that a clear versioning structure even exists. We also introduce compressed self-indexes in the comparison. These are designed for general strings (not only natural language texts) and represent the text collection plus the index structure (not an inverted index) in integrated form. We show that these techniques can compress much further, using a small fraction of the space required by our new inverted indexes. Yet, they are orders of magnitude slower. [ABSTRACT FROM AUTHOR]

The goal of an information retrieval system is to provide the information that is relevant to the user's query. In some cases the information relevant to the user request may not exist in the user's native language. Situations may also arise where the user is able to read documents in languages different from the native one, but might have difficulty in formulating queries in them. The main intention behind Multilingual Information Retrieval is to find the relevant information available irrespective of the language used in the query. [ABSTRACT FROM AUTHOR]

Rational miners in the blockchain may admit the validity of search results directly to conserve calculation overhead without executing the validation. Such phenomenon is known as the Verifier's Dilemma that the search results of the blockchain-based searchable encryption scheme would be claimed to have been verified but devoid of valid verification. Besides, the top-k query technique utilized by most existing searchable encryption schemes fails to filter the search results from the user demands during retrieval, which results in massive ineffective access lists with high complexity. Aiming at the two problems, we present a blockchain-based multi-keyword ranked searchable encryption scheme and design a fair payment protocol for multi-user scenarios. Our scheme may achieve public verifiability and user fairness by leveraging the TrueBit network, while the data owner (DO) can withdraw search privileges from the authorized data user (DU). Also, we extend the searchable encryption scheme based on inverted index to multi-DO/multi-DU scenarios, and the HashMap index is used to diminish the response time and space complexity. Moreover, the proposed scheme supports the update of index keywords and files dynamically. To enhance retrieval accuracy, we modify the information divergence algorithm in machine learning and utilize the monotonically weighted aggregation function, which improves retrieval accuracy to around 97%. Finally, Security analysis illustrates correctness and resilience against typical attacks to verify the robustness and utility of the proposed scheme. [ABSTRACT FROM AUTHOR]

The project is aimed at developing new person recognition algorithm, which deals with the problems using matching of filtered histograms of oriented gradients computed in circular sliding windows and using inverted index of histograms for efficient image retrieval. The project results have various scientific, industry and social applications, which require automatic non-cooperative indoors and outdoors person recognition at a distance using multimodal biometrics extracted from multisensory noisy data. For instance, new security and surveillance systems working under in open weather could be developed based on the proposed methods. The performance of the proposed person recognition algorithm in the actual environment is presented and discussed. The results of computer simulation obtained with the proposed algorithm are compared to those of available algorithms in terms of matching accuracy and processing time. [ABSTRACT FROM AUTHOR]

An inverted index is a core data structure of Information Retrieval systems, especially in search engines. Since the search environments have become more dynamic, many on-line index maintenance strategies have been proposed. Previous strategies were designed for HDDs. Consequently, in order to avoid expensive random access cost, Merge-based strategies have been preferred to In-place index update strategies on HDDs. However, flashSSDs have become solid alternatives to HDDs. FlashSSDs currently are adopted in a wide range of areas due to their superior features such as the short access latency, energy efficiency, and high bandwidth. In this article, we first reexamined potentials of In-place index update strategies on flashSSDs. Thanks to the insignificant access latency of flashSSDs, we discovered that In-place index update strategies outperform Merge-based strategies, since In-place index update strategies generate much less amount of I/O than Merge-based strategies despite inducing frequent random accesses. Based on this discovery, we suggest a new inverted index maintenance strategy based on an In-place index update strategy for flashSSDs, called Multipath Flash In-place Strategy (MFIS). To enhance the index maintenance performance, MFIS stores the posting list of each term non-contiguously and exploits the internal parallelism of flashSSDs. Thus, MFIS not only induces the minimum amount of I/O but also utilizes the maximum bandwidth of flashSSDs. Furthermore, MFIS is designed to show high query processing performance by utilizing the internal parallelism of flashSSDs even though the posting list of each term is stored non-contiguously. In our experiments, the index maintenance performance of MFIS was considerably better than other previous maintenance strategies. The index maintenance performance was up to 14.93, 4.04, 5.12, and 2.33 times higher than Merge-based strategies such as Immediate Merge, Geometric Partitioning, Hybrid, and SSD-aware Hybrid, respectively. The query processing performance of MFIS was up to 1.62 times higher than non-contiguous In-place. In addition, MFIS showed almost the best query processing performance as Merge-based strategies did. In conclusion, MFIS is the best on-line inverted index maintenance strategy on flashSSDs in terms of both index maintenance and query processing performance. [ABSTRACT FROM AUTHOR]

• The slope of segment and drop distance of segment head represent a waveform accurately. • New subsequence retrieval method based on the inverted index of slope range shows good efficiency. • The retrieval efficiency of different waveform shapes is found different in EAST. • Waveforms containing spiky or convex shapes are not main features of WMHD signals. The analysis technology of massive experimental data is one of the important concerns in the field of nuclear fusion research, and the waveform similarity retrieval is the basis of realizing data mining of discharge experiment. In this work, we proposed an efficient subsequence retrieval method for matching a selected waveform segment in the discharge signal waveform. This method accurately represents the waveform characteristics based on the slope of segment and the drop distance of segment head (DDSH). In order to improve the retrieval efficiency, an inverted index method based on the slope of segment was adopted to filter the discharge signal waveforms and then perform fine retrieving. The retrieval algorithm was implemented based on Python, and was verified with the plasma density and stored energy signals of historical experimental data from EAST mdsplus database. The test results show the retrieval efficiency is improved by 43.6% on average after using slope inverted index. [ABSTRACT FROM AUTHOR]

In relational database management systems (RDBMSs), an efficient join method for text retrieval using an inverted index has been developed and implemented. However, the existing crossing of the posting inverted list increases the keyword search time for large texts because of unnecessary comparisons. The relation-based search produces results by utilizing the posting list intersection. To reduce the search time for queries, a multi-way skip-merge join algorithm is proposed in this study. The proposed algorithm improves the execution speed by using a sorted inverted index posting list to minimize unnecessary comparison operations in the posting list intersection. The skip-merge join method, which minimizes unnecessary comparison operations using the aggregate function, is integrated with the multi-way join as a replacement for the existing two-way join method. The join algorithm combining skip-merge join and multi-way join shows good performance because the number of search keywords and the number of documents increase. The performance improvement of the keyword search is verified by implementing the multi-way skip-merge join algorithm in PostgreSQL, an RDBMS. [ABSTRACT FROM AUTHOR]

Abstract: During the past few years, the commercial Web search engines have augmented their underlying index structures by significantly enriching the information which describes the appearance of a word within a document Dean (2009) . This enriched information is now used in complex and effective functions which rank documents by taking into consideration hundreds of features, with respect to a user query. Despite the evolution of the search engines, the past research has mainly concentrated on improving plain Web indexes storing typical data only. In this work we study the problem of organizing an inverted index storing additional information. In particular, we examine how the physical locations of a document, called zones, can be efficiently integrated with such an index structure. We introduce TZP, an encoder which compresses these zones in combination to the positions of a word in a document, by employing a fixed number of bits for each portion of a word’s inverted list. We demonstrate that our method allows direct access to the compressed zones and positions without expensive look-ups, avoids decoding any unnecessary information, while its overall index size is analogous or even better when compared against state-of-the art schemes. Moreover, we examine how the word positions can be combined to the zones to improve retrieval effectiveness. We introduce BM25TOPF, a scheme which incorporates term proximity and zone weighting into a single ranking formula. Unlike other term proximity approaches, BM25TOPF also takes into account the ordering of the query terms by rewarding the documents containing them in the correct order. Our experiments with the Web Adhoc Task of TREC 2009 and a set of own queries show that BM25TOPF outperforms the current state-of-the-art approaches by a margin between 6% and 11%. [Copyright &y& Elsevier]

Abstract: More and more (semi) structured information is becoming available on the web in the form of documents embedding metadata (e.g., RDF, RDFa, Microformats and others). There are already hundreds of millions of such documents accessible and their number is growing rapidly. This calls for large scale systems providing effective means of searching and retrieving this semi-structured information with the ultimate goal of making it exploitable by humans and machines alike. This article examines the shift from the traditional web document model to a web data object (entity) model and studies the challenges faced in implementing a scalable and high performance system for searching semi-structured data objects over a large heterogeneous and decentralised infrastructure. Towards this goal, we define an entity retrieval model, develop novel methodologies for supporting this model and show how to achieve a high-performance entity retrieval system. We introduce an indexing methodology for semi-structured data which offers a good compromise between query expressiveness, query processing and index maintenance compared to other approaches. We address high-performance by optimisation of the index data structure using appropriate compression techniques. Finally, we demonstrate that the resulting system can index billions of data objects and provides keyword-based as well as more advanced search interfaces for retrieving relevant data objects in sub-second time. This work has been part of the Sindice search engine project at the Digital Enterprise Research Institute (DERI), NUI Galway. The Sindice system currently maintains more than 200 million pages downloaded from the web and is being used actively by many researchers within and outside of DERI. [Copyright &y& Elsevier]

Abstract: Starting with the practical significance of search engine, this paper introduced the classification of search engine and analyzed its advantages and disadvantages, focused on the basic system structure and key technologies in its individual modules, finally predicted the trends for it. [Copyright &y& Elsevier]

Abstract: The Web contains a large amount of documents and an increasing quantity of structured data in the form of RDF triples. Many of these triples are annotations associated with documents. While structured queries constitute the principal means to retrieve structured data, keyword queries are typically used for document retrieval. Clearly, a form of hybrid search that seamlessly integrates these formalisms to query both textual and structured data can address more complex information needs. However, hybrid search on the large scale Web environment faces several challenges. First, there is a need for repositories that can store and index a large amount of semantic data as well as textual data in documents, and manage them in an integrated way. Second, methods for hybrid query answering are needed to exploit the data from such an integrated repository. These methods should be fast and scalable, and in particular, they shall support flexible ranking schemes to return not all but only the most relevant results. In this paper, we present CE2, an integrated solution that leverages mature information retrieval and database technologies to support large scale hybrid search. For scalable and integrated management of data, CE2 integrates off-the-shelf database solutions with inverted indexes. Efficient hybrid query processing is supported through novel data structures and algorithms which allow advanced ranking schemes to be tightly integrated. Furthermore, a concrete ranking scheme is proposed to take features from both textual and structured data into account. Experiments conducted on DBpedia and Wikipedia show that CE2 can provide good performance in terms of both effectiveness and efficiency. [Copyright &y& Elsevier]

Abstract: Conventional keyword search engines are restricted to a given data model and cannot easily adapt to unstructured, semi-structured or structured data. In this paper, we propose an efficient and adaptive keyword search method, called EASE, for indexing and querying large collections of heterogeneous data. To achieve high efficiency in processing keyword queries, we first model unstructured, semi-structured and structured data as graphs, and then summarize the graphs and construct graph indices instead of using traditional inverted indices. We propose an extended inverted index to facilitate keyword-based search, and present a novel ranking mechanism for enhancing search effectiveness. We have conducted an extensive experimental study using real datasets, and the results show that EASE achieves both high search efficiency and high accuracy, and outperforms the existing approaches significantly. [ABSTRACT FROM AUTHOR]

Abstract: The Web of Data keeps growing rapidly. However, the full exploitation of this large amount of structured data faces numerous challenges like usability, scalability, imprecise information needs and data change. We present Semplore, an IR-based system that aims at addressing these issues. Semplore supports intuitive faceted search and complex queries both on text and structured data. It combines imprecise keyword search and precise structured query in a unified ranking scheme. Scalable query processing is supported by leveraging inverted indexes traditionally used in IR systems. This is combined with a novel block-based index structure to support efficient index update when data changes. The experimental results show that Semplore is an efficient and effective system for searching the Web of Data and can be used as a basic infrastructure for Web-scale Semantic Web search engines. [Copyright &y& Elsevier]

Abstract: To augment the information retrieval process, a model is proposed to facilitate simple contextual indexing for a large scale of standard text corpora. An edge index graph model is presented, which clusters documents based on a root index and an edge index created. Intelligent information retrieval is possible with the projected system where the process of querying provides proactive help to users through a knowledge base. The query is provided with automatic phrase completion and word suggestions. A thesaurus is used to provide meaningful search of the query. This model can be utilized for document retrieval, clustering, and phrase browsing. [Copyright &y& Elsevier]

With the popularity of cloud storage, increasing people are willing to upload their files to cloud services. They will encrypt these files before uploading to protect the privacy of files. However, encryption makes effective search very difficult and inefficient. In this paper, we design a dynamic multi-keyword searchable encryption scheme on the encrypted cloud files, called DMSE. Firstly, the outsourced files in DMSE are pre-classified and preprocessed according to their privacy to achieve classified search. Secondly, based on the inverted index, DMSE realizes multiple keywords search, which protects the privacy of outsourced files and users and greatly improves the search efficiency. In addition, our DMSE extends the inverted index to the multi-DO/multi-DU scenario, which is the distinguishable difference between DMSE and other existing multi-keyword search schemes. Thirdly, our DMSE can further support the update of keywords to cope with the mismatch dilemma between the first extracted keywords and outsourced files, thus improving the practicality of DMSE scheme. Finally, comprehensive performance evaluation shows our DMSE scheme is effective and feasible in practical application. [ABSTRACT FROM AUTHOR]

With the recent advancement of cloud storage services, many cloud customers (data owners) are migrating their data from local storage to cloud data storage to reduce maintenance cost. Once data is outsourced, the data owners loose control over their data, evolving various security and privacy issues. To resolve this issue, data encryption is an effective solution. Meanwhile, to increase the utilization of the encrypted outsourced data, searchable encryption would be an efficient tool capable enough to search over encrypted data. In this work, we propose a novel public-key based searchable encryption scheme (SEPS) using an inverted index data structure. We show that SEPS achieves better performance than previous efforts (both theoretically and experimentally). We prove the security of the proposed scheme using Bilinear Diffie–Hellman assumption. SEPS withstands keyword guessing attack and achieves trapdoor privacy. On the top, the index generation process of the proposed scheme is secure against chosen keyword attack. AVISPA tool is also used for the formal security verification of SEPS and found to be attack resistant. • Efficient secure searching scheme based on public-key cryptography for cloud • Security of SEPS is analysed using the Bilinear Diffie–Hellman assumption • Protects the privacy of the trapdoor • Efficient in terms of total computational overhead as compared with previous schemes • Formally analyzed using AVISPA model checking tool and found to be attack resistant [ABSTRACT FROM AUTHOR]