Search

Abstract Background High-throughput sequencing is a powerful tool that is extensively applied in biological studies. However, sequencers may produce low-quality bases, leading to ambiguous bases, ‘N’s. PCR duplicates introduced in library preparation are conventionally removed in genomics studies, and several deduplication tools have been developed for this purpose. Two identical reads may appear different due to ambiguous bases and the existing tools cannot address ‘N’s correctly or efficiently. Results Here we proposed and implemented TrieDedup, which uses the trie (prefix tree) data structure to compare and store sequences. TrieDedup can handle ambiguous base ‘N’s, and efficiently deduplicate at the level of raw sequences. We also reduced its memory usage by approximately 20% by implementing restrictedDict in Python. We benchmarked the performance of the algorithm and showed that TrieDedup can deduplicate reads up to 270-fold faster than pairwise comparison at a cost of 32-fold higher memory usage. Conclusions The TrieDedup algorithm may facilitate PCR deduplication, barcode or UMI assignment, and repertoire diversity analysis of large-scale high-throughput sequencing datasets with its ultra-fast algorithm that can account for ambiguous bases due to sequencing errors.

Background: High-throughput sequencing is a powerful tool that is extensively applied in biological studies. However, sequencers may produce low-quality bases, leading to ambiguous bases, 'N's. PCR duplicates introduced in library preparation are conventionally removed in genomics studies, and several deduplication tools have been developed for this purpose. Two identical reads may appear different due to ambiguous bases and the existing tools cannot address 'N's correctly or efficiently. Results: Here we proposed and implemented TrieDedup, which uses the trie (prefix tree) data structure to compare and store sequences. TrieDedup can handle ambiguous base 'N's, and efficiently deduplicate at the level of raw sequences. We also reduced its memory usage by approximately 20% by implementing restrictedDict in Python. We benchmarked the performance of the algorithm and showed that TrieDedup can deduplicate reads up to 270-fold faster than pairwise comparison at a cost of 32-fold higher memory usage. Conclusions: The TrieDedup algorithm may facilitate PCR deduplication, barcode or UMI assignment, and repertoire diversity analysis of large-scale high-throughput sequencing datasets with its ultra-fast algorithm that can account for ambiguous bases due to sequencing errors. [ABSTRACT FROM AUTHOR]

Automated log parsing is essential for many log-mining applications, as logs provide a vast range of information on events and variations within an operating system or software at runtime. Over the years, various methods have been proposed for log parsing. With improved log-parsing methods, log-mining applications can gain deeper insights into system behaviors and identify anomalies or failures promptly. However, current log parsers still face limitations, such as insufficient parsing of log templates and a lack of parallelism, as well as inaccurate log template parsing. To overcome these limitations, we have designed Polo, a parser that leverages a prefix forest composed of ternary search trees to mine templates from logs. We then conducted extensive experiments to evaluate the accuracy of Polo on nine representative system logs, achieving an average accuracy of 0.987. It is 9.93% to 40.95% faster than the state-of-the-art parsing methods. Furthermore, we evaluated our approach on a downstream log analysis task, specifically anomaly detection. The experimental results demonstrated that, in terms of F1-score, our parser outperformed Deeplog, LogAnomaly, CNN, and LogRobust by 11.5%, 4%, 1%, and 19.1%, respectively, exhibiting a promising recall score of 0.971. These results indicate the effectiveness of Polo for anomaly detection. [ABSTRACT FROM AUTHOR]

The professional contacts of a person, including their past colleagues and supervisors, can play an important role in job recommendation and intelligent human resources management. However, collecting such information on a large scale can be challenging and costly. In this paper, we propose CareerMiner, a system to automatically construct a dynamic professional network to capture people’s colleague and supervisor relationships based on inter-related work experiences in a large collection of Chinese resumes. Specifically, we extract fine-grained work affiliation and job title information from resume text using a customized Name Entity Recognition model and introduce a trie-like data structure to efficiently index the hierarchical affinities of the implicit organizational structure in work experiences. Then, the topology of leaf nodes is used to identify professional relationships. Experimental results on real-world resume data are presented to demonstrate the effectiveness of the proposed system. Additionally, we develop an open-source demo system that includes the main features of the system. We also conduct a case study to illustrate how CareerMiner can be utilized for the social science research and data mining applications of professional networks.

Automated log parsing is essential for many log-mining applications, as logs provide a vast range of information on events and variations within an operating system or software at runtime. Over the years, various methods have been proposed for log parsing. With improved log-parsing methods, log-mining applications can gain deeper insights into system behaviors and identify anomalies or failures promptly. However, current log parsers still face limitations, such as insufficient parsing of log templates and a lack of parallelism, as well as inaccurate log template parsing. To overcome these limitations, we have designed Polo, a parser that leverages a prefix forest composed of ternary search trees to mine templates from logs. We then conducted extensive experiments to evaluate the accuracy of Polo on nine representative system logs, achieving an average accuracy of 0.987. It is 9.93% to 40.95% faster than the state-of-the-art parsing methods. Furthermore, we evaluated our approach on a downstream log analysis task, specifically anomaly detection. The experimental results demonstrated that, in terms of F1-score, our parser outperformed Deeplog, LogAnomaly, CNN, and LogRobust by 11.5%, 4%, 1%, and 19.1%, respectively, exhibiting a promising recall score of 0.971. These results indicate the effectiveness of Polo for anomaly detection.

In this work we address the problem of performing an error-tolerant prefix search on a set of string keys. While the ideas presented here could be adopted in other applications, our primary target application is error-tolerant query autocompletion. Tries and their variations have been adopted as the basic data structure to implement recently proposed error-tolerant prefix search methods. However, they must require a lot of extra memory to process queries. Burst tries are alternative compact tries proposed to reduce storage costs and maintain a close performance when compared to the use of tries. Here we discuss alternatives for adapting burst tries as error-tolerant prefix search data structures. We show how to adapt state-of-the-art trie-based methods for use with burst tries. We studied the trade-off between memory usage and time performance while varying the parameters to build the burst trie index. As an example, when indexing the JusBrasil dataset, one of the datasets adopted in the experiments, the use of burst tries reduces the memory required by a full trie to 26% and increases the time performance to 116%. The possibility of balancing memory usage and time performance constitutes an advantage of the burst trie when compared to the full trie when adopted as an index for the task of performing error-tolerant prefix search. [ABSTRACT FROM AUTHOR]

The vehicle routing problem with two-dimensional loading constraints (2L-CVRP) is a practical variant of the classic capacitated vehicle routing problem. A number of algorithms have been developed for the problem, but it is very difficult for the existing exact methods to optimally solve instances featuring with large rectangular items. To address this issue, a branch-and-price-and-cut (BPC) algorithm is proposed in this study. A novel data structure and a new dominance rule are developed to build an exact pricing algorithm that takes the loading constraints into account. Several valid inequalities are used to strengthen the linear relaxation. Extensive computational experiments were conducted on the benchmark instances of the 2L-CVRP, showing that the BPC algorithm outperforms all the existing exact methods for the problem in terms of the solution quality. Fourteen instances are solved to optimality for the first time. In particular, the size of solvable instances with large items is nearly doubled. Moreover, managerial insights about the impact of respecting the last-in-first-out constraint are also obtained. [ABSTRACT FROM AUTHOR]

In the last years, the System 1/System 2 cognitive architecture, proposed by psychologist Daniel Kahneman, raised the interest of many researchers in the field. System 1 is an intuitive, automatic, and fast-thinking system working effortlessly, without conscious effort. System 2 is a deliberate, analytical, and slower-thinking system employing conscious effort and attention. This work proposes an innovative approach that exploits techniques typical of information retrieval (the trie data structure) to efficiently encode the solutions' repository at the border between System 2 and System 1. This repository stores the solutions (successful plans) the agent has already used and can re-enact to achieve the goals. System 2 conceives new plans and delegates System 1 to execute them. If the plan is successful (and so it becomes a solution), System 1 stores that in the repository to quickly retrieve any solution that may help fulfil the goals deliberated by System 2 in the future. • A step towards the definition of an interface between System 1 and System 2. • Efficient encoding of a repository of solutions and their quick access by System 1. • The repository is designed to support a goal-oriented perspective. [ABSTRACT FROM AUTHOR]

Set superset query is widely used in e-commerce processing and many other domains, particularly in cloud computing environments. Indexing is an efficient way to model e-commerce data. Many existing indexes, however, primarily focus on enhancing either query performance or space efficiency, often neglecting the need to strike a balance between these two factors. We have observed that upper nodes closer to the root of a tree are frequently accessed, while lower nodes near the leaves tend to entail expensive storage costs. To address this issue, we introduce TLI model, a trie and level-ordered unary degree sequence (LOUDS) hybrid model. The upper part of TLI is a trie, which is optimized for superior query performance. The lower part of TLI uses the LOUDS structure. TLI strikes a good balance between query performance and space utilization. To seamlessly integrate these two parts, we have developed efficient connecting strategies. Our simulation results on localhost demonstrate that TLI outperforms its competitors in terms of both space and time efficiency. Remarkably, it enhances query performance by up to 1.89 times, with a modest 6.72% increase in space overhead compared to LOUDS-based alternatives. [ABSTRACT FROM AUTHOR]

Summary: We see a resurgence of Datalog in a variety of applications, including program analysis, networking, data integration, cloud computing, and security. The large‐scale and complexity of these applications need the efficient management of data in relations. Hence, Datalog implementations require new data structures for managing relations that (1) are parallel, (2) are highly specialized for Datalog evaluation, and (3) can accommodate different workloads depending on the applications concerning memory consumption and computational efficiency. In this article, we present a data structure framework for relations that is specialized for shared‐memory parallel Datalog implementations such as the soufflé Datalog compiler. The data structure framework permits a portfolio of different data structures depending on the workload. We also introduce two concrete parallel data structures for relations, designed for various workloads. Our benchmarks demonstrate a speed‐up of up to 6× by using a portfolio of data structures compared with using a B‐tree alone, showing the advantage of our data structure framework. [ABSTRACT FROM AUTHOR]

Similarity-preserving hashing is a core technique for fast similarity searches, and it randomly maps data points in a metric space to strings of discrete symbols (i.e., sketches) in the Hamming space. While traditional hashing techniques produce binary sketches, recent ones produce integer sketches for preserving various similarity measures. However, most similarity search methods are designed for binary sketches and inefficient for integer sketches. Moreover, most methods are either inapplicable or inefficient for dynamic datasets, although modern real-world datasets are updated over time. We propose dynamic filter trie (DyFT), a dynamic similarity search method for both binary and integer sketches. An extensive experimental analysis using large real-world datasets shows that DyFT performs superiorly with respect to scalability, time performance, and memory efficiency. For example, on a huge dataset of 216 million data points, DyFT performs a similarity search 6000 times faster than a state-of-the-art method while reducing to one-thirteenth in memory. [ABSTRACT FROM AUTHOR]

A maximal repetition, or run, in a string, is a maximal periodic substring whose smallest period is at most half the length of the substring. In this paper, we consider runs that correspond to a path on a trie, or in other words, on a rooted edge-labeled tree where each edge is labeled with a single symbol, and the endpoints of the path must be a descendant/ancestor of the other. For a trie with n edges, we show that the number of runs is less than n. We also show an asymptotic lower bound on the maximum density of runs in tries: lim n → ∞ ⁡ ρ T (n) / n > 0.9932348 where ρ T (n) is the maximum number of runs in a trie with n edges. Furthermore, we also show an O (n log ⁡ log ⁡ n) time and O (n) space algorithm for finding all runs. [ABSTRACT FROM AUTHOR]

In this manuscript, an intelligent and adaptive web page recommender system is proposed that provides personalized, global, and group mode of recommendations. The authors enhance the utility of a trie node for storing relevant web access statistics. The trie node enables dynamic clustering of users based on their evolving browsing patterns and allows a user to belong to multiple groups at each navigation step. The system takes cues from the field of crowd psychology to augment two parameters for modeling group behavior: uniformity and recommendation strength. The system continuously tracks the user's responses in order to adaptively switch between different recommendation-criteria in the group and personalized modes. The experimental results illustrate that the system achieved the maximum F1 measure of 83.28% on CTI dataset, which is a significant improvement over the 70% F1 measure reported by automatic clustering-based genetic algorithm, the prior web recommender system. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we study the problem of selectivity estimation on set containment search. Given a query record Q and a record dataset $${\mathcal {S}}$$ S , we aim to accurately and efficiently estimate the selectivity of set containment search of query Q over $${\mathcal {S}}$$ S . We first extend existing distinct value estimating techniques to solve this problem and develop an inverted list and G-KMV sketch-based approach IL-GKMV. We analyze that the performance of IL-GKMV degrades with the increase in vocabulary size. Motivated by limitations of existing techniques and the inherent challenges of the problem, we resort to developing effective and efficient sampling approaches and propose an ordered trie structure-based sampling approach named OT-Sampling. OT-Sampling partitions records based on element frequency and occurrence patterns and is significantly more accurate compared with simple random sampling method and IL-GKMV. To further enhance the performance, a divide-and-conquer-based sampling approach, DC-Sampling, is presented with an inclusion/exclusion prefix to explore the pruning opportunities. Meanwhile, we consider weighted set containment selectivity estimation and devise stratified random sampling approach named StrRS. We theoretically analyze the proposed techniques regarding various accuracy estimators. Our comprehensive experiments on nine real datasets verify the effectiveness and efficiency of our proposed techniques.

Let P be a collection of d patterns { P 1 , P 2 , ... , P d } of total length n characters, which are chosen from an alphabet Σ of size σ. Given a text T (over Σ), the dictionary indexing problem is to create a data structure using which we can report all positions j (called occurrences) where at least one of the patterns P i ∈ P is a match with the same-length substring of T that starts at j. We consider this problem under the following definitions of matching. • Parameterized Matching: The characters of Σ are partitioned into static characters and parameterized characters. Two equal length strings S and S ′ are a parameterized match iff the static characters match exactly, and there exists a one-to-one function which renames the parameterized characters in S to those in S ′. • Order-Preserving Matching: The alphabet Σ is ordered. Two equal length strings S and S ′ are an order-preserving match iff for any two integers i , j ∈ [ 1 , | S | ] , S [ i ] ≺ S [ j ] ⇔ S ′ [ i ] ≺ S ′ [ j ] , where ≺ denotes the precedence order in Σ. Let ε > 0 be an arbitrarily small constant. For parameterized matching, we first present a compact O (n log ⁡ σ + d log ⁡ n) -bit index that reports all occ occurrences in O (| T | (log ⁡ σ + log σ ⁡ n) + o c c) time, and then a succinct n log ⁡ σ + o (n log ⁡ σ) + O (d log ⁡ n) -bit index that reports all occ occurrences in O (| T | (log ⁡ σ + log ε ⁡ n log σ ⁡ n) + o c c) time. For order-preserving matching, we present indexes of the same sizes, but with slightly increased query time. [ABSTRACT FROM AUTHOR]

Superset query is widely used in object-oriented databases, data mining, and many other fields. Trie is an efficient index for superset query, whereas most existing trie index aim at improving query performance while ignoring storage overheads. To solve this problem, in this paper, we propose an efficient extended Level-Ordered Unary Degree Sequence (LOUDS) index: Ext-LOUDS. Ext-LOUDS expresses a trie by 1 integer vector and 3 bit vectors directly map each NodeID to its corresponding position, thus accelerating some key operations needed for superset query. Based on Ext-LOUDS, an efficient superset query algorithm, ELOUDS-Super, is designed. Experimental results on both real and synthetic datasets show that Ext-LOUDS can decrease 50%–60% space overheads compared with trie while maintaining a relative good query performance. [ABSTRACT FROM AUTHOR]

• A classifier that is able to dynamically learn and recognize important word n-grams. • A novel text classifier having the ability to visually explain its rationale. • Support for incremental learning and text classification over text streams. • Efficient model for addressing early risk detection problems. A recently introduced classifier, called SS3, has shown to be well suited to deal with early risk detection (ERD) problems on text streams. It obtained state-of-the-art performance on early depression and anorexia detection on Reddit in the CLEF's eRisk open tasks. SS3 was designed to deal with ERD problems naturally since: it supports incremental training and classification over text streams, and it can visually explain its rationale. However, SS3 processes the input using a bag-of-word model lacking the ability to recognize important word sequences. This aspect could negatively affect the classification performance and also reduces the descriptiveness of visual explanations. In the standard document classification field, it is very common to use word n-grams to try to overcome some of these limitations. Unfortunately, when working with text streams, using n-grams is not trivial since the system must learn and recognize which n-grams are important "on the fly". This paper introduces τ -SS3, an extension of SS3 that allows it to recognize useful patterns over text streams dynamically. We evaluated our model in the eRisk 2017 and 2018 tasks on early depression and anorexia detection. Experimental results suggest that τ -SS3 is able to improve both current results and the richness of visual explanations. [ABSTRACT FROM AUTHOR]

We present the Succinct Range Filter (SuRF), a fast and compact data structure for approximate membership tests. Unlike traditional Bloom filters, SuRF supports both single-key lookups and common range queries: open-range queries, closed-range queries, and range counts. SuRF is based on a new data structure called the Fast Succinct Trie (FST) that matches the point and range query performance of state-of-the-art orderpreserving indexes, while consuming only 10 bits per trie node. The false-positive rates in SuRF for both point and range queries are tunable to satisfy different application needs. We evaluate SuRF in RocksDB as a replacement for its Bloom filters to reduce I/O by filtering requests before they access on-disk data structures. Our experiments on a 100-GB dataset show that replacing RocksDB's Bloom filters with SuRFs speeds up open-seek (without upper-bound) and closed-seek (with upper-bound) queries by up to 1.5× and 5× with a modest cost on the worst-case (all-missing) point query throughput due to slightly higher false-positive rate. [ABSTRACT FROM AUTHOR]

The B-tree and its variants are an indispensable tool for database systems and applications. Hence the efficiency of the B-tree is one of the few critical factors that determine the performance of a database system. In main-memory database systems, the computational overhead intrinsic in the B-tree algorithms for branching becomes the dominant factor in performance. In this paper, we propose yet another but disruptive variant of the B + -tree called the DB + -tree that redesigns the node structure for faster branching operations. The novel branching algorithm of the DB + -tree can be implemented in an O (1) number of SIMD and other sequential instructions, which supports fast branching, and this leads to efficient point search, range search, and update operations. • DB+ tree redesigns the node structure of B+ tree for faster branching operation • Our branching algorithm can be implemented in an O(1) number of instructions • DB+ tree performs point search 170% faster than pkB-tree. • DB+ tree performs range search 130% faster than pkB-tree. • DB+ tree performs insertions and deletions 140% faster than pkB-tree. [ABSTRACT FROM AUTHOR]

Mass spectrometers enable identifying proteins in biological samples leading to biomarkers for biological process parameters and diseases. However, bioinformatic evaluation of the mass spectrometer data needs a standardized workflow and system that stores the protein sequences. Due to its standardization and maturity, relational systems are a great fit for storing protein sequences. Hence, in this work, we present a schema for distributed column-based database management systems using a column-oriented index to store sequence data. In order to achieve a high storage performance, it was necessary to choose a well-performing strategy for transforming the protein sequence data from the FASTA format to the new schema. Therefore, we applied an in-memory map, HDDmap, database engine, and extended radix tree and evaluated their performance. The results show that our proposed extended radix tree performs best regarding memory consumption and runtime. Hence, the radix tree is a suitable data structure for transforming protein sequences into the indexed schema.